[
https://issues.apache.org/jira/browse/DISPATCH-1309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16808111#comment-16808111
]
Chuck Rolke commented on DISPATCH-1309:
---------------------------------------
I tried git bisecting from 1.5.0 to 1.6.0 release tags and got no errors. Then
eventually I got the following backtrace trying to log "Computed valid origins
{}" from python run-time code.
The crashes I saw earlier happened when a console was attached to an
http-enabled listener. During my bisect exercise I had no consoles attached.
I'll go back and try some with consoles attached where I can.
{{(gdb) bt}}
{{#0 malloc_consolidate (av=av@entry=0x7f9708000020) at malloc.c:4469}}
{{#1 0x00007f972b97d098 in _int_malloc (av=av@entry=0x7f9708000020,
bytes=bytes@entry=2288) at malloc.c:3695}}
{{#2 0x00007f972b97dd7b in _int_memalign (av=av@entry=0x7f9708000020,
alignment=alignment@entry=64, bytes=bytes@entry=2176) at malloc.c:4726}}
{{#3 0x00007f972b97ef7a in _mid_memalign (alignment=64, bytes=2176,
address=<optimized out>) at malloc.c:3306}}
{{#4 0x00007f972b98052c in __posix_memalign (size=<optimized out>,
alignment=<optimized out>, memptr=0x7f9717ffd830) at malloc.c:5401}}
{{#5 __posix_memalign (memptr=0x7f9717ffd830, alignment=<optimized out>,
size=<optimized out>) at malloc.c:5388}}
{{#6 0x00007f972be8b3c4 in qd_alloc (desc=0x7f972beb5dc0
<__desc_qd_log_entry_t>, tpool=0x7f9717fff578) at
/home/chug/git/qpid-dispatch/src/alloc_pool.c:181}}
{{#7 0x00007f972be336ab in new_qd_log_entry_t () at
/home/chug/git/qpid-dispatch/src/log.c:61}}
{{#8 0x00007f972be3491d in qd_vlog_impl (source=0x8c0bb0, level=QD_LOG_INFO, }}
{{ file=0x7f971df1d3d4
"/opt/local/lib/qpid-dispatch/python/qpid_dispatch_internal/router/node.py",
line=162, fmt=0x7f972be928df "%s", ap=0x7f9717ffd8e8)}}
{{ at /home/chug/git/qpid-dispatch/src/log.c:416}}
{{#9 0x00007f972be34ba1 in qd_log_impl (source=0x8c0bb0, level=QD_LOG_INFO, }}
{{ file=0x7f971df1d3d4
"/opt/local/lib/qpid-dispatch/python/qpid_dispatch_internal/router/node.py",
line=162, fmt=0x7f972be928df "%s")}}
{{ at /home/chug/git/qpid-dispatch/src/log.c:435}}
{{#10 0x00007f972be460a8 in qd_python_log
(self=<qpid_dispatch_internal.dispatch.LogAdapter at remote 0x7f971d71a0f0>, }}
{{ args=(4L, u'Computed valid origins: {}',
'/opt/local/lib/qpid-dispatch/python/qpid_dispatch_internal/router/node.py',
162))}}
{{ at /home/chug/git/qpid-dispatch/src/python_embedded.c:429}}
{{#11 0x00007f972bc6314b in call_function (oparg=<optimized out>,
pp_stack=0x7f9717ffdab0) at
/usr/src/debug/python2-2.7.15-11.fc29.x86_64/Python/ceval.c:4449}}
{{#12 PyEval_EvalFrameEx (}}
{{ f=f@entry=Frame 0x7f971d714bf0, for file
/opt/local/lib/qpid-dispatch/python/qpid_dispatch_internal/router/engine.py,
line 229, in log_ls
(self=<RouterEngine(node_tracker=<NodeTracker(neighbor_max_age=3,
next_maskbit=4, container=<...>, nodes_by_link_id=\{0:
<RouterNode(mobile_addresses=[u'HEC1', u'HEC2'], cost=1, log=<instancemethod at
remote 0x7f971d796f00>, parent=<...>, neighbor_refresh_time=<float at remote
0x5ac758>, keep_alive_count=0, adapter=<dispatch.RouterAdapter at remote
0x7f971d72e078>, valid_origins=[], peer_link_id=None,
link_state=<LinkState(ls_seq=2L, peers={u'INTD': 1L, u'INTB': 1L}, area=u'0',
id=u'INTC', last_seen=<float at remote 0x5ac770>) at remote 0x7f971c4952d0>,
instance=1554232196L, version=1L, next_hop_router=None,
mobile_address_sequence=8, need_mobile_request=False, id=u'INTC', maskbit=1,
need_ls_request=False) at remote 0x7f971d7c2790>}, maskbits=[True, True, True,
True, None, None, None, None, None, None, None, None, None, None, None, None,
None, None, None, None, None, None, None, None,...(truncated),
throwflag=throwflag@entry=0)}}
{{ at /usr/src/debug/python2-2.7.15-11.fc29.x86_64/Python/ceval.c:3083}}
{{#13 0x00007f972bc63902 in PyEval_EvalCodeEx (co=<optimized out>,
globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, }}
{{ argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x0,
defcount=0, closure=0x0)}}
{{ at /usr/src/debug/python2-2.7.15-11.fc29.x86_64/Python/ceval.c:3681}}
{{#14 0x00007f972bc6053c in fast_function (nk=<optimized out>, na=<optimized
out>, n=<optimized out>, pp_stack=0x7f9717ffdc70, }}
{{ func=<function at remote 0x7f971df26500>) at
/usr/src/debug/python2-2.7.15-11.fc29.x86_64/Python/ceval.c:4544}}
{{#15 call_function (oparg=<optimized out>, pp_stack=0x7f9717ffdc70) at
/usr/src/debug/python2-2.7.15-11.fc29.x86_64/Python/ceval.c:4469}}
{{#16 PyEval_EvalFrameEx (}}
{{ f=f@entry=Frame 0x7f970404cc40, for file
/opt/local/lib/qpid-dispatch/python/qpid_dispatch_internal/router/node.py, line
162, in tick (self=<NodeTracker(neighbor_max_age=3, next_maskbit=4,
container=<RouterEngine(node_tracker=<...>, domain=u'domain',
_log_hello=<qpid_dispatch_internal.dispatch.LogAdapter at remote
0x7f971d71a0d0>, area=u'0',
mobile_address_engine=<MobileAddressEngine(sent_deltas=\{1:
<MessageMAU(del_list=[], area=u'0', version=1L, mobile_seq=1L,
add_list=[u'HED2'], id=u'INTD', exist_list=None, hints=[3L]) at remote
0x7f971dde5790>, 2: <MessageMAU(del_list=[], area=u'0', version=1L,
mobile_seq=2L, add_list=[u'HED1'], id=u'INTD', exist_list=None, hints=[3L]) at
remote 0x7f971d7c26d0>}, container=<...>, deleted_addrs=[],
treatments=\{u'HED1': 3L, u'HED2': 3L}, added_addrs=[], node_tracker=<...>,
local_addrs=[u'HED2', u'HED1'], mobile_seq=2, id=u'INTD') at remote
0x7f971d70cd10>, _config=<RouterEntity(workerThreads=4, raIntervalSeconds=30,
raIntervalFluxSeconds=4, allowResumableLinkRoute=True,
_agent=<Agent(...(truncated), throwflag=throwflag@entry=0)}}
{{ at /usr/src/debug/python2-2.7.15-11.fc29.x86_64/Python/ceval.c:3083}}
{{#17 0x00007f972bc63902 in PyEval_EvalCodeEx (co=<optimized out>,
globals=<optimized out>, locals=locals@entry=0x0, args=<optimized out>, }}
{{ argcount=<optimized out>, kws=<optimized out>, kwcount=0, defs=0x0,
defcount=0, closure=0x0)}}
{{ at /usr/src/debug/python2-2.7.15-11.fc29.x86_64/Python/ceval.c:3681}}
{{#18 0x00007f972bc6053c in fast_function (nk=<optimized out>, na=<optimized
out>, n=<optimized out>, pp_stack=0x7f9717ffde30, }}
{{--Type <RET> for more, q to quit, c to continue without paging--}}
{{ func=<function at remote 0x7f971df18d70>) at
/usr/src/debug/python2-2.7.15-11.fc29.x86_64/Python/ceval.c:4544}}
{{#19 call_function (oparg=<optimized out>, pp_stack=0x7f9717ffde30) at
/usr/src/debug/python2-2.7.15-11.fc29.x86_64/Python/ceval.c:4469}}
{{#20 PyEval_EvalFrameEx (}}
{{ f=f@entry=Frame 0x7f971d75e988, for file
/opt/local/lib/qpid-dispatch/python/qpid_dispatch_internal/router/engine.py,
line 132, in handleTimerTick
(self=<RouterEngine(node_tracker=<NodeTracker(neighbor_max_age=3,
next_maskbit=4, container=<...>, nodes_by_link_id=\{0:
<RouterNode(mobile_addresses=[u'HEC1', u'HEC2'], cost=1, log=<instancemethod at
remote 0x7f971d796f00>, parent=<...>, neighbor_refresh_time=<float at remote
0x5ac758>, keep_alive_count=0, adapter=<dispatch.RouterAdapter at remote
0x7f971d72e078>, valid_origins=[], peer_link_id=None,
link_state=<LinkState(ls_seq=2L, peers={u'INTD': 1L, u'INTB': 1L}, area=u'0',
id=u'INTC', last_seen=<float at remote 0x5ac770>) at remote 0x7f971c4952d0>,
instance=1554232196L, version=1L, next_hop_router=None,
mobile_address_sequence=8, need_mobile_request=False, id=u'INTC', maskbit=1,
need_ls_request=False) at remote 0x7f971d7c2790>}, maskbits=[True, True, True,
True, None, None, None, None, None, None, None, None, None, None, None, None,
None, None, None, None, None, None, No...(truncated),
throwflag=throwflag@entry=0)}}
{{ at /usr/src/debug/python2-2.7.15-11.fc29.x86_64/Python/ceval.c:3083}}
{{#21 0x00007f972bc63902 in PyEval_EvalCodeEx (co=<optimized out>,
globals=<optimized out>, locals=locals@entry=0x0,
args=args@entry=0x7f972afb4768, }}
{{ argcount=<optimized out>, kws=kws@entry=0x0, kwcount=0, defs=0x0,
defcount=0, closure=0x0)}}
{{ at /usr/src/debug/python2-2.7.15-11.fc29.x86_64/Python/ceval.c:3681}}
{{#22 0x00007f972bb9e972 in function_call (func=func@entry=<function at remote
0x7f971df26230>, }}
{{
arg=arg@entry=(<RouterEngine(node_tracker=<NodeTracker(neighbor_max_age=3,
next_maskbit=4, container=<...>, nodes_by_link_id=\{0:
<RouterNode(mobile_addresses=[u'HEC1', u'HEC2'], cost=1, log=<instancemethod at
remote 0x7f971d796f00>, parent=<...>, neighbor_refresh_time=<float at remote
0x5ac758>, keep_alive_count=0, adapter=<dispatch.RouterAdapter at remote
0x7f971d72e078>, valid_origins=[], peer_link_id=None,
link_state=<LinkState(ls_seq=2L, peers={u'INTD': 1L, u'INTB': 1L}, area=u'0',
id=u'INTC', last_seen=<float at remote 0x5ac770>) at remote 0x7f971c4952d0>,
instance=1554232196L, version=1L, next_hop_router=None,
mobile_address_sequence=8, need_mobile_request=False, id=u'INTC', maskbit=1,
need_ls_request=False) at remote 0x7f971d7c2790>}, maskbits=[True, True, True,
True, None, None, None, None, None, None, None, None, None, None, None, None,
None, None, None, None, None, None, None, None, None, None, None, None, None,
None, None, None, None, None, None, None, None, None, None, None, None, None,
None, None, None, None, ...(truncated), kw=kw@entry=0x0)}}
{{ at /usr/src/debug/python2-2.7.15-11.fc29.x86_64/Objects/funcobject.c:523}}
{{#23 0x00007f972bb85aa7 in PyObject_Call (func=func@entry=<function at remote
0x7f971df26230>, }}
{{
arg=arg@entry=(<RouterEngine(node_tracker=<NodeTracker(neighbor_max_age=3,
next_maskbit=4, container=<...>, nodes_by_link_id=\{0:
<RouterNode(mobile_addresses=[u'HEC1', u'HEC2'], cost=1, log=<instancemethod at
remote 0x7f971d796f00>, parent=<...>, neighbor_refresh_time=<float at remote
0x5ac758>, keep_alive_count=0, adapter=<dispatch.RouterAdapter at remote
0x7f971d72e078>, valid_origins=[], peer_link_id=None,
link_state=<LinkState(ls_seq=2L, peers={u'INTD': 1L, u'INTB': 1L}, area=u'0',
id=u'INTC', last_seen=<float at remote 0x5ac770>) at remote 0x7f971c4952d0>,
instance=1554232196L, version=1L, next_hop_router=None,
mobile_address_sequence=8, need_mobile_request=False, id=u'INTC', maskbit=1,
need_ls_request=False) at remote 0x7f971d7c2790>}, maskbits=[True, True, True,
True, None, None, None, None, None, None, None, None, None, None, None, None,
None, None, None, None, None, None, None, None, None, None, None, None, None,
None, None, None, None, None, None, None, None, None, None, None, None, None,
None, None, None, None, ...(truncated), kw=kw@entry=0x0)}}
{{ at /usr/src/debug/python2-2.7.15-11.fc29.x86_64/Objects/abstract.c:2547}}
{{#24 0x00007f972bb9147e in instancemethod_call (func=<function at remote
0x7f971df26230>, func@entry=<instancemethod at remote 0x7f971d8f52d0>, }}
{{ arg=(<RouterEngine(node_tracker=<NodeTracker(neighbor_max_age=3,
next_maskbit=4, container=<...>, nodes_by_link_id=\{0:
<RouterNode(mobile_addresses=[u'HEC1', u'HEC2'], cost=1, log=<instancemethod at
remote 0x7f971d796f00>, parent=<...>, neighbor_refresh_time=<float at remote
0x5ac758>, keep_alive_count=0, adapter=<dispatch.RouterAdapter at remote
0x7f971d72e078>, valid_origins=[], peer_link_id=None,
link_state=<LinkState(ls_seq=2L, peers={u'INTD': 1L, u'INTB': 1L}, area=u'0',
id=u'INTC', last_seen=<float at remote 0x5ac770>) at remote 0x7f971c4952d0>,
instance=1554232196L, version=1L, next_hop_router=None,
mobile_address_sequence=8, need_mobile_request=False, id=u'INTC', maskbit=1,
need_ls_request=False) at remote 0x7f971d7c2790>}, maskbits=[True, True, True,
True, None, None, None, None, None, None, None, None, None, None, None, None,
None, None, None, None, None, None, None, None, None, None, None, None, None,
None, None, None, None, None, None, None, None, None, None, None, None, None,
None, None, None, None, ...(truncated), arg@entry=(), kw=0x0)}}
{{ at
/usr/src/debug/python2-2.7.15-11.fc29.x86_64/Objects/classobject.c:2600}}
{{#25 0x00007f972bb85aa7 in PyObject_Call (func=func@entry=<instancemethod at
remote 0x7f971d8f52d0>, arg=arg@entry=(), kw=<optimized out>)}}
{{ at /usr/src/debug/python2-2.7.15-11.fc29.x86_64/Objects/abstract.c:2547}}
{{#26 0x00007f972bc59cc7 in PyEval_CallObjectWithKeywords (func=<instancemethod
at remote 0x7f971d8f52d0>, arg=(), kw=<optimized out>)}}
{{ at /usr/src/debug/python2-2.7.15-11.fc29.x86_64/Python/ceval.c:4318}}
{{#27 0x00007f972be81c3d in qd_pyrouter_tick (router=0x804670) at
/home/chug/git/qpid-dispatch/src/router_pynode.c:476}}
{{--Type <RET> for more, q to quit, c to continue without paging--}}
{{#28 0x00007f972be7f902 in qd_router_timer_handler (context=0x804670) at
/home/chug/git/qpid-dispatch/src/router_node.c:1232}}
{{#29 0x00007f972be87e23 in qd_timer_visit () at
/home/chug/git/qpid-dispatch/src/timer.c:200}}
{{#30 0x00007f972be8512b in handle (qd_server=0x8da130, e=0x7f970c000b30,
pn_conn=0x0, ctx=0x0) at /home/chug/git/qpid-dispatch/src/server.c:913}}
{{#31 0x00007f972be8561a in thread_run (arg=0x8da130) at
/home/chug/git/qpid-dispatch/src/server.c:1009}}
{{#32 0x00007f972bd5658e in start_thread (arg=<optimized out>) at
pthread_create.c:486}}
{{#33 0x00007f972b9f56a3 in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95}}
> Various crashes in 1.6 release
> ------------------------------
>
> Key: DISPATCH-1309
> URL: https://issues.apache.org/jira/browse/DISPATCH-1309
> Project: Qpid Dispatch
> Issue Type: Bug
> Affects Versions: 1.6.0
> Environment: System 'unused':(
> Fedora 5.0.3-200.fc29.x86_64,
> Python 2.7.15,
> Proton master @ eab1f.
> System 'taj':(
> Fedora 4.18.16-200.fc28.x86_64,
> Python 3.6.6,
> Proton master @ 68b38
> Reporter: Chuck Rolke
> Priority: Major
> Attachments: DISPATCH-1309-backtraces.txt,
> DISPATCH-1309-gen_configs_linear.py
>
>
> qpid-dispatch master @ 51244, which is very close to the 1.6 release, has
> various crashes.
> The test network is 12 routers spread over two systems. (Configuration
> generator to be attached.) Four interior routers are in linear arrangement
> with A and C on one system ('unused'), and B and D on the other system
> ('taj'). Each system then attaches four edge routers, one to each interior
> router.
> Running lightweight tests, like proton cpp simple_send and simple_recv to
> ports on INTA and INTB interior routers leads to a crash on INTC. The crashes
> typically look like reuse of structures after they have been freed (addresses
> are 0x9999999999999999). Other crashes hint of general memory corruption
> (crashes in malloc.c).
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
