[
https://issues.apache.org/jira/browse/DISPATCH-1927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17270374#comment-17270374
]
Charles E. Rolke commented on DISPATCH-1927:
--------------------------------------------
backtrace
{code:java}
Core was generated by `qdrouterd -c A.conf'.
Program terminated with signal SIGABRT, Aborted.
warning: Unexpected size of section `.reg-xstate/3544196' in core file.
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50 return ret;
[Current thread is 1 (Thread 0x7f7fa3f4d2c0 (LWP 3544196))]
(gdb) bt
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 0x00007f7fa47558d9 in __GI_abort () at abort.c:79
#2 0x00007f7fa47557a9 in __assert_fail_base (fmt=0x7f7fa48c0af8 "%s%s%s:%u:
%s%sAssertion `%s' failed.\n%n", assertion=0x7f7fa4ea43ab
"(link->undelivered).head",
file=0x7f7fa4ea3fc8
"/home/chug/git/qpid-dispatch/src/router_core/transfer.c", line=231,
function=<optimized out>) at assert.c:92
#3 0x00007f7fa4764a66 in __GI___assert_fail (assertion=0x7f7fa4ea43ab
"(link->undelivered).head", file=0x7f7fa4ea3fc8
"/home/chug/git/qpid-dispatch/src/router_core/transfer.c", line=231,
function=0x7f7fa4ea4bc0 <__PRETTY_FUNCTION__.9815>
"qdr_link_process_deliveries") at assert.c:101
#4 0x00007f7fa4e616aa in qdr_link_process_deliveries (core=0x1cd4360,
link=0x1da5ad0, credit=2637) at
/home/chug/git/qpid-dispatch/src/router_core/transfer.c:231
#5 0x00007f7fa4e03a97 in qdr_tcp_push (context=0x1d65310, link=0x1da5ad0,
limit=2637) at /home/chug/git/qpid-dispatch/src/adaptors/tcp_adaptor.c:1056
#6 0x00007f7fa4e3e475 in qdr_connection_process (conn=0x1d92050) at
/home/chug/git/qpid-dispatch/src/router_core/connections.c:412
#7 0x00007f7fa4dffb24 in on_activate (context=0x1d7c7a0) at
/home/chug/git/qpid-dispatch/src/adaptors/tcp_adaptor.c:112
#8 0x00007f7fa4e7ee62 in qd_timer_visit () at
/home/chug/git/qpid-dispatch/src/timer.c:201
#9 0x00007f7fa4e7b9c7 in handle (qd_server=0x1ad9040, e=0x7f7f8008dd50,
pn_conn=0x0, ctx=0x0) at /home/chug/git/qpid-dispatch/src/server.c:1008
#10 0x00007f7fa4e7c026 in thread_run (arg=0x1ad9040) at
/home/chug/git/qpid-dispatch/src/server.c:1122
#11 0x00007f7fa4e7d3a4 in qd_server_run (qd=0x1abc650) at
/home/chug/git/qpid-dispatch/src/server.c:1484
#12 0x0000000000402604 in main_process (config_path=0x7ffc59461679 "A.conf",
python_pkgdir=0x404290 "/opt/local/lib/qpid-dispatch/python", test_hooks=false,
fd=2)
at /home/chug/git/qpid-dispatch/router/src/main.c:113
#13 0x00000000004030d5 in main (argc=3, argv=0x7ffc5945fa38) at
/home/chug/git/qpid-dispatch/router/src/main.c:367
(gdb) up 4
#4 0x00007f7fa4e616aa in qdr_link_process_deliveries (core=0x1cd4360,
link=0x1da5ad0, credit=2637) at
/home/chug/git/qpid-dispatch/src/router_core/transfer.c:231
231 DEQ_REMOVE_HEAD(link->undelivered);
$4 = {prev = 0x0, next = 0x7f7f8809b010, core = 0x1cd4360, identity = 1,
user_context = 0x1d7c7a0,
edge_context = 0x0, conn = 0x1d92050, link_type = QD_LINK_ENDPOINT,
link_direction = QD_OUTGOING,
work_list = {head = 0x7f7f90162590, tail = 0x7f7f90162590, scratch =
0x7f7f90047990, size = 1},
name = 0x1d7c1a0 "tcp.egress.out", disambiguated_name = 0x0, terminus_addr =
0x0, attach_count = 1,
detach_count = 0, owning_addr = 0x7f7f90014d90, phase = 0, connected_link =
0x0, core_endpoint = 0x0, ref = {
0x7f7f9002a2d0, 0x0, 0x7f7f9002a410, 0x7f7f900e4d10}, auto_link = 0x0,
undelivered = {head = 0x0,
tail = 0x0, scratch = 0x7f7f880d4350, size = 0}, unsettled = {head = 0x0,
tail = 0x0, scratch = 0x0,
size = 0}, settled = {head = 0x0, tail = 0x0, scratch = 0x0, size = 0},
updated_deliveries = {head = 0x0,
tail = 0x0, scratch = 0x0, size = 0}, oper_status = QDR_LINK_OPER_UP,
capacity = 250, credit_to_core = 3,
credit_pending = 250, credit_stored = 10, credit_reported = 10,
zero_credit_time = 0,
reported_as_blocked = false, admin_enabled = true, strip_annotations_in =
false,
strip_annotations_out = false, drain_mode = false, stalled_outbound = false,
detach_received = false,
detach_send_done = false, edge = false, processing = true, ready_to_free =
false, fallback = false,
streaming = false, in_streaming_pool = false, terminus_survives_disconnect =
false, no_route = false,
strip_prefix = 0x0, insert_prefix = 0x0, total_deliveries = 7,
presettled_deliveries = 0,
dropped_presettled_deliveries = 0, accepted_deliveries = 0,
rejected_deliveries = 0,
released_deliveries = 0, modified_deliveries = 0, deliveries_delayed_1sec =
0, deliveries_delayed_10sec = 0,
deliveries_stuck = 0, settled_deliveries = {0, 0, 0, 0, 0}, ingress_histogram
= 0x1ba88d0,
priority = 0 '\000', rate_cursor = 0 '\000', core_ticks = 0, conn_id = 1,
prevSTREAMING_POOL = 0x0,
nextSTREAMING_POOL = 0x0}
(gdb)
{code}
> TCP adaptor Assertion `(link->undelivered).head' failed.
> --------------------------------------------------------
>
> Key: DISPATCH-1927
> URL: https://issues.apache.org/jira/browse/DISPATCH-1927
> Project: Qpid Dispatch
> Issue Type: Bug
> Components: Protocol Adaptors, Router Node
> Environment: Fedora 31
> Reporter: Charles E. Rolke
> Priority: Major
>
> Test setup:
> * Proton git: branch master @ 5e7d7af8f
> * Dispatch git: branch master @ ec39e5e
> * Debug build
> * Running the configuration described in DISPATCH-1777
> Seven clients run TCP_echo_client continuously against the server:
> * Four run --size 0
> * One runs --size 100
> * One runs --size 100000
> * One runs --size 1000000
> Eventually the router running the tcpConnector fails with the assertion.
> Tail of the router log:
> {code:java}
> Two router setup with a TCP listener on one router and a TCP connector on the
> other router.
> This is the log from the second tcpConnector router
> Here comes a message from core to TCP needing a new connection to egress
> server
> 2021-01-22 09:37:57.979054 -0500 TCP_ADAPTOR (debug) [C1] on_activate
> 2021-01-22 09:37:57.979594 -0500 TCP_ADAPTOR (debug) [C1][L1] qdr_tcp_push
> 2021-01-22 09:37:57.979619 -0500 TCP_ADAPTOR (debug) [C1][L1][D46793]
> qdr_tcp_deliver Delivery event
> 2021-01-22 09:37:57.979641 -0500 TCP_ADAPTOR (debug) [C1][L1][D46793]
> tcp_adaptor initiating egress connection
> 2021-01-22 09:37:57.979671 -0500 TCP_ADAPTOR (info) [C11768] Connecting to:
> 127.0.0.1:9090
> 2021-01-22 09:37:57.979789 -0500 TCP_ADAPTOR (debug) [C11767]
> qdr_tcp_activate: waking raw connection
> 2021-01-22 09:37:57.979974 -0500 TCP_ADAPTOR (info) [C11768]
> PN_RAW_CONNECTION_CONNECTED Egress connected to 127.0.0.1:9090
> 2021-01-22 09:37:57.980003 -0500 TCP_ADAPTOR (info) [C11768] Opening
> server-side core connection 127.0.0.1:9090
> 2021-01-22 09:37:57.980057 -0500 ROUTER_CORE (info) [C11768] Connection
> Opened: dir=out host=127.0.0.1:9090 vhost= encrypted=no auth=no user=
> container_id=TcpAdaptor props=
> Delivery is handed off to new connection/link
> 2021-01-22 09:37:57.980153 -0500 TCP_ADAPTOR (debug) [C1][L1][D46793]
> initial_delivery ownership passed to [C11768][L23564][D46793]
> 2021-01-22 09:37:57.980191 -0500 TCP_ADAPTOR (debug) [C11768]
> PN_RAW_CONNECTION_NEED_WRITE_BUFFERS
> 2021-01-22 09:37:57.980215 -0500 TCP_ADAPTOR (debug) [C11768]
> PN_RAW_CONNECTION_NEED_READ_BUFFERS
> 2021-01-22 09:37:57.980243 -0500 TCP_ADAPTOR (debug) [C11768][L23564] Waiting
> for credit to initiate message
> POOF! link_process_delivery has no deliveries yet.
> This is happening too soon. Probably.
> qdrouterd: /home/chug/git/qpid-dispatch/src/router_core/transfer.c:231:
> qdr_link_process_deliveries: Assertion `(link->undelivered).head' failed.
> from gdb: delivery_id = 46793, link_id = 23564, conn_id = 11768 or:
> [C11768][L23564][D46793] for failed process
> Moments later the rest of the connection/link setup happens
> 2021-01-22 09:37:57.980312 -0500 ROUTER_CORE (info) [C11768][L23564] Link
> attached: dir=out source={foo expire:link} target={<none> expire:link}
> 2021-01-22 09:37:57.980332 -0500 TCP_ADAPTOR (debug) [C11768]
> qdr_tcp_activate: waking raw connection
> 2021-01-22 09:37:57.980356 -0500 TCP_ADAPTOR (debug) [C11767]
> qdr_tcp_activate: waking raw connection
> 2021-01-22 09:37:57.980386 -0500 TCP_ADAPTOR (debug) [C11768]
> PN_RAW_CONNECTION_WAKE
> 2021-01-22 09:37:57.980404 -0500 TCP_ADAPTOR (debug) [C11768][L23564]
> qdr_tcp_second_attach
> 2021-01-22 09:37:57.980430 -0500 TCP_ADAPTOR (debug) [C11768][L23564]
> qdr_tcp_get_credit: NOOP
> 2021-01-22 09:37:57.980453 -0500 TCP_ADAPTOR (debug) [C11768][L23564]
> qdr_tcp_push
> 2021-01-22 09:37:57.980471 -0500 TCP_ADAPTOR (debug) [C11768][L23564][D46793]
> qdr_tcp_deliver Delivery event
> 2021-01-22 09:37:57.980563 -0500 TCP_ADAPTOR (debug) [C11768][L23565] Create
> Link to amqp:/_topo/0/router-b/temp.tsMXbvjL9M01_l8
> 2021-01-22 09:37:57.980591 -0500 TCP_ADAPTOR (debug) [C11768][L23564] Waiting
> for credit to initiate message
> 2021-01-22 09:37:57.980621 -0500 TCP_ADAPTOR (debug) [C11768] Writing 2040
> bytes
> 2021-01-22 09:37:57.980640 -0500 TCP_ADAPTOR (debug) [C11768] Writing 2048
> bytes
> 2021-01-22 09:37:57.980657 -0500 TCP_ADAPTOR (debug) [C11768] Writing 2048
> bytes
> 2021-01-22 09:37:57.980674 -0500 TCP_ADAPTOR (debug) [C11768] Writing 1104
> bytes
> 2021-01-22 09:37:57.980691 -0500 TCP_ADAPTOR (debug) [C11768][L23564]
> qdr_tcp_get_credit: NOOP
> Aborted (core dumped){code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
