Jiri Daněk created DISPATCH-1937:
------------------------------------
Summary: system_tests_grpc failure due to leak from
handle_outgoing_http -> qd_iterator_copy
Key: DISPATCH-1937
URL: https://issues.apache.org/jira/browse/DISPATCH-1937
Project: Qpid Dispatch
Issue Type: Bug
Components: Tests
Affects Versions: 1.15.0
Reporter: Jiri Daněk
https://travis-ci.com/github/apache/qpid-dispatch/jobs/477987433#L7257
{noformat}
test 71
Start 71: system_tests_grpc
71: Test command: /usr/bin/python
"/home/travis/build/apache/qpid-dispatch/build/tests/run.py" "-m" "unittest"
"-v" "system_tests_grpc"
71: Test timeout computed to be: 600
71: test_grpc_01_unary (system_tests_grpc.GrpcServiceMethodsTest)
71: Validates unary request and response message ... ok
71: test_grpc_02_bidirectional_stream (system_tests_grpc.GrpcServiceMethodsTest)
71: Validates bidirectional streaming request and response messages ... ok
71: test_grpc_03_server_stream (system_tests_grpc.GrpcServiceMethodsTest)
71: Validates server streaming response messages ... ok
71: test_grpc_04_client_stream (system_tests_grpc.GrpcServiceMethodsTest)
71: Validates client streaming request messages ... ok
71:
71: Router QDR debug dump file:
71: >>>>
71: alloc.c: Items of type 'qd_buffer_t' remain allocated at shutdown: 5
(SUPPRESSED)
71: alloc.c: Items of type 'qd_message_t' remain allocated at shutdown: 2
(SUPPRESSED)
71: alloc.c: Items of type 'qd_message_content_t' remain allocated at shutdown:
2 (SUPPRESSED)
71: alloc.c: Items of type 'qdr_link_work_t' remain allocated at shutdown: 1
(SUPPRESSED)
71: alloc.c: Items of type 'qdr_delivery_t' remain allocated at shutdown: 2
(SUPPRESSED)
71:
71: <<<<
71:
71: Router QDR output file:
71: >>>>
71:
71: =================================================================
71: ==19371==ERROR: LeakSanitizer: detected memory leaks
71:
71: Direct leak of 325 byte(s) in 26 object(s) allocated from:
71: #0 0x494bcd in malloc
(/home/travis/build/apache/qpid-dispatch/build/router/qdrouterd+0x494bcd)
71: #1 0x7f52fd730ce9 in qd_iterator_copy
/home/travis/build/apache/qpid-dispatch/src/iterator.c:984:27
71: #2 0x7f52fd90dbd8 in handle_outgoing_http
/home/travis/build/apache/qpid-dispatch/src/adaptors/http2/http2_adaptor.c:1729:49
71: #3 0x7f52fd91fb03 in qdr_http_deliver
/home/travis/build/apache/qpid-dispatch/src/adaptors/http2/http2_adaptor.c:1861:25
71: #4 0x7f52fd8761e0 in qdr_link_process_deliveries
/home/travis/build/apache/qpid-dispatch/src/router_core/transfer.c:176:32
71: #5 0x7f52fd7bf3b7 in qdr_connection_process
/home/travis/build/apache/qpid-dispatch/src/router_core/connections.c:412:37
71: #6 0x7f52fd904f9c in handle_connection_event
/home/travis/build/apache/qpid-dispatch/src/adaptors/http2/http2_adaptor.c:2293:16
71: #7 0x7f52fd8e47cd in handle
/home/travis/build/apache/qpid-dispatch/src/server.c
71: #8 0x7f52fd8de0bf in thread_run
/home/travis/build/apache/qpid-dispatch/src/server.c:1122:23
71: #9 0x7f52fd354608 in start_thread
(/lib/x86_64-linux-gnu/libpthread.so.0+0x9608)
71:
71: Direct leak of 39 byte(s) in 26 object(s) allocated from:
71: #0 0x494bcd in malloc
(/home/travis/build/apache/qpid-dispatch/build/router/qdrouterd+0x494bcd)
71: #1 0x7f52fd730ce9 in qd_iterator_copy
/home/travis/build/apache/qpid-dispatch/src/iterator.c:984:27
71: #2 0x7f52fd90dc0f in handle_outgoing_http
/home/travis/build/apache/qpid-dispatch/src/adaptors/http2/http2_adaptor.c:1730:50
71: #3 0x7f52fd91fb03 in qdr_http_deliver
/home/travis/build/apache/qpid-dispatch/src/adaptors/http2/http2_adaptor.c:1861:25
71: #4 0x7f52fd8761e0 in qdr_link_process_deliveries
/home/travis/build/apache/qpid-dispatch/src/router_core/transfer.c:176:32
71: #5 0x7f52fd7bf3b7 in qdr_connection_process
/home/travis/build/apache/qpid-dispatch/src/router_core/connections.c:412:37
71: #6 0x7f52fd904f9c in handle_connection_event
/home/travis/build/apache/qpid-dispatch/src/adaptors/http2/http2_adaptor.c:2293:16
71: #7 0x7f52fd8e47cd in handle
/home/travis/build/apache/qpid-dispatch/src/server.c
71: #8 0x7f52fd8de0bf in thread_run
/home/travis/build/apache/qpid-dispatch/src/server.c:1122:23
71: #9 0x7f52fd354608 in start_thread
(/lib/x86_64-linux-gnu/libpthread.so.0+0x9608)
71:
71: -----------------------------------------------------
71: Suppressions used:
71: count bytes template
71: 2 128 sys_mutex
71: 1 56 qdr_core_subscribe
71: 536 871916 *libpython*
71: 20 3968 *libqpid-proton*
71: -----------------------------------------------------
71:
71: SUMMARY: AddressSanitizer: 364 byte(s) leaked in 52 allocation(s).
71:
71: <<<<
71:
71: Router QDR command file:
71: >>>>
71: qdrouterd -c grpc-test-router.conf -I
/home/travis/build/apache/qpid-dispatch/python
71: pid=19371
71:
71: <<<<
71:
71: Router QDR log file tail:
71: >>>>
71: 2021-01-29 05:06:37.239038 +0000 ROUTER_CORE (debug) [C3][L51][D52]
Delivery decref_CT: rc:1 qdr_delivery_anycast_update CT - dlv removed from
unsettled
(/home/travis/build/apache/qpid-dispatch/src/router_core/delivery.c:640)
71: 2021-01-29 05:06:37.239060 +0000 ROUTER_CORE (debug) [C1][L53][D51]
Delivery decref_CT: rc:3 qdr_delivery_anycast_update_CT - peer removed from
unsettled
(/home/travis/build/apache/qpid-dispatch/src/router_core/delivery.c:640)
71: 2021-01-29 05:06:37.239079 +0000 ROUTER_CORE (debug) [C1][L53][D51]
Delivery decref_CT: rc:2 qdr_delivery_anycast_update_CT - allow free of peer
(/home/travis/build/apache/qpid-dispatch/src/router_core/delivery.c:640)
71: 2021-01-29 05:06:37.239114 +0000 ROUTER_CORE (debug) [C3][L51][D52]
Delivery decref_CT: rc:0 qdr_update_delivery_CT - remove from action
(/home/travis/build/apache/qpid-dispatch/src/router_core/delivery.c:640)
71: 2021-01-29 05:06:37.239159 +0000 ROUTER_CORE (debug) [C3][L50][D49]
Delivery decref: rc:1 HTTP2 adaptor - qdr_http_delivery_update
(/home/travis/build/apache/qpid-dispatch/src/router_core/delivery.c:147)
71: 2021-01-29 05:06:37.239195 +0000 HTTP_ADAPTOR (trace) [C3][S25]
qdr_http_delivery_update, stream_data->status == QD_STREAM_FULLY_CLOSED,
calling free_http2_stream_data, send_complete(dlv)=1
(/home/travis/build/apache/qpid-dispatch/src/adaptors/http2/http2_adaptor.c:1449)
71: 2021-01-29 05:06:37.239221 +0000 ROUTER_CORE (debug) [C3][L51][D52]
Delivery outcome : is accepted (0x24)
(/home/travis/build/apache/qpid-dispatch/src/router_core/delivery.c:395)
71: 2021-01-29 05:06:37.239247 +0000 HTTP_ADAPTOR (trace) [C3][S25] Freeing
stream_data in free_http2_stream_data (61500001a4d0)
(/home/travis/build/apache/qpid-dispatch/src/adaptors/http2/http2_adaptor.c:308)
71: 2021-01-29 05:06:37.239272 +0000 ROUTER_CORE (debug) [C3][L50][D49]
Delivery decref: rc:0 qdr_connection_process - remove from updated list
(/home/travis/build/apache/qpid-dispatch/src/router_core/delivery.c:147)
71: 2021-01-29 05:06:37.239307 +0000 HTTP_ADAPTOR (trace) [C1] Activation
triggered, calling pn_raw_connection_wake()
(/home/travis/build/apache/qpid-dispatch/src/adaptors/http2/http2_adaptor.c:1524)
71: 2021-01-29 05:06:37.239331 +0000 HTTP_ADAPTOR (trace) [C3]
PN_RAW_CONNECTION_WAKE Wake-up
(/home/travis/build/apache/qpid-dispatch/src/adaptors/http2/http2_adaptor.c:2292)
71: 2021-01-29 05:06:37.239353 +0000 HTTP_ADAPTOR (trace) [C1]
PN_RAW_CONNECTION_WAKE Wake-up
(/home/travis/build/apache/qpid-dispatch/src/adaptors/http2/http2_adaptor.c:2292)
71: 2021-01-29 05:06:37.239372 +0000 HTTP_ADAPTOR (trace) [C1][S25]
qdr_http_delivery_update, stream_data->in_dlv == dlv
(/home/travis/build/apache/qpid-dispatch/src/adaptors/http2/http2_adaptor.c:1424)
71: 2021-01-29 05:06:37.239385 +0000 HTTP_ADAPTOR (trace) [C1][S25]
qdr_http_delivery_update, stream_data->status == QD_STREAM_FULLY_CLOSED
(/home/travis/build/apache/qpid-dispatch/src/adaptors/http2/http2_adaptor.c:1431)
71: 2021-01-29 05:06:37.239902 +0000 HTTP_ADAPTOR (trace) [C3]
PN_RAW_CONNECTION_WRITTEN Wrote 127 bytes, DEQ_SIZE(session_data->buffs) = 0
(/home/travis/build/apache/qpid-dispatch/src/adaptors/http2/http2_adaptor.c:2329)
71: 2021-01-29 05:06:37.239941 +0000 HTTP_ADAPTOR (trace) [C3] In
restart_streams, no stream_data, returning
(/home/travis/build/apache/qpid-dispatch/src/adaptors/http2/http2_adaptor.c:2013)
71: 2021-01-29 05:06:37.239963 +0000 ROUTER_CORE (trace) Core action
'link_detach'
(/home/travis/build/apache/qpid-dispatch/src/router_core/router_core_thread.c:238)
71: 2021-01-29 05:06:37.240010 +0000 ROUTER_CORE (trace) Core action
'link_detach'
(/home/travis/build/apache/qpid-dispatch/src/router_core/router_core_thread.c:238)
71: 2021-01-29 05:06:37.240031 +0000 ROUTER_CORE (trace) Core action
'qdr_connection_process - remove from updated list'
(/home/travis/build/apache/qpid-dispatch/src/router_core/router_core_thread.c:238)
71: 2021-01-29 05:06:37.240049 +0000 ROUTER_CORE (debug) [C3][L50][D49]
Delivery outcome : is accepted (0x24)
(/home/travis/build/apache/qpid-dispatch/src/router_core/delivery.c:395)
71: 2021-01-29 05:06:37.240083 +0000 HTTP_ADAPTOR (trace) [C3] Activation
triggered, calling pn_raw_connection_wake()
(/home/travis/build/apache/qpid-dispatch/src/adaptors/http2/http2_adaptor.c:1524)
71: 2021-01-29 05:06:37.240115 +0000 HTTP_ADAPTOR (trace) [C1][S25]
qdr_http_delivery_update, send_complete=true
(/home/travis/build/apache/qpid-dispatch/src/adaptors/http2/http2_adaptor.c:1439)
71: 2021-01-29 05:06:37.240143 +0000 HTTP_ADAPTOR (trace) [C3]
PN_RAW_CONNECTION_NEED_WRITE_BUFFERS Need write buffers
(/home/travis/build/apache/qpid-dispatch/src/adaptors/http2/http2_adaptor.c:2280)
71: 2021-01-29 05:06:37.240169 +0000 ROUTER_CORE (debug) [C1][L53][D51]
Delivery decref: rc:1 HTTP2 adaptor - qdr_http_delivery_update
(/home/travis/build/apache/qpid-dispatch/src/router_core/delivery.c:147)
71: 2021-01-29 05:06:37.240193 +0000 HTTP_ADAPTOR (trace) [C3]
PN_RAW_CONNECTION_WAKE Wake-up
(/home/travis/build/apache/qpid-dispatch/src/adaptors/http2/http2_adaptor.c:2292)
71: 2021-01-29 05:06:37.240226 +0000 HTTP_ADAPTOR (trace) [C1][S25]
qdr_http_delivery_update, stream_data->status == QD_STREAM_FULLY_CLOSED,
calling free_http2_stream_data, send_complete(dlv)=1
(/home/travis/build/apache/qpid-dispatch/src/adaptors/http2/http2_adaptor.c:1449)
71: 2021-01-29 05:06:37.240262 +0000 HTTP_ADAPTOR (trace) [C1][S25] Freeing
stream_data in free_http2_stream_data (61500000b750)
(/home/travis/build/apache/qpid-dispatch/src/adaptors/http2/http2_adaptor.c:308)
71: 2021-01-29 05:06:37.240290 +0000 ROUTER_CORE (debug) [C1][L52][D50]
Delivery decref: rc:0 HTTP2 adaptor out_dlv - free_http2_stream_data
(/home/travis/build/apache/qpid-dispatch/src/router_core/delivery.c:147)
71: 2021-01-29 05:06:37.240319 +0000 ROUTER_CORE (debug) [C1][L53][D51]
Delivery decref: rc:0 qdr_connection_process - remove from updated list
(/home/travis/build/apache/qpid-dispatch/src/router_core/delivery.c:147)
71: 2021-01-29 05:06:37.240413 +0000 SERVER (notice) Shut Down
(/home/travis/build/apache/qpid-dispatch/src/server.c:1493)
71: 2021-01-29 05:06:37.240537 +0000 CONTAINER (trace) Default node removed
(/home/travis/build/apache/qpid-dispatch/src/container.c:848)
71: 2021-01-29 05:06:37.240562 +0000 ROUTER_CORE (trace) Core action
'link_detach_sent' (discard)
(/home/travis/build/apache/qpid-dispatch/src/router_core/router_core_thread.c:238)
71: 2021-01-29 05:06:37.240573 +0000 ROUTER_CORE (trace) Core action
'link_detach_sent' (discard)
(/home/travis/build/apache/qpid-dispatch/src/router_core/router_core_thread.c:238)
71: 2021-01-29 05:06:37.240580 +0000 ROUTER_CORE (trace) Core action
'link_detach' (discard)
(/home/travis/build/apache/qpid-dispatch/src/router_core/router_core_thread.c:238)
71: 2021-01-29 05:06:37.240588 +0000 ROUTER_CORE (trace) Core action
'link_detach' (discard)
(/home/travis/build/apache/qpid-dispatch/src/router_core/router_core_thread.c:238)
71: 2021-01-29 05:06:37.240595 +0000 ROUTER_CORE (trace) Core action 'HTTP2
adaptor out_dlv - free_http2_stream_data' (discard)
(/home/travis/build/apache/qpid-dispatch/src/router_core/router_core_thread.c:238)
71: 2021-01-29 05:06:37.240602 +0000 ROUTER_CORE (trace) Core action
'qdr_connection_process - remove from updated list' (discard)
(/home/travis/build/apache/qpid-dispatch/src/router_core/router_core_thread.c:238)
71: 2021-01-29 05:06:37.240641 +0000 ROUTER_CORE (info) Router Core thread
exited
(/home/travis/build/apache/qpid-dispatch/src/router_core/router_core_thread.c:259)
71: 2021-01-29 05:06:37.240746 +0000 HTTP_ADAPTOR (trace) Shutting down HTTP2
Protocol adaptor
(/home/travis/build/apache/qpid-dispatch/src/adaptors/http2/http2_adaptor.c:2438)
71: 2021-01-29 05:06:37.240806 +0000 HTTP_ADAPTOR (info) Deleted HttpListener
for examples, 127.0.0.1:25162
(/home/travis/build/apache/qpid-dispatch/src/adaptors/http2/http2_adaptor.c:2400)
71: 2021-01-29 05:06:37.240842 +0000 HTTP_ADAPTOR (info) Deleted HttpConnector
for examples, 127.0.0.1:25161
(/home/travis/build/apache/qpid-dispatch/src/adaptors/http2/http2_adaptor.c:2372)
71: 2021-01-29 05:06:37.240885 +0000 HTTP_ADAPTOR (info) [C1] Freeing
stream_data (stream_dispatcher, qdr_http2_adaptor_final) (61500000b9d0)
(/home/travis/build/apache/qpid-dispatch/src/adaptors/http2/http2_adaptor.c:2460)
71: 2021-01-29 05:06:37.240903 +0000 HTTP_ADAPTOR (info) [C1] Freeing http2
connection (calling free_qdr_http2_connection)
(/home/travis/build/apache/qpid-dispatch/src/adaptors/http2/http2_adaptor.c:2464)
71: 2021-01-29 05:06:37.240973 +0000 HTTP_ADAPTOR (trace) [C1] Freeing http2
connection in free_qdr_http2_connection
(/home/travis/build/apache/qpid-dispatch/src/adaptors/http2/http2_adaptor.c:381)
71: 2021-01-29 05:06:37.240985 +0000 HTTP_ADAPTOR (info) [C3] Freeing http2
connection (calling free_qdr_http2_connection)
(/home/travis/build/apache/qpid-dispatch/src/adaptors/http2/http2_adaptor.c:2464)
71: 2021-01-29 05:06:37.241038 +0000 HTTP_ADAPTOR (trace) [C3] Freeing http2
connection in free_qdr_http2_connection
(/home/travis/build/apache/qpid-dispatch/src/adaptors/http2/http2_adaptor.c:381)
71: 2021-01-29 05:06:37.241069 +0000 TCP_ADAPTOR (info) Shutting down TCP
protocol adaptor
(/home/travis/build/apache/qpid-dispatch/src/adaptors/tcp_adaptor.c:1249)
71: 2021-01-29 05:06:37.241321 +0000 ROUTER_CORE (info) Finalizing core module:
streaming_link_scrubber
(/home/travis/build/apache/qpid-dispatch/src/router_core/router_core_thread.c:138)
71: 2021-01-29 05:06:37.241346 +0000 ROUTER_CORE (info) Finalizing core module:
stuck_delivery_detection
(/home/travis/build/apache/qpid-dispatch/src/router_core/router_core_thread.c:138)
71: 2021-01-29 05:06:37.241365 +0000 ROUTER_CORE (info) Finalizing core module:
address_lookup_client
(/home/travis/build/apache/qpid-dispatch/src/router_core/router_core_thread.c:138)
71:
71: <<<<
71: ERROR
71:
71: ======================================================================
71: ERROR: tearDownClass (system_tests_grpc.GrpcServiceMethodsTest)
71: ----------------------------------------------------------------------
71: Traceback (most recent call last):
71: File
"/home/travis/build/apache/qpid-dispatch/tests/system_tests_grpc.py", line 134,
in tearDownClass
71: super(GrpcServiceMethodsTest, cls).tearDownClass()
71: File "/home/travis/build/apache/qpid-dispatch/tests/system_test.py", line
839, in tearDownClass
71: cls.tester.teardown()
71: File "/home/travis/build/apache/qpid-dispatch/tests/system_test.py", line
782, in teardown
71: raise RuntimeError("Errors during teardown: \n\n%s" %
"\n\n".join([str(e) for e in errors]))
71: RuntimeError: Errors during teardown:
71:
71: Process 19371 error: exit code 1, expected 0
71: qdrouterd -c grpc-test-router.conf -I
/home/travis/build/apache/qpid-dispatch/python
71:
/home/travis/build/apache/qpid-dispatch/build/tests/system_test.dir/system_tests_grpc/GrpcServiceMethodsTest/setUpClass/grpc-test-router-1.cmd
71: >>>>
71:
71: =================================================================
71: ==19371==ERROR: LeakSanitizer: detected memory leaks
71:
71: Direct leak of 325 byte(s) in 26 object(s) allocated from:
71: #0 0x494bcd in malloc
(/home/travis/build/apache/qpid-dispatch/build/router/qdrouterd+0x494bcd)
71: #1 0x7f52fd730ce9 in qd_iterator_copy
/home/travis/build/apache/qpid-dispatch/src/iterator.c:984:27
71: #2 0x7f52fd90dbd8 in handle_outgoing_http
/home/travis/build/apache/qpid-dispatch/src/adaptors/http2/http2_adaptor.c:1729:49
71: #3 0x7f52fd91fb03 in qdr_http_deliver
/home/travis/build/apache/qpid-dispatch/src/adaptors/http2/http2_adaptor.c:1861:25
71: #4 0x7f52fd8761e0 in qdr_link_process_deliveries
/home/travis/build/apache/qpid-dispatch/src/router_core/transfer.c:176:32
71: #5 0x7f52fd7bf3b7 in qdr_connection_process
/home/travis/build/apache/qpid-dispatch/src/router_core/connections.c:412:37
71: #6 0x7f52fd904f9c in handle_connection_event
/home/travis/build/apache/qpid-dispatch/src/adaptors/http2/http2_adaptor.c:2293:16
71: #7 0x7f52fd8e47cd in handle
/home/travis/build/apache/qpid-dispatch/src/server.c
71: #8 0x7f52fd8de0bf in thread_run
/home/travis/build/apache/qpid-dispatch/src/server.c:1122:23
71: #9 0x7f52fd354608 in start_thread
(/lib/x86_64-linux-gnu/libpthread.so.0+0x9608)
71:
71: Direct leak of 39 byte(s) in 26 object(s) allocated from:
71: #0 0x494bcd in malloc
(/home/travis/build/apache/qpid-dispatch/build/router/qdrouterd+0x494bcd)
71: #1 0x7f52fd730ce9 in qd_iterator_copy
/home/travis/build/apache/qpid-dispatch/src/iterator.c:984:27
71: #2 0x7f52fd90dc0f in handle_outgoing_http
/home/travis/build/apache/qpid-dispatch/src/adaptors/http2/http2_adaptor.c:1730:50
71: #3 0x7f52fd91fb03 in qdr_http_deliver
/home/travis/build/apache/qpid-dispatch/src/adaptors/http2/http2_adaptor.c:1861:25
71: #4 0x7f52fd8761e0 in qdr_link_process_deliveries
/home/travis/build/apache/qpid-dispatch/src/router_core/transfer.c:176:32
71: #5 0x7f52fd7bf3b7 in qdr_connection_process
/home/travis/build/apache/qpid-dispatch/src/router_core/connections.c:412:37
71: #6 0x7f52fd904f9c in handle_connection_event
/home/travis/build/apache/qpid-dispatch/src/adaptors/http2/http2_adaptor.c:2293:16
71: #7 0x7f52fd8e47cd in handle
/home/travis/build/apache/qpid-dispatch/src/server.c
71: #8 0x7f52fd8de0bf in thread_run
/home/travis/build/apache/qpid-dispatch/src/server.c:1122:23
71: #9 0x7f52fd354608 in start_thread
(/lib/x86_64-linux-gnu/libpthread.so.0+0x9608)
71:
71: -----------------------------------------------------
71: Suppressions used:
71: count bytes template
71: 2 128 sys_mutex
71: 1 56 qdr_core_subscribe
71: 536 871916 *libpython*
71: 20 3968 *libqpid-proton*
71: -----------------------------------------------------
71:
71: SUMMARY: AddressSanitizer: 364 byte(s) leaked in 52 allocation(s).
71: <<<<
71:
71: ----------------------------------------------------------------------
71: Ran 4 tests in 0.382s
71:
71: FAILED (errors=1)
71/73 Test #71: system_tests_grpc .................................***Failed
0.64 sec
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
