[
https://issues.apache.org/jira/browse/PROTON-1484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiri Danek updated PROTON-1484:
-------------------------------
Description:
I've noticed multiple qdrouterd test failures with seem to come from the same
place in proactor code.
SIGSEGV in qdrouterd tests; in pconnection_cleanup (pc=0x7f...) at
/main/qpid-proton/proton-c/src/proactor/epoll.c:645
stop_polling(&pc->timer.epoll_io, pc->psocket.proactor->epollfd);
Reproducibility: the issue seems nondeterministic
I am attaching logs and backtraces from two of them below.
{noformat}
test 14
Start 14: system_tests_autolinks
14: Test command: /usr/bin/python "/main/qpid-dispatch/build/tests/run.py" "-m"
"unittest" "-v" "system_tests_autolinks"
14: Test timeout computed to be: 1500
14: test_01_autolink_attach (system_tests_autolinks.AutolinkTest) ... FAIL
14: test_02_autolink_credit (system_tests_autolinks.AutolinkTest) ... ok
14: test_03_autolink_sender (system_tests_autolinks.AutolinkTest) ... ok
14: test_04_autolink_receiver (system_tests_autolinks.AutolinkTest) ... ok
14: test_05_inter_container_transfer (system_tests_autolinks.AutolinkTest) ...
ok
14: test_06_manage_autolinks (system_tests_autolinks.AutolinkTest) ...
ERROR:root:Connectivity to the peer container was lost
14: ERROR:root:Connectivity to the peer container was lost
14: ERROR:root:Connectivity to the peer container was lost
14: ERROR:root:Connectivity to the peer container was lost
14: ERROR:root:Connectivity to the peer container was lost
14: ok
14: test_07_autolink_attach_with_ext_addr (system_tests_autolinks.AutolinkTest)
... ERROR:root:proton:io: recv: Connection reset by peer
14: ERROR:root:proton:io: recv: Connection reset by peer
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: FAIL
14: test_08_autolink_sender_with_ext_addr (system_tests_autolinks.AutolinkTest)
... ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: FAIL
14: test_09_autolink_receiver_with_ext_addr
(system_tests_autolinks.AutolinkTest) ... ERROR:root:proton:io: recv:
Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: FAIL
14: ERROR
14:
14: ======================================================================
14: ERROR: tearDownClass (system_tests_autolinks.AutolinkTest)
14: ----------------------------------------------------------------------
14: Traceback (most recent call last):
14: File "/main/qpid-dispatch/tests/system_test.py", line 605, in
tearDownClass
14: cls.tester.teardown()
14: File "/main/qpid-dispatch/tests/system_test.py", line 543, in teardown
14: raise RuntimeError("Errors during teardown: \n\n%s" %
"\n\n".join([str(e) for e in errors]))
14: RuntimeError: Errors during teardown:
14:
14: Process 81 error: exit code -11, expected -1
14: qdrouterd -c test-router.conf -I /main/qpid-dispatch/python
14:
/main/qpid-dispatch/build/tests/system_test.dir/system_tests_autolinks/AutolinkTest/setUpClass/test-router-1.cmd
14: >>>>
14: <<<<
14:
14: ======================================================================
14: FAIL: test_01_autolink_attach (system_tests_autolinks.AutolinkTest)
14: ----------------------------------------------------------------------
14: Traceback (most recent call last):
14: File "/main/qpid-dispatch/tests/system_tests_autolinks.py", line 85, in
test_01_autolink_attach
14: self.assertEqual(None, test.error)
14: AssertionError: None != 'Timeout Expired: n_rx_attach=1 n_tx_attach=1'
14:
14: ======================================================================
14: FAIL: test_07_autolink_attach_with_ext_addr
(system_tests_autolinks.AutolinkTest)
14: ----------------------------------------------------------------------
14: Traceback (most recent call last):
14: File "/main/qpid-dispatch/tests/system_tests_autolinks.py", line 146, in
test_07_autolink_attach_with_ext_addr
14: self.assertEqual(None, test.error)
14: AssertionError: None != 'Timeout Expired: n_rx_attach=0 n_tx_attach=0'
14:
14: ======================================================================
14: FAIL: test_08_autolink_sender_with_ext_addr
(system_tests_autolinks.AutolinkTest)
14: ----------------------------------------------------------------------
14: Traceback (most recent call last):
14: File "/main/qpid-dispatch/tests/system_tests_autolinks.py", line 156, in
test_08_autolink_sender_with_ext_addr
14: self.assertEqual(None, test.error)
14: AssertionError: None != 'Timeout Expired: last_action=Connected route
container n_sent=0 n_received=0 n_settled=0'
14:
14: ======================================================================
14: FAIL: test_09_autolink_receiver_with_ext_addr
(system_tests_autolinks.AutolinkTest)
14: ----------------------------------------------------------------------
14: Traceback (most recent call last):
14: File "/main/qpid-dispatch/tests/system_tests_autolinks.py", line 166, in
test_09_autolink_receiver_with_ext_addr
14: self.assertEqual(None, test.error)
14: AssertionError: None != 'Timeout Expired: last_action=Connected route
container n_sent=0 n_received=0 n_settled=0'
14:
14: ----------------------------------------------------------------------
14: Ran 9 tests in 248.688s
14:
14: FAILED (failures=4, errors=1)
14/33 Test #14: system_tests_autolinks ....................***Failed 248.75 sec
{noformat}
{noformat}
root@9622b08a07af:/main/qpid-dispatch/build# gdb -c
/main/qpid-dispatch/build/tests/system_test.dir/system_tests_autolinks/AutolinkTest/setUpClass/core
`which qdrouterd`
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/qdrouterd...(no debugging symbols found)...done.
[New LWP 81]
[New LWP 84]
[New LWP 82]
[New LWP 83]
[New LWP 85]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `qdrouterd -c test-router.conf -I
/main/qpid-dispatch/python'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007f89a496f320 in pconnection_cleanup (pc=0x55e69fe2dec0) at
/main/qpid-proton/proton-c/src/proactor/epoll.c:645
645 stop_polling(&pc->timer.epoll_io, pc->psocket.proactor->epollfd);
[Current thread is 1 (Thread 0x7f89a5245080 (LWP 81))]
(gdb) thread apply all bt
Thread 5 (Thread 0x7f899b7fe700 (LWP 85)):
#0 0x00007f89a3afaf33 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
#1 0x00007f89a4971fcd in proactor_do_epoll (p=0x55e69fd5f1a0, can_block=true)
at /main/qpid-proton/proton-c/src/proactor/epoll.c:1634
#2 0x00007f89a4972185 in pn_proactor_wait (p=0x55e69fd5f1a0) at
/main/qpid-proton/proton-c/src/proactor/epoll.c:1680
#3 0x00007f89a4e13e7a in thread_run (arg=0x55e69fb27ec0) at
/main/qpid-dispatch/src/server.c:817
#4 0x00007f89a4755494 in start_thread (arg=0x7f899b7fe700) at
pthread_create.c:333
#5 0x00007f89a3afa93f in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:97
Thread 4 (Thread 0x7f89a0a94700 (LWP 83)):
#0 0x00007f89a3afaf33 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
#1 0x00007f89a4971fcd in proactor_do_epoll (p=0x55e69fd5f1a0, can_block=true)
at /main/qpid-proton/proton-c/src/proactor/epoll.c:1634
#2 0x00007f89a4972185 in pn_proactor_wait (p=0x55e69fd5f1a0) at
/main/qpid-proton/proton-c/src/proactor/epoll.c:1680
#3 0x00007f89a4e13e7a in thread_run (arg=0x55e69fb27ec0) at
/main/qpid-dispatch/src/server.c:817
#4 0x00007f89a4755494 in start_thread (arg=0x7f89a0a94700) at
pthread_create.c:333
#5 0x00007f89a3afa93f in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:97
Thread 3 (Thread 0x7f89a1295700 (LWP 82)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at
../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f89a4df19c0 in sys_cond_wait (cond=0x55e69fdcee80,
held_mutex=0x55e69fdcef40) at /main/qpid-dispatch/src/posix/threading.c:91
#2 0x00007f89a4e070fa in router_core_thread (arg=0x55e69fdceb70) at
/main/qpid-dispatch/src/router_core/router_core_thread.c:53
#3 0x00007f89a4755494 in start_thread (arg=0x7f89a1295700) at
pthread_create.c:333
#4 0x00007f89a3afa93f in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:97
Thread 2 (Thread 0x7f899bfff700 (LWP 84)):
#0 0x00007f89a3afaf33 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
#1 0x00007f89a4971fcd in proactor_do_epoll (p=0x55e69fd5f1a0, can_block=true)
at /main/qpid-proton/proton-c/src/proactor/epoll.c:1634
#2 0x00007f89a4972185 in pn_proactor_wait (p=0x55e69fd5f1a0) at
/main/qpid-proton/proton-c/src/proactor/epoll.c:1680
#3 0x00007f89a4e13e7a in thread_run (arg=0x55e69fb27ec0) at
/main/qpid-dispatch/src/server.c:817
#4 0x00007f89a4755494 in start_thread (arg=0x7f899bfff700) at
pthread_create.c:333
---Type <return> to continue, or q <return> to quit---
#5 0x00007f89a3afa93f in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:97
Thread 1 (Thread 0x7f89a5245080 (LWP 81)):
#0 0x00007f89a496f320 in pconnection_cleanup (pc=0x55e69fe2dec0) at
/main/qpid-proton/proton-c/src/proactor/epoll.c:645
#1 0x00007f89a496fac5 in pconnection_process (pc=0x55e69fe2dec0, events=0,
timeout=false, topup=false) at
/main/qpid-proton/proton-c/src/proactor/epoll.c:843
#2 0x00007f89a49720e5 in proactor_do_epoll (p=0x55e69fd5f1a0, can_block=true)
at /main/qpid-proton/proton-c/src/proactor/epoll.c:1662
#3 0x00007f89a4972185 in pn_proactor_wait (p=0x55e69fd5f1a0) at
/main/qpid-proton/proton-c/src/proactor/epoll.c:1680
#4 0x00007f89a4e13e7a in thread_run (arg=0x55e69fb27ec0) at
/main/qpid-dispatch/src/server.c:817
#5 0x00007f89a4e14987 in qd_server_run (qd=0x55e69facc010) at
/main/qpid-dispatch/src/server.c:1031
#6 0x000055e69f5a28d0 in ?? ()
#7 0x000055e69f5a325b in main ()
(gdb)
{noformat}
========================================
{noformat}
16: Test command: /usr/bin/python "/main/qpid-dispatch/build/tests/run.py" "-m"
"unittest" "-v" "system_tests_management"
16: Test timeout computed to be: 1500
16: test_bad_query (system_tests_management.ManagementTest)
16: Test that various badly formed queries get the proper response ... ok
16: test_connection (system_tests_management.ManagementTest)
16: Verify there is at least one connection ... ok
16: test_create_fixed_address (system_tests_management.ManagementTest) ... ok
16: test_create_listener (system_tests_management.ManagementTest)
16: Create a new listener on a running router ... ok
16: test_dummy (system_tests_management.ManagementTest)
16: Test all operations on the dummy test entity ... ok
16: test_entity_names (system_tests_management.ManagementTest) ... ok
16: test_get_attributes (system_tests_management.ManagementTest) ... ok
16: test_get_operations (system_tests_management.ManagementTest) ... ok
16: test_get_schema (system_tests_management.ManagementTest) ... ok
16: test_get_types (system_tests_management.ManagementTest) ... ok
16: test_link (system_tests_management.ManagementTest)
16: Verify we can find our own reply-to address in links ... ok
16: test_log (system_tests_management.ManagementTest)
16: Create, update and query log entities ... ok
16: test_query_attributes (system_tests_management.ManagementTest)
16: Query with attributes only ... ok
16: test_query_type (system_tests_management.ManagementTest)
16: Query with type only ... ok
16: test_query_type_attributes (system_tests_management.ManagementTest)
16: Query with type and attribute names ... ok
16: test_remote_node (system_tests_management.ManagementTest)
16: Test that we can access management info of remote nodes using
get_mgmt_nodes addresses ... ok
16: test_router (system_tests_management.ManagementTest)
16: Verify router counts match entity counts ... ok
16: test_router_node (system_tests_management.ManagementTest)
16: Test node entity in a trio of linked routers ... ERROR
16: test_standalone_no_inter_router (system_tests_management.ManagementTest)
16: Verify that we do not allow inter-router connectors or listeners in
standalone mode ... ERROR
16: ERROR
16:
16: ======================================================================
16: ERROR: test_router_node (system_tests_management.ManagementTest)
16: Test node entity in a trio of linked routers
16: ----------------------------------------------------------------------
16: Traceback (most recent call last):
16: File "/main/qpid-dispatch/tests/system_tests_management.py", line 112, in
setUp
16: self.node = self.cleanup(Node.connect(self.router.addresses[0]))
16: File "/main/qpid-dispatch/python/qpid_dispatch/management/client.py",
line 100, in connect
16: return Node(Node.connection(url, router, timeout, ssl_domain, sasl))
16: File "/main/qpid-dispatch/python/qpid_dispatch/management/client.py",
line 95, in connection
16: password=str(sasl.password) if sasl else None)
16: File "/usr/lib/python2.7/dist-packages/proton/utils.py", line 213, in
__init__
16: msg="Opening connection")
16: File "/usr/lib/python2.7/dist-packages/proton/utils.py", line 279, in wait
16: "Connection %s disconnected: %s" % (self.url, self.disconnected))
16: ConnectionException: Connection amqp://0.0.0.0:29636/$management
disconnected: Condition('proton:io', 'recv: Connection reset by peer')
16:
16: ======================================================================
16: ERROR: test_standalone_no_inter_router
(system_tests_management.ManagementTest)
16: Verify that we do not allow inter-router connectors or listeners in
standalone mode
16: ----------------------------------------------------------------------
16: Traceback (most recent call last):
16: File "/main/qpid-dispatch/tests/system_tests_management.py", line 112, in
setUp
16: self.node = self.cleanup(Node.connect(self.router.addresses[0]))
16: File "/main/qpid-dispatch/python/qpid_dispatch/management/client.py",
line 100, in connect
16: return Node(Node.connection(url, router, timeout, ssl_domain, sasl))
16: File "/main/qpid-dispatch/python/qpid_dispatch/management/client.py",
line 95, in connection
16: password=str(sasl.password) if sasl else None)
16: File "/usr/lib/python2.7/dist-packages/proton/utils.py", line 213, in
__init__
16: msg="Opening connection")
16: File "/usr/lib/python2.7/dist-packages/proton/utils.py", line 279, in wait
16: "Connection %s disconnected: %s" % (self.url, self.disconnected))
16: ConnectionException: Connection amqp://0.0.0.0:29636/$management
disconnected: Condition('proton:io', 'recv: Connection refused')
16:
16: ======================================================================
16: ERROR: tearDownClass (system_tests_management.ManagementTest)
16: ----------------------------------------------------------------------
16: Traceback (most recent call last):
16: File "/main/qpid-dispatch/tests/system_test.py", line 605, in
tearDownClass
16: cls.tester.teardown()
16: File "/main/qpid-dispatch/tests/system_test.py", line 543, in teardown
16: raise RuntimeError("Errors during teardown: \n\n%s" %
"\n\n".join([str(e) for e in errors]))
16: RuntimeError: Errors during teardown:
16:
16: Process 95 error: exit code -11, expected -1
16: qdrouterd -c solo.conf -I /main/qpid-dispatch/python
16:
/main/qpid-dispatch/build/tests/system_test.dir/system_tests_management/ManagementTest/setUpClass/solo-1.cmd
16: >>>>
16: <<<<
16:
16: ----------------------------------------------------------------------
16: Ran 19 tests in 8.165s
16:
16: FAILED (errors=3)
16/33 Test #16: system_tests_management ...................***Failed 8.24 sec
{noformat}
{noformat}
root@9622b08a07af:/main/qpid-dispatch/build# gdb -c
/main/qpid-dispatch/build/tests/system_test.dir/system_tests_management/ManagementTest/setUpClass/core
`which qdrouterd`
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/qdrouterd...(no debugging symbols found)...done.
[New LWP 104]
[New LWP 100]
[New LWP 105]
[New LWP 95]
[New LWP 106]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `qdrouterd -c solo.conf -I /main/qpid-dispatch/python'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007ff2b9320320 in pconnection_cleanup (pc=0x7ff2a80af460) at
/main/qpid-proton/proton-c/src/proactor/epoll.c:645
645 stop_polling(&pc->timer.epoll_io, pc->psocket.proactor->epollfd);
[Current thread is 1 (Thread 0x7ff2b5445700 (LWP 104))]
(gdb) thread apply all bt
Thread 5 (Thread 0x7ff2affff700 (LWP 106)):
#0 0x00007ff2b84abf33 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
#1 0x00007ff2b9322fcd in proactor_do_epoll (p=0x55bda0cc1f00, can_block=true)
at /main/qpid-proton/proton-c/src/proactor/epoll.c:1634
#2 0x00007ff2b9323185 in pn_proactor_wait (p=0x55bda0cc1f00) at
/main/qpid-proton/proton-c/src/proactor/epoll.c:1680
#3 0x00007ff2b97c4e7a in thread_run (arg=0x55bda0cc1e00) at
/main/qpid-dispatch/src/server.c:817
#4 0x00007ff2b9106494 in start_thread (arg=0x7ff2affff700) at
pthread_create.c:333
#5 0x00007ff2b84ab93f in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:97
Thread 4 (Thread 0x7ff2b9bf6080 (LWP 95)):
#0 0x00007ff2b84abf33 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
#1 0x00007ff2b9322fcd in proactor_do_epoll (p=0x55bda0cc1f00, can_block=true)
at /main/qpid-proton/proton-c/src/proactor/epoll.c:1634
#2 0x00007ff2b9323185 in pn_proactor_wait (p=0x55bda0cc1f00) at
/main/qpid-proton/proton-c/src/proactor/epoll.c:1680
#3 0x00007ff2b97c4e7a in thread_run (arg=0x55bda0cc1e00) at
/main/qpid-dispatch/src/server.c:817
#4 0x00007ff2b97c5987 in qd_server_run (qd=0x55bda0a36010) at
/main/qpid-dispatch/src/server.c:1031
#5 0x000055bda002f8d0 in ?? ()
#6 0x000055bda003025b in main ()
Thread 3 (Thread 0x7ff2b4c44700 (LWP 105)):
#0 0x00007ff2b84abf33 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
#1 0x00007ff2b9322fcd in proactor_do_epoll (p=0x55bda0cc1f00, can_block=true)
at /main/qpid-proton/proton-c/src/proactor/epoll.c:1634
#2 0x00007ff2b9323185 in pn_proactor_wait (p=0x55bda0cc1f00) at
/main/qpid-proton/proton-c/src/proactor/epoll.c:1680
#3 0x00007ff2b97c4e7a in thread_run (arg=0x55bda0cc1e00) at
/main/qpid-dispatch/src/server.c:817
#4 0x00007ff2b9106494 in start_thread (arg=0x7ff2b4c44700) at
pthread_create.c:333
#5 0x00007ff2b84ab93f in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:97
Thread 2 (Thread 0x7ff2b5c46700 (LWP 100)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at
../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007ff2b97a29c0 in sys_cond_wait (cond=0x55bda0d38e00,
held_mutex=0x55bda0d38ec0) at /main/qpid-dispatch/src/posix/threading.c:91
#2 0x00007ff2b97b80fa in router_core_thread (arg=0x55bda0d38b10) at
/main/qpid-dispatch/src/router_core/router_core_thread.c:53
---Type <return> to continue, or q <return> to quit---
#3 0x00007ff2b9106494 in start_thread (arg=0x7ff2b5c46700) at
pthread_create.c:333
#4 0x00007ff2b84ab93f in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:97
Thread 1 (Thread 0x7ff2b5445700 (LWP 104)):
#0 0x00007ff2b9320320 in pconnection_cleanup (pc=0x7ff2a80af460) at
/main/qpid-proton/proton-c/src/proactor/epoll.c:645
#1 0x00007ff2b9320ac5 in pconnection_process (pc=0x7ff2a80af460, events=0,
timeout=false, topup=false) at
/main/qpid-proton/proton-c/src/proactor/epoll.c:843
#2 0x00007ff2b93230e5 in proactor_do_epoll (p=0x55bda0cc1f00, can_block=true)
at /main/qpid-proton/proton-c/src/proactor/epoll.c:1662
#3 0x00007ff2b9323185 in pn_proactor_wait (p=0x55bda0cc1f00) at
/main/qpid-proton/proton-c/src/proactor/epoll.c:1680
#4 0x00007ff2b97c4e7a in thread_run (arg=0x55bda0cc1e00) at
/main/qpid-dispatch/src/server.c:817
#5 0x00007ff2b9106494 in start_thread (arg=0x7ff2b5445700) at
pthread_create.c:333
#6 0x00007ff2b84ab93f in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:97
(gdb)
{noformat}
{noformat}
(gdb) info locals
can_free = false
{noformat}
was:
I've noticed multiple qdrouterd test failures with seem to come from the same
place in proactor code.
SIGSEGV in qdrouterd tests; in pconnection_cleanup (pc=0x7f...) at
/main/qpid-proton/proton-c/src/proactor/epoll.c:645
stop_polling(&pc->timer.epoll_io, pc->psocket.proactor->epollfd);
I am attaching logs and backtraces from two of them below.
{noformat}
test 14
Start 14: system_tests_autolinks
14: Test command: /usr/bin/python "/main/qpid-dispatch/build/tests/run.py" "-m"
"unittest" "-v" "system_tests_autolinks"
14: Test timeout computed to be: 1500
14: test_01_autolink_attach (system_tests_autolinks.AutolinkTest) ... FAIL
14: test_02_autolink_credit (system_tests_autolinks.AutolinkTest) ... ok
14: test_03_autolink_sender (system_tests_autolinks.AutolinkTest) ... ok
14: test_04_autolink_receiver (system_tests_autolinks.AutolinkTest) ... ok
14: test_05_inter_container_transfer (system_tests_autolinks.AutolinkTest) ...
ok
14: test_06_manage_autolinks (system_tests_autolinks.AutolinkTest) ...
ERROR:root:Connectivity to the peer container was lost
14: ERROR:root:Connectivity to the peer container was lost
14: ERROR:root:Connectivity to the peer container was lost
14: ERROR:root:Connectivity to the peer container was lost
14: ERROR:root:Connectivity to the peer container was lost
14: ok
14: test_07_autolink_attach_with_ext_addr (system_tests_autolinks.AutolinkTest)
... ERROR:root:proton:io: recv: Connection reset by peer
14: ERROR:root:proton:io: recv: Connection reset by peer
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: FAIL
14: test_08_autolink_sender_with_ext_addr (system_tests_autolinks.AutolinkTest)
... ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: FAIL
14: test_09_autolink_receiver_with_ext_addr
(system_tests_autolinks.AutolinkTest) ... ERROR:root:proton:io: recv:
Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: ERROR:root:proton:io: recv: Connection refused
14: FAIL
14: ERROR
14:
14: ======================================================================
14: ERROR: tearDownClass (system_tests_autolinks.AutolinkTest)
14: ----------------------------------------------------------------------
14: Traceback (most recent call last):
14: File "/main/qpid-dispatch/tests/system_test.py", line 605, in
tearDownClass
14: cls.tester.teardown()
14: File "/main/qpid-dispatch/tests/system_test.py", line 543, in teardown
14: raise RuntimeError("Errors during teardown: \n\n%s" %
"\n\n".join([str(e) for e in errors]))
14: RuntimeError: Errors during teardown:
14:
14: Process 81 error: exit code -11, expected -1
14: qdrouterd -c test-router.conf -I /main/qpid-dispatch/python
14:
/main/qpid-dispatch/build/tests/system_test.dir/system_tests_autolinks/AutolinkTest/setUpClass/test-router-1.cmd
14: >>>>
14: <<<<
14:
14: ======================================================================
14: FAIL: test_01_autolink_attach (system_tests_autolinks.AutolinkTest)
14: ----------------------------------------------------------------------
14: Traceback (most recent call last):
14: File "/main/qpid-dispatch/tests/system_tests_autolinks.py", line 85, in
test_01_autolink_attach
14: self.assertEqual(None, test.error)
14: AssertionError: None != 'Timeout Expired: n_rx_attach=1 n_tx_attach=1'
14:
14: ======================================================================
14: FAIL: test_07_autolink_attach_with_ext_addr
(system_tests_autolinks.AutolinkTest)
14: ----------------------------------------------------------------------
14: Traceback (most recent call last):
14: File "/main/qpid-dispatch/tests/system_tests_autolinks.py", line 146, in
test_07_autolink_attach_with_ext_addr
14: self.assertEqual(None, test.error)
14: AssertionError: None != 'Timeout Expired: n_rx_attach=0 n_tx_attach=0'
14:
14: ======================================================================
14: FAIL: test_08_autolink_sender_with_ext_addr
(system_tests_autolinks.AutolinkTest)
14: ----------------------------------------------------------------------
14: Traceback (most recent call last):
14: File "/main/qpid-dispatch/tests/system_tests_autolinks.py", line 156, in
test_08_autolink_sender_with_ext_addr
14: self.assertEqual(None, test.error)
14: AssertionError: None != 'Timeout Expired: last_action=Connected route
container n_sent=0 n_received=0 n_settled=0'
14:
14: ======================================================================
14: FAIL: test_09_autolink_receiver_with_ext_addr
(system_tests_autolinks.AutolinkTest)
14: ----------------------------------------------------------------------
14: Traceback (most recent call last):
14: File "/main/qpid-dispatch/tests/system_tests_autolinks.py", line 166, in
test_09_autolink_receiver_with_ext_addr
14: self.assertEqual(None, test.error)
14: AssertionError: None != 'Timeout Expired: last_action=Connected route
container n_sent=0 n_received=0 n_settled=0'
14:
14: ----------------------------------------------------------------------
14: Ran 9 tests in 248.688s
14:
14: FAILED (failures=4, errors=1)
14/33 Test #14: system_tests_autolinks ....................***Failed 248.75 sec
{noformat}
{noformat}
root@9622b08a07af:/main/qpid-dispatch/build# gdb -c
/main/qpid-dispatch/build/tests/system_test.dir/system_tests_autolinks/AutolinkTest/setUpClass/core
`which qdrouterd`
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/qdrouterd...(no debugging symbols found)...done.
[New LWP 81]
[New LWP 84]
[New LWP 82]
[New LWP 83]
[New LWP 85]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `qdrouterd -c test-router.conf -I
/main/qpid-dispatch/python'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007f89a496f320 in pconnection_cleanup (pc=0x55e69fe2dec0) at
/main/qpid-proton/proton-c/src/proactor/epoll.c:645
645 stop_polling(&pc->timer.epoll_io, pc->psocket.proactor->epollfd);
[Current thread is 1 (Thread 0x7f89a5245080 (LWP 81))]
(gdb) thread apply all bt
Thread 5 (Thread 0x7f899b7fe700 (LWP 85)):
#0 0x00007f89a3afaf33 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
#1 0x00007f89a4971fcd in proactor_do_epoll (p=0x55e69fd5f1a0, can_block=true)
at /main/qpid-proton/proton-c/src/proactor/epoll.c:1634
#2 0x00007f89a4972185 in pn_proactor_wait (p=0x55e69fd5f1a0) at
/main/qpid-proton/proton-c/src/proactor/epoll.c:1680
#3 0x00007f89a4e13e7a in thread_run (arg=0x55e69fb27ec0) at
/main/qpid-dispatch/src/server.c:817
#4 0x00007f89a4755494 in start_thread (arg=0x7f899b7fe700) at
pthread_create.c:333
#5 0x00007f89a3afa93f in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:97
Thread 4 (Thread 0x7f89a0a94700 (LWP 83)):
#0 0x00007f89a3afaf33 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
#1 0x00007f89a4971fcd in proactor_do_epoll (p=0x55e69fd5f1a0, can_block=true)
at /main/qpid-proton/proton-c/src/proactor/epoll.c:1634
#2 0x00007f89a4972185 in pn_proactor_wait (p=0x55e69fd5f1a0) at
/main/qpid-proton/proton-c/src/proactor/epoll.c:1680
#3 0x00007f89a4e13e7a in thread_run (arg=0x55e69fb27ec0) at
/main/qpid-dispatch/src/server.c:817
#4 0x00007f89a4755494 in start_thread (arg=0x7f89a0a94700) at
pthread_create.c:333
#5 0x00007f89a3afa93f in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:97
Thread 3 (Thread 0x7f89a1295700 (LWP 82)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at
../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007f89a4df19c0 in sys_cond_wait (cond=0x55e69fdcee80,
held_mutex=0x55e69fdcef40) at /main/qpid-dispatch/src/posix/threading.c:91
#2 0x00007f89a4e070fa in router_core_thread (arg=0x55e69fdceb70) at
/main/qpid-dispatch/src/router_core/router_core_thread.c:53
#3 0x00007f89a4755494 in start_thread (arg=0x7f89a1295700) at
pthread_create.c:333
#4 0x00007f89a3afa93f in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:97
Thread 2 (Thread 0x7f899bfff700 (LWP 84)):
#0 0x00007f89a3afaf33 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
#1 0x00007f89a4971fcd in proactor_do_epoll (p=0x55e69fd5f1a0, can_block=true)
at /main/qpid-proton/proton-c/src/proactor/epoll.c:1634
#2 0x00007f89a4972185 in pn_proactor_wait (p=0x55e69fd5f1a0) at
/main/qpid-proton/proton-c/src/proactor/epoll.c:1680
#3 0x00007f89a4e13e7a in thread_run (arg=0x55e69fb27ec0) at
/main/qpid-dispatch/src/server.c:817
#4 0x00007f89a4755494 in start_thread (arg=0x7f899bfff700) at
pthread_create.c:333
---Type <return> to continue, or q <return> to quit---
#5 0x00007f89a3afa93f in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:97
Thread 1 (Thread 0x7f89a5245080 (LWP 81)):
#0 0x00007f89a496f320 in pconnection_cleanup (pc=0x55e69fe2dec0) at
/main/qpid-proton/proton-c/src/proactor/epoll.c:645
#1 0x00007f89a496fac5 in pconnection_process (pc=0x55e69fe2dec0, events=0,
timeout=false, topup=false) at
/main/qpid-proton/proton-c/src/proactor/epoll.c:843
#2 0x00007f89a49720e5 in proactor_do_epoll (p=0x55e69fd5f1a0, can_block=true)
at /main/qpid-proton/proton-c/src/proactor/epoll.c:1662
#3 0x00007f89a4972185 in pn_proactor_wait (p=0x55e69fd5f1a0) at
/main/qpid-proton/proton-c/src/proactor/epoll.c:1680
#4 0x00007f89a4e13e7a in thread_run (arg=0x55e69fb27ec0) at
/main/qpid-dispatch/src/server.c:817
#5 0x00007f89a4e14987 in qd_server_run (qd=0x55e69facc010) at
/main/qpid-dispatch/src/server.c:1031
#6 0x000055e69f5a28d0 in ?? ()
#7 0x000055e69f5a325b in main ()
(gdb)
{noformat}
========================================
{noformat}
16: Test command: /usr/bin/python "/main/qpid-dispatch/build/tests/run.py" "-m"
"unittest" "-v" "system_tests_management"
16: Test timeout computed to be: 1500
16: test_bad_query (system_tests_management.ManagementTest)
16: Test that various badly formed queries get the proper response ... ok
16: test_connection (system_tests_management.ManagementTest)
16: Verify there is at least one connection ... ok
16: test_create_fixed_address (system_tests_management.ManagementTest) ... ok
16: test_create_listener (system_tests_management.ManagementTest)
16: Create a new listener on a running router ... ok
16: test_dummy (system_tests_management.ManagementTest)
16: Test all operations on the dummy test entity ... ok
16: test_entity_names (system_tests_management.ManagementTest) ... ok
16: test_get_attributes (system_tests_management.ManagementTest) ... ok
16: test_get_operations (system_tests_management.ManagementTest) ... ok
16: test_get_schema (system_tests_management.ManagementTest) ... ok
16: test_get_types (system_tests_management.ManagementTest) ... ok
16: test_link (system_tests_management.ManagementTest)
16: Verify we can find our own reply-to address in links ... ok
16: test_log (system_tests_management.ManagementTest)
16: Create, update and query log entities ... ok
16: test_query_attributes (system_tests_management.ManagementTest)
16: Query with attributes only ... ok
16: test_query_type (system_tests_management.ManagementTest)
16: Query with type only ... ok
16: test_query_type_attributes (system_tests_management.ManagementTest)
16: Query with type and attribute names ... ok
16: test_remote_node (system_tests_management.ManagementTest)
16: Test that we can access management info of remote nodes using
get_mgmt_nodes addresses ... ok
16: test_router (system_tests_management.ManagementTest)
16: Verify router counts match entity counts ... ok
16: test_router_node (system_tests_management.ManagementTest)
16: Test node entity in a trio of linked routers ... ERROR
16: test_standalone_no_inter_router (system_tests_management.ManagementTest)
16: Verify that we do not allow inter-router connectors or listeners in
standalone mode ... ERROR
16: ERROR
16:
16: ======================================================================
16: ERROR: test_router_node (system_tests_management.ManagementTest)
16: Test node entity in a trio of linked routers
16: ----------------------------------------------------------------------
16: Traceback (most recent call last):
16: File "/main/qpid-dispatch/tests/system_tests_management.py", line 112, in
setUp
16: self.node = self.cleanup(Node.connect(self.router.addresses[0]))
16: File "/main/qpid-dispatch/python/qpid_dispatch/management/client.py",
line 100, in connect
16: return Node(Node.connection(url, router, timeout, ssl_domain, sasl))
16: File "/main/qpid-dispatch/python/qpid_dispatch/management/client.py",
line 95, in connection
16: password=str(sasl.password) if sasl else None)
16: File "/usr/lib/python2.7/dist-packages/proton/utils.py", line 213, in
__init__
16: msg="Opening connection")
16: File "/usr/lib/python2.7/dist-packages/proton/utils.py", line 279, in wait
16: "Connection %s disconnected: %s" % (self.url, self.disconnected))
16: ConnectionException: Connection amqp://0.0.0.0:29636/$management
disconnected: Condition('proton:io', 'recv: Connection reset by peer')
16:
16: ======================================================================
16: ERROR: test_standalone_no_inter_router
(system_tests_management.ManagementTest)
16: Verify that we do not allow inter-router connectors or listeners in
standalone mode
16: ----------------------------------------------------------------------
16: Traceback (most recent call last):
16: File "/main/qpid-dispatch/tests/system_tests_management.py", line 112, in
setUp
16: self.node = self.cleanup(Node.connect(self.router.addresses[0]))
16: File "/main/qpid-dispatch/python/qpid_dispatch/management/client.py",
line 100, in connect
16: return Node(Node.connection(url, router, timeout, ssl_domain, sasl))
16: File "/main/qpid-dispatch/python/qpid_dispatch/management/client.py",
line 95, in connection
16: password=str(sasl.password) if sasl else None)
16: File "/usr/lib/python2.7/dist-packages/proton/utils.py", line 213, in
__init__
16: msg="Opening connection")
16: File "/usr/lib/python2.7/dist-packages/proton/utils.py", line 279, in wait
16: "Connection %s disconnected: %s" % (self.url, self.disconnected))
16: ConnectionException: Connection amqp://0.0.0.0:29636/$management
disconnected: Condition('proton:io', 'recv: Connection refused')
16:
16: ======================================================================
16: ERROR: tearDownClass (system_tests_management.ManagementTest)
16: ----------------------------------------------------------------------
16: Traceback (most recent call last):
16: File "/main/qpid-dispatch/tests/system_test.py", line 605, in
tearDownClass
16: cls.tester.teardown()
16: File "/main/qpid-dispatch/tests/system_test.py", line 543, in teardown
16: raise RuntimeError("Errors during teardown: \n\n%s" %
"\n\n".join([str(e) for e in errors]))
16: RuntimeError: Errors during teardown:
16:
16: Process 95 error: exit code -11, expected -1
16: qdrouterd -c solo.conf -I /main/qpid-dispatch/python
16:
/main/qpid-dispatch/build/tests/system_test.dir/system_tests_management/ManagementTest/setUpClass/solo-1.cmd
16: >>>>
16: <<<<
16:
16: ----------------------------------------------------------------------
16: Ran 19 tests in 8.165s
16:
16: FAILED (errors=3)
16/33 Test #16: system_tests_management ...................***Failed 8.24 sec
{noformat}
{noformat}
root@9622b08a07af:/main/qpid-dispatch/build# gdb -c
/main/qpid-dispatch/build/tests/system_test.dir/system_tests_management/ManagementTest/setUpClass/core
`which qdrouterd`
GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/qdrouterd...(no debugging symbols found)...done.
[New LWP 104]
[New LWP 100]
[New LWP 105]
[New LWP 95]
[New LWP 106]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `qdrouterd -c solo.conf -I /main/qpid-dispatch/python'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007ff2b9320320 in pconnection_cleanup (pc=0x7ff2a80af460) at
/main/qpid-proton/proton-c/src/proactor/epoll.c:645
645 stop_polling(&pc->timer.epoll_io, pc->psocket.proactor->epollfd);
[Current thread is 1 (Thread 0x7ff2b5445700 (LWP 104))]
(gdb) thread apply all bt
Thread 5 (Thread 0x7ff2affff700 (LWP 106)):
#0 0x00007ff2b84abf33 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
#1 0x00007ff2b9322fcd in proactor_do_epoll (p=0x55bda0cc1f00, can_block=true)
at /main/qpid-proton/proton-c/src/proactor/epoll.c:1634
#2 0x00007ff2b9323185 in pn_proactor_wait (p=0x55bda0cc1f00) at
/main/qpid-proton/proton-c/src/proactor/epoll.c:1680
#3 0x00007ff2b97c4e7a in thread_run (arg=0x55bda0cc1e00) at
/main/qpid-dispatch/src/server.c:817
#4 0x00007ff2b9106494 in start_thread (arg=0x7ff2affff700) at
pthread_create.c:333
#5 0x00007ff2b84ab93f in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:97
Thread 4 (Thread 0x7ff2b9bf6080 (LWP 95)):
#0 0x00007ff2b84abf33 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
#1 0x00007ff2b9322fcd in proactor_do_epoll (p=0x55bda0cc1f00, can_block=true)
at /main/qpid-proton/proton-c/src/proactor/epoll.c:1634
#2 0x00007ff2b9323185 in pn_proactor_wait (p=0x55bda0cc1f00) at
/main/qpid-proton/proton-c/src/proactor/epoll.c:1680
#3 0x00007ff2b97c4e7a in thread_run (arg=0x55bda0cc1e00) at
/main/qpid-dispatch/src/server.c:817
#4 0x00007ff2b97c5987 in qd_server_run (qd=0x55bda0a36010) at
/main/qpid-dispatch/src/server.c:1031
#5 0x000055bda002f8d0 in ?? ()
#6 0x000055bda003025b in main ()
Thread 3 (Thread 0x7ff2b4c44700 (LWP 105)):
#0 0x00007ff2b84abf33 in epoll_wait () at ../sysdeps/unix/syscall-template.S:84
#1 0x00007ff2b9322fcd in proactor_do_epoll (p=0x55bda0cc1f00, can_block=true)
at /main/qpid-proton/proton-c/src/proactor/epoll.c:1634
#2 0x00007ff2b9323185 in pn_proactor_wait (p=0x55bda0cc1f00) at
/main/qpid-proton/proton-c/src/proactor/epoll.c:1680
#3 0x00007ff2b97c4e7a in thread_run (arg=0x55bda0cc1e00) at
/main/qpid-dispatch/src/server.c:817
#4 0x00007ff2b9106494 in start_thread (arg=0x7ff2b4c44700) at
pthread_create.c:333
#5 0x00007ff2b84ab93f in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:97
Thread 2 (Thread 0x7ff2b5c46700 (LWP 100)):
#0 pthread_cond_wait@@GLIBC_2.3.2 () at
../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
#1 0x00007ff2b97a29c0 in sys_cond_wait (cond=0x55bda0d38e00,
held_mutex=0x55bda0d38ec0) at /main/qpid-dispatch/src/posix/threading.c:91
#2 0x00007ff2b97b80fa in router_core_thread (arg=0x55bda0d38b10) at
/main/qpid-dispatch/src/router_core/router_core_thread.c:53
---Type <return> to continue, or q <return> to quit---
#3 0x00007ff2b9106494 in start_thread (arg=0x7ff2b5c46700) at
pthread_create.c:333
#4 0x00007ff2b84ab93f in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:97
Thread 1 (Thread 0x7ff2b5445700 (LWP 104)):
#0 0x00007ff2b9320320 in pconnection_cleanup (pc=0x7ff2a80af460) at
/main/qpid-proton/proton-c/src/proactor/epoll.c:645
#1 0x00007ff2b9320ac5 in pconnection_process (pc=0x7ff2a80af460, events=0,
timeout=false, topup=false) at
/main/qpid-proton/proton-c/src/proactor/epoll.c:843
#2 0x00007ff2b93230e5 in proactor_do_epoll (p=0x55bda0cc1f00, can_block=true)
at /main/qpid-proton/proton-c/src/proactor/epoll.c:1662
#3 0x00007ff2b9323185 in pn_proactor_wait (p=0x55bda0cc1f00) at
/main/qpid-proton/proton-c/src/proactor/epoll.c:1680
#4 0x00007ff2b97c4e7a in thread_run (arg=0x55bda0cc1e00) at
/main/qpid-dispatch/src/server.c:817
#5 0x00007ff2b9106494 in start_thread (arg=0x7ff2b5445700) at
pthread_create.c:333
#6 0x00007ff2b84ab93f in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:97
(gdb)
{noformat}
{noformat}
(gdb) info locals
can_free = false
{noformat}
> [c proactor] SIGSEGV in pconnection_cleanup
> -------------------------------------------
>
> Key: PROTON-1484
> URL: https://issues.apache.org/jira/browse/PROTON-1484
> Project: Qpid Proton
> Issue Type: Bug
> Components: proton-c
> Affects Versions: 0.18.0
> Reporter: Jiri Danek
>
> I've noticed multiple qdrouterd test failures with seem to come from the same
> place in proactor code.
> SIGSEGV in qdrouterd tests; in pconnection_cleanup (pc=0x7f...) at
> /main/qpid-proton/proton-c/src/proactor/epoll.c:645
> stop_polling(&pc->timer.epoll_io, pc->psocket.proactor->epollfd);
> Reproducibility: the issue seems nondeterministic
> I am attaching logs and backtraces from two of them below.
> {noformat}
> test 14
> Start 14: system_tests_autolinks
> 14: Test command: /usr/bin/python "/main/qpid-dispatch/build/tests/run.py"
> "-m" "unittest" "-v" "system_tests_autolinks"
> 14: Test timeout computed to be: 1500
> 14: test_01_autolink_attach (system_tests_autolinks.AutolinkTest) ... FAIL
> 14: test_02_autolink_credit (system_tests_autolinks.AutolinkTest) ... ok
> 14: test_03_autolink_sender (system_tests_autolinks.AutolinkTest) ... ok
> 14: test_04_autolink_receiver (system_tests_autolinks.AutolinkTest) ... ok
> 14: test_05_inter_container_transfer (system_tests_autolinks.AutolinkTest)
> ... ok
> 14: test_06_manage_autolinks (system_tests_autolinks.AutolinkTest) ...
> ERROR:root:Connectivity to the peer container was lost
> 14: ERROR:root:Connectivity to the peer container was lost
> 14: ERROR:root:Connectivity to the peer container was lost
> 14: ERROR:root:Connectivity to the peer container was lost
> 14: ERROR:root:Connectivity to the peer container was lost
> 14: ok
> 14: test_07_autolink_attach_with_ext_addr
> (system_tests_autolinks.AutolinkTest) ... ERROR:root:proton:io: recv:
> Connection reset by peer
> 14: ERROR:root:proton:io: recv: Connection reset by peer
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: FAIL
> 14: test_08_autolink_sender_with_ext_addr
> (system_tests_autolinks.AutolinkTest) ... ERROR:root:proton:io: recv:
> Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: FAIL
> 14: test_09_autolink_receiver_with_ext_addr
> (system_tests_autolinks.AutolinkTest) ... ERROR:root:proton:io: recv:
> Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: ERROR:root:proton:io: recv: Connection refused
> 14: FAIL
> 14: ERROR
> 14:
> 14: ======================================================================
> 14: ERROR: tearDownClass (system_tests_autolinks.AutolinkTest)
> 14: ----------------------------------------------------------------------
> 14: Traceback (most recent call last):
> 14: File "/main/qpid-dispatch/tests/system_test.py", line 605, in
> tearDownClass
> 14: cls.tester.teardown()
> 14: File "/main/qpid-dispatch/tests/system_test.py", line 543, in teardown
> 14: raise RuntimeError("Errors during teardown: \n\n%s" %
> "\n\n".join([str(e) for e in errors]))
> 14: RuntimeError: Errors during teardown:
> 14:
> 14: Process 81 error: exit code -11, expected -1
> 14: qdrouterd -c test-router.conf -I /main/qpid-dispatch/python
> 14:
> /main/qpid-dispatch/build/tests/system_test.dir/system_tests_autolinks/AutolinkTest/setUpClass/test-router-1.cmd
> 14: >>>>
> 14: <<<<
> 14:
> 14: ======================================================================
> 14: FAIL: test_01_autolink_attach (system_tests_autolinks.AutolinkTest)
> 14: ----------------------------------------------------------------------
> 14: Traceback (most recent call last):
> 14: File "/main/qpid-dispatch/tests/system_tests_autolinks.py", line 85, in
> test_01_autolink_attach
> 14: self.assertEqual(None, test.error)
> 14: AssertionError: None != 'Timeout Expired: n_rx_attach=1 n_tx_attach=1'
> 14:
> 14: ======================================================================
> 14: FAIL: test_07_autolink_attach_with_ext_addr
> (system_tests_autolinks.AutolinkTest)
> 14: ----------------------------------------------------------------------
> 14: Traceback (most recent call last):
> 14: File "/main/qpid-dispatch/tests/system_tests_autolinks.py", line 146,
> in test_07_autolink_attach_with_ext_addr
> 14: self.assertEqual(None, test.error)
> 14: AssertionError: None != 'Timeout Expired: n_rx_attach=0 n_tx_attach=0'
> 14:
> 14: ======================================================================
> 14: FAIL: test_08_autolink_sender_with_ext_addr
> (system_tests_autolinks.AutolinkTest)
> 14: ----------------------------------------------------------------------
> 14: Traceback (most recent call last):
> 14: File "/main/qpid-dispatch/tests/system_tests_autolinks.py", line 156,
> in test_08_autolink_sender_with_ext_addr
> 14: self.assertEqual(None, test.error)
> 14: AssertionError: None != 'Timeout Expired: last_action=Connected route
> container n_sent=0 n_received=0 n_settled=0'
> 14:
> 14: ======================================================================
> 14: FAIL: test_09_autolink_receiver_with_ext_addr
> (system_tests_autolinks.AutolinkTest)
> 14: ----------------------------------------------------------------------
> 14: Traceback (most recent call last):
> 14: File "/main/qpid-dispatch/tests/system_tests_autolinks.py", line 166,
> in test_09_autolink_receiver_with_ext_addr
> 14: self.assertEqual(None, test.error)
> 14: AssertionError: None != 'Timeout Expired: last_action=Connected route
> container n_sent=0 n_received=0 n_settled=0'
> 14:
> 14: ----------------------------------------------------------------------
> 14: Ran 9 tests in 248.688s
> 14:
> 14: FAILED (failures=4, errors=1)
> 14/33 Test #14: system_tests_autolinks ....................***Failed 248.75
> sec
> {noformat}
> {noformat}
> root@9622b08a07af:/main/qpid-dispatch/build# gdb -c
> /main/qpid-dispatch/build/tests/system_test.dir/system_tests_autolinks/AutolinkTest/setUpClass/core
> `which qdrouterd`
> GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
> Copyright (C) 2016 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-linux-gnu".
> Type "show configuration" for configuration details.
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>.
> Find the GDB manual and other documentation resources online at:
> <http://www.gnu.org/software/gdb/documentation/>.
> For help, type "help".
> Type "apropos word" to search for commands related to "word"...
> Reading symbols from /usr/sbin/qdrouterd...(no debugging symbols
> found)...done.
> [New LWP 81]
> [New LWP 84]
> [New LWP 82]
> [New LWP 83]
> [New LWP 85]
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> Core was generated by `qdrouterd -c test-router.conf -I
> /main/qpid-dispatch/python'.
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0 0x00007f89a496f320 in pconnection_cleanup (pc=0x55e69fe2dec0) at
> /main/qpid-proton/proton-c/src/proactor/epoll.c:645
> 645 stop_polling(&pc->timer.epoll_io, pc->psocket.proactor->epollfd);
> [Current thread is 1 (Thread 0x7f89a5245080 (LWP 81))]
> (gdb) thread apply all bt
> Thread 5 (Thread 0x7f899b7fe700 (LWP 85)):
> #0 0x00007f89a3afaf33 in epoll_wait () at
> ../sysdeps/unix/syscall-template.S:84
> #1 0x00007f89a4971fcd in proactor_do_epoll (p=0x55e69fd5f1a0,
> can_block=true) at /main/qpid-proton/proton-c/src/proactor/epoll.c:1634
> #2 0x00007f89a4972185 in pn_proactor_wait (p=0x55e69fd5f1a0) at
> /main/qpid-proton/proton-c/src/proactor/epoll.c:1680
> #3 0x00007f89a4e13e7a in thread_run (arg=0x55e69fb27ec0) at
> /main/qpid-dispatch/src/server.c:817
> #4 0x00007f89a4755494 in start_thread (arg=0x7f899b7fe700) at
> pthread_create.c:333
> #5 0x00007f89a3afa93f in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
> Thread 4 (Thread 0x7f89a0a94700 (LWP 83)):
> #0 0x00007f89a3afaf33 in epoll_wait () at
> ../sysdeps/unix/syscall-template.S:84
> #1 0x00007f89a4971fcd in proactor_do_epoll (p=0x55e69fd5f1a0,
> can_block=true) at /main/qpid-proton/proton-c/src/proactor/epoll.c:1634
> #2 0x00007f89a4972185 in pn_proactor_wait (p=0x55e69fd5f1a0) at
> /main/qpid-proton/proton-c/src/proactor/epoll.c:1680
> #3 0x00007f89a4e13e7a in thread_run (arg=0x55e69fb27ec0) at
> /main/qpid-dispatch/src/server.c:817
> #4 0x00007f89a4755494 in start_thread (arg=0x7f89a0a94700) at
> pthread_create.c:333
> #5 0x00007f89a3afa93f in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
> Thread 3 (Thread 0x7f89a1295700 (LWP 82)):
> #0 pthread_cond_wait@@GLIBC_2.3.2 () at
> ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
> #1 0x00007f89a4df19c0 in sys_cond_wait (cond=0x55e69fdcee80,
> held_mutex=0x55e69fdcef40) at /main/qpid-dispatch/src/posix/threading.c:91
> #2 0x00007f89a4e070fa in router_core_thread (arg=0x55e69fdceb70) at
> /main/qpid-dispatch/src/router_core/router_core_thread.c:53
> #3 0x00007f89a4755494 in start_thread (arg=0x7f89a1295700) at
> pthread_create.c:333
> #4 0x00007f89a3afa93f in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
> Thread 2 (Thread 0x7f899bfff700 (LWP 84)):
> #0 0x00007f89a3afaf33 in epoll_wait () at
> ../sysdeps/unix/syscall-template.S:84
> #1 0x00007f89a4971fcd in proactor_do_epoll (p=0x55e69fd5f1a0,
> can_block=true) at /main/qpid-proton/proton-c/src/proactor/epoll.c:1634
> #2 0x00007f89a4972185 in pn_proactor_wait (p=0x55e69fd5f1a0) at
> /main/qpid-proton/proton-c/src/proactor/epoll.c:1680
> #3 0x00007f89a4e13e7a in thread_run (arg=0x55e69fb27ec0) at
> /main/qpid-dispatch/src/server.c:817
> #4 0x00007f89a4755494 in start_thread (arg=0x7f899bfff700) at
> pthread_create.c:333
> ---Type <return> to continue, or q <return> to quit---
> #5 0x00007f89a3afa93f in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
> Thread 1 (Thread 0x7f89a5245080 (LWP 81)):
> #0 0x00007f89a496f320 in pconnection_cleanup (pc=0x55e69fe2dec0) at
> /main/qpid-proton/proton-c/src/proactor/epoll.c:645
> #1 0x00007f89a496fac5 in pconnection_process (pc=0x55e69fe2dec0, events=0,
> timeout=false, topup=false) at
> /main/qpid-proton/proton-c/src/proactor/epoll.c:843
> #2 0x00007f89a49720e5 in proactor_do_epoll (p=0x55e69fd5f1a0,
> can_block=true) at /main/qpid-proton/proton-c/src/proactor/epoll.c:1662
> #3 0x00007f89a4972185 in pn_proactor_wait (p=0x55e69fd5f1a0) at
> /main/qpid-proton/proton-c/src/proactor/epoll.c:1680
> #4 0x00007f89a4e13e7a in thread_run (arg=0x55e69fb27ec0) at
> /main/qpid-dispatch/src/server.c:817
> #5 0x00007f89a4e14987 in qd_server_run (qd=0x55e69facc010) at
> /main/qpid-dispatch/src/server.c:1031
> #6 0x000055e69f5a28d0 in ?? ()
> #7 0x000055e69f5a325b in main ()
> (gdb)
> {noformat}
> ========================================
> {noformat}
> 16: Test command: /usr/bin/python "/main/qpid-dispatch/build/tests/run.py"
> "-m" "unittest" "-v" "system_tests_management"
> 16: Test timeout computed to be: 1500
> 16: test_bad_query (system_tests_management.ManagementTest)
> 16: Test that various badly formed queries get the proper response ... ok
> 16: test_connection (system_tests_management.ManagementTest)
> 16: Verify there is at least one connection ... ok
> 16: test_create_fixed_address (system_tests_management.ManagementTest) ... ok
> 16: test_create_listener (system_tests_management.ManagementTest)
> 16: Create a new listener on a running router ... ok
> 16: test_dummy (system_tests_management.ManagementTest)
> 16: Test all operations on the dummy test entity ... ok
> 16: test_entity_names (system_tests_management.ManagementTest) ... ok
> 16: test_get_attributes (system_tests_management.ManagementTest) ... ok
> 16: test_get_operations (system_tests_management.ManagementTest) ... ok
> 16: test_get_schema (system_tests_management.ManagementTest) ... ok
> 16: test_get_types (system_tests_management.ManagementTest) ... ok
> 16: test_link (system_tests_management.ManagementTest)
> 16: Verify we can find our own reply-to address in links ... ok
> 16: test_log (system_tests_management.ManagementTest)
> 16: Create, update and query log entities ... ok
> 16: test_query_attributes (system_tests_management.ManagementTest)
> 16: Query with attributes only ... ok
> 16: test_query_type (system_tests_management.ManagementTest)
> 16: Query with type only ... ok
> 16: test_query_type_attributes (system_tests_management.ManagementTest)
> 16: Query with type and attribute names ... ok
> 16: test_remote_node (system_tests_management.ManagementTest)
> 16: Test that we can access management info of remote nodes using
> get_mgmt_nodes addresses ... ok
> 16: test_router (system_tests_management.ManagementTest)
> 16: Verify router counts match entity counts ... ok
> 16: test_router_node (system_tests_management.ManagementTest)
> 16: Test node entity in a trio of linked routers ... ERROR
> 16: test_standalone_no_inter_router (system_tests_management.ManagementTest)
> 16: Verify that we do not allow inter-router connectors or listeners in
> standalone mode ... ERROR
> 16: ERROR
> 16:
> 16: ======================================================================
> 16: ERROR: test_router_node (system_tests_management.ManagementTest)
> 16: Test node entity in a trio of linked routers
> 16: ----------------------------------------------------------------------
> 16: Traceback (most recent call last):
> 16: File "/main/qpid-dispatch/tests/system_tests_management.py", line 112,
> in setUp
> 16: self.node = self.cleanup(Node.connect(self.router.addresses[0]))
> 16: File "/main/qpid-dispatch/python/qpid_dispatch/management/client.py",
> line 100, in connect
> 16: return Node(Node.connection(url, router, timeout, ssl_domain, sasl))
> 16: File "/main/qpid-dispatch/python/qpid_dispatch/management/client.py",
> line 95, in connection
> 16: password=str(sasl.password) if sasl else None)
> 16: File "/usr/lib/python2.7/dist-packages/proton/utils.py", line 213, in
> __init__
> 16: msg="Opening connection")
> 16: File "/usr/lib/python2.7/dist-packages/proton/utils.py", line 279, in
> wait
> 16: "Connection %s disconnected: %s" % (self.url, self.disconnected))
> 16: ConnectionException: Connection amqp://0.0.0.0:29636/$management
> disconnected: Condition('proton:io', 'recv: Connection reset by peer')
> 16:
> 16: ======================================================================
> 16: ERROR: test_standalone_no_inter_router
> (system_tests_management.ManagementTest)
> 16: Verify that we do not allow inter-router connectors or listeners in
> standalone mode
> 16: ----------------------------------------------------------------------
> 16: Traceback (most recent call last):
> 16: File "/main/qpid-dispatch/tests/system_tests_management.py", line 112,
> in setUp
> 16: self.node = self.cleanup(Node.connect(self.router.addresses[0]))
> 16: File "/main/qpid-dispatch/python/qpid_dispatch/management/client.py",
> line 100, in connect
> 16: return Node(Node.connection(url, router, timeout, ssl_domain, sasl))
> 16: File "/main/qpid-dispatch/python/qpid_dispatch/management/client.py",
> line 95, in connection
> 16: password=str(sasl.password) if sasl else None)
> 16: File "/usr/lib/python2.7/dist-packages/proton/utils.py", line 213, in
> __init__
> 16: msg="Opening connection")
> 16: File "/usr/lib/python2.7/dist-packages/proton/utils.py", line 279, in
> wait
> 16: "Connection %s disconnected: %s" % (self.url, self.disconnected))
> 16: ConnectionException: Connection amqp://0.0.0.0:29636/$management
> disconnected: Condition('proton:io', 'recv: Connection refused')
> 16:
> 16: ======================================================================
> 16: ERROR: tearDownClass (system_tests_management.ManagementTest)
> 16: ----------------------------------------------------------------------
> 16: Traceback (most recent call last):
> 16: File "/main/qpid-dispatch/tests/system_test.py", line 605, in
> tearDownClass
> 16: cls.tester.teardown()
> 16: File "/main/qpid-dispatch/tests/system_test.py", line 543, in teardown
> 16: raise RuntimeError("Errors during teardown: \n\n%s" %
> "\n\n".join([str(e) for e in errors]))
> 16: RuntimeError: Errors during teardown:
> 16:
> 16: Process 95 error: exit code -11, expected -1
> 16: qdrouterd -c solo.conf -I /main/qpid-dispatch/python
> 16:
> /main/qpid-dispatch/build/tests/system_test.dir/system_tests_management/ManagementTest/setUpClass/solo-1.cmd
> 16: >>>>
> 16: <<<<
> 16:
> 16: ----------------------------------------------------------------------
> 16: Ran 19 tests in 8.165s
> 16:
> 16: FAILED (errors=3)
> 16/33 Test #16: system_tests_management ...................***Failed 8.24
> sec
> {noformat}
> {noformat}
> root@9622b08a07af:/main/qpid-dispatch/build# gdb -c
> /main/qpid-dispatch/build/tests/system_test.dir/system_tests_management/ManagementTest/setUpClass/core
> `which qdrouterd`
> GNU gdb (Debian 7.12-6) 7.12.0.20161007-git
> Copyright (C) 2016 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-linux-gnu".
> Type "show configuration" for configuration details.
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>.
> Find the GDB manual and other documentation resources online at:
> <http://www.gnu.org/software/gdb/documentation/>.
> For help, type "help".
> Type "apropos word" to search for commands related to "word"...
> Reading symbols from /usr/sbin/qdrouterd...(no debugging symbols
> found)...done.
> [New LWP 104]
> [New LWP 100]
> [New LWP 105]
> [New LWP 95]
> [New LWP 106]
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
> Core was generated by `qdrouterd -c solo.conf -I /main/qpid-dispatch/python'.
> Program terminated with signal SIGSEGV, Segmentation fault.
> #0 0x00007ff2b9320320 in pconnection_cleanup (pc=0x7ff2a80af460) at
> /main/qpid-proton/proton-c/src/proactor/epoll.c:645
> 645 stop_polling(&pc->timer.epoll_io, pc->psocket.proactor->epollfd);
> [Current thread is 1 (Thread 0x7ff2b5445700 (LWP 104))]
> (gdb) thread apply all bt
> Thread 5 (Thread 0x7ff2affff700 (LWP 106)):
> #0 0x00007ff2b84abf33 in epoll_wait () at
> ../sysdeps/unix/syscall-template.S:84
> #1 0x00007ff2b9322fcd in proactor_do_epoll (p=0x55bda0cc1f00,
> can_block=true) at /main/qpid-proton/proton-c/src/proactor/epoll.c:1634
> #2 0x00007ff2b9323185 in pn_proactor_wait (p=0x55bda0cc1f00) at
> /main/qpid-proton/proton-c/src/proactor/epoll.c:1680
> #3 0x00007ff2b97c4e7a in thread_run (arg=0x55bda0cc1e00) at
> /main/qpid-dispatch/src/server.c:817
> #4 0x00007ff2b9106494 in start_thread (arg=0x7ff2affff700) at
> pthread_create.c:333
> #5 0x00007ff2b84ab93f in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
> Thread 4 (Thread 0x7ff2b9bf6080 (LWP 95)):
> #0 0x00007ff2b84abf33 in epoll_wait () at
> ../sysdeps/unix/syscall-template.S:84
> #1 0x00007ff2b9322fcd in proactor_do_epoll (p=0x55bda0cc1f00,
> can_block=true) at /main/qpid-proton/proton-c/src/proactor/epoll.c:1634
> #2 0x00007ff2b9323185 in pn_proactor_wait (p=0x55bda0cc1f00) at
> /main/qpid-proton/proton-c/src/proactor/epoll.c:1680
> #3 0x00007ff2b97c4e7a in thread_run (arg=0x55bda0cc1e00) at
> /main/qpid-dispatch/src/server.c:817
> #4 0x00007ff2b97c5987 in qd_server_run (qd=0x55bda0a36010) at
> /main/qpid-dispatch/src/server.c:1031
> #5 0x000055bda002f8d0 in ?? ()
> #6 0x000055bda003025b in main ()
> Thread 3 (Thread 0x7ff2b4c44700 (LWP 105)):
> #0 0x00007ff2b84abf33 in epoll_wait () at
> ../sysdeps/unix/syscall-template.S:84
> #1 0x00007ff2b9322fcd in proactor_do_epoll (p=0x55bda0cc1f00,
> can_block=true) at /main/qpid-proton/proton-c/src/proactor/epoll.c:1634
> #2 0x00007ff2b9323185 in pn_proactor_wait (p=0x55bda0cc1f00) at
> /main/qpid-proton/proton-c/src/proactor/epoll.c:1680
> #3 0x00007ff2b97c4e7a in thread_run (arg=0x55bda0cc1e00) at
> /main/qpid-dispatch/src/server.c:817
> #4 0x00007ff2b9106494 in start_thread (arg=0x7ff2b4c44700) at
> pthread_create.c:333
> #5 0x00007ff2b84ab93f in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
> Thread 2 (Thread 0x7ff2b5c46700 (LWP 100)):
> #0 pthread_cond_wait@@GLIBC_2.3.2 () at
> ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185
> #1 0x00007ff2b97a29c0 in sys_cond_wait (cond=0x55bda0d38e00,
> held_mutex=0x55bda0d38ec0) at /main/qpid-dispatch/src/posix/threading.c:91
> #2 0x00007ff2b97b80fa in router_core_thread (arg=0x55bda0d38b10) at
> /main/qpid-dispatch/src/router_core/router_core_thread.c:53
> ---Type <return> to continue, or q <return> to quit---
> #3 0x00007ff2b9106494 in start_thread (arg=0x7ff2b5c46700) at
> pthread_create.c:333
> #4 0x00007ff2b84ab93f in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
> Thread 1 (Thread 0x7ff2b5445700 (LWP 104)):
> #0 0x00007ff2b9320320 in pconnection_cleanup (pc=0x7ff2a80af460) at
> /main/qpid-proton/proton-c/src/proactor/epoll.c:645
> #1 0x00007ff2b9320ac5 in pconnection_process (pc=0x7ff2a80af460, events=0,
> timeout=false, topup=false) at
> /main/qpid-proton/proton-c/src/proactor/epoll.c:843
> #2 0x00007ff2b93230e5 in proactor_do_epoll (p=0x55bda0cc1f00,
> can_block=true) at /main/qpid-proton/proton-c/src/proactor/epoll.c:1662
> #3 0x00007ff2b9323185 in pn_proactor_wait (p=0x55bda0cc1f00) at
> /main/qpid-proton/proton-c/src/proactor/epoll.c:1680
> #4 0x00007ff2b97c4e7a in thread_run (arg=0x55bda0cc1e00) at
> /main/qpid-dispatch/src/server.c:817
> #5 0x00007ff2b9106494 in start_thread (arg=0x7ff2b5445700) at
> pthread_create.c:333
> #6 0x00007ff2b84ab93f in clone () at
> ../sysdeps/unix/sysv/linux/x86_64/clone.S:97
> (gdb)
> {noformat}
> {noformat}
> (gdb) info locals
> can_free = false
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
