[
https://issues.apache.org/jira/browse/DISPATCH-1936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ganesh Murthy resolved DISPATCH-1936.
-------------------------------------
Fix Version/s: 1.16.0
Resolution: Cannot Reproduce
It looks like the router crashed during the test. I ran this test in a loop for
about an hour and was unable to reproduce this crash. I will mark this as Not
Reproducible for now and we can reopen if this happens again.
> system_tests_qdmanage failing on rhel 7
> ---------------------------------------
>
> Key: DISPATCH-1936
> URL: https://issues.apache.org/jira/browse/DISPATCH-1936
> Project: Qpid Dispatch
> Issue Type: Test
> Components: Tests
> Reporter: Ganesh Murthy
> Assignee: Ganesh Murthy
> Priority: Major
> Fix For: 1.16.0
>
>
> {noformat}
> test_check_address_name (system_tests_qdmanage.QdmanageTest) ... ok
> test_check_auto_link_name (system_tests_qdmanage.QdmanageTest) ... ok
> test_check_link_route_name (system_tests_qdmanage.QdmanageTest) ... ok
> test_check_memory_usage (system_tests_qdmanage.QdmanageTest) ... ok
> test_create_address (system_tests_qdmanage.QdmanageTest) ... ok
> test_create_auto_link_with_dir (system_tests_qdmanage.QdmanageTest) ... ok
> test_create_auto_link_with_phase (system_tests_qdmanage.QdmanageTest) ... ok
> test_create_delete_address_pattern (system_tests_qdmanage.QdmanageTest) ... ok
> test_create_delete_connector (system_tests_qdmanage.QdmanageTest) ... ok
> test_create_delete_ssl_profile (system_tests_qdmanage.QdmanageTest) ... ok
> test_create_link_route_with_dir (system_tests_qdmanage.QdmanageTest) ... ok
> test_crud (system_tests_qdmanage.QdmanageTest) ... ok
> test_delete_connection (system_tests_qdmanage.QdmanageTest) ... ok
> test_get_annotations (system_tests_qdmanage.QdmanageTest) ... ok
> test_get_attributes (system_tests_qdmanage.QdmanageTest) ... ok
> test_get_log (system_tests_qdmanage.QdmanageTest) ... ok
> test_get_logstats (system_tests_qdmanage.QdmanageTest) ... ok
> test_get_operations (system_tests_qdmanage.QdmanageTest) ... ok
> test_get_schema (system_tests_qdmanage.QdmanageTest) ... ok
> test_get_ssl_profile_attributes (system_tests_qdmanage.QdmanageTest) ... ok
> test_get_ssl_profile_operations (system_tests_qdmanage.QdmanageTest) ... ok
> test_get_ssl_profile_type_attributes (system_tests_qdmanage.QdmanageTest) ...
> ok
> test_get_ssl_profile_type_operations (system_tests_qdmanage.QdmanageTest) ...
> ok
> test_get_types (system_tests_qdmanage.QdmanageTest) ... ok
> test_get_types_with_ssl_profile_type (system_tests_qdmanage.QdmanageTest) ...
> ok
> test_query (system_tests_qdmanage.QdmanageTest) ... ok
> test_specify_container_id_connection_auto_link
> (system_tests_qdmanage.QdmanageTest) ... ok
> test_specify_container_id_connection_link_route
> (system_tests_qdmanage.QdmanageTest) ... ok
> test_stdin (system_tests_qdmanage.QdmanageTest)
> Test piping from stdin ... ok
> test_update (system_tests_qdmanage.QdmanageTest) ... ERROR
> test_worker_threads (system_tests_qdmanage.QdmanageTest) ... ERROR
> test_yy_query_many_links (system_tests_qdmanage.QdmanageTest) ... ERROR
> test_zzz_add_connector (system_tests_qdmanage.QdmanageTest) ... ERROR
> test_zzz_create_delete_listener (system_tests_qdmanage.QdmanageTest) ...
> ERRORRouter R1 output file:
> >>>><<<<Router R1 command file:
> >>>>
> qdrouterd -c test_router_1.conf -I /foo/qpid-dispatch/python
> pid=4732<<<<Router R1 log file tail:
> >>>>
> 2021-01-28 13:11:39.319547 +0000 CONTAINER (debug) Aborting link
> 'ee19e7b8-bd76-4431-8fab-dcca33625f95-$management' due to parent connection
> end (/foo/qpid-dispatch/src/container.c:331)
> 2021-01-28 13:11:39.319600 +0000 CONTAINER (debug) Aborting link
> 'ee19e7b8-bd76-4431-8fab-dcca33625f95-f2923094-9da6-49f3-9472-38d1347dd292'
> due to parent connection end (/foo/qpid-dispatch/src/container.c:331)
> 2021-01-28 13:11:39.319750 +0000 ROUTER_CORE (trace) Core action
> 'link_detach' (/foo/qpid-dispatch/src/router_core/router_core_thread.c:238)
> 2021-01-28 13:11:39.319788 +0000 ROUTER_CORE (info) [C67][L157] Link lost:
> del=1 presett=0 psdrop=0 acc=1 rej=0 rel=0 mod=0 delay1=0 delay10=0
> blocked=no (/foo/qpid-dispatch/src/router_core/connections.c:1054)
> 2021-01-28 13:11:39.319834 +0000 ROUTER_CORE (trace) Core action
> 'link_detach' (/foo/qpid-dispatch/src/router_core/router_core_thread.c:238)
> 2021-01-28 13:11:39.319872 +0000 ROUTER_CORE (info) [C67][L158] Link lost:
> del=1 presett=1 psdrop=0 acc=0 rej=0 rel=0 mod=0 delay1=0 delay10=0
> blocked=no (/foo/qpid-dispatch/src/router_core/connections.c:1054)
> 2021-01-28 13:11:39.319908 +0000 ROUTER_CORE (trace) Core action
> 'connection_closed'
> (/foo/qpid-dispatch/src/router_core/router_core_thread.c:238)
> 2021-01-28 13:11:39.319939 +0000 ROUTER_CORE (info) [C67] Connection Closed
> (/foo/qpid-dispatch/src/router_core/connections.c:1531)
> 2021-01-28 13:11:39.508786 +0000 SERVER (trace) Accepting connection on
> 0.0.0.0:27834 (/foo/qpid-dispatch/src/server.c:854)
> 2021-01-28 13:11:39.508932 +0000 SERVER (trace) [C68]: Accepting incoming
> connection to '0.0.0.0:27834' (/foo/qpid-dispatch/src/server.c:610)
> 2021-01-28 13:11:39.509090 +0000 POLICY (trace) ALLOW Connection '127.0.0.1'
> based on global connection count. nConnections= 4
> (/foo/qpid-dispatch/src/policy.c:269)
> 2021-01-28 13:11:39.509130 +0000 SERVER (info) [C68] Accepted connection to
> 0.0.0.0:27834 from 127.0.0.1:56006 (/foo/qpid-dispatch/src/server.c:753)
> 2021-01-28 13:11:39.509663 +0000 PROTOCOL (trace) [C68]:FRAME: <- AMQP
> (/foo/qpid-dispatch/src/server.c:113)
> 2021-01-28 13:11:39.509728 +0000 PROTOCOL (trace) [C68]:FRAME: 0 <- @open(16)
> [container-id="4b25b26d-794d-4801-817f-5c0d536fd0aa", hostname="0.0.0.0",
> channel-max=32767] (/foo/qpid-dispatch/src/server.c:113)
> 2021-01-28 13:11:39.509892 +0000 ROUTER_CORE (info) [C68] Connection Opened:
> dir=in host=127.0.0.1:56006 vhost= encrypted=no auth=no user=anonymous
> container_id=4b25b26d-794d-4801-817f-5c0d536fd0aa props=
> (/foo/qpid-dispatch/src/router_core/connections.c:139)
> 2021-01-28 13:11:39.509952 +0000 ROUTER_CORE (trace) Core action
> 'connection_opened'
> (/foo/qpid-dispatch/src/router_core/router_core_thread.c:238)
> 2021-01-28 13:11:39.509993 +0000 PROTOCOL (trace) [C68]:FRAME: -> AMQP
> (/foo/qpid-dispatch/src/server.c:113)
> 2021-01-28 13:11:39.510078 +0000 PROTOCOL (trace) [C68]:FRAME: 0 -> @open(16)
> [container-id="R1", max-frame-size=16384, channel-max=32767,
> idle-time-out=60000, offered-capabilities=@PN_SYMBOL[:"ANONYMOUS-RELAY",
> :"qd.streaming-links"], desired-capabilities=@PN_SYMBOL[:"ANONYMOUS-RELAY",
> :"qd.streaming-links"], properties={:product="qpid-dispatch-router",
> :version="1.15.0-SNAPSHOT", :"qd.conn-id"=68}]
> (/foo/qpid-dispatch/src/server.c:113)
> 2021-01-28 13:11:39.512215 +0000 PROTOCOL (trace) [C68]:FRAME: 0 <-
> @begin(17) [next-outgoing-id=0, incoming-window=2147483647,
> outgoing-window=2147483647] (/foo/qpid-dispatch/src/server.c:113)
> 2021-01-28 13:11:39.512301 +0000 PROTOCOL (trace) [C68]:FRAME: 0 <-
> @attach(18) [name="4b25b26d-794d-4801-817f-5c0d536fd0aa-$management",
> handle=0, role=false, snd-settle-mode=2, rcv-settle-mode=0,
> source=@source(40) [durable=0, timeout=0, dynamic=false], target=@target(41)
> [address="$management", durable=0, timeout=0, dynamic=false],
> initial-delivery-count=0, max-message-size=0]
> (/foo/qpid-dispatch/src/server.c:113)
> 2021-01-28 13:11:39.512492 +0000 ROUTER_CORE (trace) Core action
> 'link_first_attach'
> (/foo/qpid-dispatch/src/router_core/router_core_thread.c:238)
> 2021-01-28 13:11:39.512544 +0000 PROTOCOL (trace) [C68]:FRAME: 0 ->
> @begin(17) [remote-channel=0, next-outgoing-id=0, incoming-window=2147483647,
> outgoing-window=2147483647] (/foo/qpid-dispatch/src/server.c:113)
> 2021-01-28 13:11:39.512585 +0000 DEFAULT (trace) Parse tree search for
> '$management' (/foo/qpid-dispatch/src/parse_tree.c:705)
> 2021-01-28 13:11:39.512622 +0000 DEFAULT (trace) Parse tree match not found
> (/foo/qpid-dispatch/src/parse_tree.c:693)
> 2021-01-28 13:11:39.512653 +0000 DEFAULT (trace) Parse tree search for
> '$management' (/foo/qpid-dispatch/src/parse_tree.c:705)
> 2021-01-28 13:11:39.512683 +0000 DEFAULT (trace) Parse tree match not found
> (/foo/qpid-dispatch/src/parse_tree.c:693)
> 2021-01-28 13:11:39.512766 +0000 ROUTER_CORE (info) [C68][L159] Link
> attached: dir=in source={<none> expire:sess} target={$management expire:sess}
> (/foo/qpid-dispatch/src/router_core/connections.c:1813)
> 2021-01-28 13:11:39.512908 +0000 PROTOCOL (trace) [C68]:FRAME: 0 ->
> @attach(18) [name="4b25b26d-794d-4801-817f-5c0d536fd0aa-$management",
> handle=0, role=true, snd-settle-mode=2, rcv-settle-mode=0, source=@source(40)
> [durable=0, expiry-policy=:"session-end", timeout=0, dynamic=false],
> target=@target(41) [address="$management", durable=0,
> expiry-policy=:"session-end", timeout=0, dynamic=false],
> initial-delivery-count=0, max-message-size=0]
> (/foo/qpid-dispatch/src/server.c:113)
> 2021-01-28 13:11:39.512970 +0000 PROTOCOL (trace) [C68]:FRAME: 0 -> @flow(19)
> [next-incoming-id=0, incoming-window=2147483647, next-outgoing-id=0,
> outgoing-window=2147483647, handle=0, delivery-count=0, link-credit=250,
> drain=false] (/foo/qpid-dispatch/src/server.c:113)
> 2021-01-28 13:11:39.514713 +0000 PROTOCOL (trace) [C68]:FRAME: 0 <-
> @attach(18)
> [name="4b25b26d-794d-4801-817f-5c0d536fd0aa-441f31e2-fd0a-40e4-8f27-7ec8b9e0b894",
> handle=1, role=true, snd-settle-mode=2, rcv-settle-mode=0,
> source=@source(40) [durable=0, timeout=0, dynamic=true], target=@target(41)
> [durable=0, timeout=0, dynamic=false], initial-delivery-count=0,
> max-message-size=0] (/foo/qpid-dispatch/src/server.c:113)
> 2021-01-28 13:11:39.514855 +0000 ROUTER_CORE (trace) Core action
> 'link_first_attach'
> (/foo/qpid-dispatch/src/router_core/router_core_thread.c:238)
> 2021-01-28 13:11:39.515011 +0000 ROUTER_CORE (info) [C68][L160] Link
> attached: dir=out source={(dyn)<none> expire:sess} target={<none>
> expire:sess} (/foo/qpid-dispatch/src/router_core/connections.c:1813)
> 2021-01-28 13:11:39.515127 +0000 PROTOCOL (trace) [C68]:FRAME: 0 ->
> @attach(18)
> [name="4b25b26d-794d-4801-817f-5c0d536fd0aa-441f31e2-fd0a-40e4-8f27-7ec8b9e0b894",
> handle=1, role=false, snd-settle-mode=2, rcv-settle-mode=0,
> source=@source(40) [address="amqp:/_topo/0/R1/temp.l0c4z3atGJGNumU",
> durable=0, expiry-policy=:"session-end", timeout=0, dynamic=true],
> target=@target(41) [durable=0, expiry-policy=:"session-end", timeout=0,
> dynamic=false], initial-delivery-count=0, max-message-size=0]
> (/foo/qpid-dispatch/src/server.c:113)
> 2021-01-28 13:11:39.517173 +0000 PROTOCOL (trace) [C68]:FRAME: 0 <- @flow(19)
> [next-incoming-id=0, incoming-window=2147483647, next-outgoing-id=0,
> outgoing-window=2147483647, handle=1, delivery-count=0, link-credit=1,
> drain=false] (/foo/qpid-dispatch/src/server.c:113)
> 2021-01-28 13:11:39.517242 +0000 PROTOCOL (trace) [C68]:FRAME: 0 <-
> @transfer(20) [handle=0, delivery-id=0, delivery-tag=b"1", message-format=0]
> (195)
> "\x00SpE\x00Ss\xd0\x00\x00\x002\x00\x00\x00\x06@@@@\xa1%amqp:/_topo/0/R1/temp.l0c4z3atGJGNumU\xa0\x011\x00St\xd1\x00\x00\x00N\x00\x00\x00\x06\xa1\x09operation\xa1\x06UPDATE\xa1\x04type\xa1\x1corg.apache.qpid.dispatch.log\xa1\x04name\xa1\x0blog/DEFAULT\x00Sw\xd1\x00\x00\x00'\x00\x00\x00\x04\xa1\x0aoutputFile\xa1\x05A.log\xa1\x06enable\xa1\x06trace+"
> (/foo/qpid-dispatch/src/server.c:113)
> 2021-01-28 13:11:39.517325 +0000 ROUTER_CORE (trace) Core action 'link_flow'
> (/foo/qpid-dispatch/src/router_core/router_core_thread.c:238)
> 2021-01-28 13:11:39.517467 +0000 ROUTER_CORE (debug) [C68][L159][D183]
> Delivery created qdr_link_deliver
> (/foo/qpid-dispatch/src/router_core/transfer.c:62)
> 2021-01-28 13:11:39.517501 +0000 ROUTER_CORE (debug) [C68][L159][D183]
> Delivery incref: rc:1 qdr_link_deliver - newly created delivery, add to
> action list (/foo/qpid-dispatch/src/router_core/delivery.c:113)
> 2021-01-28 13:11:39.517531 +0000 ROUTER_CORE (debug) [C68][L159][D183]
> Delivery incref: rc:2 qdr_link_deliver - protect returned value
> (/foo/qpid-dispatch/src/router_core/delivery.c:113)
> 2021-01-28 13:11:39.517573 +0000 ROUTER_CORE (trace) Core action
> 'link_deliver' (/foo/qpid-dispatch/src/router_core/router_core_thread.c:238)
> 2021-01-28 13:11:39.517612 +0000 ROUTER_CORE (debug) [C68][L159][D183]
> Delivery incref: rc:3 referenced by a pn_delivery
> (/foo/qpid-dispatch/src/router_core/delivery.c:113)
> 2021-01-28 13:11:39.517650 +0000 ROUTER_CORE (debug) [C68][L159][D183]
> Delivery decref: rc:3 release protection of return from deliver
> (/foo/qpid-dispatch/src/router_core/delivery.c:147)
> 2021-01-28 13:11:39.517687 +0000 ROUTER_CORE (debug) [C68][L159][D183]
> Delivery incref: rc:4 qdr_forward_on_message_CT - adding to general work
> item (/foo/qpid-dispatch/src/router_core/delivery.c:113)
> 2021-01-28 13:11:39.517778 +0000 ROUTER_CORE (debug) [C68][L159][D183]
> Delivery transfer: qdr_link_forward_CT: action-list -> unsettled-list
> (/foo/qpid-dispatch/src/router_core/transfer.c:707)
> 2021-01-28 13:11:39.518092 +0000 ROUTER_CORE (trace) Core action 'send_to'
> (/foo/qpid-dispatch/src/router_core/router_core_thread.c:238)
> 2021-01-28 13:11:39.518186 +0000 ROUTER_CORE (trace) Core action
> 'settle_subscription_delivery'
> (/foo/qpid-dispatch/src/router_core/router_core_thread.c:238)
> 2021-01-28 13:11:39.518236 +0000 ROUTER_CORE (debug) [C68][L159][D183]
> Delivery decref_CT: rc:2 qdr_settle_subscription_delivery_CT - removed from
> unsettled (/foo/qpid-dispatch/src/router_core/delivery.c:640)
> 2021-01-28 13:11:39.518269 +0000 ROUTER_CORE (debug) [C68][L159][D183]
> Delivery incref: rc:3 qdr_delivery_push_CT - add to updated list
> (/foo/qpid-dispatch/src/router_core/delivery.c:113)
> 2021-01-28 13:11:39.518326 +0000 ROUTER_CORE (debug) [C68][L159][D183]
> Delivery decref_CT: rc:2 qdr_settle_subscription_delivery_CT - removed from
> action (/foo/qpid-dispatch/src/router_core/delivery.c:640)
> 2021-01-28 13:11:39.520539 +0000 AGENT (debug) Agent request
> Message(address=None, properties={u'operation': u'UPDATE', u'type':
> u'org.apache.qpid.dispatch.log', u'name': u'log/DEFAULT'}, body={u'enable':
> u'trace+', u'outputFile': u'A.log'},
> reply_to=u'amqp:/_topo/0/R1/temp.l0c4z3atGJGNumU', correlation_id='1',
> content_type=None)
> (/foo/qpid-dispatch/python/qpid_dispatch_internal/management/agent.py:938)<<<<
> ERROR======================================================================
> ERROR: test_update (system_tests_qdmanage.QdmanageTest)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File "/foo/qpid-dispatch/tests/system_tests_qdmanage.py", line 288, in
> test_update
> output = json.loads(self.run_qdmanage("UPDATE --type
> org.apache.qpid.dispatch.log --name log/DEFAULT "
> File "/foo/qpid-dispatch/tests/system_tests_qdmanage.py", line 94, in
> run_qdmanage
> raise Exception(out if out else str(e))
> Exception: ConnectionException: Connection amqp://0.0.0.0:27834 disconnected:
> Condition('amqp:connection:framing-error', 'connection aborted')
> ======================================================================
> ERROR: test_worker_threads (system_tests_qdmanage.QdmanageTest)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File "/foo/qpid-dispatch/tests/system_tests_qdmanage.py", line 598, in
> test_worker_threads
> output = qd_manager.query('org.apache.qpid.dispatch.router')
> File "/foo/qpid-dispatch/tests/system_test.py", line 1229, in query
> return json.loads(self('QUERY --type=%s' % long_type))
> File "/foo/qpid-dispatch/tests/system_test.py", line 1199, in __call__
> raise Exception("%s\n%s" % (e, out))
> Exception: Process 5042 error: exit code 1, expected 0
> qdmanage QUERY --type=org.apache.qpid.dispatch.router --bus
> amqp://0.0.0.0:27834 --indent=-1 --timeout 60.0
> /foo/qpid-dispatch/build/tests/system_test.dir/system_tests_qdmanage/QdmanageTest/test_worker_threads/qdmanage-62.cmd
> >>>>
> <<<<
> ConnectionException: Connection amqp://0.0.0.0:27834 disconnected:
> Condition('proton.pythonio', 'Connection refused to all addresses')
> ======================================================================
> ERROR: test_yy_query_many_links (system_tests_qdmanage.QdmanageTest)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File "/foo/qpid-dispatch/tests/system_tests_qdmanage.py", line 526, in
> test_yy_query_many_links
> c = BlockingConnection(self.address())
> File
> "/foo/qpid-dispatch/install/lib64/proton/bindings/python/proton/_utils.py",
> line 362, in __init__
> msg="Opening connection")
> File
> "/foo/qpid-dispatch/install/lib64/proton/bindings/python/proton/_utils.py",
> line 487, in wait
> "Connection %s disconnected: %s" % (self.url, self.disconnected))
> ConnectionException: Connection amqp://0.0.0.0:27834 disconnected:
> Condition('proton.pythonio', 'Connection refused to all
> addresses')======================================================================
> ERROR: test_zzz_add_connector (system_tests_qdmanage.QdmanageTest)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File "/foo/qpid-dispatch/tests/system_tests_qdmanage.py", line 399, in
> test_zzz_add_connector
> output = json.loads(self.run_qdmanage(command))
> File "/foo/qpid-dispatch/tests/system_tests_qdmanage.py", line 94, in
> run_qdmanage
> raise Exception(out if out else str(e))
> Exception: ConnectionException: Connection amqp://0.0.0.0:27834 disconnected:
> Condition('proton.pythonio', 'Connection refused to all addresses')
> ======================================================================
> ERROR: test_zzz_create_delete_listener (system_tests_qdmanage.QdmanageTest)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File "/foo/qpid-dispatch/tests/system_tests_qdmanage.py", line 416, in
> test_zzz_create_delete_listener
> listener = self.create(long_type, name, str(listener_port))
> File "/foo/qpid-dispatch/tests/system_tests_qdmanage.py", line 295, in
> create
> connector = json.loads(self.run_qdmanage(create_command))
> File "/foo/qpid-dispatch/tests/system_tests_qdmanage.py", line 94, in
> run_qdmanage
> raise Exception(out if out else str(e))
> Exception: ConnectionException: Connection amqp://0.0.0.0:27834 disconnected:
> Condition('proton.pythonio', 'Connection refused to all addresses')
> ======================================================================
> ERROR: tearDownClass (system_tests_qdmanage.QdmanageTest)
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File "/foo/qpid-dispatch/tests/system_test.py", line 839, in tearDownClass
> cls.tester.teardown()
> File "/foo/qpid-dispatch/tests/system_test.py", line 782, in teardown
> raise RuntimeError("Errors during teardown: \n\n%s" % "\n\n".join([str(e)
> for e in errors]))
> RuntimeError: Errors during teardown: Process 4732 error: exit code -11,
> expected -1
> qdrouterd -c test_router_1.conf -I /foo/qpid-dispatch/python
> /foo/qpid-dispatch/build/tests/system_test.dir/system_tests_qdmanage/QdmanageTest/setUpClass/test_router_1-2.cmd
> >>>>
> <<<<----------------------------------------------------------------------
> Ran 34 tests in 18.317sFAILED (errors=6) {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
