[jira] [Updated] (DISPATCH-380) Router stops receiving messages from multiple senders publishing to multiple queues in parallel
[ https://issues.apache.org/jira/browse/DISPATCH-380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Ross updated DISPATCH-380: -- Fix Version/s: 0.8.0 > Router stops receiving messages from multiple senders publishing to multiple > queues in parallel > --- > > Key: DISPATCH-380 > URL: https://issues.apache.org/jira/browse/DISPATCH-380 > Project: Qpid Dispatch > Issue Type: Bug > Components: Routing Engine > Environment: Debian 8.3, Apache Qpid Proton 0.13.0-RC for drivers and > dependencies, Hardware: 2 CPUs, 15 GB RAM, 60 GB HDD each on 3 separate > machines >Reporter: Vishal Sharda >Priority: Critical > Fix For: 0.8.0 > > Attachments: AWS_hung_at_round_figures.png, > AWS_hung_for_4_seconds.png, AWS_uneven_start.png, Senders_1.png, > Senders_2.png, Senders_3_10_Queues.png, Senders_4_10_queues.png, > Single_router_testing_results.pdf, qdstat_wrong_output.png > > > I am running a Java Client against a cluster of 3 interior routers connected > to each other. 2-way SSL is enabled for all the connections. > There were 20 simultaneous queues with 20 senders on each queue and each > sender publishing 1000 messages. All the senders were connected to Router 1. > 20 receivers were connected to Router 2 with 1 receiver receiving from each > queue. > In the first run, router stopped receiving incoming messages after delivering > 386,339 out of 400K "Hello World!" messages. > In the second run, 388,781 messages out of 400K were delivered. > I reduced the number of queues to 10 (halving total number of messages to > 200K) and the issue occurred again. > I ran the Java client on an 8 CPU machine again with 10 queues and the issue > occurred again after delivering just 54K out of 200K messages. > All the senders were hung (still connected) with no messages flowing at all. > Connection information from qdstat: > When the messages are flowing properly and I run "qdstat -c", I see all the > senders as secure and authenticated. > After they hang and I run "qdstat -c", it erroneously shows all the clients > as insecure and unauthenticated. > Shortly after the clients hang, all the queues are deleted from the router > network but connections are still shown until I terminate the clients. > I saw this erroneous situation before also when "qdstat -c" showed some > senders as secure and authentic but some as insecure/unauthentic. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Updated] (DISPATCH-380) Router stops receiving messages from multiple senders publishing to multiple queues in parallel
[ https://issues.apache.org/jira/browse/DISPATCH-380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Ross updated DISPATCH-380: -- Fix Version/s: (was: 0.6.0) > Router stops receiving messages from multiple senders publishing to multiple > queues in parallel > --- > > Key: DISPATCH-380 > URL: https://issues.apache.org/jira/browse/DISPATCH-380 > Project: Qpid Dispatch > Issue Type: Bug > Components: Routing Engine > Environment: Debian 8.3, Apache Qpid Proton 0.13.0-RC for drivers and > dependencies, Hardware: 2 CPUs, 15 GB RAM, 60 GB HDD each on 3 separate > machines >Reporter: Vishal Sharda >Priority: Critical > Attachments: AWS_hung_at_round_figures.png, > AWS_hung_for_4_seconds.png, AWS_uneven_start.png, Senders_1.png, > Senders_2.png, Senders_3_10_Queues.png, Senders_4_10_queues.png, > Single_router_testing_results.pdf, qdstat_wrong_output.png > > > I am running a Java Client against a cluster of 3 interior routers connected > to each other. 2-way SSL is enabled for all the connections. > There were 20 simultaneous queues with 20 senders on each queue and each > sender publishing 1000 messages. All the senders were connected to Router 1. > 20 receivers were connected to Router 2 with 1 receiver receiving from each > queue. > In the first run, router stopped receiving incoming messages after delivering > 386,339 out of 400K "Hello World!" messages. > In the second run, 388,781 messages out of 400K were delivered. > I reduced the number of queues to 10 (halving total number of messages to > 200K) and the issue occurred again. > I ran the Java client on an 8 CPU machine again with 10 queues and the issue > occurred again after delivering just 54K out of 200K messages. > All the senders were hung (still connected) with no messages flowing at all. > Connection information from qdstat: > When the messages are flowing properly and I run "qdstat -c", I see all the > senders as secure and authenticated. > After they hang and I run "qdstat -c", it erroneously shows all the clients > as insecure and unauthenticated. > Shortly after the clients hang, all the queues are deleted from the router > network but connections are still shown until I terminate the clients. > I saw this erroneous situation before also when "qdstat -c" showed some > senders as secure and authentic but some as insecure/unauthentic. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Updated] (DISPATCH-380) Router stops receiving messages from multiple senders publishing to multiple queues in parallel
[ https://issues.apache.org/jira/browse/DISPATCH-380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vishal Sharda updated DISPATCH-380: --- Attachment: Single_router_testing_results.pdf Attached a file Single_router_testing_results.pdf containing several results of testing a single router. > Router stops receiving messages from multiple senders publishing to multiple > queues in parallel > --- > > Key: DISPATCH-380 > URL: https://issues.apache.org/jira/browse/DISPATCH-380 > Project: Qpid Dispatch > Issue Type: Bug > Components: Routing Engine > Environment: Debian 8.3, Apache Qpid Proton 0.13.0-RC for drivers and > dependencies, Hardware: 2 CPUs, 15 GB RAM, 60 GB HDD each on 3 separate > machines >Reporter: Vishal Sharda >Priority: Critical > Fix For: 0.6.0 > > Attachments: AWS_hung_at_round_figures.png, > AWS_hung_for_4_seconds.png, AWS_uneven_start.png, Senders_1.png, > Senders_2.png, Senders_3_10_Queues.png, Senders_4_10_queues.png, > Single_router_testing_results.pdf, qdstat_wrong_output.png > > > I am running a Java Client against a cluster of 3 interior routers connected > to each other. 2-way SSL is enabled for all the connections. > There were 20 simultaneous queues with 20 senders on each queue and each > sender publishing 1000 messages. All the senders were connected to Router 1. > 20 receivers were connected to Router 2 with 1 receiver receiving from each > queue. > In the first run, router stopped receiving incoming messages after delivering > 386,339 out of 400K "Hello World!" messages. > In the second run, 388,781 messages out of 400K were delivered. > I reduced the number of queues to 10 (halving total number of messages to > 200K) and the issue occurred again. > I ran the Java client on an 8 CPU machine again with 10 queues and the issue > occurred again after delivering just 54K out of 200K messages. > All the senders were hung (still connected) with no messages flowing at all. > Connection information from qdstat: > When the messages are flowing properly and I run "qdstat -c", I see all the > senders as secure and authenticated. > After they hang and I run "qdstat -c", it erroneously shows all the clients > as insecure and unauthenticated. > Shortly after the clients hang, all the queues are deleted from the router > network but connections are still shown until I terminate the clients. > I saw this erroneous situation before also when "qdstat -c" showed some > senders as secure and authentic but some as insecure/unauthentic. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Updated] (DISPATCH-380) Router stops receiving messages from multiple senders publishing to multiple queues in parallel
[ https://issues.apache.org/jira/browse/DISPATCH-380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vishal Sharda updated DISPATCH-380: --- Attachment: AWS_uneven_start.png AWS_hung_for_4_seconds.png AWS_hung_at_round_figures.png I ran single instance of ADR already built on an AWS running. This was an interior router with no connections to other routers. I used Proton-J Messenger API based Java client that uses 2-way SSL. I started 10 receivers first with each receiver listening on a different endpoint (total 10 endpoints created). Then I started 20 senders on each endpoint (total 200 senders) each publishing 1000 “Hello World!” messages. Total 200K messages were to be delivered but I saw that all the senders hung after only 70K messages were delivered. In the second exact run, they hung after delivery of 67K messages. The attached screenshots show the following: 1.) AWS_uneven_start.png: Uneven start on different endpoints. 2.) AWS_hung_at_round_figures.png: Just before my senders hung, the number of messages sent/received on each endpoint became a round figure. 3.) AWS_hung_for_4_seconds.png: After 4 seconds, no more messages went through and all the counts remained intact. Finally, all the endpoints were deleted from the router but qdstat –c still showed connections from my senders but wrongly as insecure and inauthentic. > Router stops receiving messages from multiple senders publishing to multiple > queues in parallel > --- > > Key: DISPATCH-380 > URL: https://issues.apache.org/jira/browse/DISPATCH-380 > Project: Qpid Dispatch > Issue Type: Bug > Components: Routing Engine > Environment: Debian 8.3, Apache Qpid Proton 0.13.0-RC for drivers and > dependencies, Hardware: 2 CPUs, 15 GB RAM, 60 GB HDD each on 3 separate > machines >Reporter: Vishal Sharda >Priority: Critical > Fix For: 0.6.0 > > Attachments: AWS_hung_at_round_figures.png, > AWS_hung_for_4_seconds.png, AWS_uneven_start.png, Senders_1.png, > Senders_2.png, Senders_3_10_Queues.png, Senders_4_10_queues.png, > qdstat_wrong_output.png > > > I am running a Java Client against a cluster of 3 interior routers connected > to each other. 2-way SSL is enabled for all the connections. > There were 20 simultaneous queues with 20 senders on each queue and each > sender publishing 1000 messages. All the senders were connected to Router 1. > 20 receivers were connected to Router 2 with 1 receiver receiving from each > queue. > In the first run, router stopped receiving incoming messages after delivering > 386,339 out of 400K "Hello World!" messages. > In the second run, 388,781 messages out of 400K were delivered. > I reduced the number of queues to 10 (halving total number of messages to > 200K) and the issue occurred again. > I ran the Java client on an 8 CPU machine again with 10 queues and the issue > occurred again after delivering just 54K out of 200K messages. > All the senders were hung (still connected) with no messages flowing at all. > Connection information from qdstat: > When the messages are flowing properly and I run "qdstat -c", I see all the > senders as secure and authenticated. > After they hang and I run "qdstat -c", it erroneously shows all the clients > as insecure and unauthenticated. > Shortly after the clients hang, all the queues are deleted from the router > network but connections are still shown until I terminate the clients. > I saw this erroneous situation before also when "qdstat -c" showed some > senders as secure and authentic but some as insecure/unauthentic. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Updated] (DISPATCH-380) Router stops receiving messages from multiple senders publishing to multiple queues in parallel
[ https://issues.apache.org/jira/browse/DISPATCH-380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vishal Sharda updated DISPATCH-380: --- Attachment: qdstat_wrong_output.png Senders_4_10_queues.png Senders_3_10_Queues.png Senders_2.png Senders_1.png Screenshots showing how qdstat shows senders connected but insecure and unauthenticated after they hang. > Router stops receiving messages from multiple senders publishing to multiple > queues in parallel > --- > > Key: DISPATCH-380 > URL: https://issues.apache.org/jira/browse/DISPATCH-380 > Project: Qpid Dispatch > Issue Type: Bug > Components: Routing Engine > Environment: Debian 8.3, Apache Qpid Proton 0.13.0-RC for drivers and > dependencies, Hardware: 2 CPUs, 15 GB RAM, 60 GB HDD each on 3 separate > machines >Reporter: Vishal Sharda >Priority: Critical > Fix For: 0.6.0 > > Attachments: Senders_1.png, Senders_2.png, Senders_3_10_Queues.png, > Senders_4_10_queues.png, qdstat_wrong_output.png > > > I am running a Java Client against a cluster of 3 interior routers connected > to each other. 2-way SSL is enabled for all the connections. > There were 20 simultaneous queues with 20 senders on each queue and each > sender publishing 1000 messages. All the senders were connected to Router 1. > 20 receivers were connected to Router 2 with 1 receiver receiving from each > queue. > In the first run, router stopped receiving incoming messages after delivering > 386,339 out of 400K "Hello World!" messages. > In the second run, 388,781 messages out of 400K were delivered. > I reduced the number of queues to 10 (halving total number of messages to > 200K) and the issue occurred again. > I ran the Java client on an 8 CPU machine again with 10 queues and the issue > occurred again after delivering just 54K out of 200K messages. > All the senders were hung (still connected) with no messages flowing at all. > Connection information from qdstat: > When the messages are flowing properly and I run "qdstat -c", I see all the > senders as secure and authenticated. > After they hang and I run "qdstat -c", it erroneously shows all the clients > as insecure and unauthenticated. > Shortly after the clients hang, all the queues are deleted from the router > network but connections are still shown until I terminate the clients. > I saw this erroneous situation before also when "qdstat -c" showed some > senders as secure and authentic but some as insecure/unauthentic. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
