[jira] [Commented] (DISPATCH-332) Heavy message loss happening with 2 interconnected routers
[ https://issues.apache.org/jira/browse/DISPATCH-332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15281502#comment-15281502 ] Ganesh Murthy commented on DISPATCH-332: Vishal tried Ted's suggestion and it works. Marking this as Not a Bug > Heavy message loss happening with 2 interconnected routers > -- > > Key: DISPATCH-332 > URL: https://issues.apache.org/jira/browse/DISPATCH-332 > Project: Qpid Dispatch > Issue Type: Bug > Components: Routing Engine >Affects Versions: 0.6.0 > Environment: Debian 8.3, Qpid Proton 0.12.2 for drivers and > dependency for Qpid Dispatch, Hardware: 2 CPUs, 15 GB RAM, 30 GB HDD. >Reporter: Vishal Sharda >Assignee: Ted Ross >Priority: Blocker > Fix For: 0.6.0 > > Attachments: config1.conf, config2.conf, output.txt > > > We are running two Dispatch Routers each configured for interior mode and the > second router's configuration includes a connector to the first router with > inter-router role. > When we connect one sender to one router and one receiver to the other router > both listening to the same queue, we see all messages (20,000 in our test) > being transmitted. > As soon as we start a second sender connected to the same router to which the > first sender connects and sending to the same queue, we start seeing heavy > message loss. Around 20% of messages are lost with each sender attempting to > send 20,000 messages on its own (40,000 in total) and running in parallel > with the other sender. The message loss happens regardless of the message > size. > We tried with simple_send.py, simple_recv.py as well as send and recv C > executable files from Qpid Proton 0.12.2. > We even saw a crash in the router with the following message: > qdrouterd: /home/vsharda/qpid-dispatch/src/posix/threading.c:71: > sys_mutex_lock: Assertion `result == 0' failed. > Aborted > The message loss was observed with the 0.6.0 SNAPSHOT taken on May 9 as well > as the one taken on March 3 before the router core refactoring. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (DISPATCH-332) Heavy message loss happening with 2 interconnected routers
[ https://issues.apache.org/jira/browse/DISPATCH-332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15281469#comment-15281469 ] Ganesh Murthy commented on DISPATCH-332: Vishal, did you try out Ted's suggestion? Did that work out for you? > Heavy message loss happening with 2 interconnected routers > -- > > Key: DISPATCH-332 > URL: https://issues.apache.org/jira/browse/DISPATCH-332 > Project: Qpid Dispatch > Issue Type: Bug > Components: Routing Engine >Affects Versions: 0.6.0 > Environment: Debian 8.3, Qpid Proton 0.12.2 for drivers and > dependency for Qpid Dispatch, Hardware: 2 CPUs, 15 GB RAM, 30 GB HDD. >Reporter: Vishal Sharda >Assignee: Ted Ross >Priority: Blocker > Fix For: 0.6.0 > > Attachments: config1.conf, config2.conf, output.txt > > > We are running two Dispatch Routers each configured for interior mode and the > second router's configuration includes a connector to the first router with > inter-router role. > When we connect one sender to one router and one receiver to the other router > both listening to the same queue, we see all messages (20,000 in our test) > being transmitted. > As soon as we start a second sender connected to the same router to which the > first sender connects and sending to the same queue, we start seeing heavy > message loss. Around 20% of messages are lost with each sender attempting to > send 20,000 messages on its own (40,000 in total) and running in parallel > with the other sender. The message loss happens regardless of the message > size. > We tried with simple_send.py, simple_recv.py as well as send and recv C > executable files from Qpid Proton 0.12.2. > We even saw a crash in the router with the following message: > qdrouterd: /home/vsharda/qpid-dispatch/src/posix/threading.c:71: > sys_mutex_lock: Assertion `result == 0' failed. > Aborted > The message loss was observed with the 0.6.0 SNAPSHOT taken on May 9 as well > as the one taken on March 3 before the router core refactoring. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (DISPATCH-332) Heavy message loss happening with 2 interconnected routers
[
https://issues.apache.org/jira/browse/DISPATCH-332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15280552#comment-15280552
]
Ted Ross commented on DISPATCH-332:
---
I tried to reproduce your symptom and didn't see any problem.
If you are using simple_send and simple_recv for this test, you will have a
problem with multiple senders on the same address. Simple_recv ignores
duplicate messages so it's possible that the problem you are seeign is a result
of ignored duplicates (two instances of simple_send will send messages with the
same message-id and the receiver will detect/ignore duplicates.
Try removing the first three lines of on_message in simple_recv.py and testing
again:
{noformat}
def on_message(self, event):
- if event.message.id and event.message.id < self.received:
- # ignore duplicate message
- return
if self.expected == 0 or self.received < self.expected:
print event.message.body
self.received += 1
if self.received == self.expected:
event.receiver.close()
event.connection.close()
{noformat}
> Heavy message loss happening with 2 interconnected routers
> --
>
> Key: DISPATCH-332
> URL: https://issues.apache.org/jira/browse/DISPATCH-332
> Project: Qpid Dispatch
> Issue Type: Bug
> Components: Routing Engine
>Affects Versions: 0.6.0
> Environment: Debian 8.3, Qpid Proton 0.12.2 for drivers and
> dependency for Qpid Dispatch, Hardware: 2 CPUs, 15 GB RAM, 30 GB HDD.
>Reporter: Vishal Sharda
>Assignee: Ted Ross
>Priority: Blocker
> Fix For: 0.6.0
>
> Attachments: config1.conf, config2.conf, output.txt
>
>
> We are running two Dispatch Routers each configured for interior mode and the
> second router's configuration includes a connector to the first router with
> inter-router role.
> When we connect one sender to one router and one receiver to the other router
> both listening to the same queue, we see all messages (20,000 in our test)
> being transmitted.
> As soon as we start a second sender connected to the same router to which the
> first sender connects and sending to the same queue, we start seeing heavy
> message loss. Around 20% of messages are lost with each sender attempting to
> send 20,000 messages on its own (40,000 in total) and running in parallel
> with the other sender. The message loss happens regardless of the message
> size.
> We tried with simple_send.py, simple_recv.py as well as send and recv C
> executable files from Qpid Proton 0.12.2.
> We even saw a crash in the router with the following message:
> qdrouterd: /home/vsharda/qpid-dispatch/src/posix/threading.c:71:
> sys_mutex_lock: Assertion `result == 0' failed.
> Aborted
> The message loss was observed with the 0.6.0 SNAPSHOT taken on May 9 as well
> as the one taken on March 3 before the router core refactoring.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (DISPATCH-332) Heavy message loss happening with 2 interconnected routers
[ https://issues.apache.org/jira/browse/DISPATCH-332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15280432#comment-15280432 ] Ganesh Murthy commented on DISPATCH-332: Can you please attach the two router config files you are using to this Jira? Thanks. > Heavy message loss happening with 2 interconnected routers > -- > > Key: DISPATCH-332 > URL: https://issues.apache.org/jira/browse/DISPATCH-332 > Project: Qpid Dispatch > Issue Type: Bug > Components: Routing Engine >Affects Versions: 0.6.0 > Environment: Debian 8.3, Qpid Proton 0.12.2 for drivers and > dependency for Qpid Dispatch, Hardware: 2 CPUs, 15 GB RAM, 30 GB HDD. >Reporter: Vishal Sharda >Assignee: Ted Ross >Priority: Blocker > Fix For: 0.6.0 > > > We are running two Dispatch Routers each configured for interior mode and the > second router's configuration includes a connector to the first router with > inter-router role. > When we connect one sender to one router and one receiver to the other router > both listening to the same queue, we see all messages (20,000 in our test) > being transmitted. > As soon as we start a second sender connected to the same router to which the > first sender connects and sending to the same queue, we start seeing heavy > message loss. Around 20% of messages are lost with each sender attempting to > send 20,000 messages on its own (40,000 in total) and running in parallel > with the other sender. The message loss happens regardless of the message > size. > We tried with simple_send.py, simple_recv.py as well as send and recv C > executable files from Qpid Proton 0.12.2. > We even saw a crash in the router with the following message: > qdrouterd: /home/vsharda/qpid-dispatch/src/posix/threading.c:71: > sys_mutex_lock: Assertion `result == 0' failed. > Aborted > The message loss was observed with the 0.6.0 SNAPSHOT taken on May 9 as well > as the one taken on March 3 before the router core refactoring. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (DISPATCH-332) Heavy message loss happening with 2 interconnected routers
[
https://issues.apache.org/jira/browse/DISPATCH-332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15280386#comment-15280386
]
Vishal Sharda commented on DISPATCH-332:
I used the following fixedAddress in both the configuration files.
fixedAddress {
prefix: /
fanout: single
bias: closest
}
Insecure port 5672 was used for all the communication.
Everything is working fine if the 2 senders and 1 receiver are all attached to
the same router and also if 1 sender and 1 receiver are each connected to the
two interconnected routers. The issue occurs only when we start a second
parallel sender on the same router where one sender is already active.
Increasing the number of parallel senders and receivers further increases the
percentage of messages lost.
> Heavy message loss happening with 2 interconnected routers
> --
>
> Key: DISPATCH-332
> URL: https://issues.apache.org/jira/browse/DISPATCH-332
> Project: Qpid Dispatch
> Issue Type: Bug
> Components: Routing Engine
>Affects Versions: 0.6.0
> Environment: Debian 8.3, Qpid Proton 0.12.2 for drivers and
> dependency for Qpid Dispatch, Hardware: 2 CPUs, 15 GB RAM, 30 GB HDD.
>Reporter: Vishal Sharda
>Assignee: Ted Ross
>Priority: Blocker
> Fix For: 0.6.0
>
>
> We are running two Dispatch Routers each configured for interior mode and the
> second router's configuration includes a connector to the first router with
> inter-router role.
> When we connect one sender to one router and one receiver to the other router
> both listening to the same queue, we see all messages (20,000 in our test)
> being transmitted.
> As soon as we start a second sender connected to the same router to which the
> first sender connects and sending to the same queue, we start seeing heavy
> message loss. Around 20% of messages are lost with each sender attempting to
> send 20,000 messages on its own (40,000 in total) and running in parallel
> with the other sender. The message loss happens regardless of the message
> size.
> We tried with simple_send.py, simple_recv.py as well as send and recv C
> executable files from Qpid Proton 0.12.2.
> We even saw a crash in the router with the following message:
> qdrouterd: /home/vsharda/qpid-dispatch/src/posix/threading.c:71:
> sys_mutex_lock: Assertion `result == 0' failed.
> Aborted
> The message loss was observed with the 0.6.0 SNAPSHOT taken on May 9 as well
> as the one taken on March 3 before the router core refactoring.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org
For additional commands, e-mail: dev-h...@qpid.apache.org
[jira] [Commented] (DISPATCH-332) Heavy message loss happening with 2 interconnected routers
[ https://issues.apache.org/jira/browse/DISPATCH-332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15280288#comment-15280288 ] Ted Ross commented on DISPATCH-332: --- Can you provide information about what distribution settings you were using in your test? What addresses did you use? Did you provide any configuration for those addresses? Were they multicast, closest, or balanced? -Ted > Heavy message loss happening with 2 interconnected routers > -- > > Key: DISPATCH-332 > URL: https://issues.apache.org/jira/browse/DISPATCH-332 > Project: Qpid Dispatch > Issue Type: Bug > Components: Routing Engine >Affects Versions: 0.6.0 > Environment: Debian 8.3, Qpid Proton 0.12.2 for drivers and > dependency for Qpid Dispatch, Hardware: 2 CPUs, 15 GB RAM, 30 GB HDD. >Reporter: Vishal Sharda >Assignee: Ted Ross >Priority: Blocker > Fix For: 0.6.0 > > > We are running two Dispatch Routers each configured for interior mode and the > second router's configuration includes a connector to the first router with > inter-router role. > When we connect one sender to one router and one receiver to the other router > both listening to the same queue, we see all messages (20,000 in our test) > being transmitted. > As soon as we start a second sender connected to the same router to which the > first sender connects and sending to the same queue, we start seeing heavy > message loss. Around 20% of messages are lost with each sender attempting to > send 20,000 messages on its own (40,000 in total) and running in parallel > with the other sender. The message loss happens regardless of the message > size. > We tried with simple_send.py, simple_recv.py as well as send and recv C > executable files from Qpid Proton 0.12.2. > We even saw a crash in the router with the following message: > qdrouterd: /home/vsharda/qpid-dispatch/src/posix/threading.c:71: > sys_mutex_lock: Assertion `result == 0' failed. > Aborted > The message loss was observed with the 0.6.0 SNAPSHOT taken on May 9 as well > as the one taken on March 3 before the router core refactoring. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
