Re: Java Broker (7.0.6) stops delivering queue/consumer messages after 4 GB data transfer
On Sat, 4 Aug 2018 at 00:11, Dyslin, Mike wrote: > Rob, > > We do not know how our consumer is managing credit, or anything about > credit flow, but have started looking for information on this. It sounds > like the right place to look. If we could figure out how to turn off the > flow control limits, that may do it. We inherited most of this code, and > the original authors of the code are long gone, so we don't have the > experience of putting it all together. Perhaps the defaults for this > credit flow is different between the C++ and Java brokers. > > OK, we'll try to figure out how to get protocol logging turned on and get > a log file to look at. > Thanks - I had a quick look at the Java Broker code for managing credit and didn't immediately see any obvious errors that would hit a properly functioning client, *however* it does look like that if the client is improperly managing credit there would be an overflow error where a value that should be an unsigned integral value (the amount of remaining credit) will turn negative. The broker is storing the amount of outstanding credit as a (signed) Java long value. AMQP 0-10 defines the message.flow command as "This command controls the flow of message data to a given destination. It is used by the recipient of messages to dynamically match the incoming rate of message flow to its processing or forwarding capacity. Upon receipt of this command, the sender must add "value" number of the specified unit to the available credit balance for the specified destination." So if the client is repeatedly adding more credit than it needs the stored "limit" in the broker might overflow and turn negative. At this point the broker would stop sending messages. -- Rob > > Thanks, > Mike > > -Original Message- > From: Rob Godfrey [mailto:rob.j.godf...@gmail.com] > Sent: Friday, August 3, 2018 11:11 AM > To: users@qpid.apache.org > Cc: Mears, David B ; Herren, Elaine < > elaine.her...@hpe.com>; Rao, Shobha (NonStop) > > Subject: Re: Java Broker (7.0.6) stops delivering queue/consumer messages > after 4 GB data transfer > > On Fri, 3 Aug 2018 at 19:18, Dyslin, Mike wrote: > > > Rob, > > > > Our consumer client version is 1.37: > > > >bash 0 3: rpm -qa | grep -i qpid > >qpid-cpp-client-devel-1.37.0-1.el7.x86_64 > >python-qpid-1.37.0-1.el7.noarch > >qpid-qmf-1.37.0-1.el7.x86_64 > >qpid-cpp-client-1.37.0-1.el7.x86_64 > >python-qpid-qmf-1.37.0-1.el7.x86_64 > >qpid-cpp-server-1.37.0-1.el7.x86_64 > >qpid-proton-c-0.18.1-1.el7.x86_64 > >qpid-tools-1.37.0-1.el7.noarch > > > > Our producer client version is 6.1.5: > > -rw-r--r--1 NSDA.NSDA NSDA 570873 Jul 26 15:42 > > qpid-client-6.1.5.jar > > -rw-r--r--1 NSDA.NSDA NSDA 864493 Jul 26 15:42 > > qpid-common-6.1.5.jar > > > > Both producer and consumer clients use AMQP_0_10 protocol. > > > > OK - so my assumption here is that there is some issue in the management > of credit in the broker (or possibly in the client). AMQP 0-10 has two > distinct credit flow modes "credit" and "window" and also allows the > consumer to separately set limits for both "message" and "byte" credit. > I'm not very familiar with the C++ API, but do you know how your consumer > is managing credit? > > One thing that would be very helpful in trying to diagnose this problem is > getting protocol logging for the consumer (at least for the start of the > consumer where it sets up the credit flow mode, and towards the end where > it would be interesting to see the credit being allocated just before > message flow stops). > > -- Rob > > > > > > Thanks for your assistance, > > Mike > > > > FYI - There may be a typo on the past releases web page. I believe > "2017" > > should be "2018" in "Qpid JMS AMQP 0-x 6.3.2, July 2017". URL: > > https://qpid.apache.org/releases/index.html#past-releases > > > > -Original Message- > > From: Rob Godfrey [mailto:rob.j.godf...@gmail.com] > > Sent: Thursday, August 2, 2018 5:57 PM > > To: users@qpid.apache.org > > Cc: Mears, David B ; Herren, Elaine < > > elaine.her...@hpe.com>; Rao, Shobha (NonStop) > > > > > > Subject: Re: Java Broker (7.0.6) stops delivering queue/consumer > > messages after 4 GB data transfer > > > > Hi Mike, > > > > On Fri, 3 Aug 2018 at 01:25, Dyslin, Mike wrote: > > > > > This is my first submit to this email group. Hopefully this is the > > > correct place to post this problem. > > > > > > > > This is exactly the right place to post this problem. > > > > > > > We are running a continuous stream of message (about 5K each) from > > > producer to consumer over a single java broker queue at a rate of > > > about 600 messages/second. Outbound message flow stops after > > > transferring 4 GB of message data (about 770,000 messages in 25 > > > minutes). The Web Management Console page for our consumer > > > connection > > shows the total "Outbound Bytes" > > > growing steadily until it reaches 4.0 GB
RE: Java Broker (7.0.6) stops delivering queue/consumer messages after 4 GB data transfer
Rob, We do not know how our consumer is managing credit, or anything about credit flow, but have started looking for information on this. It sounds like the right place to look. If we could figure out how to turn off the flow control limits, that may do it. We inherited most of this code, and the original authors of the code are long gone, so we don't have the experience of putting it all together. Perhaps the defaults for this credit flow is different between the C++ and Java brokers. OK, we'll try to figure out how to get protocol logging turned on and get a log file to look at. Thanks, Mike -Original Message- From: Rob Godfrey [mailto:rob.j.godf...@gmail.com] Sent: Friday, August 3, 2018 11:11 AM To: users@qpid.apache.org Cc: Mears, David B ; Herren, Elaine ; Rao, Shobha (NonStop) Subject: Re: Java Broker (7.0.6) stops delivering queue/consumer messages after 4 GB data transfer On Fri, 3 Aug 2018 at 19:18, Dyslin, Mike wrote: > Rob, > > Our consumer client version is 1.37: > >bash 0 3: rpm -qa | grep -i qpid >qpid-cpp-client-devel-1.37.0-1.el7.x86_64 >python-qpid-1.37.0-1.el7.noarch >qpid-qmf-1.37.0-1.el7.x86_64 >qpid-cpp-client-1.37.0-1.el7.x86_64 >python-qpid-qmf-1.37.0-1.el7.x86_64 >qpid-cpp-server-1.37.0-1.el7.x86_64 >qpid-proton-c-0.18.1-1.el7.x86_64 >qpid-tools-1.37.0-1.el7.noarch > > Our producer client version is 6.1.5: > -rw-r--r--1 NSDA.NSDA NSDA 570873 Jul 26 15:42 > qpid-client-6.1.5.jar > -rw-r--r--1 NSDA.NSDA NSDA 864493 Jul 26 15:42 > qpid-common-6.1.5.jar > > Both producer and consumer clients use AMQP_0_10 protocol. > OK - so my assumption here is that there is some issue in the management of credit in the broker (or possibly in the client). AMQP 0-10 has two distinct credit flow modes "credit" and "window" and also allows the consumer to separately set limits for both "message" and "byte" credit. I'm not very familiar with the C++ API, but do you know how your consumer is managing credit? One thing that would be very helpful in trying to diagnose this problem is getting protocol logging for the consumer (at least for the start of the consumer where it sets up the credit flow mode, and towards the end where it would be interesting to see the credit being allocated just before message flow stops). -- Rob > > Thanks for your assistance, > Mike > > FYI - There may be a typo on the past releases web page. I believe "2017" > should be "2018" in "Qpid JMS AMQP 0-x 6.3.2, July 2017". URL: > https://qpid.apache.org/releases/index.html#past-releases > > -Original Message- > From: Rob Godfrey [mailto:rob.j.godf...@gmail.com] > Sent: Thursday, August 2, 2018 5:57 PM > To: users@qpid.apache.org > Cc: Mears, David B ; Herren, Elaine < > elaine.her...@hpe.com>; Rao, Shobha (NonStop) > > > Subject: Re: Java Broker (7.0.6) stops delivering queue/consumer > messages after 4 GB data transfer > > Hi Mike, > > On Fri, 3 Aug 2018 at 01:25, Dyslin, Mike wrote: > > > This is my first submit to this email group. Hopefully this is the > > correct place to post this problem. > > > > > This is exactly the right place to post this problem. > > > > We are running a continuous stream of message (about 5K each) from > > producer to consumer over a single java broker queue at a rate of > > about 600 messages/second. Outbound message flow stops after > > transferring 4 GB of message data (about 770,000 messages in 25 > > minutes). The Web Management Console page for our consumer > > connection > shows the total "Outbound Bytes" > > growing steadily until it reaches 4.0 GB and stops with "Last I/O time" > > unchanging thereafter. > > > > After outbound messages stop: > > Inbound messages continue on the producer connection (well past 4.0 > > GB) and are kept in the queue until they expire with a time-to-live > > value of 3 minutes. The queue grows until is stabilizes with a > > steady > > 600 m/s inbound, and 600 m/s expiring and being deleted from the > > queue (as expected). The Web Management Console shows that the > > consumer connection remains open and is a consumer on the queue, and > > the queue shows the connection as a consumer on the queue. > > > > If I run the exact same test replacing the Java Broker with a C++ > > broker (1.37.0), message flow continues well past the 4 GB barrier. > > I kept it running for about 17 hours reaching about 37 million > > messages, about 180 GB data transferred on the queue. > > > > Since the only difference seems to be the broker, this seems to > > point to a problem with the Java Broker, and not issues with our > > producer, consumer or network issues. Could there be some problem > > with our java broker configuration that would explain this behaviour? > > > > Unfortunately this sounds like it may be a bug in the Java Broker :-( > > > > > > Has anyone out there experienced more than 4 GB of outbound data on
Re: Java Broker (7.0.6) stops delivering queue/consumer messages after 4 GB data transfer
On Fri, 3 Aug 2018 at 20:10, Rob Godfrey wrote: > > > On Fri, 3 Aug 2018 at 19:18, Dyslin, Mike wrote: > >> Rob, >> >> Our consumer client version is 1.37: >> >>bash 0 3: rpm -qa | grep -i qpid >>qpid-cpp-client-devel-1.37.0-1.el7.x86_64 >>python-qpid-1.37.0-1.el7.noarch >>qpid-qmf-1.37.0-1.el7.x86_64 >>qpid-cpp-client-1.37.0-1.el7.x86_64 >>python-qpid-qmf-1.37.0-1.el7.x86_64 >>qpid-cpp-server-1.37.0-1.el7.x86_64 >>qpid-proton-c-0.18.1-1.el7.x86_64 >>qpid-tools-1.37.0-1.el7.noarch >> >> Our producer client version is 6.1.5: >> -rw-r--r--1 NSDA.NSDA NSDA 570873 Jul 26 15:42 >> qpid-client-6.1.5.jar >> -rw-r--r--1 NSDA.NSDA NSDA 864493 Jul 26 15:42 >> qpid-common-6.1.5.jar >> >> Both producer and consumer clients use AMQP_0_10 protocol. >> > > OK - so my assumption here is that there is some issue in the management > of credit in the broker (or possibly in the client). AMQP 0-10 has two > distinct credit flow modes "credit" and "window" and also allows the > consumer to separately set limits for both "message" and "byte" credit. > I'm not very familiar with the C++ API, but do you know how your consumer > is managing credit? > > One thing that would be very helpful in trying to diagnose this problem is > getting protocol logging for the consumer (at least for the start of the > consumer where it sets up the credit flow mode, and towards the end where > it would be interesting to see the credit being allocated just before > message flow stops). > As an aside, to enable broker protocol logging for AMQP 0-10 you would need to enable DEBUG logging for the logger org.apache.qpid.server.protocol.v0_10.ServerConnection , instructions on how to configure logging can be found here: https://qpid.apache.org/releases/qpid-broker-j-7.0.6/book/Java-Broker-Runtime.html#Java-Broker-Runtime-Logging If you are able to configure logging (and ensuring you have removed any confidential information from the logs), snippets from the logs showing the first parts (say the first 1000 messages or so) and the last part (making sure to include the last few MessageFlow commands sent by the client to the broker) would be super useful (note that the mailing list tends to remove attachments so the the easiest way is normally to create a JIRA for the issue and attach them there). Thanks again, Rob > -- Rob > > >> >> Thanks for your assistance, >> Mike >> >> FYI - There may be a typo on the past releases web page. I believe >> "2017" should be "2018" in "Qpid JMS AMQP 0-x 6.3.2, July 2017". URL: >> https://qpid.apache.org/releases/index.html#past-releases >> >> -Original Message- >> From: Rob Godfrey [mailto:rob.j.godf...@gmail.com] >> Sent: Thursday, August 2, 2018 5:57 PM >> To: users@qpid.apache.org >> Cc: Mears, David B ; Herren, Elaine < >> elaine.her...@hpe.com>; Rao, Shobha (NonStop) < >> shobha.jayatheer...@hpe.com> >> Subject: Re: Java Broker (7.0.6) stops delivering queue/consumer messages >> after 4 GB data transfer >> >> Hi Mike, >> >> On Fri, 3 Aug 2018 at 01:25, Dyslin, Mike wrote: >> >> > This is my first submit to this email group. Hopefully this is the >> > correct place to post this problem. >> > >> > >> This is exactly the right place to post this problem. >> >> >> > We are running a continuous stream of message (about 5K each) from >> > producer to consumer over a single java broker queue at a rate of >> > about 600 messages/second. Outbound message flow stops after >> > transferring 4 GB of message data (about 770,000 messages in 25 >> > minutes). The Web Management Console page for our consumer connection >> shows the total "Outbound Bytes" >> > growing steadily until it reaches 4.0 GB and stops with "Last I/O time" >> > unchanging thereafter. >> > >> > After outbound messages stop: >> > Inbound messages continue on the producer connection (well past 4.0 >> > GB) and are kept in the queue until they expire with a time-to-live >> > value of 3 minutes. The queue grows until is stabilizes with a steady >> > 600 m/s inbound, and 600 m/s expiring and being deleted from the queue >> > (as expected). The Web Management Console shows that the consumer >> > connection remains open and is a consumer on the queue, and the queue >> > shows the connection as a consumer on the queue. >> > >> > If I run the exact same test replacing the Java Broker with a C++ >> > broker (1.37.0), message flow continues well past the 4 GB barrier. I >> > kept it running for about 17 hours reaching about 37 million messages, >> > about 180 GB data transferred on the queue. >> > >> > Since the only difference seems to be the broker, this seems to point >> > to a problem with the Java Broker, and not issues with our producer, >> > consumer or network issues. Could there be some problem with our java >> > broker configuration that would explain this behaviour? >> > >> >> Unfortunately this sounds like it may be a bug in the Java Broker :-( >> >> >> >
Re: Java Broker (7.0.6) stops delivering queue/consumer messages after 4 GB data transfer
On Fri, 3 Aug 2018 at 19:18, Dyslin, Mike wrote: > Rob, > > Our consumer client version is 1.37: > >bash 0 3: rpm -qa | grep -i qpid >qpid-cpp-client-devel-1.37.0-1.el7.x86_64 >python-qpid-1.37.0-1.el7.noarch >qpid-qmf-1.37.0-1.el7.x86_64 >qpid-cpp-client-1.37.0-1.el7.x86_64 >python-qpid-qmf-1.37.0-1.el7.x86_64 >qpid-cpp-server-1.37.0-1.el7.x86_64 >qpid-proton-c-0.18.1-1.el7.x86_64 >qpid-tools-1.37.0-1.el7.noarch > > Our producer client version is 6.1.5: > -rw-r--r--1 NSDA.NSDA NSDA 570873 Jul 26 15:42 > qpid-client-6.1.5.jar > -rw-r--r--1 NSDA.NSDA NSDA 864493 Jul 26 15:42 > qpid-common-6.1.5.jar > > Both producer and consumer clients use AMQP_0_10 protocol. > OK - so my assumption here is that there is some issue in the management of credit in the broker (or possibly in the client). AMQP 0-10 has two distinct credit flow modes "credit" and "window" and also allows the consumer to separately set limits for both "message" and "byte" credit. I'm not very familiar with the C++ API, but do you know how your consumer is managing credit? One thing that would be very helpful in trying to diagnose this problem is getting protocol logging for the consumer (at least for the start of the consumer where it sets up the credit flow mode, and towards the end where it would be interesting to see the credit being allocated just before message flow stops). -- Rob > > Thanks for your assistance, > Mike > > FYI - There may be a typo on the past releases web page. I believe "2017" > should be "2018" in "Qpid JMS AMQP 0-x 6.3.2, July 2017". URL: > https://qpid.apache.org/releases/index.html#past-releases > > -Original Message- > From: Rob Godfrey [mailto:rob.j.godf...@gmail.com] > Sent: Thursday, August 2, 2018 5:57 PM > To: users@qpid.apache.org > Cc: Mears, David B ; Herren, Elaine < > elaine.her...@hpe.com>; Rao, Shobha (NonStop) > > Subject: Re: Java Broker (7.0.6) stops delivering queue/consumer messages > after 4 GB data transfer > > Hi Mike, > > On Fri, 3 Aug 2018 at 01:25, Dyslin, Mike wrote: > > > This is my first submit to this email group. Hopefully this is the > > correct place to post this problem. > > > > > This is exactly the right place to post this problem. > > > > We are running a continuous stream of message (about 5K each) from > > producer to consumer over a single java broker queue at a rate of > > about 600 messages/second. Outbound message flow stops after > > transferring 4 GB of message data (about 770,000 messages in 25 > > minutes). The Web Management Console page for our consumer connection > shows the total "Outbound Bytes" > > growing steadily until it reaches 4.0 GB and stops with "Last I/O time" > > unchanging thereafter. > > > > After outbound messages stop: > > Inbound messages continue on the producer connection (well past 4.0 > > GB) and are kept in the queue until they expire with a time-to-live > > value of 3 minutes. The queue grows until is stabilizes with a steady > > 600 m/s inbound, and 600 m/s expiring and being deleted from the queue > > (as expected). The Web Management Console shows that the consumer > > connection remains open and is a consumer on the queue, and the queue > > shows the connection as a consumer on the queue. > > > > If I run the exact same test replacing the Java Broker with a C++ > > broker (1.37.0), message flow continues well past the 4 GB barrier. I > > kept it running for about 17 hours reaching about 37 million messages, > > about 180 GB data transferred on the queue. > > > > Since the only difference seems to be the broker, this seems to point > > to a problem with the Java Broker, and not issues with our producer, > > consumer or network issues. Could there be some problem with our java > > broker configuration that would explain this behaviour? > > > > Unfortunately this sounds like it may be a bug in the Java Broker :-( > > > > > > Has anyone out there experienced more than 4 GB of outbound data on a > > single java broker connection or queue? > > > > > Can you confirm which client you are using, and which version of AMQP is > in use (as you have identified I don't expect this to be a client problem, > but knowing the client will help us track down the issue in the broker)? > > > > Any help would be appreciated. > > > > Other comments/observations: > > > > I do not know if the 4 GB barrier is associated with the connection > > and/or the queue because all our message traffic is over one consumer > > connection and one queue. I could determine this by changing our > > consumer code to spread message traffic over one connection and multiple > queues. > > > > > > > > We are using the heartbeat feature with a 5 minute timeout. Since the > > connection stays open beyond the 5 minute timeout after the messages > > stop, I assume the heartbeat messages are still being sent between > > consumer and broker, indicating that the consumer and broker are
Re: [RESULT][VOTE] Release Apache qpid-interop-test 0.2.0 (RC3)
I am closing this vote. Although no explicit -1 was received, I will
treat the issue found by jross as an implicit -1. I will fix this, then
open a vote on an RC4.
On 08/01/2018 08:47 AM, Justin Ross wrote:
I can confirm that I see the same issue with RC3.
On Wed, Aug 1, 2018 at 5:42 AM Justin Ross wrote:
I just discovered I missed the RC3. I'll try again with that.
On Wed, Aug 1, 2018 at 5:41 AM Justin Ross wrote:
In my testing, QIT 0.2 rc2 is generating a build/config.sh that has
python2.7 libraries in the PYTHON3PATH. I don't have any preexisting
PYTHONPATH or PYTHON3PATH.
[jross@localhost ~]$ echo $PYTHON3PATH
/home/jross/.local/lib64/proton/bindings/python3:/home/jross/.local/lib/python2.7/site-packages:/home/jross/.local/libexec/qpid_interop_test/shims/qpid-proton-python:
This then triggers lots of test failures like this one:
==
ERROR: test_uuid_ProtonPython3->ProtonPython3 (__main__.UuidTestCase)
--
Traceback (most recent call last):
File
"/home/jross/.local/lib/python2.7/site-packages/qpid_interop_test/amqp_types_test.py",
line 424, in inner_test_method
timeout)
File
"/home/jross/.local/lib/python2.7/site-packages/qpid_interop_test/amqp_types_test.py",
line 345, in run_test
raise InteropTestError('Send shim \'%s\':\n%s' % (send_shim.NAME,
send_obj))
InteropTestError: Send shim 'ProtonPython3':
Return code 1
stderr=Traceback (most recent call last):
File
"/home/jross/.local/libexec/qpid_interop_test/shims/qpid-proton-python/amqp_types_test/Sender.py",
line 26, in
import json
File "/usr/lib64/python3.6/json/__init__.py", line 106, in
from .decoder import JSONDecoder, JSONDecodeError
File "/usr/lib64/python3.6/json/decoder.py", line 3, in
import re
File "/usr/lib64/python3.6/re.py", line 122, in
import enum
File "/usr/lib64/python3.6/enum.py", line 2, in
from types import MappingProxyType, DynamicClassAttribute
File "/usr/lib64/python3.6/types.py", line 171, in
import functools as _functools
File "/usr/lib64/python3.6/functools.py", line 21, in
from collections import namedtuple
File "/usr/lib64/python3.6/collections/__init__.py", line 32, in
from reprlib import recursive_repr as _recursive_repr
File
"/home/jross/.local/lib/python2.7/site-packages/reprlib/__init__.py", line
7, in
raise ImportError('This package should not be accessible on Python 3.
'
ImportError: This package should not be accessible on Python 3. Either
you are trying to run from the python-future src folder or your
installation of python-future is corrupted.
stdout=
On Sat, Jun 30, 2018 at 9:24 AM Kim van der Riet
wrote:
Hi,
I have assembled a proposed release for Apache Qpid Interop Test 0.2.0
(RC2). The issues found in RC1 were fixed. Please try it out and test
it, then vote accordingly on this thread.
The files can be obtained from:
https://dist.apache.org/repos/dist/dev/qpid/interop-test/0.2.0-rc2/
The JIRAs included in this release are:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12318621=12342084
This release is tagged in git as 0.2.0-rc2.
Thanks,
Kim
-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org
-
To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org
For additional commands, e-mail: users-h...@qpid.apache.org
RE: Java Broker (7.0.6) stops delivering queue/consumer messages after 4 GB data transfer
Rob, Our consumer client version is 1.37: bash 0 3: rpm -qa | grep -i qpid qpid-cpp-client-devel-1.37.0-1.el7.x86_64 python-qpid-1.37.0-1.el7.noarch qpid-qmf-1.37.0-1.el7.x86_64 qpid-cpp-client-1.37.0-1.el7.x86_64 python-qpid-qmf-1.37.0-1.el7.x86_64 qpid-cpp-server-1.37.0-1.el7.x86_64 qpid-proton-c-0.18.1-1.el7.x86_64 qpid-tools-1.37.0-1.el7.noarch Our producer client version is 6.1.5: -rw-r--r--1 NSDA.NSDA NSDA 570873 Jul 26 15:42 qpid-client-6.1.5.jar -rw-r--r--1 NSDA.NSDA NSDA 864493 Jul 26 15:42 qpid-common-6.1.5.jar Both producer and consumer clients use AMQP_0_10 protocol. Thanks for your assistance, Mike FYI - There may be a typo on the past releases web page. I believe "2017" should be "2018" in "Qpid JMS AMQP 0-x 6.3.2, July 2017". URL: https://qpid.apache.org/releases/index.html#past-releases -Original Message- From: Rob Godfrey [mailto:rob.j.godf...@gmail.com] Sent: Thursday, August 2, 2018 5:57 PM To: users@qpid.apache.org Cc: Mears, David B ; Herren, Elaine ; Rao, Shobha (NonStop) Subject: Re: Java Broker (7.0.6) stops delivering queue/consumer messages after 4 GB data transfer Hi Mike, On Fri, 3 Aug 2018 at 01:25, Dyslin, Mike wrote: > This is my first submit to this email group. Hopefully this is the > correct place to post this problem. > > This is exactly the right place to post this problem. > We are running a continuous stream of message (about 5K each) from > producer to consumer over a single java broker queue at a rate of > about 600 messages/second. Outbound message flow stops after > transferring 4 GB of message data (about 770,000 messages in 25 > minutes). The Web Management Console page for our consumer connection shows > the total "Outbound Bytes" > growing steadily until it reaches 4.0 GB and stops with "Last I/O time" > unchanging thereafter. > > After outbound messages stop: > Inbound messages continue on the producer connection (well past 4.0 > GB) and are kept in the queue until they expire with a time-to-live > value of 3 minutes. The queue grows until is stabilizes with a steady > 600 m/s inbound, and 600 m/s expiring and being deleted from the queue > (as expected). The Web Management Console shows that the consumer > connection remains open and is a consumer on the queue, and the queue > shows the connection as a consumer on the queue. > > If I run the exact same test replacing the Java Broker with a C++ > broker (1.37.0), message flow continues well past the 4 GB barrier. I > kept it running for about 17 hours reaching about 37 million messages, > about 180 GB data transferred on the queue. > > Since the only difference seems to be the broker, this seems to point > to a problem with the Java Broker, and not issues with our producer, > consumer or network issues. Could there be some problem with our java > broker configuration that would explain this behaviour? > Unfortunately this sounds like it may be a bug in the Java Broker :-( > > Has anyone out there experienced more than 4 GB of outbound data on a > single java broker connection or queue? > > Can you confirm which client you are using, and which version of AMQP is in use (as you have identified I don't expect this to be a client problem, but knowing the client will help us track down the issue in the broker)? > Any help would be appreciated. > > Other comments/observations: > > I do not know if the 4 GB barrier is associated with the connection > and/or the queue because all our message traffic is over one consumer > connection and one queue. I could determine this by changing our > consumer code to spread message traffic over one connection and multiple > queues. > > We are using the heartbeat feature with a 5 minute timeout. Since the > connection stays open beyond the 5 minute timeout after the messages > stop, I assume the heartbeat messages are still being sent between > consumer and broker, indicating that the consumer and broker are able > to communicate over the socket. It has been awhile since I have > tested that the heartbeat feature is working correctly. > > If I close the consumer connection from the Web Management Console, > the broker deletes the queue (I believe) and our consumer detects the > closed connection, establishes a new connection and new queue, and > messages start flowing again until . . . we reach the 4 GB barrier and > messages stop being delivered once again. > > We have run with the Java Broker on both Linux (RHEL 7.4) and > proprietary NonStop POSIX platform with the same results. > Unfortunately, the C++ broker is not yet an option on the NonStop > POSIX platform where we require the broker to be. > > Hopefully we can quickly track down the issue in the Java Broker and push out a fix, > Thanks, > Mike > > Apologies you've run into this issue, Rob > - >
Re: [VOTE] Release Apache Qpid Proton-J 0.27.3 (RC2)
+1 * Verified signatures and checksums * Built and ran tests from source bundle with environment variable PN_TRACE_FRM=true * Ran successfully Qpid Broker-J integration tests using proton-j staged artefacts On 2 August 2018 at 18:19, Robbie Gemmell wrote: > I have put together a second spin for a Qpid Proton-J 0.27.3 release, > please test it and vote accordingly. > > The files can be grabbed from: > https://dist.apache.org/repos/dist/dev/qpid/proton-j/0.27.3-rc2/ > > The maven artifacts are staged for now at: > https://repository.apache.org/content/repositories/orgapacheqpid-1153 > > The JIRAs assigned are: > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12313720=12343802 > > If you are looking at the tag (0.27.3), remember that it has been > deleted and replaced since RC1, you may need to refresh your checkout. > > Regards, > Robbie > > P.S. If you want to test things out using maven with your own build > you can temporarily add this to your poms to access the staging repo: > > > > staging > > https://repository.apache.org/content/repositories/orgapacheqpid-1153 > > > > The dependency for proton-j would then be: > > > org.apache.qpid > proton-j > 0.27.3 > > > - > To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org > For additional commands, e-mail: users-h...@qpid.apache.org > - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
Re: [VOTE] Release Apache Qpid Proton-J 0.28.1 (RC2)
+1 * Verified signatures and checksums * Built and ran tests from sources with environment variable PN_TRACE_FRM=true * Ran successfully Qpid Broker-J integration tests using proton-j staged artefacts On 2 August 2018 at 18:20, Robbie Gemmell wrote: > Hi folks, > > I have put together a second spin for a Qpid Proton-J 0.28.1 release, > please test it and vote accordingly. > > The files can be grabbed from: > https://dist.apache.org/repos/dist/dev/qpid/proton-j/0.28.1-rc2/ > > The maven artifacts are staged for now at: > https://repository.apache.org/content/repositories/orgapacheqpid-1154 > > The JIRAs assigned are: > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12313720=12343744 > > If you are looking at the tag (0.28.1), remember that it has been > replaced since RC1, you may need to refresh your checkout. > > Regards, > Robbie > > P.S. If you want to test things out using maven with your own build > you can temporarily add this to your poms to access the staging repo: > > > > staging > > https://repository.apache.org/content/repositories/orgapacheqpid-1154 > > > > The dependency for proton-j would then be: > > > org.apache.qpid > proton-j > 0.28.1 > > > - > To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org > For additional commands, e-mail: users-h...@qpid.apache.org > - To unsubscribe, e-mail: users-unsubscr...@qpid.apache.org For additional commands, e-mail: users-h...@qpid.apache.org
