Hi Mike, On Fri, 3 Aug 2018 at 01:25, Dyslin, Mike <[email protected]> wrote:
> This is my first submit to this email group. Hopefully this is the > correct place to post this problem. > > This is exactly the right place to post this problem. > We are running a continuous stream of message (about 5K each) from > producer to consumer over a single java broker queue at a rate of about 600 > messages/second. Outbound message flow stops after transferring 4 GB of > message data (about 770,000 messages in 25 minutes). The Web Management > Console page for our consumer connection shows the total "Outbound Bytes" > growing steadily until it reaches 4.0 GB and stops with "Last I/O time" > unchanging thereafter. > > After outbound messages stop: > Inbound messages continue on the producer connection (well past 4.0 GB) > and are kept in the queue until they expire with a time-to-live value of 3 > minutes. The queue grows until is stabilizes with a steady 600 m/s > inbound, and 600 m/s expiring and being deleted from the queue (as > expected). The Web Management Console shows that the consumer connection > remains open and is a consumer on the queue, and the queue shows the > connection as a consumer on the queue. > > If I run the exact same test replacing the Java Broker with a C++ broker > (1.37.0), message flow continues well past the 4 GB barrier. I kept it > running for about 17 hours reaching about 37 million messages, about 180 GB > data transferred on the queue. > > Since the only difference seems to be the broker, this seems to point to a > problem with the Java Broker, and not issues with our producer, consumer or > network issues. Could there be some problem with our java broker > configuration that would explain this behaviour? > Unfortunately this sounds like it may be a bug in the Java Broker :-( > > Has anyone out there experienced more than 4 GB of outbound data on a > single java broker connection or queue? > > Can you confirm which client you are using, and which version of AMQP is in use (as you have identified I don't expect this to be a client problem, but knowing the client will help us track down the issue in the broker)? > Any help would be appreciated. > > Other comments/observations: > > I do not know if the 4 GB barrier is associated with the connection and/or > the queue because all our message traffic is over one consumer connection > and one queue. I could determine this by changing our consumer code to > spread message traffic over one connection and multiple queues. > > We are using the heartbeat feature with a 5 minute timeout. Since the > connection stays open beyond the 5 minute timeout after the messages stop, > I assume the heartbeat messages are still being sent between consumer and > broker, indicating that the consumer and broker are able to communicate > over the socket. It has been awhile since I have tested that the heartbeat > feature is working correctly. > > If I close the consumer connection from the Web Management Console, the > broker deletes the queue (I believe) and our consumer detects the closed > connection, establishes a new connection and new queue, and messages start > flowing again until . . . we reach the 4 GB barrier and messages stop being > delivered once again. > > We have run with the Java Broker on both Linux (RHEL 7.4) and proprietary > NonStop POSIX platform with the same results. Unfortunately, the C++ > broker is not yet an option on the NonStop POSIX platform where we require > the broker to be. > > Hopefully we can quickly track down the issue in the Java Broker and push out a fix, > Thanks, > Mike > > Apologies you've run into this issue, Rob > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
