Hello Adel Since v6, the Java Broker keeps messages in direct memory rather than heap. There is some explanation here:
https://qpid.apache.org/releases/qpid-java-6.0.4/java-broker/book/Java-Broker-Runtime-Memory.html Internally the Broker for Java uses a direct-memory caching strategy that aims to avoid the underlying JVM having to release/free chunks of direct memory frequently. In the AMQP 1.0 implementation within the Broker there are currently some compromises around this caching strategy that mean it sometimes relies on a JVM GC to free direct memory. Now, the JVM normally does not trigger a GC until there is pressure on the _heap_. I speculate that as your heap is so large, that the GC events are not occurring sufficiently frequently and meaning direct memory is not released and allowing direct memory to became exhausted. Try experimenting by reducing the heap. If you can tell us a little more about your test case, we may be able to make other suggestions. How exactly are the 5,000,000 messages arriving at the Broker? Are they arriving on a short period of time? How connections are used to publish the messages? Are transactions in use? Batching? Kind regards, Keith. On 28 July 2016 at 10:58, Adel Boutros <[email protected]> wrote: > Hello, > > While sending messages to the Java broker which is backed by a Berkley DB > message store, I am getting the below exception. However, the heap memory is > stable so I assume the problem is coming from elsewhere. > I am sending 5 000 000 messages which contain a byte array of 100KB of data. > I have the error at around 4 000 000 messages. Please note that the messages > are all being consumed by consumers which are not congested. > > I have also monitored the heap and it is never above 1 GB. > > So I have the following questions: > What parts of the broker uses direct memory?Does the fact using BDB means > that each message which is referenced by BDB has memory allocated on the > direct memory even if the message is consumed? > How can I analyse this OutOfMemory exception? (There is no heap dump So I > assume the error is not related to GC and I personally suspect the BDB to be > the culprit) > > > Broker Console output > Info: QPID_JAVA_GC not set. Defaulting to JAVA_GC -XX:+UseConcMarkSweepGC > -XX:+HeapDumpOnOutOfMemoryError > Using QPID_JAVA_MEM setting -Xmx16g -Xms2g > [Broker] BRK-1006 : Using configuration : > /dell445srv/apps/messaging_server/brokers/work1/config.json > [Broker] BRK-1001 : Startup : Version: 6.0.1 Build: 1731621 > [Broker] BRK-1010 : Platform : JVM : Oracle Corporation version: 1.7.0_80-b15 > OS : Linux version: 2.6.32-358.el6.x86_64 arch: amd64 cores: 20 > [Broker] BRK-1011 : Maximum Memory : Heap : 17,049,059,328 bytes Direct : > 17,049,059,328 bytes > [Broker] BRK-1017 : Process : PID : 26179 > [Broker] BRK-1002 : Starting : Listening on TCP port 10255 > [Broker] MNG-1001 : Web Management Startup > [Broker] MNG-1002 : Starting : HTTP : Listening on TCP port 10256 > [Broker] MNG-1004 : Web Management Ready > [Broker] BRK-1004 : Qpid Broker Ready > ######################################################################## > # > # Unhandled Exception java.lang.OutOfMemoryError: Direct buffer memory in > Thread IO-/10.25.6.48:54546 > # > # Exiting > # > ######################################################################## > java.lang.OutOfMemoryError: Direct buffer memory > ... > > > > Qpid Log > 2016-07-28 11:41:50,143 ERROR [IO-/10.25.6.48:54546] (o.a.q.s.Main) - > Uncaught exception, shutting down. > java.lang.OutOfMemoryError: Direct buffer memory > at java.nio.Bits.reserveMemory(Bits.java:658) ~[na:1.7.0_80] > at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) > ~[na:1.7.0_80] > at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306) > ~[na:1.7.0_80] > at > org.apache.qpid.bytebuffer.QpidByteBuffer.allocateDirect(QpidByteBuffer.java:469) > ~[qpid-common-6.0.1.jar:6.0.1] > at > org.apache.qpid.bytebuffer.QpidByteBuffer.allocateDirect(QpidByteBuffer.java:482) > ~[qpid-common-6.0.1.jar:6.0.1] > at > org.apache.qpid.amqp_1_0.framing.FrameHandler.parse(FrameHandler.java:149) > ~[qpid-broker-plugins-amqp-1-0-protocol-6.0.1.jar:6.0.1] > at > org.apache.qpid.server.protocol.v1_0.AMQPConnection_1_0$5.run(AMQPConnection_1_0.java:440) > ~[qpid-broker-plugins-amqp-1-0-protocol-6.0.1.jar:6.0.1] > at > org.apache.qpid.server.protocol.v1_0.AMQPConnection_1_0$5.run(AMQPConnection_1_0.java:436) > ~[qpid-broker-plugins-amqp-1-0-protocol-6.0.1.jar:6.0.1] > at java.security.AccessController.doPrivileged(Native Method) > ~[na:1.7.0_80] > at > org.apache.qpid.server.protocol.v1_0.AMQPConnection_1_0.received(AMQPConnection_1_0.java:435) > ~[qpid-broker-plugins-amqp-1-0-protocol-6.0.1.jar:6.0.1] > at > org.apache.qpid.server.transport.MultiVersionProtocolEngine.received(MultiVersionProtocolEngine.java:142) > ~[qpid-broker-core-6.0.1.jar:6.0.1] > at > org.apache.qpid.server.transport.NonBlockingConnection.processAmqpData(NonBlockingConnection.java:564) > ~[qpid-broker-core-6.0.1.jar:6.0.1] > at > org.apache.qpid.server.transport.NonBlockingConnectionPlainDelegate.processData(NonBlockingConnectionPlainDelegate.java:58) > ~[qpid-broker-core-6.0.1.jar:6.0.1] > at > org.apache.qpid.server.transport.NonBlockingConnection.doRead(NonBlockingConnection.java:463) > ~[qpid-broker-core-6.0.1.jar:6.0.1] > at > org.apache.qpid.server.transport.NonBlockingConnection.doWork(NonBlockingConnection.java:259) > ~[qpid-broker-core-6.0.1.jar:6.0.1] > at > org.apache.qpid.server.transport.NetworkConnectionScheduler.processConnection(NetworkConnectionScheduler.java:108) > ~[qpid-broker-core-6.0.1.jar:6.0.1] > at > org.apache.qpid.server.transport.SelectorThread$ConnectionProcessor.processConnection(SelectorThread.java:505) > ~[qpid-broker-core-6.0.1.jar:6.0.1] > at > org.apache.qpid.server.transport.SelectorThread$SelectionTask.performSelect(SelectorThread.java:338) > ~[qpid-broker-core-6.0.1.jar:6.0.1] > at > org.apache.qpid.server.transport.SelectorThread$SelectionTask.run(SelectorThread.java:87) > ~[qpid-broker-core-6.0.1.jar:6.0.1] > at > org.apache.qpid.server.transport.SelectorThread.run(SelectorThread.java:463) > ~[qpid-broker-core-6.0.1.jar:6.0.1] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > ~[na:1.7.0_80] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > ~[na:1.7.0_80] > at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_80] > > > > > > > Regards, > Adel > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
