Hello again, I re-launched the tests with 2 configurations and they both failed very quick: Xmx1g ---> Failed after sending just 12 000 messages Xmx2g ---> Failed after sending just 25 000 messages
Regards, Adel > From: [email protected] > To: [email protected] > Subject: RE: [Java Broker - 6.0.1] OutOfMemory in the directy memory buffer > Date: Fri, 29 Jul 2016 15:15:24 +0200 > > Hello Keith, > > As stated in this thread[1], we are actually doing some benchmarks to test > the performance of the broker and the dispatch router. In the failing test > case, I try to send the messages using 4 JMS producers connected directly to > the broker each having its own connection (1 JVM per producer). The messages > are sent in synchronous mode with AUTO_ACKNOWLEDGE. We try to send them as > fast as possible but the test actually takes around 2 hours. > > There are no transactions and no batching. Each producer sends a message > continuously in the main thread. > > In order for the test to pass, I had to set the maximum heap to 32 GB which > resulted in augmenting the direct memory to 32 GB also. > > Because the test takes a long time, I have to run it at night. So I cannot > experiment different heap sizes now. > > [1]: > http://qpid.2158936.n2.nabble.com/Performance-Benchmarking-Qpid-dispatch-router-0-6-0-with-Qpid-Java-Broker-6-0-0-td7648076.html > > Regards, > Adel > > > From: [email protected] > > Date: Fri, 29 Jul 2016 11:44:57 +0100 > > Subject: Re: [Java Broker - 6.0.1] OutOfMemory in the directy memory buffer > > To: [email protected] > > > > Hello Adel > > > > Since v6, the Java Broker keeps messages in direct memory rather than > > heap. There is some explanation here: > > > > https://qpid.apache.org/releases/qpid-java-6.0.4/java-broker/book/Java-Broker-Runtime-Memory.html > > > > Internally the Broker for Java uses a direct-memory caching strategy > > that aims to avoid the underlying JVM having to release/free chunks of > > direct memory frequently. In the AMQP 1.0 implementation within the > > Broker there are currently some compromises around this caching > > strategy that mean it sometimes relies on a JVM GC to free direct > > memory. Now, the JVM normally does not trigger a GC until there is > > pressure on the _heap_. I speculate that as your heap is so large, > > that the GC events are not occurring sufficiently frequently and > > meaning direct memory is not released and allowing direct memory to > > became exhausted. Try experimenting by reducing the heap. > > > > If you can tell us a little more about your test case, we may be able > > to make other suggestions. How exactly are the 5,000,000 messages > > arriving at the Broker? Are they arriving on a short period of time? > > How connections are used to publish the messages? Are transactions in > > use? Batching? > > > > > > Kind regards, Keith. > > > > On 28 July 2016 at 10:58, Adel Boutros <[email protected]> wrote: > > > Hello, > > > > > > While sending messages to the Java broker which is backed by a Berkley DB > > > message store, I am getting the below exception. However, the heap memory > > > is stable so I assume the problem is coming from elsewhere. > > > I am sending 5 000 000 messages which contain a byte array of 100KB of > > > data. I have the error at around 4 000 000 messages. Please note that the > > > messages are all being consumed by consumers which are not congested. > > > > > > I have also monitored the heap and it is never above 1 GB. > > > > > > So I have the following questions: > > > What parts of the broker uses direct memory?Does the fact using BDB means > > > that each message which is referenced by BDB has memory allocated on the > > > direct memory even if the message is consumed? > > > How can I analyse this OutOfMemory exception? (There is no heap dump So I > > > assume the error is not related to GC and I personally suspect the BDB to > > > be the culprit) > > > > > > > > > Broker Console output > > > Info: QPID_JAVA_GC not set. Defaulting to JAVA_GC -XX:+UseConcMarkSweepGC > > > -XX:+HeapDumpOnOutOfMemoryError > > > Using QPID_JAVA_MEM setting -Xmx16g -Xms2g > > > [Broker] BRK-1006 : Using configuration : > > > /dell445srv/apps/messaging_server/brokers/work1/config.json > > > [Broker] BRK-1001 : Startup : Version: 6.0.1 Build: 1731621 > > > [Broker] BRK-1010 : Platform : JVM : Oracle Corporation version: > > > 1.7.0_80-b15 OS : Linux version: 2.6.32-358.el6.x86_64 arch: amd64 cores: > > > 20 > > > [Broker] BRK-1011 : Maximum Memory : Heap : 17,049,059,328 bytes Direct : > > > 17,049,059,328 bytes > > > [Broker] BRK-1017 : Process : PID : 26179 > > > [Broker] BRK-1002 : Starting : Listening on TCP port 10255 > > > [Broker] MNG-1001 : Web Management Startup > > > [Broker] MNG-1002 : Starting : HTTP : Listening on TCP port 10256 > > > [Broker] MNG-1004 : Web Management Ready > > > [Broker] BRK-1004 : Qpid Broker Ready > > > ######################################################################## > > > # > > > # Unhandled Exception java.lang.OutOfMemoryError: Direct buffer memory in > > > Thread IO-/10.25.6.48:54546 > > > # > > > # Exiting > > > # > > > ######################################################################## > > > java.lang.OutOfMemoryError: Direct buffer memory > > > ... > > > > > > > > > > > > Qpid Log > > > 2016-07-28 11:41:50,143 ERROR [IO-/10.25.6.48:54546] (o.a.q.s.Main) - > > > Uncaught exception, shutting down. > > > java.lang.OutOfMemoryError: Direct buffer memory > > > at java.nio.Bits.reserveMemory(Bits.java:658) ~[na:1.7.0_80] > > > at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123) > > > ~[na:1.7.0_80] > > > at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306) > > > ~[na:1.7.0_80] > > > at > > > org.apache.qpid.bytebuffer.QpidByteBuffer.allocateDirect(QpidByteBuffer.java:469) > > > ~[qpid-common-6.0.1.jar:6.0.1] > > > at > > > org.apache.qpid.bytebuffer.QpidByteBuffer.allocateDirect(QpidByteBuffer.java:482) > > > ~[qpid-common-6.0.1.jar:6.0.1] > > > at > > > org.apache.qpid.amqp_1_0.framing.FrameHandler.parse(FrameHandler.java:149) > > > ~[qpid-broker-plugins-amqp-1-0-protocol-6.0.1.jar:6.0.1] > > > at > > > org.apache.qpid.server.protocol.v1_0.AMQPConnection_1_0$5.run(AMQPConnection_1_0.java:440) > > > ~[qpid-broker-plugins-amqp-1-0-protocol-6.0.1.jar:6.0.1] > > > at > > > org.apache.qpid.server.protocol.v1_0.AMQPConnection_1_0$5.run(AMQPConnection_1_0.java:436) > > > ~[qpid-broker-plugins-amqp-1-0-protocol-6.0.1.jar:6.0.1] > > > at java.security.AccessController.doPrivileged(Native Method) > > > ~[na:1.7.0_80] > > > at > > > org.apache.qpid.server.protocol.v1_0.AMQPConnection_1_0.received(AMQPConnection_1_0.java:435) > > > ~[qpid-broker-plugins-amqp-1-0-protocol-6.0.1.jar:6.0.1] > > > at > > > org.apache.qpid.server.transport.MultiVersionProtocolEngine.received(MultiVersionProtocolEngine.java:142) > > > ~[qpid-broker-core-6.0.1.jar:6.0.1] > > > at > > > org.apache.qpid.server.transport.NonBlockingConnection.processAmqpData(NonBlockingConnection.java:564) > > > ~[qpid-broker-core-6.0.1.jar:6.0.1] > > > at > > > org.apache.qpid.server.transport.NonBlockingConnectionPlainDelegate.processData(NonBlockingConnectionPlainDelegate.java:58) > > > ~[qpid-broker-core-6.0.1.jar:6.0.1] > > > at > > > org.apache.qpid.server.transport.NonBlockingConnection.doRead(NonBlockingConnection.java:463) > > > ~[qpid-broker-core-6.0.1.jar:6.0.1] > > > at > > > org.apache.qpid.server.transport.NonBlockingConnection.doWork(NonBlockingConnection.java:259) > > > ~[qpid-broker-core-6.0.1.jar:6.0.1] > > > at > > > org.apache.qpid.server.transport.NetworkConnectionScheduler.processConnection(NetworkConnectionScheduler.java:108) > > > ~[qpid-broker-core-6.0.1.jar:6.0.1] > > > at > > > org.apache.qpid.server.transport.SelectorThread$ConnectionProcessor.processConnection(SelectorThread.java:505) > > > ~[qpid-broker-core-6.0.1.jar:6.0.1] > > > at > > > org.apache.qpid.server.transport.SelectorThread$SelectionTask.performSelect(SelectorThread.java:338) > > > ~[qpid-broker-core-6.0.1.jar:6.0.1] > > > at > > > org.apache.qpid.server.transport.SelectorThread$SelectionTask.run(SelectorThread.java:87) > > > ~[qpid-broker-core-6.0.1.jar:6.0.1] > > > at > > > org.apache.qpid.server.transport.SelectorThread.run(SelectorThread.java:463) > > > ~[qpid-broker-core-6.0.1.jar:6.0.1] > > > at > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > > ~[na:1.7.0_80] > > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > > ~[na:1.7.0_80] > > > at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_80] > > > > > > > > > > > > > > > > > > > > > Regards, > > > Adel > > > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: [email protected] > > For additional commands, e-mail: [email protected] > > >
