Hello,
While sending messages to the Java broker which is backed by a Berkley DB
message store, I am getting the below exception. However, the heap memory is
stable so I assume the problem is coming from elsewhere.
I am sending 5 000 000 messages which contain a byte array of 100KB of data. I
have the error at around 4 000 000 messages. Please note that the messages are
all being consumed by consumers which are not congested.
I have also monitored the heap and it is never above 1 GB.
So I have the following questions:
What parts of the broker uses direct memory?Does the fact using BDB means that
each message which is referenced by BDB has memory allocated on the direct
memory even if the message is consumed?
How can I analyse this OutOfMemory exception? (There is no heap dump So I
assume the error is not related to GC and I personally suspect the BDB to be
the culprit)
Broker Console output
Info: QPID_JAVA_GC not set. Defaulting to JAVA_GC -XX:+UseConcMarkSweepGC
-XX:+HeapDumpOnOutOfMemoryError
Using QPID_JAVA_MEM setting -Xmx16g -Xms2g
[Broker] BRK-1006 : Using configuration :
/dell445srv/apps/messaging_server/brokers/work1/config.json
[Broker] BRK-1001 : Startup : Version: 6.0.1 Build: 1731621
[Broker] BRK-1010 : Platform : JVM : Oracle Corporation version: 1.7.0_80-b15
OS : Linux version: 2.6.32-358.el6.x86_64 arch: amd64 cores: 20
[Broker] BRK-1011 : Maximum Memory : Heap : 17,049,059,328 bytes Direct :
17,049,059,328 bytes
[Broker] BRK-1017 : Process : PID : 26179
[Broker] BRK-1002 : Starting : Listening on TCP port 10255
[Broker] MNG-1001 : Web Management Startup
[Broker] MNG-1002 : Starting : HTTP : Listening on TCP port 10256
[Broker] MNG-1004 : Web Management Ready
[Broker] BRK-1004 : Qpid Broker Ready
########################################################################
#
# Unhandled Exception java.lang.OutOfMemoryError: Direct buffer memory in
Thread IO-/10.25.6.48:54546
#
# Exiting
#
########################################################################
java.lang.OutOfMemoryError: Direct buffer memory
...
Qpid Log
2016-07-28 11:41:50,143 ERROR [IO-/10.25.6.48:54546] (o.a.q.s.Main) - Uncaught
exception, shutting down.
java.lang.OutOfMemoryError: Direct buffer memory
at java.nio.Bits.reserveMemory(Bits.java:658) ~[na:1.7.0_80]
at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
~[na:1.7.0_80]
at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)
~[na:1.7.0_80]
at
org.apache.qpid.bytebuffer.QpidByteBuffer.allocateDirect(QpidByteBuffer.java:469)
~[qpid-common-6.0.1.jar:6.0.1]
at
org.apache.qpid.bytebuffer.QpidByteBuffer.allocateDirect(QpidByteBuffer.java:482)
~[qpid-common-6.0.1.jar:6.0.1]
at
org.apache.qpid.amqp_1_0.framing.FrameHandler.parse(FrameHandler.java:149)
~[qpid-broker-plugins-amqp-1-0-protocol-6.0.1.jar:6.0.1]
at
org.apache.qpid.server.protocol.v1_0.AMQPConnection_1_0$5.run(AMQPConnection_1_0.java:440)
~[qpid-broker-plugins-amqp-1-0-protocol-6.0.1.jar:6.0.1]
at
org.apache.qpid.server.protocol.v1_0.AMQPConnection_1_0$5.run(AMQPConnection_1_0.java:436)
~[qpid-broker-plugins-amqp-1-0-protocol-6.0.1.jar:6.0.1]
at java.security.AccessController.doPrivileged(Native Method)
~[na:1.7.0_80]
at
org.apache.qpid.server.protocol.v1_0.AMQPConnection_1_0.received(AMQPConnection_1_0.java:435)
~[qpid-broker-plugins-amqp-1-0-protocol-6.0.1.jar:6.0.1]
at
org.apache.qpid.server.transport.MultiVersionProtocolEngine.received(MultiVersionProtocolEngine.java:142)
~[qpid-broker-core-6.0.1.jar:6.0.1]
at
org.apache.qpid.server.transport.NonBlockingConnection.processAmqpData(NonBlockingConnection.java:564)
~[qpid-broker-core-6.0.1.jar:6.0.1]
at
org.apache.qpid.server.transport.NonBlockingConnectionPlainDelegate.processData(NonBlockingConnectionPlainDelegate.java:58)
~[qpid-broker-core-6.0.1.jar:6.0.1]
at
org.apache.qpid.server.transport.NonBlockingConnection.doRead(NonBlockingConnection.java:463)
~[qpid-broker-core-6.0.1.jar:6.0.1]
at
org.apache.qpid.server.transport.NonBlockingConnection.doWork(NonBlockingConnection.java:259)
~[qpid-broker-core-6.0.1.jar:6.0.1]
at
org.apache.qpid.server.transport.NetworkConnectionScheduler.processConnection(NetworkConnectionScheduler.java:108)
~[qpid-broker-core-6.0.1.jar:6.0.1]
at
org.apache.qpid.server.transport.SelectorThread$ConnectionProcessor.processConnection(SelectorThread.java:505)
~[qpid-broker-core-6.0.1.jar:6.0.1]
at
org.apache.qpid.server.transport.SelectorThread$SelectionTask.performSelect(SelectorThread.java:338)
~[qpid-broker-core-6.0.1.jar:6.0.1]
at
org.apache.qpid.server.transport.SelectorThread$SelectionTask.run(SelectorThread.java:87)
~[qpid-broker-core-6.0.1.jar:6.0.1]
at
org.apache.qpid.server.transport.SelectorThread.run(SelectorThread.java:463)
~[qpid-broker-core-6.0.1.jar:6.0.1]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_80]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
~[na:1.7.0_80]
at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_80]
Regards,
Adel
