Hello,
We are doing some performance tests with a Qpid Java Broker connected to a Qpid
dispatcher.
We have noticed that after some time, the broker dies with an OutOfMemory
exception and java heap is dumped.
After analyzing the heap dump and checking the broker, we have noticed 2 weird
behaviors:
A queue is created on the broker with a weird Name bound on our topic
(perf.topic) with a binding key "#"
It will receive all messages.
This queue seems to be created only when the dispatcher is present and is
configured.
As no consumer is connected to this queue, all messages are kept in-memory
which causes the OutOfMemory exception in the broker.
Weird Queue definition
Name: 18bb86dd-2953-4c6f-8eb2-56e5128963f3
Type: standard
State: ACTIVE
Durable: false
Lifespan: DELETE_ON_NO_OUTBOUND_LINKS
Persist Messages: NEVER
Inbound: 0 msg/s (0.00 B/s)
Outbound: 0 msg/s (0.00 B/s)
Size: 2374 msgs (0.00 B)
Pre-fetched: 0 msgs (0.00 B)
Oldest Message Age: 356.064 secs
Enforced Max. Ttl(ms): 0
Enforced Min. Ttl(ms): 0
Exclusive: LINK
Consumer connected to that queue (Is it the dispatcher?):
Name: qdlink.OyY41QAJRZ4JGGg
Mode: MOVE
All messages even those consumed successfully by the consumers are still
present on the broker
>From the heap dump, we can see the Berkley DB is keeping a reference to all
>messages. Is this also coming from the above weird queue?
PS: If we only use the dispatcher instead, we have none of the weird behaviors
Extract from the heap dump (Object holding
reference to one of the message header. "Validated" is one the message
header fields we set and which is already received by a consumer)
char[9] @ 0xf59ff3d8 VALIDATED
'- value java.lang.String @ 0xf59ff3c0 VALIDATED
'- value java.util.HashMap$Entry @ 0xf59ff310
'- [3] java.util.HashMap$Entry[8] @ 0xf59ff2c0
'- table java.util.HashMap @ 0xf59ff188
'- _appProperties
org.apache.qpid.server.protocol.v1_0.MessageMetaData_1_0 @ 0xf59fef88
'- _metaData
org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore$MessageDataSoftRef
@ 0xf59fef70
'- _messageDataRef
org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore$StoredBDBMessage
@ 0xf59fef20
'- _handle org.apache.qpid.server.protocol.v1_0.Message_1_0 @
0xf59feef0
'- _message
org.apache.qpid.server.message.AbstractServerMessageImpl$Reference @ 0xf59621c8
'- _message org.apache.qpid.server.queue.StandardQueueEntry @
0xf5962188
'- _next org.apache.qpid.server.queue.StandardQueueEntry @
0xf5962130
- this$0
org.apache.qpid.server.protocol.v1_0.MessageMetaData_1_0$MessageHeader_1_0 @
0xf59ff1f0
Broker Config
1 Virtual Host Node and 1 Virtual Host configured with Berkley DB types (BDB
with the default configuration)
1 Topic (perf.topic)
bound to a queue (perfQueue) with a binding key and a jms-selector filterbound
to an "alternate exchange" of a type fanout which is also bound to a queue but
with no binding key (empty string)
Dispatcher Config
qdmanage -b amqp://localhost:10454 create --type=address prefix=perfQueue
waypoint=true name=perf.queue.addr
qdmanage -b amqp://localhost:10454 create --type=address prefix=perf.topic
waypoint=true name=perf.topic.addr
qdmanage -b amqp://localhost:10454 create --type=connector role=route-container
addr=localhost port=10455 name=localhost.broker.10455.connector
qdmanage -b amqp://localhost:10454 create --type=autoLink addr=perfQueue
dir=out connection=localhost.broker.10455.connector
name=localhost.broker.10455.perfQueue.out
qdmanage -b amqp://localhost:10454 create --type=autoLink addr=perfQueue dir=in
connection=localhost.broker.10455.connector
name=localhost.broker.10455.perfQueue.in
qdmanage -b amqp://localhost:10454 create --type=autoLink addr=perf.topic
dir=out connection=localhost.broker.10455.connector
name=localhost.broker.10455.perf.topic.out
qdmanage -b amqp://localhost:10454 create --type=autoLink addr=perf.topic
dir=in connection=localhost.broker.10455.connector
name=localhost.broker.10455.perf.topic.in
Clients config
3 JMS Consumers connected each to perfQueue on the dispatcher2 JMS producers
connected each to perf.topic on the dispatcher
With the above config, we send a number of messages of which only 1/3 will be
routed to the "alternate exchange" and never consumed.
Versions
Qpid Java Broker: 6.0.0Qpid Dispatch: 0.6.0JMS: 0.9.0
Regards,
Adel
