On 07/08/2016 10:40 AM, Adel Boutros wrote:
Hello,
We are doing some performance tests with a Qpid Java Broker connected to a Qpid
dispatcher.
We have noticed that after some time, the broker dies with an OutOfMemory
exception and java heap is dumped.
After analyzing the heap dump and checking the broker, we have noticed 2 weird
behaviors:
A queue is created on the broker with a weird Name bound on our topic
(perf.topic) with a binding key "#"
It will receive all messages.
This queue seems to be created only when the dispatcher is present and is
configured.
As no consumer is connected to this queue, all messages are kept in-memory
which causes the OutOfMemory exception in the broker.
Weird Queue definition
Name: 18bb86dd-2953-4c6f-8eb2-56e5128963f3
Type: standard
State: ACTIVE
Durable: false
Lifespan: DELETE_ON_NO_OUTBOUND_LINKS
Persist Messages: NEVER
Inbound: 0 msg/s (0.00 B/s)
Outbound: 0 msg/s (0.00 B/s)
Size: 2374 msgs (0.00 B)
Pre-fetched: 0 msgs (0.00 B)
Oldest Message Age: 356.064 secs
Enforced Max. Ttl(ms): 0
Enforced Min. Ttl(ms): 0
Exclusive: LINK
Consumer connected to that queue (Is it the dispatcher?):
Name: qdlink.OyY41QAJRZ4JGGg
Mode: MOVE
The queue with the UUID name is created when Dispatch Router (it's not
called "dispatcher") attaches a listener to the "perf.topic" address.
This is because the broker only allows consumers from queues, not
exchanges. This is a temporary queue that exists to hold messages
coming through the exchange on the way to that consumer. A copy of
every message delivered to that exchange will be put on that queue.
All messages even those consumed successfully by the consumers are still
present on the broker
In which queue?
From the heap dump, we can see the Berkley DB is keeping a reference to all
messages. Is this also coming from the above weird queue?
PS: If we only use the dispatcher instead, we have none of the weird behaviors
Extract from the heap dump (Object holding
reference to one of the message header. "Validated" is one the message
header fields we set and which is already received by a consumer)
char[9] @ 0xf59ff3d8 VALIDATED
'- value java.lang.String @ 0xf59ff3c0 VALIDATED
'- value java.util.HashMap$Entry @ 0xf59ff310
'- [3] java.util.HashMap$Entry[8] @ 0xf59ff2c0
'- table java.util.HashMap @ 0xf59ff188
'- _appProperties
org.apache.qpid.server.protocol.v1_0.MessageMetaData_1_0 @ 0xf59fef88
'- _metaData
org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore$MessageDataSoftRef
@ 0xf59fef70
'- _messageDataRef
org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore$StoredBDBMessage
@ 0xf59fef20
'- _handle org.apache.qpid.server.protocol.v1_0.Message_1_0 @
0xf59feef0
'- _message
org.apache.qpid.server.message.AbstractServerMessageImpl$Reference @ 0xf59621c8
'- _message org.apache.qpid.server.queue.StandardQueueEntry @
0xf5962188
'- _next org.apache.qpid.server.queue.StandardQueueEntry @
0xf5962130
- this$0
org.apache.qpid.server.protocol.v1_0.MessageMetaData_1_0$MessageHeader_1_0 @
0xf59ff1f0
Broker Config
1 Virtual Host Node and 1 Virtual Host configured with Berkley DB types (BDB
with the default configuration)
1 Topic (perf.topic)
bound to a queue (perfQueue) with a binding key and a jms-selector filterbound to an
"alternate exchange" of a type fanout which is also bound to a queue but with
no binding key (empty string)
You don't say what the binding key is for perfQueue, but all messages
that match that binding are also enqueued on the temporary queue. I
expect that you are seeing the same messages delivered twice, once
through perfQueue and once through perf.topic (via the temporary queue).
Is it true that the broker is out-of-memory because a queue filled up?
Which queue is full?
Dispatcher Config
qdmanage -b amqp://localhost:10454 create --type=address prefix=perfQueue
waypoint=true name=perf.queue.addr
qdmanage -b amqp://localhost:10454 create --type=address prefix=perf.topic
waypoint=true name=perf.topic.addr
qdmanage -b amqp://localhost:10454 create --type=connector role=route-container
addr=localhost port=10455 name=localhost.broker.10455.connector
qdmanage -b amqp://localhost:10454 create --type=autoLink addr=perfQueue
dir=out connection=localhost.broker.10455.connector
name=localhost.broker.10455.perfQueue.out
qdmanage -b amqp://localhost:10454 create --type=autoLink addr=perfQueue dir=in
connection=localhost.broker.10455.connector
name=localhost.broker.10455.perfQueue.in
qdmanage -b amqp://localhost:10454 create --type=autoLink addr=perf.topic
dir=out connection=localhost.broker.10455.connector
name=localhost.broker.10455.perf.topic.out
qdmanage -b amqp://localhost:10454 create --type=autoLink addr=perf.topic
dir=in connection=localhost.broker.10455.connector
name=localhost.broker.10455.perf.topic.in
Clients config
3 JMS Consumers connected each to perfQueue on the dispatcher2 JMS producers
connected each to perf.topic on the dispatcher
With the above config, we send a number of messages of which only 1/3 will be routed to
the "alternate exchange" and never consumed.
Do the 1/3 account for the memory expansion?
Versions
Qpid Java Broker: 6.0.0Qpid Dispatch: 0.6.0JMS: 0.9.0
Regards,
Adel
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
