On 07/08/2016 11:42 AM, Olivier Mallassi wrote:
Hi Rob
We have the following topology:
* Publisher -> Dispatcher -> Broker1 or Broker 2 (Java) -> Dispatcher (same
instance) -> Consumer
* each Broker is configured (before the run)
- with an exchange perf.topic (amqp topic)
- with a queue (perf.queue) that has a binding (binding key + header based
filtering) to the perf.topic exchange
* the Dispatcher is configured with autoLink
* The publisher publish in perf.topic (kind of distributed topic)
* The consumer consume from perf.queue (distributed queue)
* Dispatch version is 0.6.0 and Broker is in version 6.0.0.
I think your autoLink _in_ from perf.topic is spurious. I suspect that
there are no consumers on the router consuming from "perf.topic" and
therefore the listener attached to the temporary queue is never issued
credit and it accumulates all perf.topic messages.
We have the expected behavior : published messages are load-balanced on
brokers and consumed as expected.
Yet, and we observed the OOM while running the test for a longer time, when
the dispatcher connect, it creates a queue with a UUID (more or less) name.
that queue is bind to perf.topic with # as a binding key (so every messages
sent by the publisher fill up this queue thus the OOM because it is in
memory).
It looks to be "expected".
Is there a way to avoid this? are we missing something?
Many thx.
oliv/
On Fri, Jul 8, 2016 at 5:16 PM, Rob Godfrey <[email protected]> wrote:
Hi Adel,
I'm a bit confused... you say
As no consumer is connected to this queue, all messages are kept
in-memory which causes the OutOfMemory exception in the broker.
but then also say
Consumer connected to that queue (Is it the dispatcher?):
Name: qdlink.OyY41QAJRZ4JGGg
Mode: MOVE
Does the queue have a consumer or not?
As to holding on to the messages... the heap dump shows that the message is
in the queue (which we know) and has not been evicted by flow to disk (we'd
need to work out why)...
On average what size are the messages you are sending? Do you see messages
flow through this topic at all?
As to the "weird" queue name - what is happening is that we are creating an
internal temporary queue when you subscribe to an exchange object in the
broker... for that we use a UUID as the queue name.
-- Rob
On 8 July 2016 at 15:48, Adel Boutros <[email protected]> wrote:
I forgot to add that when I execute a qdmanage command to delete the
"connector" to the broker, the weird queue disappears.
Adel
Date: Fri, 8 Jul 2016 07:45:49 -0700
From: [email protected]
To: [email protected]
Subject: Re: OutOfMemory exception with a Qpid Java Broker linked to
qpid dispatcher
Fixing bad formatting on Nabbledue to Hotmail:
Hello,
We are doing some performance tests with a Qpid Java Broker connected
to
a
Qpid dispatcher.
We have noticed that after some time, the broker dies with an
OutOfMemory
exception and java heap is dumped.
After analyzing the heap dump and checking the broker, we have noticed
2
weird behaviors:
1) A queue is created on the broker with a weird Name bound on our
topic
(perf.topic) with a binding key "#". It will receive all messages.
This queue seems to be created only when the dispatcher is present and
is
configured.
As no consumer is connected to this queue, all messages are kept
in-memory
which causes the OutOfMemory exception in the broker.
Weird Queue definition
Name: 18bb86dd-2953-4c6f-8eb2-56e5128963f3
Type: standard
State: ACTIVE
Durable: false
Lifespan: DELETE_ON_NO_OUTBOUND_LINKS
Persist Messages: NEVER
Inbound: 0 msg/s (0.00 B/s)
Outbound: 0 msg/s (0.00 B/s)
Size: 2374 msgs (0.00 B)
Pre-fetched: 0 msgs (0.00 B)
Oldest Message Age: 356.064 secs
Enforced Max. Ttl(ms): 0
Enforced Min. Ttl(ms): 0
Exclusive: LINK
Consumer connected to that queue (Is it the dispatcher?):
Name: qdlink.OyY41QAJRZ4JGGg
Mode: MOVE
2) All messages even those consumed successfully by the consumers are
still
present on the broker
From the heap dump, we can see the Berkley DB is keeping a reference to
all
messages. Is this also coming from the above weird queue?
PS: If we only use the dispatcher instead, we have none of the weird
behaviors
Extract from the heap dump (Object holding
reference to one of the message header. "Validated" is one the message
header fields we set and which is already received by a consumer)
char[9] @ 0xf59ff3d8 VALIDATED
'- value java.lang.String @ 0xf59ff3c0 VALIDATED
'- value java.util.HashMap$Entry @ 0xf59ff310
'- [3] java.util.HashMap$Entry[8] @ 0xf59ff2c0
'- table java.util.HashMap @ 0xf59ff188
'- _appProperties
org.apache.qpid.server.protocol.v1_0.MessageMetaData_1_0 @ 0xf59fef88
'- _metaData
org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore$MessageDataSoftRef
@ 0xf59fef70
'- _messageDataRef
org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore$StoredBDBMessage
@ 0xf59fef20
'- _handle
org.apache.qpid.server.protocol.v1_0.Message_1_0 @
0xf59feef0
'- _message
org.apache.qpid.server.message.AbstractServerMessageImpl$Reference @
0xf59621c8
'- _message
org.apache.qpid.server.queue.StandardQueueEntry @ 0xf5962188
'- _next
org.apache.qpid.server.queue.StandardQueueEntry
@ 0xf5962130
- this$0
org.apache.qpid.server.protocol.v1_0.MessageMetaData_1_0$MessageHeader_1_0 @
0xf59ff1f0
Broker Config
1 Virtual Host Node and 1 Virtual Host configured with Berkley DB types
(BDB
with the default configuration).
1 Topic (perf.topic)
bound to a queue (perfQueue) with a binding key and a jms-selector
filter
bound to an "alternate exchange" of a type fanout which is also
bound
to a queue but with no binding key (empty string)
Dispatcher Config
qdmanage -b amqp://localhost:10454 create --type=address
prefix=perfQueue
waypoint=true name=perf.queue.addr
qdmanage -b amqp://localhost:10454 create --type=address
prefix=perf.topic
waypoint=true name=perf.topic.addr
qdmanage -b amqp://localhost:10454 create --type=connector
role=route-container addr=localhost port=10455
name=localhost.broker.10455.connector
qdmanage -b amqp://localhost:10454 create --type=autoLink
addr=perfQueue
dir=out connection=localhost.broker.10455.connector
name=localhost.broker.10455.perfQueue.out
qdmanage -b amqp://localhost:10454 create --type=autoLink
addr=perfQueue
dir=in connection=localhost.broker.10455.connector
name=localhost.broker.10455.perfQueue.in
qdmanage -b amqp://localhost:10454 create --type=autoLink
addr=perf.topic
dir=out connection=localhost.broker.10455.connector
name=localhost.broker.10455.perf.topic.out
qdmanage -b amqp://localhost:10454 create --type=autoLink
addr=perf.topic
dir=in connection=localhost.broker.10455.connector
name=localhost.broker.10455.perf.topic.in
Clients config
3 JMS Consumers connected each to perfQueue on the dispatcher
2 JMS producers connected each to perf.topic on the dispatcher
With the above config, we send a number of messages of which only 1/3
will
be routed to the "alternate exchange" and never consumed.
Versions
Qpid Java Broker: 6.0.0
Qpid Dispatch: 0.6.0
JMS: 0.9.0
Regards,
Adel
--
View this message in context:
http://qpid.2158936.n2.nabble.com/OutOfMemory-exception-with-a-Qpid-Java-Broker-linked-to-qpid-dispatcher-tp7647065p7647066.html
Sent from the Apache Qpid users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
