Ted, yes. this is as you describe the queue that fills up is the "UUID name" queue (created by the Dispatch Router ).
On Fri, Jul 8, 2016 at 5:42 PM, Olivier Mallassi <[email protected] > wrote: > Hi Rob > > We have the following topology: > > * Publisher -> Dispatcher -> Broker1 or Broker 2 (Java) -> Dispatcher > (same instance) -> Consumer > * each Broker is configured (before the run) > - with an exchange perf.topic (amqp topic) > - with a queue (perf.queue) that has a binding (binding key + header > based filtering) to the perf.topic exchange > > * the Dispatcher is configured with autoLink > * The publisher publish in perf.topic (kind of distributed topic) > * The consumer consume from perf.queue (distributed queue) > * Dispatch version is 0.6.0 and Broker is in version 6.0.0. > > We have the expected behavior : published messages are load-balanced on > brokers and consumed as expected. > > Yet, and we observed the OOM while running the test for a longer time, > when the dispatcher connect, it creates a queue with a UUID (more or less) > name. that queue is bind to perf.topic with # as a binding key (so every > messages sent by the publisher fill up this queue thus the OOM because it > is in memory). > > It looks to be "expected". > Is there a way to avoid this? are we missing something? > > Many thx. > > oliv/ > > > > > > > On Fri, Jul 8, 2016 at 5:16 PM, Rob Godfrey <[email protected]> > wrote: > >> Hi Adel, >> >> I'm a bit confused... you say >> >> > As no consumer is connected to this queue, all messages are kept >> in-memory which causes the OutOfMemory exception in the broker. >> >> but then also say >> >> > Consumer connected to that queue (Is it the dispatcher?): >> > Name: qdlink.OyY41QAJRZ4JGGg >> > Mode: MOVE >> >> Does the queue have a consumer or not? >> >> As to holding on to the messages... the heap dump shows that the message >> is >> in the queue (which we know) and has not been evicted by flow to disk >> (we'd >> need to work out why)... >> >> On average what size are the messages you are sending? Do you see >> messages >> flow through this topic at all? >> >> As to the "weird" queue name - what is happening is that we are creating >> an >> internal temporary queue when you subscribe to an exchange object in the >> broker... for that we use a UUID as the queue name. >> >> -- Rob >> >> On 8 July 2016 at 15:48, Adel Boutros <[email protected]> wrote: >> >> > I forgot to add that when I execute a qdmanage command to delete the >> > "connector" to the broker, the weird queue disappears. >> > >> > Adel >> > >> > > Date: Fri, 8 Jul 2016 07:45:49 -0700 >> > > From: [email protected] >> > > To: [email protected] >> > > Subject: Re: OutOfMemory exception with a Qpid Java Broker linked to >> > qpid dispatcher >> > > >> > > Fixing bad formatting on Nabbledue to Hotmail: >> > > >> > > Hello, >> > > >> > > We are doing some performance tests with a Qpid Java Broker connected >> to >> > a >> > > Qpid dispatcher. >> > > We have noticed that after some time, the broker dies with an >> OutOfMemory >> > > exception and java heap is dumped. >> > > After analyzing the heap dump and checking the broker, we have >> noticed 2 >> > > weird behaviors: >> > > >> > > 1) A queue is created on the broker with a weird Name bound on our >> topic >> > > (perf.topic) with a binding key "#". It will receive all messages. >> > > This queue seems to be created only when the dispatcher is present >> and is >> > > configured. >> > > As no consumer is connected to this queue, all messages are kept >> > in-memory >> > > which causes the OutOfMemory exception in the broker. >> > > >> > > Weird Queue definition >> > > Name: 18bb86dd-2953-4c6f-8eb2-56e5128963f3 >> > > Type: standard >> > > State: ACTIVE >> > > Durable: false >> > > Lifespan: DELETE_ON_NO_OUTBOUND_LINKS >> > > Persist Messages: NEVER >> > > Inbound: 0 msg/s (0.00 B/s) >> > > Outbound: 0 msg/s (0.00 B/s) >> > > Size: 2374 msgs (0.00 B) >> > > Pre-fetched: 0 msgs (0.00 B) >> > > Oldest Message Age: 356.064 secs >> > > Enforced Max. Ttl(ms): 0 >> > > Enforced Min. Ttl(ms): 0 >> > > Exclusive: LINK >> > > >> > > Consumer connected to that queue (Is it the dispatcher?): >> > > Name: qdlink.OyY41QAJRZ4JGGg >> > > Mode: MOVE >> > > >> > > >> > > 2) All messages even those consumed successfully by the consumers are >> > still >> > > present on the broker >> > > >> > > From the heap dump, we can see the Berkley DB is keeping a reference >> to >> > all >> > > messages. Is this also coming from the above weird queue? >> > > >> > > >> > > PS: If we only use the dispatcher instead, we have none of the weird >> > > behaviors >> > > >> > > Extract from the heap dump (Object holding >> > > reference to one of the message header. "Validated" is one the message >> > > header fields we set and which is already received by a consumer) >> > > >> > > char[9] @ 0xf59ff3d8 VALIDATED >> > > '- value java.lang.String @ 0xf59ff3c0 VALIDATED >> > > '- value java.util.HashMap$Entry @ 0xf59ff310 >> > > '- [3] java.util.HashMap$Entry[8] @ 0xf59ff2c0 >> > > '- table java.util.HashMap @ 0xf59ff188 >> > > '- _appProperties >> > > org.apache.qpid.server.protocol.v1_0.MessageMetaData_1_0 @ 0xf59fef88 >> > > '- _metaData >> > > >> > >> org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore$MessageDataSoftRef >> > > @ 0xf59fef70 >> > > '- _messageDataRef >> > > >> > >> org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore$StoredBDBMessage >> > > @ 0xf59fef20 >> > > '- _handle >> > org.apache.qpid.server.protocol.v1_0.Message_1_0 @ >> > > 0xf59feef0 >> > > '- _message >> > > org.apache.qpid.server.message.AbstractServerMessageImpl$Reference @ >> > > 0xf59621c8 >> > > '- _message >> > > org.apache.qpid.server.queue.StandardQueueEntry @ 0xf5962188 >> > > '- _next >> > org.apache.qpid.server.queue.StandardQueueEntry >> > > @ 0xf5962130 >> > > - this$0 >> > > >> > >> org.apache.qpid.server.protocol.v1_0.MessageMetaData_1_0$MessageHeader_1_0 @ >> > > 0xf59ff1f0 >> > > >> > > Broker Config >> > > 1 Virtual Host Node and 1 Virtual Host configured with Berkley DB >> types >> > (BDB >> > > with the default configuration). >> > > 1 Topic (perf.topic) >> > > bound to a queue (perfQueue) with a binding key and a >> jms-selector >> > > filter >> > > bound to an "alternate exchange" of a type fanout which is also >> > bound >> > > to a queue but with no binding key (empty string) >> > > >> > > Dispatcher Config >> > > >> > > qdmanage -b amqp://localhost:10454 create --type=address >> prefix=perfQueue >> > > waypoint=true name=perf.queue.addr >> > > qdmanage -b amqp://localhost:10454 create --type=address >> > prefix=perf.topic >> > > waypoint=true name=perf.topic.addr >> > > qdmanage -b amqp://localhost:10454 create --type=connector >> > > role=route-container addr=localhost port=10455 >> > > name=localhost.broker.10455.connector >> > > qdmanage -b amqp://localhost:10454 create --type=autoLink >> addr=perfQueue >> > > dir=out connection=localhost.broker.10455.connector >> > > name=localhost.broker.10455.perfQueue.out >> > > qdmanage -b amqp://localhost:10454 create --type=autoLink >> addr=perfQueue >> > > dir=in connection=localhost.broker.10455.connector >> > > name=localhost.broker.10455.perfQueue.in >> > > qdmanage -b amqp://localhost:10454 create --type=autoLink >> addr=perf.topic >> > > dir=out connection=localhost.broker.10455.connector >> > > name=localhost.broker.10455.perf.topic.out >> > > qdmanage -b amqp://localhost:10454 create --type=autoLink >> addr=perf.topic >> > > dir=in connection=localhost.broker.10455.connector >> > > name=localhost.broker.10455.perf.topic.in >> > > >> > > Clients config >> > > 3 JMS Consumers connected each to perfQueue on the dispatcher >> > > 2 JMS producers connected each to perf.topic on the dispatcher >> > > >> > > With the above config, we send a number of messages of which only 1/3 >> > will >> > > be routed to the "alternate exchange" and never consumed. >> > > >> > > >> > > Versions >> > > Qpid Java Broker: 6.0.0 >> > > Qpid Dispatch: 0.6.0 >> > > JMS: 0.9.0 >> > > >> > > Regards, >> > > Adel >> > > >> > > >> > > >> > > -- >> > > View this message in context: >> > >> http://qpid.2158936.n2.nabble.com/OutOfMemory-exception-with-a-Qpid-Java-Broker-linked-to-qpid-dispatcher-tp7647065p7647066.html >> > > Sent from the Apache Qpid users mailing list archive at Nabble.com. >> > > >> > > --------------------------------------------------------------------- >> > > To unsubscribe, e-mail: [email protected] >> > > For additional commands, e-mail: [email protected] >> > > >> > >> > >> > >
