Thank you Ted, Configuring the perf.topic/in and perfQueue/out solved the OOM exception.
Regards, Adel > Subject: Re: OutOfMemory exception with a Qpid Java Broker linked to qpid > dispatcher > To: [email protected] > From: [email protected] > Date: Fri, 8 Jul 2016 13:17:13 -0400 > > From what I understand of your setup, I think you should remove two > autoLinks from your router config. The perf.topic/in and the > perfQueue/out links are not necessary and the perf.topic/in link is > actually causing problems because of the side-effect of the creation of > the temporary queue. > > -Ted > > On 07/08/2016 01:11 PM, Olivier Mallassi wrote: > > We need to run the tests again but you are right. Looking at the Dispatch > > Router config, the links are in and out. > > thx. > > > > > > On Fri, Jul 8, 2016 at 5:47 PM, Ted Ross <[email protected]> wrote: > > > >> > >> > >> On 07/08/2016 11:42 AM, Olivier Mallassi wrote: > >> > >>> Hi Rob > >>> > >>> We have the following topology: > >>> > >>> * Publisher -> Dispatcher -> Broker1 or Broker 2 (Java) -> Dispatcher > >>> (same > >>> instance) -> Consumer > >>> * each Broker is configured (before the run) > >>> - with an exchange perf.topic (amqp topic) > >>> - with a queue (perf.queue) that has a binding (binding key + header > >>> based > >>> filtering) to the perf.topic exchange > >>> > >>> * the Dispatcher is configured with autoLink > >>> * The publisher publish in perf.topic (kind of distributed topic) > >>> * The consumer consume from perf.queue (distributed queue) > >>> * Dispatch version is 0.6.0 and Broker is in version 6.0.0. > >>> > >> > >> I think your autoLink _in_ from perf.topic is spurious. I suspect that > >> there are no consumers on the router consuming from "perf.topic" and > >> therefore the listener attached to the temporary queue is never issued > >> credit and it accumulates all perf.topic messages. > >> > >> > >> > >>> We have the expected behavior : published messages are load-balanced on > >>> brokers and consumed as expected. > >>> > >>> Yet, and we observed the OOM while running the test for a longer time, > >>> when > >>> the dispatcher connect, it creates a queue with a UUID (more or less) > >>> name. > >>> that queue is bind to perf.topic with # as a binding key (so every > >>> messages > >>> sent by the publisher fill up this queue thus the OOM because it is in > >>> memory). > >>> > >>> It looks to be "expected". > >>> Is there a way to avoid this? are we missing something? > >>> > >>> Many thx. > >>> > >>> oliv/ > >>> > >>> > >>> > >>> > >>> > >>> > >>> On Fri, Jul 8, 2016 at 5:16 PM, Rob Godfrey <[email protected]> > >>> wrote: > >>> > >>> Hi Adel, > >>>> > >>>> I'm a bit confused... you say > >>>> > >>>> As no consumer is connected to this queue, all messages are kept > >>>>> > >>>> in-memory which causes the OutOfMemory exception in the broker. > >>>> > >>>> but then also say > >>>> > >>>> Consumer connected to that queue (Is it the dispatcher?): > >>>>> Name: qdlink.OyY41QAJRZ4JGGg > >>>>> Mode: MOVE > >>>>> > >>>> > >>>> Does the queue have a consumer or not? > >>>> > >>>> As to holding on to the messages... the heap dump shows that the message > >>>> is > >>>> in the queue (which we know) and has not been evicted by flow to disk > >>>> (we'd > >>>> need to work out why)... > >>>> > >>>> On average what size are the messages you are sending? Do you see > >>>> messages > >>>> flow through this topic at all? > >>>> > >>>> As to the "weird" queue name - what is happening is that we are creating > >>>> an > >>>> internal temporary queue when you subscribe to an exchange object in the > >>>> broker... for that we use a UUID as the queue name. > >>>> > >>>> -- Rob > >>>> > >>>> On 8 July 2016 at 15:48, Adel Boutros <[email protected]> wrote: > >>>> > >>>> I forgot to add that when I execute a qdmanage command to delete the > >>>>> "connector" to the broker, the weird queue disappears. > >>>>> > >>>>> Adel > >>>>> > >>>>> Date: Fri, 8 Jul 2016 07:45:49 -0700 > >>>>>> From: [email protected] > >>>>>> To: [email protected] > >>>>>> Subject: Re: OutOfMemory exception with a Qpid Java Broker linked to > >>>>>> > >>>>> qpid dispatcher > >>>>> > >>>>>> > >>>>>> Fixing bad formatting on Nabbledue to Hotmail: > >>>>>> > >>>>>> Hello, > >>>>>> > >>>>>> We are doing some performance tests with a Qpid Java Broker connected > >>>>>> > >>>>> to > >>>> > >>>>> a > >>>>> > >>>>>> Qpid dispatcher. > >>>>>> We have noticed that after some time, the broker dies with an > >>>>>> > >>>>> OutOfMemory > >>>> > >>>>> exception and java heap is dumped. > >>>>>> After analyzing the heap dump and checking the broker, we have noticed > >>>>>> > >>>>> 2 > >>>> > >>>>> weird behaviors: > >>>>>> > >>>>>> 1) A queue is created on the broker with a weird Name bound on our > >>>>>> > >>>>> topic > >>>> > >>>>> (perf.topic) with a binding key "#". It will receive all messages. > >>>>>> This queue seems to be created only when the dispatcher is present and > >>>>>> > >>>>> is > >>>> > >>>>> configured. > >>>>>> As no consumer is connected to this queue, all messages are kept > >>>>>> > >>>>> in-memory > >>>>> > >>>>>> which causes the OutOfMemory exception in the broker. > >>>>>> > >>>>>> Weird Queue definition > >>>>>> Name: 18bb86dd-2953-4c6f-8eb2-56e5128963f3 > >>>>>> Type: standard > >>>>>> State: ACTIVE > >>>>>> Durable: false > >>>>>> Lifespan: DELETE_ON_NO_OUTBOUND_LINKS > >>>>>> Persist Messages: NEVER > >>>>>> Inbound: 0 msg/s (0.00 B/s) > >>>>>> Outbound: 0 msg/s (0.00 B/s) > >>>>>> Size: 2374 msgs (0.00 B) > >>>>>> Pre-fetched: 0 msgs (0.00 B) > >>>>>> Oldest Message Age: 356.064 secs > >>>>>> Enforced Max. Ttl(ms): 0 > >>>>>> Enforced Min. Ttl(ms): 0 > >>>>>> Exclusive: LINK > >>>>>> > >>>>>> Consumer connected to that queue (Is it the dispatcher?): > >>>>>> Name: qdlink.OyY41QAJRZ4JGGg > >>>>>> Mode: MOVE > >>>>>> > >>>>>> > >>>>>> 2) All messages even those consumed successfully by the consumers are > >>>>>> > >>>>> still > >>>>> > >>>>>> present on the broker > >>>>>> > >>>>>> From the heap dump, we can see the Berkley DB is keeping a reference to > >>>>>> > >>>>> all > >>>>> > >>>>>> messages. Is this also coming from the above weird queue? > >>>>>> > >>>>>> > >>>>>> PS: If we only use the dispatcher instead, we have none of the weird > >>>>>> behaviors > >>>>>> > >>>>>> Extract from the heap dump (Object holding > >>>>>> reference to one of the message header. "Validated" is one the message > >>>>>> header fields we set and which is already received by a consumer) > >>>>>> > >>>>>> char[9] @ 0xf59ff3d8 VALIDATED > >>>>>> '- value java.lang.String @ 0xf59ff3c0 VALIDATED > >>>>>> '- value java.util.HashMap$Entry @ 0xf59ff310 > >>>>>> '- [3] java.util.HashMap$Entry[8] @ 0xf59ff2c0 > >>>>>> '- table java.util.HashMap @ 0xf59ff188 > >>>>>> '- _appProperties > >>>>>> org.apache.qpid.server.protocol.v1_0.MessageMetaData_1_0 @ 0xf59fef88 > >>>>>> '- _metaData > >>>>>> > >>>>>> > >>>>> > >>>> org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore$MessageDataSoftRef > >>>> > >>>>> @ 0xf59fef70 > >>>>>> '- _messageDataRef > >>>>>> > >>>>>> > >>>>> > >>>> org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore$StoredBDBMessage > >>>> > >>>>> @ 0xf59fef20 > >>>>>> '- _handle > >>>>>> > >>>>> org.apache.qpid.server.protocol.v1_0.Message_1_0 @ > >>>>> > >>>>>> 0xf59feef0 > >>>>>> '- _message > >>>>>> org.apache.qpid.server.message.AbstractServerMessageImpl$Reference @ > >>>>>> 0xf59621c8 > >>>>>> '- _message > >>>>>> org.apache.qpid.server.queue.StandardQueueEntry @ 0xf5962188 > >>>>>> '- _next > >>>>>> > >>>>> org.apache.qpid.server.queue.StandardQueueEntry > >>>>> > >>>>>> @ 0xf5962130 > >>>>>> - this$0 > >>>>>> > >>>>>> > >>>>> org.apache.qpid.server.protocol.v1_0.MessageMetaData_1_0$MessageHeader_1_0 > >>>> @ > >>>> > >>>>> 0xf59ff1f0 > >>>>>> > >>>>>> Broker Config > >>>>>> 1 Virtual Host Node and 1 Virtual Host configured with Berkley DB types > >>>>>> > >>>>> (BDB > >>>>> > >>>>>> with the default configuration). > >>>>>> 1 Topic (perf.topic) > >>>>>> bound to a queue (perfQueue) with a binding key and a jms-selector > >>>>>> filter > >>>>>> bound to an "alternate exchange" of a type fanout which is also > >>>>>> > >>>>> bound > >>>>> > >>>>>> to a queue but with no binding key (empty string) > >>>>>> > >>>>>> Dispatcher Config > >>>>>> > >>>>>> qdmanage -b amqp://localhost:10454 create --type=address > >>>>>> > >>>>> prefix=perfQueue > >>>> > >>>>> waypoint=true name=perf.queue.addr > >>>>>> qdmanage -b amqp://localhost:10454 create --type=address > >>>>>> > >>>>> prefix=perf.topic > >>>>> > >>>>>> waypoint=true name=perf.topic.addr > >>>>>> qdmanage -b amqp://localhost:10454 create --type=connector > >>>>>> role=route-container addr=localhost port=10455 > >>>>>> name=localhost.broker.10455.connector > >>>>>> qdmanage -b amqp://localhost:10454 create --type=autoLink > >>>>>> > >>>>> addr=perfQueue > >>>> > >>>>> dir=out connection=localhost.broker.10455.connector > >>>>>> name=localhost.broker.10455.perfQueue.out > >>>>>> qdmanage -b amqp://localhost:10454 create --type=autoLink > >>>>>> > >>>>> addr=perfQueue > >>>> > >>>>> dir=in connection=localhost.broker.10455.connector > >>>>>> name=localhost.broker.10455.perfQueue.in > >>>>>> qdmanage -b amqp://localhost:10454 create --type=autoLink > >>>>>> > >>>>> addr=perf.topic > >>>> > >>>>> dir=out connection=localhost.broker.10455.connector > >>>>>> name=localhost.broker.10455.perf.topic.out > >>>>>> qdmanage -b amqp://localhost:10454 create --type=autoLink > >>>>>> > >>>>> addr=perf.topic > >>>> > >>>>> dir=in connection=localhost.broker.10455.connector > >>>>>> name=localhost.broker.10455.perf.topic.in > >>>>>> > >>>>>> Clients config > >>>>>> 3 JMS Consumers connected each to perfQueue on the dispatcher > >>>>>> 2 JMS producers connected each to perf.topic on the dispatcher > >>>>>> > >>>>>> With the above config, we send a number of messages of which only 1/3 > >>>>>> > >>>>> will > >>>>> > >>>>>> be routed to the "alternate exchange" and never consumed. > >>>>>> > >>>>>> > >>>>>> Versions > >>>>>> Qpid Java Broker: 6.0.0 > >>>>>> Qpid Dispatch: 0.6.0 > >>>>>> JMS: 0.9.0 > >>>>>> > >>>>>> Regards, > >>>>>> Adel > >>>>>> > >>>>>> > >>>>>> > >>>>>> -- > >>>>>> View this message in context: > >>>>>> > >>>>> > >>>>> > >>>> http://qpid.2158936.n2.nabble.com/OutOfMemory-exception-with-a-Qpid-Java-Broker-linked-to-qpid-dispatcher-tp7647065p7647066.html > >>>> > >>>>> Sent from the Apache Qpid users mailing list archive at Nabble.com. > >>>>>> > >>>>>> --------------------------------------------------------------------- > >>>>>> To unsubscribe, e-mail: [email protected] > >>>>>> For additional commands, e-mail: [email protected] > >>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: [email protected] > >> For additional commands, e-mail: [email protected] > >> > >> > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] >
