Hi Rob We have the following topology:
* Publisher -> Dispatcher -> Broker1 or Broker 2 (Java) -> Dispatcher (same instance) -> Consumer * each Broker is configured (before the run) - with an exchange perf.topic (amqp topic) - with a queue (perf.queue) that has a binding (binding key + header based filtering) to the perf.topic exchange * the Dispatcher is configured with autoLink * The publisher publish in perf.topic (kind of distributed topic) * The consumer consume from perf.queue (distributed queue) * Dispatch version is 0.6.0 and Broker is in version 6.0.0. We have the expected behavior : published messages are load-balanced on brokers and consumed as expected. Yet, and we observed the OOM while running the test for a longer time, when the dispatcher connect, it creates a queue with a UUID (more or less) name. that queue is bind to perf.topic with # as a binding key (so every messages sent by the publisher fill up this queue thus the OOM because it is in memory). It looks to be "expected". Is there a way to avoid this? are we missing something? Many thx. oliv/ On Fri, Jul 8, 2016 at 5:16 PM, Rob Godfrey <[email protected]> wrote: > Hi Adel, > > I'm a bit confused... you say > > > As no consumer is connected to this queue, all messages are kept > in-memory which causes the OutOfMemory exception in the broker. > > but then also say > > > Consumer connected to that queue (Is it the dispatcher?): > > Name: qdlink.OyY41QAJRZ4JGGg > > Mode: MOVE > > Does the queue have a consumer or not? > > As to holding on to the messages... the heap dump shows that the message is > in the queue (which we know) and has not been evicted by flow to disk (we'd > need to work out why)... > > On average what size are the messages you are sending? Do you see messages > flow through this topic at all? > > As to the "weird" queue name - what is happening is that we are creating an > internal temporary queue when you subscribe to an exchange object in the > broker... for that we use a UUID as the queue name. > > -- Rob > > On 8 July 2016 at 15:48, Adel Boutros <[email protected]> wrote: > > > I forgot to add that when I execute a qdmanage command to delete the > > "connector" to the broker, the weird queue disappears. > > > > Adel > > > > > Date: Fri, 8 Jul 2016 07:45:49 -0700 > > > From: [email protected] > > > To: [email protected] > > > Subject: Re: OutOfMemory exception with a Qpid Java Broker linked to > > qpid dispatcher > > > > > > Fixing bad formatting on Nabbledue to Hotmail: > > > > > > Hello, > > > > > > We are doing some performance tests with a Qpid Java Broker connected > to > > a > > > Qpid dispatcher. > > > We have noticed that after some time, the broker dies with an > OutOfMemory > > > exception and java heap is dumped. > > > After analyzing the heap dump and checking the broker, we have noticed > 2 > > > weird behaviors: > > > > > > 1) A queue is created on the broker with a weird Name bound on our > topic > > > (perf.topic) with a binding key "#". It will receive all messages. > > > This queue seems to be created only when the dispatcher is present and > is > > > configured. > > > As no consumer is connected to this queue, all messages are kept > > in-memory > > > which causes the OutOfMemory exception in the broker. > > > > > > Weird Queue definition > > > Name: 18bb86dd-2953-4c6f-8eb2-56e5128963f3 > > > Type: standard > > > State: ACTIVE > > > Durable: false > > > Lifespan: DELETE_ON_NO_OUTBOUND_LINKS > > > Persist Messages: NEVER > > > Inbound: 0 msg/s (0.00 B/s) > > > Outbound: 0 msg/s (0.00 B/s) > > > Size: 2374 msgs (0.00 B) > > > Pre-fetched: 0 msgs (0.00 B) > > > Oldest Message Age: 356.064 secs > > > Enforced Max. Ttl(ms): 0 > > > Enforced Min. Ttl(ms): 0 > > > Exclusive: LINK > > > > > > Consumer connected to that queue (Is it the dispatcher?): > > > Name: qdlink.OyY41QAJRZ4JGGg > > > Mode: MOVE > > > > > > > > > 2) All messages even those consumed successfully by the consumers are > > still > > > present on the broker > > > > > > From the heap dump, we can see the Berkley DB is keeping a reference to > > all > > > messages. Is this also coming from the above weird queue? > > > > > > > > > PS: If we only use the dispatcher instead, we have none of the weird > > > behaviors > > > > > > Extract from the heap dump (Object holding > > > reference to one of the message header. "Validated" is one the message > > > header fields we set and which is already received by a consumer) > > > > > > char[9] @ 0xf59ff3d8 VALIDATED > > > '- value java.lang.String @ 0xf59ff3c0 VALIDATED > > > '- value java.util.HashMap$Entry @ 0xf59ff310 > > > '- [3] java.util.HashMap$Entry[8] @ 0xf59ff2c0 > > > '- table java.util.HashMap @ 0xf59ff188 > > > '- _appProperties > > > org.apache.qpid.server.protocol.v1_0.MessageMetaData_1_0 @ 0xf59fef88 > > > '- _metaData > > > > > > org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore$MessageDataSoftRef > > > @ 0xf59fef70 > > > '- _messageDataRef > > > > > > org.apache.qpid.server.store.berkeleydb.AbstractBDBMessageStore$StoredBDBMessage > > > @ 0xf59fef20 > > > '- _handle > > org.apache.qpid.server.protocol.v1_0.Message_1_0 @ > > > 0xf59feef0 > > > '- _message > > > org.apache.qpid.server.message.AbstractServerMessageImpl$Reference @ > > > 0xf59621c8 > > > '- _message > > > org.apache.qpid.server.queue.StandardQueueEntry @ 0xf5962188 > > > '- _next > > org.apache.qpid.server.queue.StandardQueueEntry > > > @ 0xf5962130 > > > - this$0 > > > > > > org.apache.qpid.server.protocol.v1_0.MessageMetaData_1_0$MessageHeader_1_0 @ > > > 0xf59ff1f0 > > > > > > Broker Config > > > 1 Virtual Host Node and 1 Virtual Host configured with Berkley DB types > > (BDB > > > with the default configuration). > > > 1 Topic (perf.topic) > > > bound to a queue (perfQueue) with a binding key and a jms-selector > > > filter > > > bound to an "alternate exchange" of a type fanout which is also > > bound > > > to a queue but with no binding key (empty string) > > > > > > Dispatcher Config > > > > > > qdmanage -b amqp://localhost:10454 create --type=address > prefix=perfQueue > > > waypoint=true name=perf.queue.addr > > > qdmanage -b amqp://localhost:10454 create --type=address > > prefix=perf.topic > > > waypoint=true name=perf.topic.addr > > > qdmanage -b amqp://localhost:10454 create --type=connector > > > role=route-container addr=localhost port=10455 > > > name=localhost.broker.10455.connector > > > qdmanage -b amqp://localhost:10454 create --type=autoLink > addr=perfQueue > > > dir=out connection=localhost.broker.10455.connector > > > name=localhost.broker.10455.perfQueue.out > > > qdmanage -b amqp://localhost:10454 create --type=autoLink > addr=perfQueue > > > dir=in connection=localhost.broker.10455.connector > > > name=localhost.broker.10455.perfQueue.in > > > qdmanage -b amqp://localhost:10454 create --type=autoLink > addr=perf.topic > > > dir=out connection=localhost.broker.10455.connector > > > name=localhost.broker.10455.perf.topic.out > > > qdmanage -b amqp://localhost:10454 create --type=autoLink > addr=perf.topic > > > dir=in connection=localhost.broker.10455.connector > > > name=localhost.broker.10455.perf.topic.in > > > > > > Clients config > > > 3 JMS Consumers connected each to perfQueue on the dispatcher > > > 2 JMS producers connected each to perf.topic on the dispatcher > > > > > > With the above config, we send a number of messages of which only 1/3 > > will > > > be routed to the "alternate exchange" and never consumed. > > > > > > > > > Versions > > > Qpid Java Broker: 6.0.0 > > > Qpid Dispatch: 0.6.0 > > > JMS: 0.9.0 > > > > > > Regards, > > > Adel > > > > > > > > > > > > -- > > > View this message in context: > > > http://qpid.2158936.n2.nabble.com/OutOfMemory-exception-with-a-Qpid-Java-Broker-linked-to-qpid-dispatcher-tp7647065p7647066.html > > > Sent from the Apache Qpid users mailing list archive at Nabble.com. > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: [email protected] > > > For additional commands, e-mail: [email protected] > > > > > > > >
