On 8 November 2008 05:15, Keith Chow <[email protected]> wrote: > We applied the server side queue limits, server/client lowered prefetech > high/low marks, and simplified our test case as follows: > > 1) One fast producer at 200 msg/s, msg size ~250bytes, non-transactional, > non-persistent, no-ack mode, TTL = 5s. > 2) 2 to 3 extremely slow and/or suspended consumers subscribed to the same > topic. > > We also modified broker's expire message task to remove any node with > expired = true, ignoring durable / acquire condition (to make sure they're > purged from the message store). > > Result: > > Broker size old generation heap would still reach gigabytes in size in less > than 10 minutes. JConsole showed no significant messages had built up in any > queues much higher the prefetech count. > > Profiling showed the gigabytes of byte[] were referenced by the broker's > pool event Job queue. And the events themselves where referenced by > underlying mina's SocketImpl. > > The cause is similar to this TCP congestion issue from the apache mina users > list, http://mina.markmail.org/message/6q5t5gwdozypm6dk?q=byte%5B%5D+gc > > Is this expected behaviour with M3 java broker with slow client? > > As an interim solution, we've modified the broker to detect slow topic > consumers (by inspecting expiry timestamp for our usecase) and kill them off > (with mina protocol's close session call). This allowed > GC to reclaim the dead client's memory resource. > > Keith
Hi Keith, I'm going to take a look at this slow consumer issue in terms of reducing the data retained by the broker. I was surprised to read that you were able to cause the broker memory issues in the mina layers due to the slow consumer. I would have thought that the prefetch limit would have protected you. I was thinking of doing something similar to what you decided to do, disconnecting the client when they are detected to be 'slow'. Are you currently using this change in a production system? Do you have any feedback that you would like me to incorporate in the design of this feature? I will be putting a design together for this next week. Regards Martin > On Thu, Nov 6, 2008 at 5:14 AM, Gordon Sim <[email protected]> wrote: > >> Robert Greig wrote: >> >>> 2008/11/4 Gordon Sim <[email protected]>: >>> >>> Can someone from the C++ side indicate whether the C++ broker does >>>>> this? If not I shall raise enhancement requests for both brokers. >>>>> >>>> The c++ broker allows a hard limit to be set for a queue or a system wide >>>> default. >>>> >>> >>> What actions can you configure when the limit is hit? It occurs to me >>> that there are two main cases: >>> >>> 1) "regular" queue - in this case you want to limit the publisher >>> >>> 2) private temporary queue bound to a topic exchange (or indeed other >>> exchange types) - in this case you probably want to kill the slow >>> consumer >>> >>> Thoughts? >>> >> >> I agree. >> >> At present the c++ broker is much more limited and can either: kill the >> publisher, flow to disk, or discard the oldest message(s) to make room. >> > -- Martin Ritchie --------------------------------------------------------------------- Apache Qpid - AMQP Messaging Implementation Project: http://qpid.apache.org Use/Interact: mailto:[email protected]
