@Shazni, Thank you for the suggestion. We are planning to implement a global limit initially. It will be a bit challenging to implement a queue wise limit. I mean the configuration part of it. Two options I see is that we can define a broker argument to be used when declaring queue using AMQP, or we can define it in the main configuration file and lookup when creating queues. We will consider this in later iterations. This is tracked with https://github.com/wso2/message-broker/issues/241.
@Waruna Currently the limit is based on message count. We will consuder this in a later iteration. This is tracked with https://github.com/wso2/message-broker/issues/242. On Sat, Feb 10, 2018 at 12:09 AM, Waruna Jayaweera <[email protected]> wrote: > Hi, > > On Tue, Feb 6, 2018 at 11:34 AM, Hasitha Hiranya <[email protected]> > wrote: > >> Hi Asanka, >> >> +1 for the idea. >> >> Better if we can define the queue depth queue wise. If depth is >> exceeded, we have two options >> >> 1. drop the message (Qpid in-memory queues did this) >> 2. spill data out. >> >> If we think about message is a bean, it will have fetch metadata and >> fetch content methods. For inmemory messages those two methods will get >> data from memory (inside that message object itself). For spilled out ones >> we keep a id, so when loading to memory when queue moves, same method will >> be called and DB is queried. But yeah, there is a performance issue if we >> load message my message back to memory as we query DB message by message. >> > > In that case we can load message data back to memory as batches, so > number of DB calls will be less. > > @Asanka is this buffer limit is based on message count or total messages > size ? IMO total messages size of queue would be better limitation. > >> >> Thanks >> >> On Tue, Feb 6, 2018 at 10:11 AM, Asanka Abeyweera <[email protected]> >> wrote: >> >>> Hi all, >>> >>> >>> >>> *Problem* >>> Currently, all the messages for queues are kept in memory without a >>> limit. If we accumulate too many messages, the broker can go out of memory >>> which we need to avoid. Therefore as a solution, we need to keep only a >>> portion of the messages when a certain limit exceeds for a queue. That is, >>> there will be two modes of operation. >>> >>> 1. Default mode - Messages are parallely persisted and put in to a in >>> memory queue. The message delivery is done directly from messages published >>> to in-memory queue storage >>> 2. Spill mode - Published messages are directly persisted to database >>> without putting in to in memory queue. Separately, messages are read from >>> the database for delivery. >>> >>> But this change will introduce following issues. >>> >>> - Since we are keeping all the messages in memory, the reference >>> counting was easy. But with this change, there can be a situation where >>> we >>> do not have all copies of the same message in memory and we will have to >>> go >>> for the database to see if there are any message references, which is not >>> very efficient. >>> - Since we now have two modes of operation and switching between the >>> two modes need to be handled carefully. Otherwise, the message order can >>> be >>> messed up. >>> >>> >>> >>> *Proposed Solution* >>> I am planning to keep the message-queue reference information (queue >>> attachment info) in memory along with the message ordering for all the >>> messages for a queue. This will not add much memory footprint since we are >>> only keeping the message ID and the queue attachment info. >>> >>> >>> 1. In each queue buffer, we will put message with message data until we >>> reach the buffer limit. Then only the message ID and queue attachement >>> information is stored in the queue buffer to avoid filling memory. >>> 2. When one of the messages in the queue buffer is acknowledged we, will >>> proceed the buffer limit cursor and send message data read requests to fill >>> the message data. >>> 3. When the message data is read from database and filled, the >>> deliverable limit cursor will be progressed to make that message >>> deliverable. >>> >>> When we start a node only the message id and queue attachment >>> information is loaded. The handling logic will then asynchronously load >>> message data to memory. >>> >>> WDYT? >>> >>> -- >>> Asanka Abeyweera >>> Associate Technical Lead >>> WSO2 Inc. >>> >>> Phone: +94 712228648 <+94%2071%20222%208648> >>> Blog: a5anka.github.io >>> >>> <https://wso2.com/signature> >>> >> >> >> >> -- >> *Hasitha Abeykoon* >> Associate Technical Lead; WSO2, Inc.; http://wso2.com >> *cell:* *+94 719363063* >> *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com> >> >> >> _______________________________________________ >> Architecture mailing list >> [email protected] >> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture >> >> > > > -- > Regards, > > Waruna Lakshitha Jayaweera > Senior Software Engineer > WSO2 Inc; http://wso2.com > phone: +94713255198 <+94%2071%20325%205198> > http://waruapz.blogspot.com/ > > -- Asanka Abeyweera Associate Technical Lead WSO2 Inc. Phone: +94 712228648 Blog: a5anka.github.io <https://wso2.com/signature>
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
