Hi Asanka, +1 for the idea. I think this will bring down the memory footprint significantly.
I would also like to suggest to have queue wise configurable in memory limit, if that's not in your plan as of now. Say we have two queues A and B. And we know queue A will be heavily populated and consumed compared to B. So having a memory limit allocated to A being larger than B would justify the demand for A's frequency of population and consumption. On Tue, Feb 6, 2018 at 12:40 AM, Asanka Abeyweera <[email protected]> wrote: > Hi Hasitha, > > On Tue, Feb 6, 2018 at 11:34 AM, Hasitha Hiranya <[email protected]> > wrote: > >> Hi Asanka, >> >> +1 for the idea. >> >> Better if we can define the queue depth queue wise. If depth is >> exceeded, we have two options >> > Added an issue to track this (https://github.com/wso2/ > message-broker/issues/223) > >> >> 1. drop the message (Qpid in-memory queues did this) >> 2. spill data out. >> >> We do not drop messages in a durable queue. We just partially sync > messages to memory from DB. > > >> If we think about message is a bean, it will have fetch metadata and >> fetch content methods. For inmemory messages those two methods will get >> data from memory (inside that message object itself). For spilled out ones >> we keep a id, so when loading to memory when queue moves, same method will >> be called and DB is queried. But yeah, there is a performance issue if we >> load message my message back to memory as we query DB message by message. >> >> Thanks >> >> On Tue, Feb 6, 2018 at 10:11 AM, Asanka Abeyweera <[email protected]> >> wrote: >> >>> Hi all, >>> >>> >>> >>> *Problem* >>> Currently, all the messages for queues are kept in memory without a >>> limit. If we accumulate too many messages, the broker can go out of memory >>> which we need to avoid. Therefore as a solution, we need to keep only a >>> portion of the messages when a certain limit exceeds for a queue. That is, >>> there will be two modes of operation. >>> >>> 1. Default mode - Messages are parallely persisted and put in to a in >>> memory queue. The message delivery is done directly from messages published >>> to in-memory queue storage >>> 2. Spill mode - Published messages are directly persisted to database >>> without putting in to in memory queue. Separately, messages are read from >>> the database for delivery. >>> >>> But this change will introduce following issues. >>> >>> - Since we are keeping all the messages in memory, the reference >>> counting was easy. But with this change, there can be a situation where >>> we >>> do not have all copies of the same message in memory and we will have to >>> go >>> for the database to see if there are any message references, which is not >>> very efficient. >>> - Since we now have two modes of operation and switching between the >>> two modes need to be handled carefully. Otherwise, the message order can >>> be >>> messed up. >>> >>> >>> >>> *Proposed Solution* >>> I am planning to keep the message-queue reference information (queue >>> attachment info) in memory along with the message ordering for all the >>> messages for a queue. This will not add much memory footprint since we are >>> only keeping the message ID and the queue attachment info. >>> >>> >>> 1. In each queue buffer, we will put message with message data until we >>> reach the buffer limit. Then only the message ID and queue attachement >>> information is stored in the queue buffer to avoid filling memory. >>> 2. When one of the messages in the queue buffer is acknowledged we, will >>> proceed the buffer limit cursor and send message data read requests to fill >>> the message data. >>> 3. When the message data is read from database and filled, the >>> deliverable limit cursor will be progressed to make that message >>> deliverable. >>> >>> When we start a node only the message id and queue attachment >>> information is loaded. The handling logic will then asynchronously load >>> message data to memory. >>> >>> WDYT? >>> >>> -- >>> Asanka Abeyweera >>> Associate Technical Lead >>> WSO2 Inc. >>> >>> Phone: +94 712228648 <+94%2071%20222%208648> >>> Blog: a5anka.github.io >>> >>> <https://wso2.com/signature> >>> >> >> >> >> -- >> *Hasitha Abeykoon* >> Associate Technical Lead; WSO2, Inc.; http://wso2.com >> *cell:* *+94 719363063* >> *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com> >> >> > > > -- > Asanka Abeyweera > Associate Technical Lead > WSO2 Inc. > > Phone: +94 712228648 <+94%2071%20222%208648> > Blog: a5anka.github.io > > <https://wso2.com/signature> > > _______________________________________________ > Architecture mailing list > [email protected] > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- Shazni Nazeer Mob : +94 777737331 LinkedIn : http://lk.linkedin.com/in/shazninazeer Blogs : https://medium.com/@mshazninazeer http://shazninazeer.blogspot.com <http://wso2.com/signature>
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
