Hi, On Tue, Feb 6, 2018 at 11:34 AM, Hasitha Hiranya <hasit...@wso2.com> wrote:
> Hi Asanka, > > +1 for the idea. > > Better if we can define the queue depth queue wise. If depth is exceeded, > we have two options > > 1. drop the message (Qpid in-memory queues did this) > 2. spill data out. > > If we think about message is a bean, it will have fetch metadata and fetch > content methods. For inmemory messages those two methods will get data from > memory (inside that message object itself). For spilled out ones we keep a > id, so when loading to memory when queue moves, same method will be called > and DB is queried. But yeah, there is a performance issue if we load > message my message back to memory as we query DB message by message. > In that case we can load message data back to memory as batches, so number of DB calls will be less. @Asanka is this buffer limit is based on message count or total messages size ? IMO total messages size of queue would be better limitation. > > Thanks > > On Tue, Feb 6, 2018 at 10:11 AM, Asanka Abeyweera <asank...@wso2.com> > wrote: > >> Hi all, >> >> >> >> *Problem* >> Currently, all the messages for queues are kept in memory without a >> limit. If we accumulate too many messages, the broker can go out of memory >> which we need to avoid. Therefore as a solution, we need to keep only a >> portion of the messages when a certain limit exceeds for a queue. That is, >> there will be two modes of operation. >> >> 1. Default mode - Messages are parallely persisted and put in to a in >> memory queue. The message delivery is done directly from messages published >> to in-memory queue storage >> 2. Spill mode - Published messages are directly persisted to database >> without putting in to in memory queue. Separately, messages are read from >> the database for delivery. >> >> But this change will introduce following issues. >> >> - Since we are keeping all the messages in memory, the reference >> counting was easy. But with this change, there can be a situation where we >> do not have all copies of the same message in memory and we will have to >> go >> for the database to see if there are any message references, which is not >> very efficient. >> - Since we now have two modes of operation and switching between the >> two modes need to be handled carefully. Otherwise, the message order can >> be >> messed up. >> >> >> >> *Proposed Solution* >> I am planning to keep the message-queue reference information (queue >> attachment info) in memory along with the message ordering for all the >> messages for a queue. This will not add much memory footprint since we are >> only keeping the message ID and the queue attachment info. >> >> >> 1. In each queue buffer, we will put message with message data until we >> reach the buffer limit. Then only the message ID and queue attachement >> information is stored in the queue buffer to avoid filling memory. >> 2. When one of the messages in the queue buffer is acknowledged we, will >> proceed the buffer limit cursor and send message data read requests to fill >> the message data. >> 3. When the message data is read from database and filled, the >> deliverable limit cursor will be progressed to make that message >> deliverable. >> >> When we start a node only the message id and queue attachment information >> is loaded. The handling logic will then asynchronously load message data to >> memory. >> >> WDYT? >> >> -- >> Asanka Abeyweera >> Associate Technical Lead >> WSO2 Inc. >> >> Phone: +94 712228648 <+94%2071%20222%208648> >> Blog: a5anka.github.io >> >> <https://wso2.com/signature> >> > > > > -- > *Hasitha Abeykoon* > Associate Technical Lead; WSO2, Inc.; http://wso2.com > *cell:* *+94 719363063* > *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com> > > > _______________________________________________ > Architecture mailing list > Architecture@wso2.org > https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture > > -- Regards, Waruna Lakshitha Jayaweera Senior Software Engineer WSO2 Inc; http://wso2.com phone: +94713255198 <+94%2071%20325%205198> http://waruapz.blogspot.com/
_______________________________________________ Architecture mailing list Architecture@wso2.org https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
