Hi Asanka, +1 for the idea.
Better if we can define the queue depth queue wise. If depth is exceeded, we have two options 1. drop the message (Qpid in-memory queues did this) 2. spill data out. If we think about message is a bean, it will have fetch metadata and fetch content methods. For inmemory messages those two methods will get data from memory (inside that message object itself). For spilled out ones we keep a id, so when loading to memory when queue moves, same method will be called and DB is queried. But yeah, there is a performance issue if we load message my message back to memory as we query DB message by message. Thanks On Tue, Feb 6, 2018 at 10:11 AM, Asanka Abeyweera <[email protected]> wrote: > Hi all, > > > > *Problem* > Currently, all the messages for queues are kept in memory without a limit. > If we accumulate too many messages, the broker can go out of memory which > we need to avoid. Therefore as a solution, we need to keep only a portion > of the messages when a certain limit exceeds for a queue. That is, there > will be two modes of operation. > > 1. Default mode - Messages are parallely persisted and put in to a in > memory queue. The message delivery is done directly from messages published > to in-memory queue storage > 2. Spill mode - Published messages are directly persisted to database > without putting in to in memory queue. Separately, messages are read from > the database for delivery. > > But this change will introduce following issues. > > - Since we are keeping all the messages in memory, the reference > counting was easy. But with this change, there can be a situation where we > do not have all copies of the same message in memory and we will have to go > for the database to see if there are any message references, which is not > very efficient. > - Since we now have two modes of operation and switching between the > two modes need to be handled carefully. Otherwise, the message order can be > messed up. > > > > *Proposed Solution* > I am planning to keep the message-queue reference information (queue > attachment info) in memory along with the message ordering for all the > messages for a queue. This will not add much memory footprint since we are > only keeping the message ID and the queue attachment info. > > > 1. In each queue buffer, we will put message with message data until we > reach the buffer limit. Then only the message ID and queue attachement > information is stored in the queue buffer to avoid filling memory. > 2. When one of the messages in the queue buffer is acknowledged we, will > proceed the buffer limit cursor and send message data read requests to fill > the message data. > 3. When the message data is read from database and filled, the deliverable > limit cursor will be progressed to make that message deliverable. > > When we start a node only the message id and queue attachment information > is loaded. The handling logic will then asynchronously load message data to > memory. > > WDYT? > > -- > Asanka Abeyweera > Associate Technical Lead > WSO2 Inc. > > Phone: +94 712228648 > Blog: a5anka.github.io > > <https://wso2.com/signature> > -- *Hasitha Abeykoon* Associate Technical Lead; WSO2, Inc.; http://wso2.com *cell:* *+94 719363063* *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com>
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
