Hi all,
*Problem* Currently, all the messages for queues are kept in memory without a limit. If we accumulate too many messages, the broker can go out of memory which we need to avoid. Therefore as a solution, we need to keep only a portion of the messages when a certain limit exceeds for a queue. That is, there will be two modes of operation. 1. Default mode - Messages are parallely persisted and put in to a in memory queue. The message delivery is done directly from messages published to in-memory queue storage 2. Spill mode - Published messages are directly persisted to database without putting in to in memory queue. Separately, messages are read from the database for delivery. But this change will introduce following issues. - Since we are keeping all the messages in memory, the reference counting was easy. But with this change, there can be a situation where we do not have all copies of the same message in memory and we will have to go for the database to see if there are any message references, which is not very efficient. - Since we now have two modes of operation and switching between the two modes need to be handled carefully. Otherwise, the message order can be messed up. *Proposed Solution* I am planning to keep the message-queue reference information (queue attachment info) in memory along with the message ordering for all the messages for a queue. This will not add much memory footprint since we are only keeping the message ID and the queue attachment info. 1. In each queue buffer, we will put message with message data until we reach the buffer limit. Then only the message ID and queue attachement information is stored in the queue buffer to avoid filling memory. 2. When one of the messages in the queue buffer is acknowledged we, will proceed the buffer limit cursor and send message data read requests to fill the message data. 3. When the message data is read from database and filled, the deliverable limit cursor will be progressed to make that message deliverable. When we start a node only the message id and queue attachment information is loaded. The handling logic will then asynchronously load message data to memory. WDYT? -- Asanka Abeyweera Associate Technical Lead WSO2 Inc. Phone: +94 712228648 Blog: a5anka.github.io <https://wso2.com/signature>
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
