Hi,

On Tue, Feb 6, 2018 at 11:34 AM, Hasitha Hiranya <hasit...@wso2.com> wrote:

> Hi Asanka,
>
> +1 for the idea.
>
> Better if we can define the queue depth queue wise. If depth is exceeded,
> we have two options
>
> 1. drop the message (Qpid in-memory queues did this)
> 2. spill data out.
>
> If we think about message is a bean, it will have fetch metadata and fetch
> content methods. For inmemory messages those two methods will get data from
> memory (inside that message object itself). For spilled out ones we keep a
> id, so when loading to memory when queue moves, same method will be called
> and DB is queried. But yeah, there is a performance issue if we load
> message my message back to memory as we query DB message by message.
>

In that case we can load message data back to memory as batches,  so number
of DB calls will be less.

@Asanka is this buffer limit  is based on message count or total messages
size ? IMO total messages size of queue would be better limitation.

>
> Thanks
>
> On Tue, Feb 6, 2018 at 10:11 AM, Asanka Abeyweera <asank...@wso2.com>
> wrote:
>
>> Hi all,
>>
>>
>>
>> *Problem*
>> Currently, all the messages for queues are kept in memory without a
>> limit. If we accumulate too many messages, the broker can go out of memory
>> which we need to avoid. Therefore as a solution, we need to keep only a
>> portion of the messages when a certain limit exceeds for a queue. That is,
>> there will be two modes of operation.
>>
>> 1. Default mode - Messages are parallely persisted and put in to a in
>> memory queue. The message delivery is done directly from messages published
>> to in-memory queue storage
>> 2. Spill mode - Published messages are directly persisted to database
>> without putting in to in memory queue. Separately, messages are read from
>> the database for delivery.
>>
>> But this change will introduce following issues.
>>
>>    - Since we are keeping all the messages in memory, the reference
>>    counting was easy. But with this change, there can be a situation where we
>>    do not have all copies of the same message in memory and we will have to 
>> go
>>    for the database to see if there are any message references, which is not
>>    very efficient.
>>    - Since we now have two modes of operation and switching between the
>>    two modes need to be handled carefully. Otherwise, the message order can 
>> be
>>    messed up.
>>
>>
>>
>> *Proposed Solution*
>> I am planning to keep the message-queue reference information (queue
>> attachment info) in memory along with the message ordering for all the
>> messages for a queue. This will not add much memory footprint since we are
>> only keeping the message ID and the queue attachment info.
>>
>>
>> 1. In each queue buffer, we will put message with message data until we
>> reach the buffer limit. Then only the message ID and queue attachement
>> information is stored in the queue buffer to avoid filling memory.
>> 2. When one of the messages in the queue buffer is acknowledged we, will
>> proceed the buffer limit cursor and send message data read requests to fill
>> the message data.
>> 3. When the message data is read from database and filled, the
>> deliverable limit cursor will be progressed to make that message
>> deliverable.
>>
>> When we start a node only the message id and queue attachment information
>> is loaded. The handling logic will then asynchronously load message data to
>> memory.
>>
>> WDYT?
>> ‚Äč
>> --
>> Asanka Abeyweera
>> Associate Technical Lead
>> WSO2 Inc.
>>
>> Phone: +94 712228648 <+94%2071%20222%208648>
>> Blog: a5anka.github.io
>>
>> <https://wso2.com/signature>
>>
>
>
>
> --
> *Hasitha Abeykoon*
> Associate Technical Lead; WSO2, Inc.; http://wso2.com
> *cell:* *+94 719363063*
> *blog: **abeykoon.blogspot.com* <http://abeykoon.blogspot.com>
>
>
> _______________________________________________
> Architecture mailing list
> Architecture@wso2.org
> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>
>


-- 
Regards,

Waruna Lakshitha Jayaweera
Senior Software Engineer
WSO2 Inc; http://wso2.com
phone: +94713255198 <+94%2071%20325%205198>
http://waruapz.blogspot.com/
_______________________________________________
Architecture mailing list
Architecture@wso2.org
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture

Reply via email to