Hello yukon,

Thank you for the inputs. I was able to look at the ~/store and kind of
understand the storage structure. I also looked at the
DefaultMQProducerImpl and DefaultMQPullConsumerImpl, used in the examples.
Now I understand why you proposed a merge sort like approach for performing
global ordering. Since the proposals are open, I am finalizing my draft and
will have it up for review very soon.

thanks,
Sowmya

On Fri, Mar 9, 2018 at 7:03 AM, yukon <yu...@apache.org> wrote:

> Hi Sowmya,
>
> ```
> Also, it would be great if you can at a high level, help me understand how
> the messages in the message queues are stored before the consumer reads
> them.
> ```
>
>
>
> As shown in this figure, messages are sent to brokers by producer and
> stored in commit log[1], then messages are dispatched to ConsumeQueue by
> topic, the consumer pulls messages from the queue.
>
> I recommend you run a broker and send/consume some messages, then check
> out the ~/store directory for details.
>
> Regards,
> yukon
>
> On Thu, Mar 8, 2018 at 12:09 PM, sowmya s <sowmya9...@gmail.com> wrote:
>
>> Hello yukon,
>>
>> Currently FIFO can be achieved with a producer sending to one message
>> queue, and when global ordering is required, multiple producers have to
>> send to a single topic queue.
>>
>> We want to allow multiple producers to send messages on a topic to
>> multiple message queues and still provide ordering guarantees to the
>> consumer, so that all consumers see the same order of data and also the
>> data is delivered in an ordered fashion.
>>
>> 1) Your idea of using a merge sort with the assumption that the first
>> arriving message is treated as the first message to be delivered, however,
>> I want to propose an approach where when the producer sends a message to a
>> message queue, it must be done in a synchronous fashion and the response
>> will be that the message is accepted, which means that the message follows
>> the convention that all messages delivered prior by that producer have been
>> stored across groups and if not the producer will need to resend the
>> message.
>>
>> We can use a variant of total causal ordering in the layer between the
>> message queue and store.
>>
>> I have been busy with my class project so I couldn't make a lot of
>> progress in detailing my approach. Also, it would be great if you can at a
>> high level, help me understand how the messages in the message queues are
>> stored before the consumer reads them.
>>
>> Does the consumer read directly from the message queue that the producer
>> sends data to? does the broker receive the queued producer messages, store
>> them and then pushes them to another queue for the consumer to read from?
>>
>> For reference on total causal ordering: https://www.cl.cam.ac.uk/teach
>> ing/0910/ConcDistS/10b-ProcGp-order.pdf
>>
>> thanks,
>> Sowmya
>>
>> On Mon, Mar 5, 2018 at 4:26 AM, yukon <yu...@apache.org> wrote:
>>
>>> Hi Sowmya,
>>>
>>> Sorry for the late reply, do you have any update on this project?
>>>
>>> In RocketMQ, one message queue is a FIFO queue natively, so I proposed a
>>> simple solution that performs merge sort on multiple queues to improve
>>> performance and scalability.
>>>
>>> While the order issue across producers is difficult, we could assume
>>> that the message first arrives the broker should be consumed firstly.
>>>
>>> But it would be wonderful if you have a real design about the order
>>> issue across producers based on the RocketMQ design and the storage
>>> structure.
>>>
>>> Looking forward your design ~
>>>
>>> Regards,
>>> yukon
>>>
>>> On Fri, Mar 2, 2018 at 2:50 AM, sowmya s <sowmya9...@gmail.com> wrote:
>>>
>>>> Hello all,
>>>>
>>>> Adding a few more thoughts on the problem of establishing an order of
>>>> messages across producers.
>>>>
>>>> Consider the Scenario
>>>>
>>>> Producer-1 produces messages a1, b1 and c1 into a queue Queue1
>>>> Producer-2 produces messages a2, b2 and c2 into queue Queue2.
>>>>
>>>> If we assume that time(a1) < time(b1) < time(c1) and similarly time(a2)
>>>> < time(b2) < time(c2)
>>>>
>>>> Are the following orders acceptable to the consumer?
>>>>
>>>> > a1, a2, b1, b2, c1, c2
>>>> > a1, b1, c1, a2, b2, c2
>>>> > a1 b1, b2, a2, a3, b3
>>>>
>>>> and an order displayed at one consumer is consistent across all
>>>> consumers.
>>>>
>>>> This can be achieved using Total Causal Ordering at the Producer or
>>>> Queue level, using Leslie Lamport's clock and synchronization approach.
>>>>
>>>> For reference is the paper attached, http://lamport.azurewebsites.n
>>>> et/pubs/time-clocks.pdf
>>>>
>>>>
>>>> --
>>>> Regards,
>>>>
>>>> Sowmya
>>>>
>>>> On Sun, Feb 25, 2018 at 7:37 PM, sowmya s <sowmya9...@gmail.com> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> I'm trying to work on the issue ROCKETMQ-122
>>>>> <https://issues.apache.org/jira/browse/ROCKETMQ-122?filter=12343065> as
>>>>> a part of Google Summer of Code 2018. I've been spending some time to
>>>>> understand the system, architecture and the existing Messaging Patterns.
>>>>> I still have a few questions and would like to clarify my assumptions.
>>>>>
>>>>>    - Is the current FIFO order example limited to one message queue
>>>>>    per producer? Can the producer send the same message to multiple 
>>>>> queues?
>>>>>    Will the consumers of the queues be able to read messages in Order?
>>>>>    - Can I assume that each producer will send messages to one queue?
>>>>>    - Global Order is to be identified across all
>>>>>    GlobalOrderedProducer (a new producer that is to be used for global 
>>>>> order)
>>>>>    instances that are running.
>>>>>    - I think using a global clock can help establish the order
>>>>>    between 2 or more producers, however using some form of vector
>>>>>    clock might also help identify the global order of messages between the
>>>>>    producers.
>>>>>    - A GlobalOrderedConsumer ( consumer that knows how to read
>>>>>    globally ordered messages) can then compare messages across all message
>>>>>    queues from the corresponding producers and extract the messages. [ 
>>>>> this is
>>>>>    the approach recommended by yukon on the issue page ]
>>>>>    - We can also potentially have another layer in the Message Queue
>>>>>    which accumulates all messages sent from producers and provides one 
>>>>> ordered
>>>>>    message queue for consumers to read from.
>>>>>
>>>>> Thank you for your patience and please let me know if my understanding
>>>>> of the problem and the assumptions are right.
>>>>>
>>>>> Best Regards,
>>>>> Sowmya
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>>
>>>> Sowmya
>>>>
>>>>
>>>
>>
>>
>> --
>> Regards,
>>
>> Sowmya
>>
>>
>


-- 
Regards,

Sowmya

Reply via email to