Hi Sowmya, ``` Also, it would be great if you can at a high level, help me understand how the messages in the message queues are stored before the consumer reads them. ```
As shown in this figure, messages are sent to brokers by producer and stored in commit log[1], then messages are dispatched to ConsumeQueue by topic, the consumer pulls messages from the queue. I recommend you run a broker and send/consume some messages, then check out the ~/store directory for details. Regards, yukon On Thu, Mar 8, 2018 at 12:09 PM, sowmya s <[email protected]> wrote: > Hello yukon, > > Currently FIFO can be achieved with a producer sending to one message > queue, and when global ordering is required, multiple producers have to > send to a single topic queue. > > We want to allow multiple producers to send messages on a topic to > multiple message queues and still provide ordering guarantees to the > consumer, so that all consumers see the same order of data and also the > data is delivered in an ordered fashion. > > 1) Your idea of using a merge sort with the assumption that the first > arriving message is treated as the first message to be delivered, however, > I want to propose an approach where when the producer sends a message to a > message queue, it must be done in a synchronous fashion and the response > will be that the message is accepted, which means that the message follows > the convention that all messages delivered prior by that producer have been > stored across groups and if not the producer will need to resend the > message. > > We can use a variant of total causal ordering in the layer between the > message queue and store. > > I have been busy with my class project so I couldn't make a lot of > progress in detailing my approach. Also, it would be great if you can at a > high level, help me understand how the messages in the message queues are > stored before the consumer reads them. > > Does the consumer read directly from the message queue that the producer > sends data to? does the broker receive the queued producer messages, store > them and then pushes them to another queue for the consumer to read from? > > For reference on total causal ordering: https://www.cl.cam.ac.uk/ > teaching/0910/ConcDistS/10b-ProcGp-order.pdf > > thanks, > Sowmya > > On Mon, Mar 5, 2018 at 4:26 AM, yukon <[email protected]> wrote: > >> Hi Sowmya, >> >> Sorry for the late reply, do you have any update on this project? >> >> In RocketMQ, one message queue is a FIFO queue natively, so I proposed a >> simple solution that performs merge sort on multiple queues to improve >> performance and scalability. >> >> While the order issue across producers is difficult, we could assume that >> the message first arrives the broker should be consumed firstly. >> >> But it would be wonderful if you have a real design about the order >> issue across producers based on the RocketMQ design and the storage >> structure. >> >> Looking forward your design ~ >> >> Regards, >> yukon >> >> On Fri, Mar 2, 2018 at 2:50 AM, sowmya s <[email protected]> wrote: >> >>> Hello all, >>> >>> Adding a few more thoughts on the problem of establishing an order of >>> messages across producers. >>> >>> Consider the Scenario >>> >>> Producer-1 produces messages a1, b1 and c1 into a queue Queue1 >>> Producer-2 produces messages a2, b2 and c2 into queue Queue2. >>> >>> If we assume that time(a1) < time(b1) < time(c1) and similarly time(a2) >>> < time(b2) < time(c2) >>> >>> Are the following orders acceptable to the consumer? >>> >>> > a1, a2, b1, b2, c1, c2 >>> > a1, b1, c1, a2, b2, c2 >>> > a1 b1, b2, a2, a3, b3 >>> >>> and an order displayed at one consumer is consistent across all >>> consumers. >>> >>> This can be achieved using Total Causal Ordering at the Producer or >>> Queue level, using Leslie Lamport's clock and synchronization approach. >>> >>> For reference is the paper attached, http://lamport.azurewebsites.n >>> et/pubs/time-clocks.pdf >>> >>> >>> -- >>> Regards, >>> >>> Sowmya >>> >>> On Sun, Feb 25, 2018 at 7:37 PM, sowmya s <[email protected]> wrote: >>> >>>> Hello, >>>> >>>> I'm trying to work on the issue ROCKETMQ-122 >>>> <https://issues.apache.org/jira/browse/ROCKETMQ-122?filter=12343065> as >>>> a part of Google Summer of Code 2018. I've been spending some time to >>>> understand the system, architecture and the existing Messaging Patterns. >>>> I still have a few questions and would like to clarify my assumptions. >>>> >>>> - Is the current FIFO order example limited to one message queue >>>> per producer? Can the producer send the same message to multiple queues? >>>> Will the consumers of the queues be able to read messages in Order? >>>> - Can I assume that each producer will send messages to one queue? >>>> - Global Order is to be identified across all GlobalOrderedProducer >>>> (a new producer that is to be used for global order) instances that are >>>> running. >>>> - I think using a global clock can help establish the order between >>>> 2 or more producers, however using some form of vector clock might >>>> also help identify the global order of messages between the producers. >>>> - A GlobalOrderedConsumer ( consumer that knows how to read >>>> globally ordered messages) can then compare messages across all message >>>> queues from the corresponding producers and extract the messages. [ >>>> this is >>>> the approach recommended by yukon on the issue page ] >>>> - We can also potentially have another layer in the Message Queue >>>> which accumulates all messages sent from producers and provides one >>>> ordered >>>> message queue for consumers to read from. >>>> >>>> Thank you for your patience and please let me know if my understanding >>>> of the problem and the assumptions are right. >>>> >>>> Best Regards, >>>> Sowmya >>>> >>>> >>> >>> >>> -- >>> Regards, >>> >>> Sowmya >>> >>> >> > > > -- > Regards, > > Sowmya > >
