Hello yukon, Thank you for the inputs. I was able to look at the ~/store and kind of understand the storage structure. I also looked at the DefaultMQProducerImpl and DefaultMQPullConsumerImpl, used in the examples. Now I understand why you proposed a merge sort like approach for performing global ordering. Since the proposals are open, I am finalizing my draft and will have it up for review very soon.
thanks, Sowmya On Fri, Mar 9, 2018 at 7:03 AM, yukon <[email protected]> wrote: > Hi Sowmya, > > ``` > Also, it would be great if you can at a high level, help me understand how > the messages in the message queues are stored before the consumer reads > them. > ``` > > > > As shown in this figure, messages are sent to brokers by producer and > stored in commit log[1], then messages are dispatched to ConsumeQueue by > topic, the consumer pulls messages from the queue. > > I recommend you run a broker and send/consume some messages, then check > out the ~/store directory for details. > > Regards, > yukon > > On Thu, Mar 8, 2018 at 12:09 PM, sowmya s <[email protected]> wrote: > >> Hello yukon, >> >> Currently FIFO can be achieved with a producer sending to one message >> queue, and when global ordering is required, multiple producers have to >> send to a single topic queue. >> >> We want to allow multiple producers to send messages on a topic to >> multiple message queues and still provide ordering guarantees to the >> consumer, so that all consumers see the same order of data and also the >> data is delivered in an ordered fashion. >> >> 1) Your idea of using a merge sort with the assumption that the first >> arriving message is treated as the first message to be delivered, however, >> I want to propose an approach where when the producer sends a message to a >> message queue, it must be done in a synchronous fashion and the response >> will be that the message is accepted, which means that the message follows >> the convention that all messages delivered prior by that producer have been >> stored across groups and if not the producer will need to resend the >> message. >> >> We can use a variant of total causal ordering in the layer between the >> message queue and store. >> >> I have been busy with my class project so I couldn't make a lot of >> progress in detailing my approach. Also, it would be great if you can at a >> high level, help me understand how the messages in the message queues are >> stored before the consumer reads them. >> >> Does the consumer read directly from the message queue that the producer >> sends data to? does the broker receive the queued producer messages, store >> them and then pushes them to another queue for the consumer to read from? >> >> For reference on total causal ordering: https://www.cl.cam.ac.uk/teach >> ing/0910/ConcDistS/10b-ProcGp-order.pdf >> >> thanks, >> Sowmya >> >> On Mon, Mar 5, 2018 at 4:26 AM, yukon <[email protected]> wrote: >> >>> Hi Sowmya, >>> >>> Sorry for the late reply, do you have any update on this project? >>> >>> In RocketMQ, one message queue is a FIFO queue natively, so I proposed a >>> simple solution that performs merge sort on multiple queues to improve >>> performance and scalability. >>> >>> While the order issue across producers is difficult, we could assume >>> that the message first arrives the broker should be consumed firstly. >>> >>> But it would be wonderful if you have a real design about the order >>> issue across producers based on the RocketMQ design and the storage >>> structure. >>> >>> Looking forward your design ~ >>> >>> Regards, >>> yukon >>> >>> On Fri, Mar 2, 2018 at 2:50 AM, sowmya s <[email protected]> wrote: >>> >>>> Hello all, >>>> >>>> Adding a few more thoughts on the problem of establishing an order of >>>> messages across producers. >>>> >>>> Consider the Scenario >>>> >>>> Producer-1 produces messages a1, b1 and c1 into a queue Queue1 >>>> Producer-2 produces messages a2, b2 and c2 into queue Queue2. >>>> >>>> If we assume that time(a1) < time(b1) < time(c1) and similarly time(a2) >>>> < time(b2) < time(c2) >>>> >>>> Are the following orders acceptable to the consumer? >>>> >>>> > a1, a2, b1, b2, c1, c2 >>>> > a1, b1, c1, a2, b2, c2 >>>> > a1 b1, b2, a2, a3, b3 >>>> >>>> and an order displayed at one consumer is consistent across all >>>> consumers. >>>> >>>> This can be achieved using Total Causal Ordering at the Producer or >>>> Queue level, using Leslie Lamport's clock and synchronization approach. >>>> >>>> For reference is the paper attached, http://lamport.azurewebsites.n >>>> et/pubs/time-clocks.pdf >>>> >>>> >>>> -- >>>> Regards, >>>> >>>> Sowmya >>>> >>>> On Sun, Feb 25, 2018 at 7:37 PM, sowmya s <[email protected]> wrote: >>>> >>>>> Hello, >>>>> >>>>> I'm trying to work on the issue ROCKETMQ-122 >>>>> <https://issues.apache.org/jira/browse/ROCKETMQ-122?filter=12343065> as >>>>> a part of Google Summer of Code 2018. I've been spending some time to >>>>> understand the system, architecture and the existing Messaging Patterns. >>>>> I still have a few questions and would like to clarify my assumptions. >>>>> >>>>> - Is the current FIFO order example limited to one message queue >>>>> per producer? Can the producer send the same message to multiple >>>>> queues? >>>>> Will the consumers of the queues be able to read messages in Order? >>>>> - Can I assume that each producer will send messages to one queue? >>>>> - Global Order is to be identified across all >>>>> GlobalOrderedProducer (a new producer that is to be used for global >>>>> order) >>>>> instances that are running. >>>>> - I think using a global clock can help establish the order >>>>> between 2 or more producers, however using some form of vector >>>>> clock might also help identify the global order of messages between the >>>>> producers. >>>>> - A GlobalOrderedConsumer ( consumer that knows how to read >>>>> globally ordered messages) can then compare messages across all message >>>>> queues from the corresponding producers and extract the messages. [ >>>>> this is >>>>> the approach recommended by yukon on the issue page ] >>>>> - We can also potentially have another layer in the Message Queue >>>>> which accumulates all messages sent from producers and provides one >>>>> ordered >>>>> message queue for consumers to read from. >>>>> >>>>> Thank you for your patience and please let me know if my understanding >>>>> of the problem and the assumptions are right. >>>>> >>>>> Best Regards, >>>>> Sowmya >>>>> >>>>> >>>> >>>> >>>> -- >>>> Regards, >>>> >>>> Sowmya >>>> >>>> >>> >> >> >> -- >> Regards, >> >> Sowmya >> >> > -- Regards, Sowmya
