Thank you yukon, I'm done with my coursework for this semester and have more time now to improve my proposal. I'm looking at the BrokerController and MessageStore implementation and hooks to understand where the merge logic will best fit. So far I've looked at the codebase from a Producer and Consumer perspective and looked at the DefaultMQProducerImpl and DefaultMQPushConsumerImpl for understanding the link between how Producers send and Consumers receive messages.
thanks, Sowmya On Fri, Mar 16, 2018 at 8:07 PM, yukon <[email protected]> wrote: > Cool, let's focus on it and see whether is there anything can be polished. > > Regards > > On Fri, Mar 16, 2018 at 11:54 PM, sowmya s <[email protected]> wrote: > > > Hey yukon, > > I submitted my draft on the summer of code homepage a couple of days ago, > > also attaching the link here for reference, > > > > https://docs.google.com/file/d/1nXktUO_TF9-rSHSnGj5z5QZoHzMhosxm/edit? > > usp=docslist_api&filetype=msword > > > > Thanks, > > Sowmya > > > > On Thu, Mar 15, 2018 at 2:08 AM yukon <[email protected]> wrote: > > > > > Hi, > > > > > > Of course, we can work together to finetune your design draft. > > > > > > Regards, > > > yukon > > > > > > On Thu, Mar 15, 2018 at 5:33 AM, sowmya s <[email protected]> > wrote: > > > > > > > Hello, yukon and Von, > > > > > > > > I've shared my GSOC - 18 draft of the project. I'm looking forward to > > > > working with all of you to finetune the proposal. I will be > allocating > > 20 > > > > hours per week from now to the proposal acceptance phase to address > > > > questions and dive deep into any suggestions that you provide. > > > > I am really looking forward to work on this project. > > > > > > > > thanks, > > > > Sowmya > > > > > > > > On Tue, Mar 13, 2018 at 11:47 PM, sowmya s <[email protected]> > > wrote: > > > > > > > > > Hello yukon, > > > > > > > > > > Thank you for the inputs. I was able to look at the ~/store and > kind > > of > > > > > understand the storage structure. I also looked at the > > > > > DefaultMQProducerImpl and DefaultMQPullConsumerImpl, used in the > > > > examples. > > > > > Now I understand why you proposed a merge sort like approach for > > > > > performing global ordering. Since the proposals are open, I am > > > finalizing > > > > > my draft and will have it up for review very soon. > > > > > > > > > > thanks, > > > > > Sowmya > > > > > > > > > > On Fri, Mar 9, 2018 at 7:03 AM, yukon <[email protected]> wrote: > > > > > > > > > >> Hi Sowmya, > > > > >> > > > > >> ``` > > > > >> Also, it would be great if you can at a high level, help me > > understand > > > > >> how the messages in the message queues are stored before the > > consumer > > > > reads > > > > >> them. > > > > >> ``` > > > > >> > > > > >> > > > > >> > > > > >> As shown in this figure, messages are sent to brokers by producer > > and > > > > >> stored in commit log[1], then messages are dispatched to > > ConsumeQueue > > > by > > > > >> topic, the consumer pulls messages from the queue. > > > > >> > > > > >> I recommend you run a broker and send/consume some messages, then > > > check > > > > >> out the ~/store directory for details. > > > > >> > > > > >> Regards, > > > > >> yukon > > > > >> > > > > >> On Thu, Mar 8, 2018 at 12:09 PM, sowmya s <[email protected]> > > > wrote: > > > > >> > > > > >>> Hello yukon, > > > > >>> > > > > >>> Currently FIFO can be achieved with a producer sending to one > > message > > > > >>> queue, and when global ordering is required, multiple producers > > have > > > to > > > > >>> send to a single topic queue. > > > > >>> > > > > >>> We want to allow multiple producers to send messages on a topic > to > > > > >>> multiple message queues and still provide ordering guarantees to > > the > > > > >>> consumer, so that all consumers see the same order of data and > also > > > the > > > > >>> data is delivered in an ordered fashion. > > > > >>> > > > > >>> 1) Your idea of using a merge sort with the assumption that the > > first > > > > >>> arriving message is treated as the first message to be delivered, > > > > however, > > > > >>> I want to propose an approach where when the producer sends a > > message > > > > to a > > > > >>> message queue, it must be done in a synchronous fashion and the > > > > response > > > > >>> will be that the message is accepted, which means that the > message > > > > follows > > > > >>> the convention that all messages delivered prior by that producer > > > have > > > > been > > > > >>> stored across groups and if not the producer will need to resend > > the > > > > >>> message. > > > > >>> > > > > >>> We can use a variant of total causal ordering in the layer > between > > > the > > > > >>> message queue and store. > > > > >>> > > > > >>> I have been busy with my class project so I couldn't make a lot > of > > > > >>> progress in detailing my approach. Also, it would be great if you > > can > > > > at a > > > > >>> high level, help me understand how the messages in the message > > queues > > > > are > > > > >>> stored before the consumer reads them. > > > > >>> > > > > >>> Does the consumer read directly from the message queue that the > > > > producer > > > > >>> sends data to? does the broker receive the queued producer > > messages, > > > > store > > > > >>> them and then pushes them to another queue for the consumer to > read > > > > from? > > > > >>> > > > > >>> For reference on total causal ordering: > > > https://www.cl.cam.ac.uk/teach > > > > >>> ing/0910/ConcDistS/10b-ProcGp-order.pdf > > > > >>> > > > > >>> thanks, > > > > >>> Sowmya > > > > >>> > > > > >>> On Mon, Mar 5, 2018 at 4:26 AM, yukon <[email protected]> wrote: > > > > >>> > > > > >>>> Hi Sowmya, > > > > >>>> > > > > >>>> Sorry for the late reply, do you have any update on this > project? > > > > >>>> > > > > >>>> In RocketMQ, one message queue is a FIFO queue natively, so I > > > proposed > > > > >>>> a simple solution that performs merge sort on multiple queues to > > > > improve > > > > >>>> performance and scalability. > > > > >>>> > > > > >>>> While the order issue across producers is difficult, we could > > assume > > > > >>>> that the message first arrives the broker should be consumed > > > firstly. > > > > >>>> > > > > >>>> But it would be wonderful if you have a real design about the > > order > > > > >>>> issue across producers based on the RocketMQ design and the > > storage > > > > >>>> structure. > > > > >>>> > > > > >>>> Looking forward your design ~ > > > > >>>> > > > > >>>> Regards, > > > > >>>> yukon > > > > >>>> > > > > >>>> On Fri, Mar 2, 2018 at 2:50 AM, sowmya s <[email protected]> > > > > wrote: > > > > >>>> > > > > >>>>> Hello all, > > > > >>>>> > > > > >>>>> Adding a few more thoughts on the problem of establishing an > > order > > > of > > > > >>>>> messages across producers. > > > > >>>>> > > > > >>>>> Consider the Scenario > > > > >>>>> > > > > >>>>> Producer-1 produces messages a1, b1 and c1 into a queue Queue1 > > > > >>>>> Producer-2 produces messages a2, b2 and c2 into queue Queue2. > > > > >>>>> > > > > >>>>> If we assume that time(a1) < time(b1) < time(c1) and similarly > > > > >>>>> time(a2) < time(b2) < time(c2) > > > > >>>>> > > > > >>>>> Are the following orders acceptable to the consumer? > > > > >>>>> > > > > >>>>> > a1, a2, b1, b2, c1, c2 > > > > >>>>> > a1, b1, c1, a2, b2, c2 > > > > >>>>> > a1 b1, b2, a2, a3, b3 > > > > >>>>> > > > > >>>>> and an order displayed at one consumer is consistent across all > > > > >>>>> consumers. > > > > >>>>> > > > > >>>>> This can be achieved using Total Causal Ordering at the > Producer > > or > > > > >>>>> Queue level, using Leslie Lamport's clock and synchronization > > > > approach. > > > > >>>>> > > > > >>>>> For reference is the paper attached, > > > http://lamport.azurewebsites.n > > > > >>>>> et/pubs/time-clocks.pdf > > > > >>>>> > > > > >>>>> > > > > >>>>> -- > > > > >>>>> Regards, > > > > >>>>> > > > > >>>>> Sowmya > > > > >>>>> > > > > >>>>> On Sun, Feb 25, 2018 at 7:37 PM, sowmya s < > [email protected]> > > > > >>>>> wrote: > > > > >>>>> > > > > >>>>>> Hello, > > > > >>>>>> > > > > >>>>>> I'm trying to work on the issue ROCKETMQ-122 > > > > >>>>>> < > > > https://issues.apache.org/jira/browse/ROCKETMQ-122?filter=12343065> > > > > as > > > > >>>>>> a part of Google Summer of Code 2018. I've been spending some > > time > > > > to > > > > >>>>>> understand the system, architecture and the existing Messaging > > > > Patterns. > > > > >>>>>> I still have a few questions and would like to clarify my > > > > assumptions. > > > > >>>>>> > > > > >>>>>> - Is the current FIFO order example limited to one message > > > queue > > > > >>>>>> per producer? Can the producer send the same message to > > > multiple > > > > queues? > > > > >>>>>> Will the consumers of the queues be able to read messages > in > > > > Order? > > > > >>>>>> - Can I assume that each producer will send messages to one > > > > queue? > > > > >>>>>> - Global Order is to be identified across all > > > > >>>>>> GlobalOrderedProducer (a new producer that is to be used > for > > > > global order) > > > > >>>>>> instances that are running. > > > > >>>>>> - I think using a global clock can help establish the order > > > > >>>>>> between 2 or more producers, however using some form of > > vector > > > > >>>>>> clock might also help identify the global order of messages > > > > between the > > > > >>>>>> producers. > > > > >>>>>> - A GlobalOrderedConsumer ( consumer that knows how to read > > > > >>>>>> globally ordered messages) can then compare messages across > > all > > > > message > > > > >>>>>> queues from the corresponding producers and extract the > > > > messages. [ this is > > > > >>>>>> the approach recommended by yukon on the issue page ] > > > > >>>>>> - We can also potentially have another layer in the Message > > > Queue > > > > >>>>>> which accumulates all messages sent from producers and > > provides > > > > one ordered > > > > >>>>>> message queue for consumers to read from. > > > > >>>>>> > > > > >>>>>> Thank you for your patience and please let me know if my > > > > >>>>>> understanding of the problem and the assumptions are right. > > > > >>>>>> > > > > >>>>>> Best Regards, > > > > >>>>>> Sowmya > > > > >>>>>> > > > > >>>>>> > > > > >>>>> > > > > >>>>> > > > > >>>>> -- > > > > >>>>> Regards, > > > > >>>>> > > > > >>>>> Sowmya > > > > >>>>> > > > > >>>>> > > > > >>>> > > > > >>> > > > > >>> > > > > >>> -- > > > > >>> Regards, > > > > >>> > > > > >>> Sowmya > > > > >>> > > > > >>> > > > > >> > > > > > > > > > > > > > > > -- > > > > > Regards, > > > > > > > > > > Sowmya > > > > > > > > > > > > > > > > > > > > > > -- > > > > Regards, > > > > > > > > Sowmya > > > > > > > > > -- > > Regards, > > > > Sowmya > > > -- Regards, Sowmya
