Cool, let's focus on it and see whether is there anything can be polished. Regards
On Fri, Mar 16, 2018 at 11:54 PM, sowmya s <[email protected]> wrote: > Hey yukon, > I submitted my draft on the summer of code homepage a couple of days ago, > also attaching the link here for reference, > > https://docs.google.com/file/d/1nXktUO_TF9-rSHSnGj5z5QZoHzMhosxm/edit? > usp=docslist_api&filetype=msword > > Thanks, > Sowmya > > On Thu, Mar 15, 2018 at 2:08 AM yukon <[email protected]> wrote: > > > Hi, > > > > Of course, we can work together to finetune your design draft. > > > > Regards, > > yukon > > > > On Thu, Mar 15, 2018 at 5:33 AM, sowmya s <[email protected]> wrote: > > > > > Hello, yukon and Von, > > > > > > I've shared my GSOC - 18 draft of the project. I'm looking forward to > > > working with all of you to finetune the proposal. I will be allocating > 20 > > > hours per week from now to the proposal acceptance phase to address > > > questions and dive deep into any suggestions that you provide. > > > I am really looking forward to work on this project. > > > > > > thanks, > > > Sowmya > > > > > > On Tue, Mar 13, 2018 at 11:47 PM, sowmya s <[email protected]> > wrote: > > > > > > > Hello yukon, > > > > > > > > Thank you for the inputs. I was able to look at the ~/store and kind > of > > > > understand the storage structure. I also looked at the > > > > DefaultMQProducerImpl and DefaultMQPullConsumerImpl, used in the > > > examples. > > > > Now I understand why you proposed a merge sort like approach for > > > > performing global ordering. Since the proposals are open, I am > > finalizing > > > > my draft and will have it up for review very soon. > > > > > > > > thanks, > > > > Sowmya > > > > > > > > On Fri, Mar 9, 2018 at 7:03 AM, yukon <[email protected]> wrote: > > > > > > > >> Hi Sowmya, > > > >> > > > >> ``` > > > >> Also, it would be great if you can at a high level, help me > understand > > > >> how the messages in the message queues are stored before the > consumer > > > reads > > > >> them. > > > >> ``` > > > >> > > > >> > > > >> > > > >> As shown in this figure, messages are sent to brokers by producer > and > > > >> stored in commit log[1], then messages are dispatched to > ConsumeQueue > > by > > > >> topic, the consumer pulls messages from the queue. > > > >> > > > >> I recommend you run a broker and send/consume some messages, then > > check > > > >> out the ~/store directory for details. > > > >> > > > >> Regards, > > > >> yukon > > > >> > > > >> On Thu, Mar 8, 2018 at 12:09 PM, sowmya s <[email protected]> > > wrote: > > > >> > > > >>> Hello yukon, > > > >>> > > > >>> Currently FIFO can be achieved with a producer sending to one > message > > > >>> queue, and when global ordering is required, multiple producers > have > > to > > > >>> send to a single topic queue. > > > >>> > > > >>> We want to allow multiple producers to send messages on a topic to > > > >>> multiple message queues and still provide ordering guarantees to > the > > > >>> consumer, so that all consumers see the same order of data and also > > the > > > >>> data is delivered in an ordered fashion. > > > >>> > > > >>> 1) Your idea of using a merge sort with the assumption that the > first > > > >>> arriving message is treated as the first message to be delivered, > > > however, > > > >>> I want to propose an approach where when the producer sends a > message > > > to a > > > >>> message queue, it must be done in a synchronous fashion and the > > > response > > > >>> will be that the message is accepted, which means that the message > > > follows > > > >>> the convention that all messages delivered prior by that producer > > have > > > been > > > >>> stored across groups and if not the producer will need to resend > the > > > >>> message. > > > >>> > > > >>> We can use a variant of total causal ordering in the layer between > > the > > > >>> message queue and store. > > > >>> > > > >>> I have been busy with my class project so I couldn't make a lot of > > > >>> progress in detailing my approach. Also, it would be great if you > can > > > at a > > > >>> high level, help me understand how the messages in the message > queues > > > are > > > >>> stored before the consumer reads them. > > > >>> > > > >>> Does the consumer read directly from the message queue that the > > > producer > > > >>> sends data to? does the broker receive the queued producer > messages, > > > store > > > >>> them and then pushes them to another queue for the consumer to read > > > from? > > > >>> > > > >>> For reference on total causal ordering: > > https://www.cl.cam.ac.uk/teach > > > >>> ing/0910/ConcDistS/10b-ProcGp-order.pdf > > > >>> > > > >>> thanks, > > > >>> Sowmya > > > >>> > > > >>> On Mon, Mar 5, 2018 at 4:26 AM, yukon <[email protected]> wrote: > > > >>> > > > >>>> Hi Sowmya, > > > >>>> > > > >>>> Sorry for the late reply, do you have any update on this project? > > > >>>> > > > >>>> In RocketMQ, one message queue is a FIFO queue natively, so I > > proposed > > > >>>> a simple solution that performs merge sort on multiple queues to > > > improve > > > >>>> performance and scalability. > > > >>>> > > > >>>> While the order issue across producers is difficult, we could > assume > > > >>>> that the message first arrives the broker should be consumed > > firstly. > > > >>>> > > > >>>> But it would be wonderful if you have a real design about the > order > > > >>>> issue across producers based on the RocketMQ design and the > storage > > > >>>> structure. > > > >>>> > > > >>>> Looking forward your design ~ > > > >>>> > > > >>>> Regards, > > > >>>> yukon > > > >>>> > > > >>>> On Fri, Mar 2, 2018 at 2:50 AM, sowmya s <[email protected]> > > > wrote: > > > >>>> > > > >>>>> Hello all, > > > >>>>> > > > >>>>> Adding a few more thoughts on the problem of establishing an > order > > of > > > >>>>> messages across producers. > > > >>>>> > > > >>>>> Consider the Scenario > > > >>>>> > > > >>>>> Producer-1 produces messages a1, b1 and c1 into a queue Queue1 > > > >>>>> Producer-2 produces messages a2, b2 and c2 into queue Queue2. > > > >>>>> > > > >>>>> If we assume that time(a1) < time(b1) < time(c1) and similarly > > > >>>>> time(a2) < time(b2) < time(c2) > > > >>>>> > > > >>>>> Are the following orders acceptable to the consumer? > > > >>>>> > > > >>>>> > a1, a2, b1, b2, c1, c2 > > > >>>>> > a1, b1, c1, a2, b2, c2 > > > >>>>> > a1 b1, b2, a2, a3, b3 > > > >>>>> > > > >>>>> and an order displayed at one consumer is consistent across all > > > >>>>> consumers. > > > >>>>> > > > >>>>> This can be achieved using Total Causal Ordering at the Producer > or > > > >>>>> Queue level, using Leslie Lamport's clock and synchronization > > > approach. > > > >>>>> > > > >>>>> For reference is the paper attached, > > http://lamport.azurewebsites.n > > > >>>>> et/pubs/time-clocks.pdf > > > >>>>> > > > >>>>> > > > >>>>> -- > > > >>>>> Regards, > > > >>>>> > > > >>>>> Sowmya > > > >>>>> > > > >>>>> On Sun, Feb 25, 2018 at 7:37 PM, sowmya s <[email protected]> > > > >>>>> wrote: > > > >>>>> > > > >>>>>> Hello, > > > >>>>>> > > > >>>>>> I'm trying to work on the issue ROCKETMQ-122 > > > >>>>>> < > > https://issues.apache.org/jira/browse/ROCKETMQ-122?filter=12343065> > > > as > > > >>>>>> a part of Google Summer of Code 2018. I've been spending some > time > > > to > > > >>>>>> understand the system, architecture and the existing Messaging > > > Patterns. > > > >>>>>> I still have a few questions and would like to clarify my > > > assumptions. > > > >>>>>> > > > >>>>>> - Is the current FIFO order example limited to one message > > queue > > > >>>>>> per producer? Can the producer send the same message to > > multiple > > > queues? > > > >>>>>> Will the consumers of the queues be able to read messages in > > > Order? > > > >>>>>> - Can I assume that each producer will send messages to one > > > queue? > > > >>>>>> - Global Order is to be identified across all > > > >>>>>> GlobalOrderedProducer (a new producer that is to be used for > > > global order) > > > >>>>>> instances that are running. > > > >>>>>> - I think using a global clock can help establish the order > > > >>>>>> between 2 or more producers, however using some form of > vector > > > >>>>>> clock might also help identify the global order of messages > > > between the > > > >>>>>> producers. > > > >>>>>> - A GlobalOrderedConsumer ( consumer that knows how to read > > > >>>>>> globally ordered messages) can then compare messages across > all > > > message > > > >>>>>> queues from the corresponding producers and extract the > > > messages. [ this is > > > >>>>>> the approach recommended by yukon on the issue page ] > > > >>>>>> - We can also potentially have another layer in the Message > > Queue > > > >>>>>> which accumulates all messages sent from producers and > provides > > > one ordered > > > >>>>>> message queue for consumers to read from. > > > >>>>>> > > > >>>>>> Thank you for your patience and please let me know if my > > > >>>>>> understanding of the problem and the assumptions are right. > > > >>>>>> > > > >>>>>> Best Regards, > > > >>>>>> Sowmya > > > >>>>>> > > > >>>>>> > > > >>>>> > > > >>>>> > > > >>>>> -- > > > >>>>> Regards, > > > >>>>> > > > >>>>> Sowmya > > > >>>>> > > > >>>>> > > > >>>> > > > >>> > > > >>> > > > >>> -- > > > >>> Regards, > > > >>> > > > >>> Sowmya > > > >>> > > > >>> > > > >> > > > > > > > > > > > > -- > > > > Regards, > > > > > > > > Sowmya > > > > > > > > > > > > > > > > > -- > > > Regards, > > > > > > Sowmya > > > > > > -- > Regards, > > Sowmya >
