Hello, yukon and Von, I've shared my GSOC - 18 draft of the project. I'm looking forward to working with all of you to finetune the proposal. I will be allocating 20 hours per week from now to the proposal acceptance phase to address questions and dive deep into any suggestions that you provide. I am really looking forward to work on this project.
thanks, Sowmya On Tue, Mar 13, 2018 at 11:47 PM, sowmya s <sowmya9...@gmail.com> wrote: > Hello yukon, > > Thank you for the inputs. I was able to look at the ~/store and kind of > understand the storage structure. I also looked at the > DefaultMQProducerImpl and DefaultMQPullConsumerImpl, used in the examples. > Now I understand why you proposed a merge sort like approach for > performing global ordering. Since the proposals are open, I am finalizing > my draft and will have it up for review very soon. > > thanks, > Sowmya > > On Fri, Mar 9, 2018 at 7:03 AM, yukon <yu...@apache.org> wrote: > >> Hi Sowmya, >> >> ``` >> Also, it would be great if you can at a high level, help me understand >> how the messages in the message queues are stored before the consumer reads >> them. >> ``` >> >> >> >> As shown in this figure, messages are sent to brokers by producer and >> stored in commit log[1], then messages are dispatched to ConsumeQueue by >> topic, the consumer pulls messages from the queue. >> >> I recommend you run a broker and send/consume some messages, then check >> out the ~/store directory for details. >> >> Regards, >> yukon >> >> On Thu, Mar 8, 2018 at 12:09 PM, sowmya s <sowmya9...@gmail.com> wrote: >> >>> Hello yukon, >>> >>> Currently FIFO can be achieved with a producer sending to one message >>> queue, and when global ordering is required, multiple producers have to >>> send to a single topic queue. >>> >>> We want to allow multiple producers to send messages on a topic to >>> multiple message queues and still provide ordering guarantees to the >>> consumer, so that all consumers see the same order of data and also the >>> data is delivered in an ordered fashion. >>> >>> 1) Your idea of using a merge sort with the assumption that the first >>> arriving message is treated as the first message to be delivered, however, >>> I want to propose an approach where when the producer sends a message to a >>> message queue, it must be done in a synchronous fashion and the response >>> will be that the message is accepted, which means that the message follows >>> the convention that all messages delivered prior by that producer have been >>> stored across groups and if not the producer will need to resend the >>> message. >>> >>> We can use a variant of total causal ordering in the layer between the >>> message queue and store. >>> >>> I have been busy with my class project so I couldn't make a lot of >>> progress in detailing my approach. Also, it would be great if you can at a >>> high level, help me understand how the messages in the message queues are >>> stored before the consumer reads them. >>> >>> Does the consumer read directly from the message queue that the producer >>> sends data to? does the broker receive the queued producer messages, store >>> them and then pushes them to another queue for the consumer to read from? >>> >>> For reference on total causal ordering: https://www.cl.cam.ac.uk/teach >>> ing/0910/ConcDistS/10b-ProcGp-order.pdf >>> >>> thanks, >>> Sowmya >>> >>> On Mon, Mar 5, 2018 at 4:26 AM, yukon <yu...@apache.org> wrote: >>> >>>> Hi Sowmya, >>>> >>>> Sorry for the late reply, do you have any update on this project? >>>> >>>> In RocketMQ, one message queue is a FIFO queue natively, so I proposed >>>> a simple solution that performs merge sort on multiple queues to improve >>>> performance and scalability. >>>> >>>> While the order issue across producers is difficult, we could assume >>>> that the message first arrives the broker should be consumed firstly. >>>> >>>> But it would be wonderful if you have a real design about the order >>>> issue across producers based on the RocketMQ design and the storage >>>> structure. >>>> >>>> Looking forward your design ~ >>>> >>>> Regards, >>>> yukon >>>> >>>> On Fri, Mar 2, 2018 at 2:50 AM, sowmya s <sowmya9...@gmail.com> wrote: >>>> >>>>> Hello all, >>>>> >>>>> Adding a few more thoughts on the problem of establishing an order of >>>>> messages across producers. >>>>> >>>>> Consider the Scenario >>>>> >>>>> Producer-1 produces messages a1, b1 and c1 into a queue Queue1 >>>>> Producer-2 produces messages a2, b2 and c2 into queue Queue2. >>>>> >>>>> If we assume that time(a1) < time(b1) < time(c1) and similarly >>>>> time(a2) < time(b2) < time(c2) >>>>> >>>>> Are the following orders acceptable to the consumer? >>>>> >>>>> > a1, a2, b1, b2, c1, c2 >>>>> > a1, b1, c1, a2, b2, c2 >>>>> > a1 b1, b2, a2, a3, b3 >>>>> >>>>> and an order displayed at one consumer is consistent across all >>>>> consumers. >>>>> >>>>> This can be achieved using Total Causal Ordering at the Producer or >>>>> Queue level, using Leslie Lamport's clock and synchronization approach. >>>>> >>>>> For reference is the paper attached, http://lamport.azurewebsites.n >>>>> et/pubs/time-clocks.pdf >>>>> >>>>> >>>>> -- >>>>> Regards, >>>>> >>>>> Sowmya >>>>> >>>>> On Sun, Feb 25, 2018 at 7:37 PM, sowmya s <sowmya9...@gmail.com> >>>>> wrote: >>>>> >>>>>> Hello, >>>>>> >>>>>> I'm trying to work on the issue ROCKETMQ-122 >>>>>> <https://issues.apache.org/jira/browse/ROCKETMQ-122?filter=12343065> as >>>>>> a part of Google Summer of Code 2018. I've been spending some time to >>>>>> understand the system, architecture and the existing Messaging Patterns. >>>>>> I still have a few questions and would like to clarify my assumptions. >>>>>> >>>>>> - Is the current FIFO order example limited to one message queue >>>>>> per producer? Can the producer send the same message to multiple >>>>>> queues? >>>>>> Will the consumers of the queues be able to read messages in Order? >>>>>> - Can I assume that each producer will send messages to one queue? >>>>>> - Global Order is to be identified across all >>>>>> GlobalOrderedProducer (a new producer that is to be used for global >>>>>> order) >>>>>> instances that are running. >>>>>> - I think using a global clock can help establish the order >>>>>> between 2 or more producers, however using some form of vector >>>>>> clock might also help identify the global order of messages between >>>>>> the >>>>>> producers. >>>>>> - A GlobalOrderedConsumer ( consumer that knows how to read >>>>>> globally ordered messages) can then compare messages across all >>>>>> message >>>>>> queues from the corresponding producers and extract the messages. [ >>>>>> this is >>>>>> the approach recommended by yukon on the issue page ] >>>>>> - We can also potentially have another layer in the Message Queue >>>>>> which accumulates all messages sent from producers and provides one >>>>>> ordered >>>>>> message queue for consumers to read from. >>>>>> >>>>>> Thank you for your patience and please let me know if my >>>>>> understanding of the problem and the assumptions are right. >>>>>> >>>>>> Best Regards, >>>>>> Sowmya >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> Regards, >>>>> >>>>> Sowmya >>>>> >>>>> >>>> >>> >>> >>> -- >>> Regards, >>> >>> Sowmya >>> >>> >> > > > -- > Regards, > > Sowmya > > -- Regards, Sowmya
