Cool, thanks for your efforts. On Wed, Mar 28, 2018 at 9:02 AM, sowmya s <[email protected]> wrote:
> Hello Yukon, > I've submitted my final proposal addressing your comments as well. Hoping > to work on this project for the summer. In the meantime i'm working on > prototyping the global ordering approach i talked about in my proposal. > > thanks, > Sowmya > > On Fri, Mar 23, 2018 at 4:48 AM, yukon <[email protected]> wrote: > > > Hi, > > > > I added some comments, but it seems your google doc link isn't shared: > > > > ``` > > To resolve this issue, we need a global ordered messaging mechanism to > have > > a good scalability without hot-pot issue. > > > > A practical proposal is that sending messages to distributed > > queue/partition with an ordered key. Messages are ordered in the same > > queue, then implement an OrderConsumer to pull messages from these queues > > then perform merge sort, finally deliver these ordered messages to users. > > ``` > > As mentioned in ROCKETMQ-122, we should address the hot-pot issue. > > > > That means the messages can be sent to multiple distributed > > queues/partitions and still can be consumed in order manner. > > > > A simple explanation of hot-pot issue: > > > > Assume that all the generated messages in a hot shop of taobao/amazon > > should be consumed in order, in the current state, these massive messages > > should be sent to a fixed queue/partition, surely to a fixed server which > > may cause high pressure in this server. You can easily infer that we > can't > > resolve this issue through extending the scale of broker cluster. > > > > Could your arch resolve this hot-pot issue? > > > > Regards, > > yukon > > > > > > On Thu, Mar 22, 2018 at 2:15 PM, sowmya s <[email protected]> wrote: > > > > > It's going great. > > > In the DefaultMessageStore, there is a class > > > CommitLogDispatcherBuildConsumeQueue, I think that a variant of that > > class > > > or another implementation which are merged from the commitlog, > > > Since the DispachRequest has a store timestamp, the queue can be > > populated > > > based on that time stamp. > > > > > > Also, I have some more questions, > > > Does the fleet of broker slaves all store the various commit logs that > > are > > > received by them or is the commit log store, replication managed by the > > > broker controller? > > > > > > The other direction I was thinking about is adding/modifying the > > > PullAPIWrapper, since the DefaultMQPushConsumerImpl uses the > > > DefaultMQPullConsumerImpl and that uses the PullAPIWrapper, maybe > adding > > > this change there will solve the merge issue for the consumers. > > > > > > Do you think I am on the right track? > > > > > > thanks, > > > Sowmya > > > > > > > > > > > > On Mon, Mar 19, 2018 at 8:15 PM, yukon <[email protected]> wrote: > > > > > > > Hi, > > > > > > > > Sorry for the late reply. > > > > > > > > As for: > > > > > > > > ``` > > > > I'm looking at the BrokerController and MessageStore implementation > and > > > > hooks to understand where the merge logic will best fit. > > > > ``` > > > > > > > > how is it going? > > > > > > > > Regards > > > > > > > > On Sat, Mar 17, 2018 at 11:30 AM, sowmya s <[email protected]> > > wrote: > > > > > > > > > Thank you yukon, > > > > > > > > > > I'm done with my coursework for this semester and have more time > now > > to > > > > > improve my proposal. > > > > > I'm looking at the BrokerController and MessageStore implementation > > and > > > > > hooks to understand where the merge logic will best fit. So far > I've > > > > looked > > > > > at the codebase from a Producer and Consumer perspective and looked > > at > > > > the > > > > > DefaultMQProducerImpl and DefaultMQPushConsumerImpl for > > understanding > > > > the > > > > > link between how Producers send and Consumers receive messages. > > > > > > > > > > thanks, > > > > > Sowmya > > > > > > > > > > On Fri, Mar 16, 2018 at 8:07 PM, yukon <[email protected]> wrote: > > > > > > > > > > > Cool, let's focus on it and see whether is there anything can be > > > > > polished. > > > > > > > > > > > > Regards > > > > > > > > > > > > On Fri, Mar 16, 2018 at 11:54 PM, sowmya s <[email protected] > > > > > > wrote: > > > > > > > > > > > > > Hey yukon, > > > > > > > I submitted my draft on the summer of code homepage a couple of > > > days > > > > > ago, > > > > > > > also attaching the link here for reference, > > > > > > > > > > > > > > https://docs.google.com/file/d/1nXktUO_TF9- > > > > rSHSnGj5z5QZoHzMhosxm/edit? > > > > > > > usp=docslist_api&filetype=msword > > > > > > > > > > > > > > Thanks, > > > > > > > Sowmya > > > > > > > > > > > > > > On Thu, Mar 15, 2018 at 2:08 AM yukon <[email protected]> > wrote: > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > Of course, we can work together to finetune your design > draft. > > > > > > > > > > > > > > > > Regards, > > > > > > > > yukon > > > > > > > > > > > > > > > > On Thu, Mar 15, 2018 at 5:33 AM, sowmya s < > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > > > > > Hello, yukon and Von, > > > > > > > > > > > > > > > > > > I've shared my GSOC - 18 draft of the project. I'm looking > > > > forward > > > > > to > > > > > > > > > working with all of you to finetune the proposal. I will be > > > > > > allocating > > > > > > > 20 > > > > > > > > > hours per week from now to the proposal acceptance phase to > > > > address > > > > > > > > > questions and dive deep into any suggestions that you > > provide. > > > > > > > > > I am really looking forward to work on this project. > > > > > > > > > > > > > > > > > > thanks, > > > > > > > > > Sowmya > > > > > > > > > > > > > > > > > > On Tue, Mar 13, 2018 at 11:47 PM, sowmya s < > > > [email protected] > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hello yukon, > > > > > > > > > > > > > > > > > > > > Thank you for the inputs. I was able to look at the > ~/store > > > and > > > > > > kind > > > > > > > of > > > > > > > > > > understand the storage structure. I also looked at the > > > > > > > > > > DefaultMQProducerImpl and DefaultMQPullConsumerImpl, used > > in > > > > the > > > > > > > > > examples. > > > > > > > > > > Now I understand why you proposed a merge sort like > > approach > > > > for > > > > > > > > > > performing global ordering. Since the proposals are > open, I > > > am > > > > > > > > finalizing > > > > > > > > > > my draft and will have it up for review very soon. > > > > > > > > > > > > > > > > > > > > thanks, > > > > > > > > > > Sowmya > > > > > > > > > > > > > > > > > > > > On Fri, Mar 9, 2018 at 7:03 AM, yukon <[email protected]> > > > > wrote: > > > > > > > > > > > > > > > > > > > >> Hi Sowmya, > > > > > > > > > >> > > > > > > > > > >> ``` > > > > > > > > > >> Also, it would be great if you can at a high level, help > > me > > > > > > > understand > > > > > > > > > >> how the messages in the message queues are stored before > > the > > > > > > > consumer > > > > > > > > > reads > > > > > > > > > >> them. > > > > > > > > > >> ``` > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > > >> As shown in this figure, messages are sent to brokers by > > > > > producer > > > > > > > and > > > > > > > > > >> stored in commit log[1], then messages are dispatched to > > > > > > > ConsumeQueue > > > > > > > > by > > > > > > > > > >> topic, the consumer pulls messages from the queue. > > > > > > > > > >> > > > > > > > > > >> I recommend you run a broker and send/consume some > > messages, > > > > > then > > > > > > > > check > > > > > > > > > >> out the ~/store directory for details. > > > > > > > > > >> > > > > > > > > > >> Regards, > > > > > > > > > >> yukon > > > > > > > > > >> > > > > > > > > > >> On Thu, Mar 8, 2018 at 12:09 PM, sowmya s < > > > > [email protected] > > > > > > > > > > > > > > wrote: > > > > > > > > > >> > > > > > > > > > >>> Hello yukon, > > > > > > > > > >>> > > > > > > > > > >>> Currently FIFO can be achieved with a producer sending > to > > > one > > > > > > > message > > > > > > > > > >>> queue, and when global ordering is required, multiple > > > > producers > > > > > > > have > > > > > > > > to > > > > > > > > > >>> send to a single topic queue. > > > > > > > > > >>> > > > > > > > > > >>> We want to allow multiple producers to send messages > on a > > > > topic > > > > > > to > > > > > > > > > >>> multiple message queues and still provide ordering > > > guarantees > > > > > to > > > > > > > the > > > > > > > > > >>> consumer, so that all consumers see the same order of > > data > > > > and > > > > > > also > > > > > > > > the > > > > > > > > > >>> data is delivered in an ordered fashion. > > > > > > > > > >>> > > > > > > > > > >>> 1) Your idea of using a merge sort with the assumption > > that > > > > the > > > > > > > first > > > > > > > > > >>> arriving message is treated as the first message to be > > > > > delivered, > > > > > > > > > however, > > > > > > > > > >>> I want to propose an approach where when the producer > > > sends a > > > > > > > message > > > > > > > > > to a > > > > > > > > > >>> message queue, it must be done in a synchronous fashion > > and > > > > the > > > > > > > > > response > > > > > > > > > >>> will be that the message is accepted, which means that > > the > > > > > > message > > > > > > > > > follows > > > > > > > > > >>> the convention that all messages delivered prior by > that > > > > > producer > > > > > > > > have > > > > > > > > > been > > > > > > > > > >>> stored across groups and if not the producer will need > to > > > > > resend > > > > > > > the > > > > > > > > > >>> message. > > > > > > > > > >>> > > > > > > > > > >>> We can use a variant of total causal ordering in the > > layer > > > > > > between > > > > > > > > the > > > > > > > > > >>> message queue and store. > > > > > > > > > >>> > > > > > > > > > >>> I have been busy with my class project so I couldn't > > make a > > > > lot > > > > > > of > > > > > > > > > >>> progress in detailing my approach. Also, it would be > > great > > > if > > > > > you > > > > > > > can > > > > > > > > > at a > > > > > > > > > >>> high level, help me understand how the messages in the > > > > message > > > > > > > queues > > > > > > > > > are > > > > > > > > > >>> stored before the consumer reads them. > > > > > > > > > >>> > > > > > > > > > >>> Does the consumer read directly from the message queue > > that > > > > the > > > > > > > > > producer > > > > > > > > > >>> sends data to? does the broker receive the queued > > producer > > > > > > > messages, > > > > > > > > > store > > > > > > > > > >>> them and then pushes them to another queue for the > > consumer > > > > to > > > > > > read > > > > > > > > > from? > > > > > > > > > >>> > > > > > > > > > >>> For reference on total causal ordering: > > > > > > > > https://www.cl.cam.ac.uk/teach > > > > > > > > > >>> ing/0910/ConcDistS/10b-ProcGp-order.pdf > > > > > > > > > >>> > > > > > > > > > >>> thanks, > > > > > > > > > >>> Sowmya > > > > > > > > > >>> > > > > > > > > > >>> On Mon, Mar 5, 2018 at 4:26 AM, yukon < > [email protected]> > > > > > wrote: > > > > > > > > > >>> > > > > > > > > > >>>> Hi Sowmya, > > > > > > > > > >>>> > > > > > > > > > >>>> Sorry for the late reply, do you have any update on > this > > > > > > project? > > > > > > > > > >>>> > > > > > > > > > >>>> In RocketMQ, one message queue is a FIFO queue > natively, > > > so > > > > I > > > > > > > > proposed > > > > > > > > > >>>> a simple solution that performs merge sort on multiple > > > > queues > > > > > to > > > > > > > > > improve > > > > > > > > > >>>> performance and scalability. > > > > > > > > > >>>> > > > > > > > > > >>>> While the order issue across producers is difficult, > we > > > > could > > > > > > > assume > > > > > > > > > >>>> that the message first arrives the broker should be > > > consumed > > > > > > > > firstly. > > > > > > > > > >>>> > > > > > > > > > >>>> But it would be wonderful if you have a real design > > about > > > > the > > > > > > > order > > > > > > > > > >>>> issue across producers based on the RocketMQ design > and > > > the > > > > > > > storage > > > > > > > > > >>>> structure. > > > > > > > > > >>>> > > > > > > > > > >>>> Looking forward your design ~ > > > > > > > > > >>>> > > > > > > > > > >>>> Regards, > > > > > > > > > >>>> yukon > > > > > > > > > >>>> > > > > > > > > > >>>> On Fri, Mar 2, 2018 at 2:50 AM, sowmya s < > > > > > [email protected]> > > > > > > > > > wrote: > > > > > > > > > >>>> > > > > > > > > > >>>>> Hello all, > > > > > > > > > >>>>> > > > > > > > > > >>>>> Adding a few more thoughts on the problem of > > establishing > > > > an > > > > > > > order > > > > > > > > of > > > > > > > > > >>>>> messages across producers. > > > > > > > > > >>>>> > > > > > > > > > >>>>> Consider the Scenario > > > > > > > > > >>>>> > > > > > > > > > >>>>> Producer-1 produces messages a1, b1 and c1 into a > queue > > > > > Queue1 > > > > > > > > > >>>>> Producer-2 produces messages a2, b2 and c2 into queue > > > > Queue2. > > > > > > > > > >>>>> > > > > > > > > > >>>>> If we assume that time(a1) < time(b1) < time(c1) and > > > > > similarly > > > > > > > > > >>>>> time(a2) < time(b2) < time(c2) > > > > > > > > > >>>>> > > > > > > > > > >>>>> Are the following orders acceptable to the consumer? > > > > > > > > > >>>>> > > > > > > > > > >>>>> > a1, a2, b1, b2, c1, c2 > > > > > > > > > >>>>> > a1, b1, c1, a2, b2, c2 > > > > > > > > > >>>>> > a1 b1, b2, a2, a3, b3 > > > > > > > > > >>>>> > > > > > > > > > >>>>> and an order displayed at one consumer is consistent > > > across > > > > > all > > > > > > > > > >>>>> consumers. > > > > > > > > > >>>>> > > > > > > > > > >>>>> This can be achieved using Total Causal Ordering at > the > > > > > > Producer > > > > > > > or > > > > > > > > > >>>>> Queue level, using Leslie Lamport's clock and > > > > synchronization > > > > > > > > > approach. > > > > > > > > > >>>>> > > > > > > > > > >>>>> For reference is the paper attached, > > > > > > > > http://lamport.azurewebsites.n > > > > > > > > > >>>>> et/pubs/time-clocks.pdf > > > > > > > > > >>>>> > > > > > > > > > >>>>> > > > > > > > > > >>>>> -- > > > > > > > > > >>>>> Regards, > > > > > > > > > >>>>> > > > > > > > > > >>>>> Sowmya > > > > > > > > > >>>>> > > > > > > > > > >>>>> On Sun, Feb 25, 2018 at 7:37 PM, sowmya s < > > > > > > [email protected]> > > > > > > > > > >>>>> wrote: > > > > > > > > > >>>>> > > > > > > > > > >>>>>> Hello, > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> I'm trying to work on the issue ROCKETMQ-122 > > > > > > > > > >>>>>> < > > > > > > > > https://issues.apache.org/jira/browse/ROCKETMQ-122? > > > filter=12343065 > > > > > > > > > > > > > > as > > > > > > > > > >>>>>> a part of Google Summer of Code 2018. I've been > > spending > > > > > some > > > > > > > time > > > > > > > > > to > > > > > > > > > >>>>>> understand the system, architecture and the existing > > > > > Messaging > > > > > > > > > Patterns. > > > > > > > > > >>>>>> I still have a few questions and would like to > clarify > > > my > > > > > > > > > assumptions. > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> - Is the current FIFO order example limited to > one > > > > > message > > > > > > > > queue > > > > > > > > > >>>>>> per producer? Can the producer send the same > > message > > > to > > > > > > > > multiple > > > > > > > > > queues? > > > > > > > > > >>>>>> Will the consumers of the queues be able to read > > > > messages > > > > > > in > > > > > > > > > Order? > > > > > > > > > >>>>>> - Can I assume that each producer will send > > messages > > > to > > > > > one > > > > > > > > > queue? > > > > > > > > > >>>>>> - Global Order is to be identified across all > > > > > > > > > >>>>>> GlobalOrderedProducer (a new producer that is to > be > > > > used > > > > > > for > > > > > > > > > global order) > > > > > > > > > >>>>>> instances that are running. > > > > > > > > > >>>>>> - I think using a global clock can help establish > > the > > > > > order > > > > > > > > > >>>>>> between 2 or more producers, however using some > > form > > > of > > > > > > > vector > > > > > > > > > >>>>>> clock might also help identify the global order > of > > > > > messages > > > > > > > > > between the > > > > > > > > > >>>>>> producers. > > > > > > > > > >>>>>> - A GlobalOrderedConsumer ( consumer that knows > how > > > to > > > > > read > > > > > > > > > >>>>>> globally ordered messages) can then compare > > messages > > > > > across > > > > > > > all > > > > > > > > > message > > > > > > > > > >>>>>> queues from the corresponding producers and > extract > > > the > > > > > > > > > messages. [ this is > > > > > > > > > >>>>>> the approach recommended by yukon on the issue > > page ] > > > > > > > > > >>>>>> - We can also potentially have another layer in > the > > > > > Message > > > > > > > > Queue > > > > > > > > > >>>>>> which accumulates all messages sent from > producers > > > and > > > > > > > provides > > > > > > > > > one ordered > > > > > > > > > >>>>>> message queue for consumers to read from. > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> Thank you for your patience and please let me know > if > > my > > > > > > > > > >>>>>> understanding of the problem and the assumptions are > > > > right. > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> Best Regards, > > > > > > > > > >>>>>> Sowmya > > > > > > > > > >>>>>> > > > > > > > > > >>>>>> > > > > > > > > > >>>>> > > > > > > > > > >>>>> > > > > > > > > > >>>>> -- > > > > > > > > > >>>>> Regards, > > > > > > > > > >>>>> > > > > > > > > > >>>>> Sowmya > > > > > > > > > >>>>> > > > > > > > > > >>>>> > > > > > > > > > >>>> > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > > >>> -- > > > > > > > > > >>> Regards, > > > > > > > > > >>> > > > > > > > > > >>> Sowmya > > > > > > > > > >>> > > > > > > > > > >>> > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Regards, > > > > > > > > > > > > > > > > > > > > Sowmya > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > Regards, > > > > > > > > > > > > > > > > > > Sowmya > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > Regards, > > > > > > > > > > > > > > Sowmya > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Regards, > > > > > > > > > > Sowmya > > > > > > > > > > > > > > > > > > > > > -- > > > Regards, > > > > > > Sowmya > > > > > > > > > -- > Regards, > > Sowmya >
