Hi yukon, I'm working on updating the design for the hot-pot issue.
Thanks for the direction, from a consumer perspective, when the context is chosen as orderly, there is a specific Rebalance interface that orders the messages by Topic. this is used by the DefaultMQPushConsumerImpl to send messages to the consumer. The RebalanceImpl is the one that actually fetches the messages by topic from brokers and organizes by topic. Along with a How about having a custom RebalanceImpl that can be injected into the Consumer, so that the pull/push can be created based on the global order rebalance strategy. In the rebalanceImpl class, there is an implementation to do rebalance where messages from all brokers are fetched and ordered. thanks, Sowmya On Fri, Mar 23, 2018 at 5:00 AM, yukon <[email protected]> wrote: > Hi, > > ``` > Does the fleet of broker slaves all store the various commit logs that are > received by them or is the commit log store, replication managed by the > broker controller? > ``` > Yes, the slave broker will all the commit logs synced from the master > broker. > > ``` > The other direction I was thinking about is adding/modifying the > PullAPIWrapper, since the DefaultMQPushConsumerImpl uses the > DefaultMQPullConsumerImpl and that uses the PullAPIWrapper, maybe adding > this change there will solve the merge issue for the consumers. > ``` > > How about adding another consumer type? > > On Fri, Mar 23, 2018 at 7:48 PM, yukon <[email protected]> wrote: > > > Hi, > > > > I added some comments, but it seems your google doc link isn't shared: > > > > ``` > > To resolve this issue, we need a global ordered messaging mechanism to > > have a good scalability without hot-pot issue. > > > > A practical proposal is that sending messages to distributed > > queue/partition with an ordered key. Messages are ordered in the same > > queue, then implement an OrderConsumer to pull messages from these queues > > then perform merge sort, finally deliver these ordered messages to users. > > ``` > > As mentioned in ROCKETMQ-122, we should address the hot-pot issue. > > > > That means the messages can be sent to multiple distributed > > queues/partitions and still can be consumed in order manner. > > > > A simple explanation of hot-pot issue: > > > > Assume that all the generated messages in a hot shop of taobao/amazon > > should be consumed in order, in the current state, these massive messages > > should be sent to a fixed queue/partition, surely to a fixed server which > > may cause high pressure in this server. You can easily infer that we > can't > > resolve this issue through extending the scale of broker cluster. > > > > Could your arch resolve this hot-pot issue? > > > > Regards, > > yukon > > > > > > On Thu, Mar 22, 2018 at 2:15 PM, sowmya s <[email protected]> wrote: > > > >> It's going great. > >> In the DefaultMessageStore, there is a class > >> CommitLogDispatcherBuildConsumeQueue, I think that a variant of that > >> class > >> or another implementation which are merged from the commitlog, > >> Since the DispachRequest has a store timestamp, the queue can be > populated > >> based on that time stamp. > >> > >> Also, I have some more questions, > >> Does the fleet of broker slaves all store the various commit logs that > are > >> received by them or is the commit log store, replication managed by the > >> broker controller? > >> > >> The other direction I was thinking about is adding/modifying the > >> PullAPIWrapper, since the DefaultMQPushConsumerImpl uses the > >> DefaultMQPullConsumerImpl and that uses the PullAPIWrapper, maybe adding > >> this change there will solve the merge issue for the consumers. > >> > >> Do you think I am on the right track? > >> > >> thanks, > >> Sowmya > >> > >> > >> > >> On Mon, Mar 19, 2018 at 8:15 PM, yukon <[email protected]> wrote: > >> > >> > Hi, > >> > > >> > Sorry for the late reply. > >> > > >> > As for: > >> > > >> > ``` > >> > I'm looking at the BrokerController and MessageStore implementation > and > >> > hooks to understand where the merge logic will best fit. > >> > ``` > >> > > >> > how is it going? > >> > > >> > Regards > >> > > >> > On Sat, Mar 17, 2018 at 11:30 AM, sowmya s <[email protected]> > >> wrote: > >> > > >> > > Thank you yukon, > >> > > > >> > > I'm done with my coursework for this semester and have more time now > >> to > >> > > improve my proposal. > >> > > I'm looking at the BrokerController and MessageStore implementation > >> and > >> > > hooks to understand where the merge logic will best fit. So far I've > >> > looked > >> > > at the codebase from a Producer and Consumer perspective and looked > at > >> > the > >> > > DefaultMQProducerImpl and DefaultMQPushConsumerImpl for > understanding > >> > the > >> > > link between how Producers send and Consumers receive messages. > >> > > > >> > > thanks, > >> > > Sowmya > >> > > > >> > > On Fri, Mar 16, 2018 at 8:07 PM, yukon <[email protected]> wrote: > >> > > > >> > > > Cool, let's focus on it and see whether is there anything can be > >> > > polished. > >> > > > > >> > > > Regards > >> > > > > >> > > > On Fri, Mar 16, 2018 at 11:54 PM, sowmya s <[email protected]> > >> > wrote: > >> > > > > >> > > > > Hey yukon, > >> > > > > I submitted my draft on the summer of code homepage a couple of > >> days > >> > > ago, > >> > > > > also attaching the link here for reference, > >> > > > > > >> > > > > https://docs.google.com/file/d/1nXktUO_TF9- > >> > rSHSnGj5z5QZoHzMhosxm/edit? > >> > > > > usp=docslist_api&filetype=msword > >> > > > > > >> > > > > Thanks, > >> > > > > Sowmya > >> > > > > > >> > > > > On Thu, Mar 15, 2018 at 2:08 AM yukon <[email protected]> wrote: > >> > > > > > >> > > > > > Hi, > >> > > > > > > >> > > > > > Of course, we can work together to finetune your design draft. > >> > > > > > > >> > > > > > Regards, > >> > > > > > yukon > >> > > > > > > >> > > > > > On Thu, Mar 15, 2018 at 5:33 AM, sowmya s < > [email protected] > >> > > >> > > > wrote: > >> > > > > > > >> > > > > > > Hello, yukon and Von, > >> > > > > > > > >> > > > > > > I've shared my GSOC - 18 draft of the project. I'm looking > >> > forward > >> > > to > >> > > > > > > working with all of you to finetune the proposal. I will be > >> > > > allocating > >> > > > > 20 > >> > > > > > > hours per week from now to the proposal acceptance phase to > >> > address > >> > > > > > > questions and dive deep into any suggestions that you > provide. > >> > > > > > > I am really looking forward to work on this project. > >> > > > > > > > >> > > > > > > thanks, > >> > > > > > > Sowmya > >> > > > > > > > >> > > > > > > On Tue, Mar 13, 2018 at 11:47 PM, sowmya s < > >> [email protected] > >> > > > >> > > > > wrote: > >> > > > > > > > >> > > > > > > > Hello yukon, > >> > > > > > > > > >> > > > > > > > Thank you for the inputs. I was able to look at the > ~/store > >> and > >> > > > kind > >> > > > > of > >> > > > > > > > understand the storage structure. I also looked at the > >> > > > > > > > DefaultMQProducerImpl and DefaultMQPullConsumerImpl, used > in > >> > the > >> > > > > > > examples. > >> > > > > > > > Now I understand why you proposed a merge sort like > approach > >> > for > >> > > > > > > > performing global ordering. Since the proposals are open, > I > >> am > >> > > > > > finalizing > >> > > > > > > > my draft and will have it up for review very soon. > >> > > > > > > > > >> > > > > > > > thanks, > >> > > > > > > > Sowmya > >> > > > > > > > > >> > > > > > > > On Fri, Mar 9, 2018 at 7:03 AM, yukon <[email protected]> > >> > wrote: > >> > > > > > > > > >> > > > > > > >> Hi Sowmya, > >> > > > > > > >> > >> > > > > > > >> ``` > >> > > > > > > >> Also, it would be great if you can at a high level, help > me > >> > > > > understand > >> > > > > > > >> how the messages in the message queues are stored before > >> the > >> > > > > consumer > >> > > > > > > reads > >> > > > > > > >> them. > >> > > > > > > >> ``` > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > > >> As shown in this figure, messages are sent to brokers by > >> > > producer > >> > > > > and > >> > > > > > > >> stored in commit log[1], then messages are dispatched to > >> > > > > ConsumeQueue > >> > > > > > by > >> > > > > > > >> topic, the consumer pulls messages from the queue. > >> > > > > > > >> > >> > > > > > > >> I recommend you run a broker and send/consume some > >> messages, > >> > > then > >> > > > > > check > >> > > > > > > >> out the ~/store directory for details. > >> > > > > > > >> > >> > > > > > > >> Regards, > >> > > > > > > >> yukon > >> > > > > > > >> > >> > > > > > > >> On Thu, Mar 8, 2018 at 12:09 PM, sowmya s < > >> > [email protected] > >> > > > > >> > > > > > wrote: > >> > > > > > > >> > >> > > > > > > >>> Hello yukon, > >> > > > > > > >>> > >> > > > > > > >>> Currently FIFO can be achieved with a producer sending > to > >> one > >> > > > > message > >> > > > > > > >>> queue, and when global ordering is required, multiple > >> > producers > >> > > > > have > >> > > > > > to > >> > > > > > > >>> send to a single topic queue. > >> > > > > > > >>> > >> > > > > > > >>> We want to allow multiple producers to send messages on > a > >> > topic > >> > > > to > >> > > > > > > >>> multiple message queues and still provide ordering > >> guarantees > >> > > to > >> > > > > the > >> > > > > > > >>> consumer, so that all consumers see the same order of > data > >> > and > >> > > > also > >> > > > > > the > >> > > > > > > >>> data is delivered in an ordered fashion. > >> > > > > > > >>> > >> > > > > > > >>> 1) Your idea of using a merge sort with the assumption > >> that > >> > the > >> > > > > first > >> > > > > > > >>> arriving message is treated as the first message to be > >> > > delivered, > >> > > > > > > however, > >> > > > > > > >>> I want to propose an approach where when the producer > >> sends a > >> > > > > message > >> > > > > > > to a > >> > > > > > > >>> message queue, it must be done in a synchronous fashion > >> and > >> > the > >> > > > > > > response > >> > > > > > > >>> will be that the message is accepted, which means that > the > >> > > > message > >> > > > > > > follows > >> > > > > > > >>> the convention that all messages delivered prior by that > >> > > producer > >> > > > > > have > >> > > > > > > been > >> > > > > > > >>> stored across groups and if not the producer will need > to > >> > > resend > >> > > > > the > >> > > > > > > >>> message. > >> > > > > > > >>> > >> > > > > > > >>> We can use a variant of total causal ordering in the > layer > >> > > > between > >> > > > > > the > >> > > > > > > >>> message queue and store. > >> > > > > > > >>> > >> > > > > > > >>> I have been busy with my class project so I couldn't > make > >> a > >> > lot > >> > > > of > >> > > > > > > >>> progress in detailing my approach. Also, it would be > >> great if > >> > > you > >> > > > > can > >> > > > > > > at a > >> > > > > > > >>> high level, help me understand how the messages in the > >> > message > >> > > > > queues > >> > > > > > > are > >> > > > > > > >>> stored before the consumer reads them. > >> > > > > > > >>> > >> > > > > > > >>> Does the consumer read directly from the message queue > >> that > >> > the > >> > > > > > > producer > >> > > > > > > >>> sends data to? does the broker receive the queued > producer > >> > > > > messages, > >> > > > > > > store > >> > > > > > > >>> them and then pushes them to another queue for the > >> consumer > >> > to > >> > > > read > >> > > > > > > from? > >> > > > > > > >>> > >> > > > > > > >>> For reference on total causal ordering: > >> > > > > > https://www.cl.cam.ac.uk/teach > >> > > > > > > >>> ing/0910/ConcDistS/10b-ProcGp-order.pdf > >> > > > > > > >>> > >> > > > > > > >>> thanks, > >> > > > > > > >>> Sowmya > >> > > > > > > >>> > >> > > > > > > >>> On Mon, Mar 5, 2018 at 4:26 AM, yukon <[email protected] > > > >> > > wrote: > >> > > > > > > >>> > >> > > > > > > >>>> Hi Sowmya, > >> > > > > > > >>>> > >> > > > > > > >>>> Sorry for the late reply, do you have any update on > this > >> > > > project? > >> > > > > > > >>>> > >> > > > > > > >>>> In RocketMQ, one message queue is a FIFO queue > natively, > >> so > >> > I > >> > > > > > proposed > >> > > > > > > >>>> a simple solution that performs merge sort on multiple > >> > queues > >> > > to > >> > > > > > > improve > >> > > > > > > >>>> performance and scalability. > >> > > > > > > >>>> > >> > > > > > > >>>> While the order issue across producers is difficult, we > >> > could > >> > > > > assume > >> > > > > > > >>>> that the message first arrives the broker should be > >> consumed > >> > > > > > firstly. > >> > > > > > > >>>> > >> > > > > > > >>>> But it would be wonderful if you have a real design > about > >> > the > >> > > > > order > >> > > > > > > >>>> issue across producers based on the RocketMQ design and > >> the > >> > > > > storage > >> > > > > > > >>>> structure. > >> > > > > > > >>>> > >> > > > > > > >>>> Looking forward your design ~ > >> > > > > > > >>>> > >> > > > > > > >>>> Regards, > >> > > > > > > >>>> yukon > >> > > > > > > >>>> > >> > > > > > > >>>> On Fri, Mar 2, 2018 at 2:50 AM, sowmya s < > >> > > [email protected]> > >> > > > > > > wrote: > >> > > > > > > >>>> > >> > > > > > > >>>>> Hello all, > >> > > > > > > >>>>> > >> > > > > > > >>>>> Adding a few more thoughts on the problem of > >> establishing > >> > an > >> > > > > order > >> > > > > > of > >> > > > > > > >>>>> messages across producers. > >> > > > > > > >>>>> > >> > > > > > > >>>>> Consider the Scenario > >> > > > > > > >>>>> > >> > > > > > > >>>>> Producer-1 produces messages a1, b1 and c1 into a > queue > >> > > Queue1 > >> > > > > > > >>>>> Producer-2 produces messages a2, b2 and c2 into queue > >> > Queue2. > >> > > > > > > >>>>> > >> > > > > > > >>>>> If we assume that time(a1) < time(b1) < time(c1) and > >> > > similarly > >> > > > > > > >>>>> time(a2) < time(b2) < time(c2) > >> > > > > > > >>>>> > >> > > > > > > >>>>> Are the following orders acceptable to the consumer? > >> > > > > > > >>>>> > >> > > > > > > >>>>> > a1, a2, b1, b2, c1, c2 > >> > > > > > > >>>>> > a1, b1, c1, a2, b2, c2 > >> > > > > > > >>>>> > a1 b1, b2, a2, a3, b3 > >> > > > > > > >>>>> > >> > > > > > > >>>>> and an order displayed at one consumer is consistent > >> across > >> > > all > >> > > > > > > >>>>> consumers. > >> > > > > > > >>>>> > >> > > > > > > >>>>> This can be achieved using Total Causal Ordering at > the > >> > > > Producer > >> > > > > or > >> > > > > > > >>>>> Queue level, using Leslie Lamport's clock and > >> > synchronization > >> > > > > > > approach. > >> > > > > > > >>>>> > >> > > > > > > >>>>> For reference is the paper attached, > >> > > > > > http://lamport.azurewebsites.n > >> > > > > > > >>>>> et/pubs/time-clocks.pdf > >> > > > > > > >>>>> > >> > > > > > > >>>>> > >> > > > > > > >>>>> -- > >> > > > > > > >>>>> Regards, > >> > > > > > > >>>>> > >> > > > > > > >>>>> Sowmya > >> > > > > > > >>>>> > >> > > > > > > >>>>> On Sun, Feb 25, 2018 at 7:37 PM, sowmya s < > >> > > > [email protected]> > >> > > > > > > >>>>> wrote: > >> > > > > > > >>>>> > >> > > > > > > >>>>>> Hello, > >> > > > > > > >>>>>> > >> > > > > > > >>>>>> I'm trying to work on the issue ROCKETMQ-122 > >> > > > > > > >>>>>> < > >> > > > > > https://issues.apache.org/jira/browse/ROCKETMQ-122?filter= > >> 12343065 > >> > > > >> > > > > > > as > >> > > > > > > >>>>>> a part of Google Summer of Code 2018. I've been > >> spending > >> > > some > >> > > > > time > >> > > > > > > to > >> > > > > > > >>>>>> understand the system, architecture and the existing > >> > > Messaging > >> > > > > > > Patterns. > >> > > > > > > >>>>>> I still have a few questions and would like to > clarify > >> my > >> > > > > > > assumptions. > >> > > > > > > >>>>>> > >> > > > > > > >>>>>> - Is the current FIFO order example limited to one > >> > > message > >> > > > > > queue > >> > > > > > > >>>>>> per producer? Can the producer send the same > >> message to > >> > > > > > multiple > >> > > > > > > queues? > >> > > > > > > >>>>>> Will the consumers of the queues be able to read > >> > messages > >> > > > in > >> > > > > > > Order? > >> > > > > > > >>>>>> - Can I assume that each producer will send > >> messages to > >> > > one > >> > > > > > > queue? > >> > > > > > > >>>>>> - Global Order is to be identified across all > >> > > > > > > >>>>>> GlobalOrderedProducer (a new producer that is to > be > >> > used > >> > > > for > >> > > > > > > global order) > >> > > > > > > >>>>>> instances that are running. > >> > > > > > > >>>>>> - I think using a global clock can help establish > >> the > >> > > order > >> > > > > > > >>>>>> between 2 or more producers, however using some > >> form of > >> > > > > vector > >> > > > > > > >>>>>> clock might also help identify the global order of > >> > > messages > >> > > > > > > between the > >> > > > > > > >>>>>> producers. > >> > > > > > > >>>>>> - A GlobalOrderedConsumer ( consumer that knows > how > >> to > >> > > read > >> > > > > > > >>>>>> globally ordered messages) can then compare > messages > >> > > across > >> > > > > all > >> > > > > > > message > >> > > > > > > >>>>>> queues from the corresponding producers and > extract > >> the > >> > > > > > > messages. [ this is > >> > > > > > > >>>>>> the approach recommended by yukon on the issue > page > >> ] > >> > > > > > > >>>>>> - We can also potentially have another layer in > the > >> > > Message > >> > > > > > Queue > >> > > > > > > >>>>>> which accumulates all messages sent from producers > >> and > >> > > > > provides > >> > > > > > > one ordered > >> > > > > > > >>>>>> message queue for consumers to read from. > >> > > > > > > >>>>>> > >> > > > > > > >>>>>> Thank you for your patience and please let me know if > >> my > >> > > > > > > >>>>>> understanding of the problem and the assumptions are > >> > right. > >> > > > > > > >>>>>> > >> > > > > > > >>>>>> Best Regards, > >> > > > > > > >>>>>> Sowmya > >> > > > > > > >>>>>> > >> > > > > > > >>>>>> > >> > > > > > > >>>>> > >> > > > > > > >>>>> > >> > > > > > > >>>>> -- > >> > > > > > > >>>>> Regards, > >> > > > > > > >>>>> > >> > > > > > > >>>>> Sowmya > >> > > > > > > >>>>> > >> > > > > > > >>>>> > >> > > > > > > >>>> > >> > > > > > > >>> > >> > > > > > > >>> > >> > > > > > > >>> -- > >> > > > > > > >>> Regards, > >> > > > > > > >>> > >> > > > > > > >>> Sowmya > >> > > > > > > >>> > >> > > > > > > >>> > >> > > > > > > >> > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > -- > >> > > > > > > > Regards, > >> > > > > > > > > >> > > > > > > > Sowmya > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > -- > >> > > > > > > Regards, > >> > > > > > > > >> > > > > > > Sowmya > >> > > > > > > > >> > > > > > > >> > > > > -- > >> > > > > Regards, > >> > > > > > >> > > > > Sowmya > >> > > > > > >> > > > > >> > > > >> > > > >> > > > >> > > -- > >> > > Regards, > >> > > > >> > > Sowmya > >> > > > >> > > >> > >> > >> > >> -- > >> Regards, > >> > >> Sowmya > >> > > > > > -- Regards, Sowmya
