Hi, ``` Does the fleet of broker slaves all store the various commit logs that are received by them or is the commit log store, replication managed by the broker controller? ``` Yes, the slave broker will all the commit logs synced from the master broker.
``` The other direction I was thinking about is adding/modifying the PullAPIWrapper, since the DefaultMQPushConsumerImpl uses the DefaultMQPullConsumerImpl and that uses the PullAPIWrapper, maybe adding this change there will solve the merge issue for the consumers. ``` How about adding another consumer type? On Fri, Mar 23, 2018 at 7:48 PM, yukon <[email protected]> wrote: > Hi, > > I added some comments, but it seems your google doc link isn't shared: > > ``` > To resolve this issue, we need a global ordered messaging mechanism to > have a good scalability without hot-pot issue. > > A practical proposal is that sending messages to distributed > queue/partition with an ordered key. Messages are ordered in the same > queue, then implement an OrderConsumer to pull messages from these queues > then perform merge sort, finally deliver these ordered messages to users. > ``` > As mentioned in ROCKETMQ-122, we should address the hot-pot issue. > > That means the messages can be sent to multiple distributed > queues/partitions and still can be consumed in order manner. > > A simple explanation of hot-pot issue: > > Assume that all the generated messages in a hot shop of taobao/amazon > should be consumed in order, in the current state, these massive messages > should be sent to a fixed queue/partition, surely to a fixed server which > may cause high pressure in this server. You can easily infer that we can't > resolve this issue through extending the scale of broker cluster. > > Could your arch resolve this hot-pot issue? > > Regards, > yukon > > > On Thu, Mar 22, 2018 at 2:15 PM, sowmya s <[email protected]> wrote: > >> It's going great. >> In the DefaultMessageStore, there is a class >> CommitLogDispatcherBuildConsumeQueue, I think that a variant of that >> class >> or another implementation which are merged from the commitlog, >> Since the DispachRequest has a store timestamp, the queue can be populated >> based on that time stamp. >> >> Also, I have some more questions, >> Does the fleet of broker slaves all store the various commit logs that are >> received by them or is the commit log store, replication managed by the >> broker controller? >> >> The other direction I was thinking about is adding/modifying the >> PullAPIWrapper, since the DefaultMQPushConsumerImpl uses the >> DefaultMQPullConsumerImpl and that uses the PullAPIWrapper, maybe adding >> this change there will solve the merge issue for the consumers. >> >> Do you think I am on the right track? >> >> thanks, >> Sowmya >> >> >> >> On Mon, Mar 19, 2018 at 8:15 PM, yukon <[email protected]> wrote: >> >> > Hi, >> > >> > Sorry for the late reply. >> > >> > As for: >> > >> > ``` >> > I'm looking at the BrokerController and MessageStore implementation and >> > hooks to understand where the merge logic will best fit. >> > ``` >> > >> > how is it going? >> > >> > Regards >> > >> > On Sat, Mar 17, 2018 at 11:30 AM, sowmya s <[email protected]> >> wrote: >> > >> > > Thank you yukon, >> > > >> > > I'm done with my coursework for this semester and have more time now >> to >> > > improve my proposal. >> > > I'm looking at the BrokerController and MessageStore implementation >> and >> > > hooks to understand where the merge logic will best fit. So far I've >> > looked >> > > at the codebase from a Producer and Consumer perspective and looked at >> > the >> > > DefaultMQProducerImpl and DefaultMQPushConsumerImpl for understanding >> > the >> > > link between how Producers send and Consumers receive messages. >> > > >> > > thanks, >> > > Sowmya >> > > >> > > On Fri, Mar 16, 2018 at 8:07 PM, yukon <[email protected]> wrote: >> > > >> > > > Cool, let's focus on it and see whether is there anything can be >> > > polished. >> > > > >> > > > Regards >> > > > >> > > > On Fri, Mar 16, 2018 at 11:54 PM, sowmya s <[email protected]> >> > wrote: >> > > > >> > > > > Hey yukon, >> > > > > I submitted my draft on the summer of code homepage a couple of >> days >> > > ago, >> > > > > also attaching the link here for reference, >> > > > > >> > > > > https://docs.google.com/file/d/1nXktUO_TF9- >> > rSHSnGj5z5QZoHzMhosxm/edit? >> > > > > usp=docslist_api&filetype=msword >> > > > > >> > > > > Thanks, >> > > > > Sowmya >> > > > > >> > > > > On Thu, Mar 15, 2018 at 2:08 AM yukon <[email protected]> wrote: >> > > > > >> > > > > > Hi, >> > > > > > >> > > > > > Of course, we can work together to finetune your design draft. >> > > > > > >> > > > > > Regards, >> > > > > > yukon >> > > > > > >> > > > > > On Thu, Mar 15, 2018 at 5:33 AM, sowmya s <[email protected] >> > >> > > > wrote: >> > > > > > >> > > > > > > Hello, yukon and Von, >> > > > > > > >> > > > > > > I've shared my GSOC - 18 draft of the project. I'm looking >> > forward >> > > to >> > > > > > > working with all of you to finetune the proposal. I will be >> > > > allocating >> > > > > 20 >> > > > > > > hours per week from now to the proposal acceptance phase to >> > address >> > > > > > > questions and dive deep into any suggestions that you provide. >> > > > > > > I am really looking forward to work on this project. >> > > > > > > >> > > > > > > thanks, >> > > > > > > Sowmya >> > > > > > > >> > > > > > > On Tue, Mar 13, 2018 at 11:47 PM, sowmya s < >> [email protected] >> > > >> > > > > wrote: >> > > > > > > >> > > > > > > > Hello yukon, >> > > > > > > > >> > > > > > > > Thank you for the inputs. I was able to look at the ~/store >> and >> > > > kind >> > > > > of >> > > > > > > > understand the storage structure. I also looked at the >> > > > > > > > DefaultMQProducerImpl and DefaultMQPullConsumerImpl, used in >> > the >> > > > > > > examples. >> > > > > > > > Now I understand why you proposed a merge sort like approach >> > for >> > > > > > > > performing global ordering. Since the proposals are open, I >> am >> > > > > > finalizing >> > > > > > > > my draft and will have it up for review very soon. >> > > > > > > > >> > > > > > > > thanks, >> > > > > > > > Sowmya >> > > > > > > > >> > > > > > > > On Fri, Mar 9, 2018 at 7:03 AM, yukon <[email protected]> >> > wrote: >> > > > > > > > >> > > > > > > >> Hi Sowmya, >> > > > > > > >> >> > > > > > > >> ``` >> > > > > > > >> Also, it would be great if you can at a high level, help me >> > > > > understand >> > > > > > > >> how the messages in the message queues are stored before >> the >> > > > > consumer >> > > > > > > reads >> > > > > > > >> them. >> > > > > > > >> ``` >> > > > > > > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > > >> As shown in this figure, messages are sent to brokers by >> > > producer >> > > > > and >> > > > > > > >> stored in commit log[1], then messages are dispatched to >> > > > > ConsumeQueue >> > > > > > by >> > > > > > > >> topic, the consumer pulls messages from the queue. >> > > > > > > >> >> > > > > > > >> I recommend you run a broker and send/consume some >> messages, >> > > then >> > > > > > check >> > > > > > > >> out the ~/store directory for details. >> > > > > > > >> >> > > > > > > >> Regards, >> > > > > > > >> yukon >> > > > > > > >> >> > > > > > > >> On Thu, Mar 8, 2018 at 12:09 PM, sowmya s < >> > [email protected] >> > > > >> > > > > > wrote: >> > > > > > > >> >> > > > > > > >>> Hello yukon, >> > > > > > > >>> >> > > > > > > >>> Currently FIFO can be achieved with a producer sending to >> one >> > > > > message >> > > > > > > >>> queue, and when global ordering is required, multiple >> > producers >> > > > > have >> > > > > > to >> > > > > > > >>> send to a single topic queue. >> > > > > > > >>> >> > > > > > > >>> We want to allow multiple producers to send messages on a >> > topic >> > > > to >> > > > > > > >>> multiple message queues and still provide ordering >> guarantees >> > > to >> > > > > the >> > > > > > > >>> consumer, so that all consumers see the same order of data >> > and >> > > > also >> > > > > > the >> > > > > > > >>> data is delivered in an ordered fashion. >> > > > > > > >>> >> > > > > > > >>> 1) Your idea of using a merge sort with the assumption >> that >> > the >> > > > > first >> > > > > > > >>> arriving message is treated as the first message to be >> > > delivered, >> > > > > > > however, >> > > > > > > >>> I want to propose an approach where when the producer >> sends a >> > > > > message >> > > > > > > to a >> > > > > > > >>> message queue, it must be done in a synchronous fashion >> and >> > the >> > > > > > > response >> > > > > > > >>> will be that the message is accepted, which means that the >> > > > message >> > > > > > > follows >> > > > > > > >>> the convention that all messages delivered prior by that >> > > producer >> > > > > > have >> > > > > > > been >> > > > > > > >>> stored across groups and if not the producer will need to >> > > resend >> > > > > the >> > > > > > > >>> message. >> > > > > > > >>> >> > > > > > > >>> We can use a variant of total causal ordering in the layer >> > > > between >> > > > > > the >> > > > > > > >>> message queue and store. >> > > > > > > >>> >> > > > > > > >>> I have been busy with my class project so I couldn't make >> a >> > lot >> > > > of >> > > > > > > >>> progress in detailing my approach. Also, it would be >> great if >> > > you >> > > > > can >> > > > > > > at a >> > > > > > > >>> high level, help me understand how the messages in the >> > message >> > > > > queues >> > > > > > > are >> > > > > > > >>> stored before the consumer reads them. >> > > > > > > >>> >> > > > > > > >>> Does the consumer read directly from the message queue >> that >> > the >> > > > > > > producer >> > > > > > > >>> sends data to? does the broker receive the queued producer >> > > > > messages, >> > > > > > > store >> > > > > > > >>> them and then pushes them to another queue for the >> consumer >> > to >> > > > read >> > > > > > > from? >> > > > > > > >>> >> > > > > > > >>> For reference on total causal ordering: >> > > > > > https://www.cl.cam.ac.uk/teach >> > > > > > > >>> ing/0910/ConcDistS/10b-ProcGp-order.pdf >> > > > > > > >>> >> > > > > > > >>> thanks, >> > > > > > > >>> Sowmya >> > > > > > > >>> >> > > > > > > >>> On Mon, Mar 5, 2018 at 4:26 AM, yukon <[email protected]> >> > > wrote: >> > > > > > > >>> >> > > > > > > >>>> Hi Sowmya, >> > > > > > > >>>> >> > > > > > > >>>> Sorry for the late reply, do you have any update on this >> > > > project? >> > > > > > > >>>> >> > > > > > > >>>> In RocketMQ, one message queue is a FIFO queue natively, >> so >> > I >> > > > > > proposed >> > > > > > > >>>> a simple solution that performs merge sort on multiple >> > queues >> > > to >> > > > > > > improve >> > > > > > > >>>> performance and scalability. >> > > > > > > >>>> >> > > > > > > >>>> While the order issue across producers is difficult, we >> > could >> > > > > assume >> > > > > > > >>>> that the message first arrives the broker should be >> consumed >> > > > > > firstly. >> > > > > > > >>>> >> > > > > > > >>>> But it would be wonderful if you have a real design about >> > the >> > > > > order >> > > > > > > >>>> issue across producers based on the RocketMQ design and >> the >> > > > > storage >> > > > > > > >>>> structure. >> > > > > > > >>>> >> > > > > > > >>>> Looking forward your design ~ >> > > > > > > >>>> >> > > > > > > >>>> Regards, >> > > > > > > >>>> yukon >> > > > > > > >>>> >> > > > > > > >>>> On Fri, Mar 2, 2018 at 2:50 AM, sowmya s < >> > > [email protected]> >> > > > > > > wrote: >> > > > > > > >>>> >> > > > > > > >>>>> Hello all, >> > > > > > > >>>>> >> > > > > > > >>>>> Adding a few more thoughts on the problem of >> establishing >> > an >> > > > > order >> > > > > > of >> > > > > > > >>>>> messages across producers. >> > > > > > > >>>>> >> > > > > > > >>>>> Consider the Scenario >> > > > > > > >>>>> >> > > > > > > >>>>> Producer-1 produces messages a1, b1 and c1 into a queue >> > > Queue1 >> > > > > > > >>>>> Producer-2 produces messages a2, b2 and c2 into queue >> > Queue2. >> > > > > > > >>>>> >> > > > > > > >>>>> If we assume that time(a1) < time(b1) < time(c1) and >> > > similarly >> > > > > > > >>>>> time(a2) < time(b2) < time(c2) >> > > > > > > >>>>> >> > > > > > > >>>>> Are the following orders acceptable to the consumer? >> > > > > > > >>>>> >> > > > > > > >>>>> > a1, a2, b1, b2, c1, c2 >> > > > > > > >>>>> > a1, b1, c1, a2, b2, c2 >> > > > > > > >>>>> > a1 b1, b2, a2, a3, b3 >> > > > > > > >>>>> >> > > > > > > >>>>> and an order displayed at one consumer is consistent >> across >> > > all >> > > > > > > >>>>> consumers. >> > > > > > > >>>>> >> > > > > > > >>>>> This can be achieved using Total Causal Ordering at the >> > > > Producer >> > > > > or >> > > > > > > >>>>> Queue level, using Leslie Lamport's clock and >> > synchronization >> > > > > > > approach. >> > > > > > > >>>>> >> > > > > > > >>>>> For reference is the paper attached, >> > > > > > http://lamport.azurewebsites.n >> > > > > > > >>>>> et/pubs/time-clocks.pdf >> > > > > > > >>>>> >> > > > > > > >>>>> >> > > > > > > >>>>> -- >> > > > > > > >>>>> Regards, >> > > > > > > >>>>> >> > > > > > > >>>>> Sowmya >> > > > > > > >>>>> >> > > > > > > >>>>> On Sun, Feb 25, 2018 at 7:37 PM, sowmya s < >> > > > [email protected]> >> > > > > > > >>>>> wrote: >> > > > > > > >>>>> >> > > > > > > >>>>>> Hello, >> > > > > > > >>>>>> >> > > > > > > >>>>>> I'm trying to work on the issue ROCKETMQ-122 >> > > > > > > >>>>>> < >> > > > > > https://issues.apache.org/jira/browse/ROCKETMQ-122?filter= >> 12343065 >> > > >> > > > > > > as >> > > > > > > >>>>>> a part of Google Summer of Code 2018. I've been >> spending >> > > some >> > > > > time >> > > > > > > to >> > > > > > > >>>>>> understand the system, architecture and the existing >> > > Messaging >> > > > > > > Patterns. >> > > > > > > >>>>>> I still have a few questions and would like to clarify >> my >> > > > > > > assumptions. >> > > > > > > >>>>>> >> > > > > > > >>>>>> - Is the current FIFO order example limited to one >> > > message >> > > > > > queue >> > > > > > > >>>>>> per producer? Can the producer send the same >> message to >> > > > > > multiple >> > > > > > > queues? >> > > > > > > >>>>>> Will the consumers of the queues be able to read >> > messages >> > > > in >> > > > > > > Order? >> > > > > > > >>>>>> - Can I assume that each producer will send >> messages to >> > > one >> > > > > > > queue? >> > > > > > > >>>>>> - Global Order is to be identified across all >> > > > > > > >>>>>> GlobalOrderedProducer (a new producer that is to be >> > used >> > > > for >> > > > > > > global order) >> > > > > > > >>>>>> instances that are running. >> > > > > > > >>>>>> - I think using a global clock can help establish >> the >> > > order >> > > > > > > >>>>>> between 2 or more producers, however using some >> form of >> > > > > vector >> > > > > > > >>>>>> clock might also help identify the global order of >> > > messages >> > > > > > > between the >> > > > > > > >>>>>> producers. >> > > > > > > >>>>>> - A GlobalOrderedConsumer ( consumer that knows how >> to >> > > read >> > > > > > > >>>>>> globally ordered messages) can then compare messages >> > > across >> > > > > all >> > > > > > > message >> > > > > > > >>>>>> queues from the corresponding producers and extract >> the >> > > > > > > messages. [ this is >> > > > > > > >>>>>> the approach recommended by yukon on the issue page >> ] >> > > > > > > >>>>>> - We can also potentially have another layer in the >> > > Message >> > > > > > Queue >> > > > > > > >>>>>> which accumulates all messages sent from producers >> and >> > > > > provides >> > > > > > > one ordered >> > > > > > > >>>>>> message queue for consumers to read from. >> > > > > > > >>>>>> >> > > > > > > >>>>>> Thank you for your patience and please let me know if >> my >> > > > > > > >>>>>> understanding of the problem and the assumptions are >> > right. >> > > > > > > >>>>>> >> > > > > > > >>>>>> Best Regards, >> > > > > > > >>>>>> Sowmya >> > > > > > > >>>>>> >> > > > > > > >>>>>> >> > > > > > > >>>>> >> > > > > > > >>>>> >> > > > > > > >>>>> -- >> > > > > > > >>>>> Regards, >> > > > > > > >>>>> >> > > > > > > >>>>> Sowmya >> > > > > > > >>>>> >> > > > > > > >>>>> >> > > > > > > >>>> >> > > > > > > >>> >> > > > > > > >>> >> > > > > > > >>> -- >> > > > > > > >>> Regards, >> > > > > > > >>> >> > > > > > > >>> Sowmya >> > > > > > > >>> >> > > > > > > >>> >> > > > > > > >> >> > > > > > > > >> > > > > > > > >> > > > > > > > -- >> > > > > > > > Regards, >> > > > > > > > >> > > > > > > > Sowmya >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > -- >> > > > > > > Regards, >> > > > > > > >> > > > > > > Sowmya >> > > > > > > >> > > > > > >> > > > > -- >> > > > > Regards, >> > > > > >> > > > > Sowmya >> > > > > >> > > > >> > > >> > > >> > > >> > > -- >> > > Regards, >> > > >> > > Sowmya >> > > >> > >> >> >> >> -- >> Regards, >> >> Sowmya >> > >
