Re: [GSOC|ROCKETMQ-122] Support Global Ordered Messaging

[yukon]

Cool, thanks for your efforts.

On Wed, Mar 28, 2018 at 9:02 AM, sowmya s  wrote:

> Hello Yukon,
> I've submitted my final proposal addressing your comments as well. Hoping
> to work on this project for the summer. In the meantime i'm working on
> prototyping the global ordering approach i talked about in my proposal.
>
> thanks,
> Sowmya
>
> On Fri, Mar 23, 2018 at 4:48 AM, yukon  wrote:
>
> > Hi,
> >
> > I added some comments, but it seems your google doc link isn't shared:
> >
> > ```
> > To resolve this issue, we need a global ordered messaging mechanism to
> have
> > a good scalability without hot-pot issue.
> >
> > A practical proposal is that sending messages to distributed
> > queue/partition with an ordered key. Messages are ordered in the same
> > queue, then implement an OrderConsumer to pull messages from these queues
> > then perform merge sort, finally deliver these ordered messages to users.
> > ```
> > As mentioned in ROCKETMQ-122, we should address the hot-pot issue.
> >
> > That means the messages can be sent to multiple distributed
> > queues/partitions and still can be consumed in order manner.
> >
> > A simple explanation of hot-pot issue:
> >
> > Assume that all the generated messages in a hot shop of taobao/amazon
> > should be consumed in order, in the current state, these massive messages
> > should be sent to a fixed queue/partition, surely to a fixed server which
> > may cause high pressure in this server. You can easily infer that we
> can't
> > resolve this issue through extending the scale of broker cluster.
> >
> > Could your arch resolve this hot-pot issue?
> >
> > Regards,
> > yukon
> >
> >
> > On Thu, Mar 22, 2018 at 2:15 PM, sowmya s  wrote:
> >
> > > It's going great.
> > > In the DefaultMessageStore, there is a class
> > > CommitLogDispatcherBuildConsumeQueue, I think that a variant of that
> > class
> > > or another implementation which are merged from the commitlog,
> > > Since the DispachRequest has a store timestamp, the queue can be
> > populated
> > > based on that time stamp.
> > >
> > > Also, I have some more questions,
> > > Does the fleet of broker slaves all store the various commit logs that
> > are
> > > received by them or is the commit log store, replication managed by the
> > > broker controller?
> > >
> > > The other direction I was thinking about is adding/modifying the
> > > PullAPIWrapper, since the DefaultMQPushConsumerImpl uses the
> > > DefaultMQPullConsumerImpl and that uses the PullAPIWrapper, maybe
> adding
> > > this change there will solve the merge issue for the consumers.
> > >
> > > Do you think I am on the right track?
> > >
> > > thanks,
> > > Sowmya
> > >
> > >
> > >
> > > On Mon, Mar 19, 2018 at 8:15 PM, yukon  wrote:
> > >
> > > > Hi,
> > > >
> > > > Sorry for the late reply.
> > > >
> > > > As for:
> > > >
> > > > ```
> > > > I'm looking at the BrokerController and MessageStore implementation
> and
> > > > hooks to understand where the merge logic will best fit.
> > > > ```
> > > >
> > > > how is it going?
> > > >
> > > > Regards
> > > >
> > > > On Sat, Mar 17, 2018 at 11:30 AM, sowmya s 
> > wrote:
> > > >
> > > > > Thank you yukon,
> > > > >
> > > > > I'm done with my coursework for this semester and have more time
> now
> > to
> > > > > improve my proposal.
> > > > > I'm looking at the BrokerController and MessageStore implementation
> > and
> > > > > hooks to understand where the merge logic will best fit. So far
> I've
> > > > looked
> > > > > at the codebase from a Producer and Consumer perspective and looked
> > at
> > > > the
> > > > > DefaultMQProducerImpl and  DefaultMQPushConsumerImpl for
> > understanding
> > > > the
> > > > > link between how Producers send and Consumers receive messages.
> > > > >
> > > > > thanks,
> > > > > Sowmya
> > > > >
> > > > > On Fri, Mar 16, 2018 at 8:07 PM, yukon  wrote:
> > > > >
> > > > > > Cool, let's focus on it and see whether is there anything can be
> > > > > polished.
> > > > > >
> > > > > > Regards
> > > > > >
> > > > > > On Fri, Mar 16, 2018 at 11:54 PM, sowmya s  >
> > > > wrote:
> > > > > >
> > > > > > > Hey yukon,
> > > > > > > I submitted my draft on the summer of code homepage a couple of
> > > days
> > > > > ago,
> > > > > > > also attaching the link here for reference,
> > > > > > >
> > > > > > > https://docs.google.com/file/d/1nXktUO_TF9-
> > > > rSHSnGj5z5QZoHzMhosxm/edit?
> > > > > > > usp=docslist_api&filetype=msword
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Sowmya
> > > > > > >
> > > > > > > On Thu, Mar 15, 2018 at 2:08 AM yukon 
> wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > Of course, we can work together to finetune your design
> draft.
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > yukon
> > > > > > > >
> > > > > > > > On Thu, Mar 15, 2018 at 5:33 AM, sowmya s <
> > sowmya9...@gmail.com>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hello, yukon and Von,
> > > > > > > > >
> > > > > > > > > I've shared my GSOC - 18 

Re: [GSOC|ROCKETMQ-122] Support Global Ordered Messaging

[sowmya s]

Hello Yukon,
I've submitted my final proposal addressing your comments as well. Hoping
to work on this project for the summer. In the meantime i'm working on
prototyping the global ordering approach i talked about in my proposal.

thanks,
Sowmya

On Fri, Mar 23, 2018 at 4:48 AM, yukon  wrote:

> Hi,
>
> I added some comments, but it seems your google doc link isn't shared:
>
> ```
> To resolve this issue, we need a global ordered messaging mechanism to have
> a good scalability without hot-pot issue.
>
> A practical proposal is that sending messages to distributed
> queue/partition with an ordered key. Messages are ordered in the same
> queue, then implement an OrderConsumer to pull messages from these queues
> then perform merge sort, finally deliver these ordered messages to users.
> ```
> As mentioned in ROCKETMQ-122, we should address the hot-pot issue.
>
> That means the messages can be sent to multiple distributed
> queues/partitions and still can be consumed in order manner.
>
> A simple explanation of hot-pot issue:
>
> Assume that all the generated messages in a hot shop of taobao/amazon
> should be consumed in order, in the current state, these massive messages
> should be sent to a fixed queue/partition, surely to a fixed server which
> may cause high pressure in this server. You can easily infer that we can't
> resolve this issue through extending the scale of broker cluster.
>
> Could your arch resolve this hot-pot issue?
>
> Regards,
> yukon
>
>
> On Thu, Mar 22, 2018 at 2:15 PM, sowmya s  wrote:
>
> > It's going great.
> > In the DefaultMessageStore, there is a class
> > CommitLogDispatcherBuildConsumeQueue, I think that a variant of that
> class
> > or another implementation which are merged from the commitlog,
> > Since the DispachRequest has a store timestamp, the queue can be
> populated
> > based on that time stamp.
> >
> > Also, I have some more questions,
> > Does the fleet of broker slaves all store the various commit logs that
> are
> > received by them or is the commit log store, replication managed by the
> > broker controller?
> >
> > The other direction I was thinking about is adding/modifying the
> > PullAPIWrapper, since the DefaultMQPushConsumerImpl uses the
> > DefaultMQPullConsumerImpl and that uses the PullAPIWrapper, maybe adding
> > this change there will solve the merge issue for the consumers.
> >
> > Do you think I am on the right track?
> >
> > thanks,
> > Sowmya
> >
> >
> >
> > On Mon, Mar 19, 2018 at 8:15 PM, yukon  wrote:
> >
> > > Hi,
> > >
> > > Sorry for the late reply.
> > >
> > > As for:
> > >
> > > ```
> > > I'm looking at the BrokerController and MessageStore implementation and
> > > hooks to understand where the merge logic will best fit.
> > > ```
> > >
> > > how is it going?
> > >
> > > Regards
> > >
> > > On Sat, Mar 17, 2018 at 11:30 AM, sowmya s 
> wrote:
> > >
> > > > Thank you yukon,
> > > >
> > > > I'm done with my coursework for this semester and have more time now
> to
> > > > improve my proposal.
> > > > I'm looking at the BrokerController and MessageStore implementation
> and
> > > > hooks to understand where the merge logic will best fit. So far I've
> > > looked
> > > > at the codebase from a Producer and Consumer perspective and looked
> at
> > > the
> > > > DefaultMQProducerImpl and  DefaultMQPushConsumerImpl for
> understanding
> > > the
> > > > link between how Producers send and Consumers receive messages.
> > > >
> > > > thanks,
> > > > Sowmya
> > > >
> > > > On Fri, Mar 16, 2018 at 8:07 PM, yukon  wrote:
> > > >
> > > > > Cool, let's focus on it and see whether is there anything can be
> > > > polished.
> > > > >
> > > > > Regards
> > > > >
> > > > > On Fri, Mar 16, 2018 at 11:54 PM, sowmya s 
> > > wrote:
> > > > >
> > > > > > Hey yukon,
> > > > > > I submitted my draft on the summer of code homepage a couple of
> > days
> > > > ago,
> > > > > > also attaching the link here for reference,
> > > > > >
> > > > > > https://docs.google.com/file/d/1nXktUO_TF9-
> > > rSHSnGj5z5QZoHzMhosxm/edit?
> > > > > > usp=docslist_api&filetype=msword
> > > > > >
> > > > > > Thanks,
> > > > > > Sowmya
> > > > > >
> > > > > > On Thu, Mar 15, 2018 at 2:08 AM yukon  wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > Of course, we can work together to finetune your design draft.
> > > > > > >
> > > > > > > Regards,
> > > > > > > yukon
> > > > > > >
> > > > > > > On Thu, Mar 15, 2018 at 5:33 AM, sowmya s <
> sowmya9...@gmail.com>
> > > > > wrote:
> > > > > > >
> > > > > > > > Hello, yukon and Von,
> > > > > > > >
> > > > > > > > I've shared my GSOC - 18 draft of the project. I'm looking
> > > forward
> > > > to
> > > > > > > > working with all of you to finetune the proposal. I will be
> > > > > allocating
> > > > > > 20
> > > > > > > > hours per week from now to the proposal acceptance phase to
> > > address
> > > > > > > > questions and dive deep into any suggestions that you
> provide.
> > > > > > > > I am really looking forward to work on t

Re: [GSOC|ROCKETMQ-122] Support Global Ordered Messaging

[sowmya s]

Hi Yukon,

I am sorry for the confusion regarding the google doc, here is the link
again, https://drive.google.com/file/d/1nXktUO_TF9-
rSHSnGj5z5QZoHzMhosxm/view?usp=sharing

Regarding the hotpot issue, In my document, under the section Optimizing
for performance, I had a similar approach to what you mentioned, If the
Producer or the implementation of a Global Ordered Producer, attaches to
the Message or message properties, an additional attribute which is an
atomic counter that gets incremented for every message sent, in combination
with the topic and the unique producer group, the messages get to
potentially different brokers under different queues.

>From the consumer perspective, having a custom RebalanceImplementation and
also a custom Implementation for a MQConsumer which can be a Pull or push
consumer that orders based on the property we set above in the Global
Ordered Producer and sort by producer group, topic all the messages.

thanks,
Sowmya

On Fri, Mar 23, 2018 at 4:48 AM, yukon  wrote:

> Hi,
>
> I added some comments, but it seems your google doc link isn't shared:
>
> ```
> To resolve this issue, we need a global ordered messaging mechanism to have
> a good scalability without hot-pot issue.
>
> A practical proposal is that sending messages to distributed
> queue/partition with an ordered key. Messages are ordered in the same
> queue, then implement an OrderConsumer to pull messages from these queues
> then perform merge sort, finally deliver these ordered messages to users.
> ```
> As mentioned in ROCKETMQ-122, we should address the hot-pot issue.
>
> That means the messages can be sent to multiple distributed
> queues/partitions and still can be consumed in order manner.
>
> A simple explanation of hot-pot issue:
>
> Assume that all the generated messages in a hot shop of taobao/amazon
> should be consumed in order, in the current state, these massive messages
> should be sent to a fixed queue/partition, surely to a fixed server which
> may cause high pressure in this server. You can easily infer that we can't
> resolve this issue through extending the scale of broker cluster.
>
> Could your arch resolve this hot-pot issue?
>
> Regards,
> yukon
>
>
> On Thu, Mar 22, 2018 at 2:15 PM, sowmya s  wrote:
>
> > It's going great.
> > In the DefaultMessageStore, there is a class
> > CommitLogDispatcherBuildConsumeQueue, I think that a variant of that
> class
> > or another implementation which are merged from the commitlog,
> > Since the DispachRequest has a store timestamp, the queue can be
> populated
> > based on that time stamp.
> >
> > Also, I have some more questions,
> > Does the fleet of broker slaves all store the various commit logs that
> are
> > received by them or is the commit log store, replication managed by the
> > broker controller?
> >
> > The other direction I was thinking about is adding/modifying the
> > PullAPIWrapper, since the DefaultMQPushConsumerImpl uses the
> > DefaultMQPullConsumerImpl and that uses the PullAPIWrapper, maybe adding
> > this change there will solve the merge issue for the consumers.
> >
> > Do you think I am on the right track?
> >
> > thanks,
> > Sowmya
> >
> >
> >
> > On Mon, Mar 19, 2018 at 8:15 PM, yukon  wrote:
> >
> > > Hi,
> > >
> > > Sorry for the late reply.
> > >
> > > As for:
> > >
> > > ```
> > > I'm looking at the BrokerController and MessageStore implementation and
> > > hooks to understand where the merge logic will best fit.
> > > ```
> > >
> > > how is it going?
> > >
> > > Regards
> > >
> > > On Sat, Mar 17, 2018 at 11:30 AM, sowmya s 
> wrote:
> > >
> > > > Thank you yukon,
> > > >
> > > > I'm done with my coursework for this semester and have more time now
> to
> > > > improve my proposal.
> > > > I'm looking at the BrokerController and MessageStore implementation
> and
> > > > hooks to understand where the merge logic will best fit. So far I've
> > > looked
> > > > at the codebase from a Producer and Consumer perspective and looked
> at
> > > the
> > > > DefaultMQProducerImpl and  DefaultMQPushConsumerImpl for
> understanding
> > > the
> > > > link between how Producers send and Consumers receive messages.
> > > >
> > > > thanks,
> > > > Sowmya
> > > >
> > > > On Fri, Mar 16, 2018 at 8:07 PM, yukon  wrote:
> > > >
> > > > > Cool, let's focus on it and see whether is there anything can be
> > > > polished.
> > > > >
> > > > > Regards
> > > > >
> > > > > On Fri, Mar 16, 2018 at 11:54 PM, sowmya s 
> > > wrote:
> > > > >
> > > > > > Hey yukon,
> > > > > > I submitted my draft on the summer of code homepage a couple of
> > days
> > > > ago,
> > > > > > also attaching the link here for reference,
> > > > > >
> > > > > > https://docs.google.com/file/d/1nXktUO_TF9-
> > > rSHSnGj5z5QZoHzMhosxm/edit?
> > > > > > usp=docslist_api&filetype=msword
> > > > > >
> > > > > > Thanks,
> > > > > > Sowmya
> > > > > >
> > > > > > On Thu, Mar 15, 2018 at 2:08 AM yukon  wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > Of course, we can wo

Re: [GSOC|ROCKETMQ-122] Support Global Ordered Messaging

[sowmya s]

Hi yukon,

I'm working on updating the design for the hot-pot issue.

Thanks for the direction, from a consumer perspective, when the context is
chosen as orderly, there is a specific Rebalance interface that orders the
messages by Topic. this is used by the  DefaultMQPushConsumerImpl to send
messages to the consumer. The RebalanceImpl is the one that actually
fetches the messages by topic from brokers and organizes by topic.
Along with a How about having a custom RebalanceImpl that can be injected
into the Consumer, so that the pull/push can be created based on the global
order rebalance strategy.
In the rebalanceImpl class, there is an implementation to do rebalance
where messages from all brokers are fetched and ordered.

thanks,
Sowmya

On Fri, Mar 23, 2018 at 5:00 AM, yukon  wrote:

> Hi,
>
> ```
> Does the fleet of broker slaves all store the various commit logs that are
> received by them or is the commit log store, replication managed by the
> broker controller?
> ```
> Yes, the slave broker will all the commit logs synced from the master
> broker.
>
> ```
> The other direction I was thinking about is adding/modifying the
> PullAPIWrapper, since the DefaultMQPushConsumerImpl uses the
> DefaultMQPullConsumerImpl and that uses the PullAPIWrapper, maybe adding
> this change there will solve the merge issue for the consumers.
> ```
>
> How about adding another consumer type?
>
> On Fri, Mar 23, 2018 at 7:48 PM, yukon  wrote:
>
> > Hi,
> >
> > I added some comments, but it seems your google doc link isn't shared:
> >
> > ```
> > To resolve this issue, we need a global ordered messaging mechanism to
> > have a good scalability without hot-pot issue.
> >
> > A practical proposal is that sending messages to distributed
> > queue/partition with an ordered key. Messages are ordered in the same
> > queue, then implement an OrderConsumer to pull messages from these queues
> > then perform merge sort, finally deliver these ordered messages to users.
> > ```
> > As mentioned in ROCKETMQ-122, we should address the hot-pot issue.
> >
> > That means the messages can be sent to multiple distributed
> > queues/partitions and still can be consumed in order manner.
> >
> > A simple explanation of hot-pot issue:
> >
> > Assume that all the generated messages in a hot shop of taobao/amazon
> > should be consumed in order, in the current state, these massive messages
> > should be sent to a fixed queue/partition, surely to a fixed server which
> > may cause high pressure in this server. You can easily infer that we
> can't
> > resolve this issue through extending the scale of broker cluster.
> >
> > Could your arch resolve this hot-pot issue?
> >
> > Regards,
> > yukon
> >
> >
> > On Thu, Mar 22, 2018 at 2:15 PM, sowmya s  wrote:
> >
> >> It's going great.
> >> In the DefaultMessageStore, there is a class
> >> CommitLogDispatcherBuildConsumeQueue, I think that a variant of that
> >> class
> >> or another implementation which are merged from the commitlog,
> >> Since the DispachRequest has a store timestamp, the queue can be
> populated
> >> based on that time stamp.
> >>
> >> Also, I have some more questions,
> >> Does the fleet of broker slaves all store the various commit logs that
> are
> >> received by them or is the commit log store, replication managed by the
> >> broker controller?
> >>
> >> The other direction I was thinking about is adding/modifying the
> >> PullAPIWrapper, since the DefaultMQPushConsumerImpl uses the
> >> DefaultMQPullConsumerImpl and that uses the PullAPIWrapper, maybe adding
> >> this change there will solve the merge issue for the consumers.
> >>
> >> Do you think I am on the right track?
> >>
> >> thanks,
> >> Sowmya
> >>
> >>
> >>
> >> On Mon, Mar 19, 2018 at 8:15 PM, yukon  wrote:
> >>
> >> > Hi,
> >> >
> >> > Sorry for the late reply.
> >> >
> >> > As for:
> >> >
> >> > ```
> >> > I'm looking at the BrokerController and MessageStore implementation
> and
> >> > hooks to understand where the merge logic will best fit.
> >> > ```
> >> >
> >> > how is it going?
> >> >
> >> > Regards
> >> >
> >> > On Sat, Mar 17, 2018 at 11:30 AM, sowmya s 
> >> wrote:
> >> >
> >> > > Thank you yukon,
> >> > >
> >> > > I'm done with my coursework for this semester and have more time now
> >> to
> >> > > improve my proposal.
> >> > > I'm looking at the BrokerController and MessageStore implementation
> >> and
> >> > > hooks to understand where the merge logic will best fit. So far I've
> >> > looked
> >> > > at the codebase from a Producer and Consumer perspective and looked
> at
> >> > the
> >> > > DefaultMQProducerImpl and  DefaultMQPushConsumerImpl for
> understanding
> >> > the
> >> > > link between how Producers send and Consumers receive messages.
> >> > >
> >> > > thanks,
> >> > > Sowmya
> >> > >
> >> > > On Fri, Mar 16, 2018 at 8:07 PM, yukon  wrote:
> >> > >
> >> > > > Cool, let's focus on it and see whether is there anything can be
> >> > > polished.
> >> > > >
> >> > > > Regards
> >> > > >
> >> > > >

Re: [GSOC|ROCKETMQ-122] Support Global Ordered Messaging

[yukon]

Hi,

```
Does the fleet of broker slaves all store the various commit logs that are
received by them or is the commit log store, replication managed by the
broker controller?
```
Yes, the slave broker will all the commit logs synced from the master
broker.

```
The other direction I was thinking about is adding/modifying the
PullAPIWrapper, since the DefaultMQPushConsumerImpl uses the
DefaultMQPullConsumerImpl and that uses the PullAPIWrapper, maybe adding
this change there will solve the merge issue for the consumers.
```

How about adding another consumer type?

On Fri, Mar 23, 2018 at 7:48 PM, yukon  wrote:

> Hi,
>
> I added some comments, but it seems your google doc link isn't shared:
>
> ```
> To resolve this issue, we need a global ordered messaging mechanism to
> have a good scalability without hot-pot issue.
>
> A practical proposal is that sending messages to distributed
> queue/partition with an ordered key. Messages are ordered in the same
> queue, then implement an OrderConsumer to pull messages from these queues
> then perform merge sort, finally deliver these ordered messages to users.
> ```
> As mentioned in ROCKETMQ-122, we should address the hot-pot issue.
>
> That means the messages can be sent to multiple distributed
> queues/partitions and still can be consumed in order manner.
>
> A simple explanation of hot-pot issue:
>
> Assume that all the generated messages in a hot shop of taobao/amazon
> should be consumed in order, in the current state, these massive messages
> should be sent to a fixed queue/partition, surely to a fixed server which
> may cause high pressure in this server. You can easily infer that we can't
> resolve this issue through extending the scale of broker cluster.
>
> Could your arch resolve this hot-pot issue?
>
> Regards,
> yukon
>
>
> On Thu, Mar 22, 2018 at 2:15 PM, sowmya s  wrote:
>
>> It's going great.
>> In the DefaultMessageStore, there is a class
>> CommitLogDispatcherBuildConsumeQueue, I think that a variant of that
>> class
>> or another implementation which are merged from the commitlog,
>> Since the DispachRequest has a store timestamp, the queue can be populated
>> based on that time stamp.
>>
>> Also, I have some more questions,
>> Does the fleet of broker slaves all store the various commit logs that are
>> received by them or is the commit log store, replication managed by the
>> broker controller?
>>
>> The other direction I was thinking about is adding/modifying the
>> PullAPIWrapper, since the DefaultMQPushConsumerImpl uses the
>> DefaultMQPullConsumerImpl and that uses the PullAPIWrapper, maybe adding
>> this change there will solve the merge issue for the consumers.
>>
>> Do you think I am on the right track?
>>
>> thanks,
>> Sowmya
>>
>>
>>
>> On Mon, Mar 19, 2018 at 8:15 PM, yukon  wrote:
>>
>> > Hi,
>> >
>> > Sorry for the late reply.
>> >
>> > As for:
>> >
>> > ```
>> > I'm looking at the BrokerController and MessageStore implementation and
>> > hooks to understand where the merge logic will best fit.
>> > ```
>> >
>> > how is it going?
>> >
>> > Regards
>> >
>> > On Sat, Mar 17, 2018 at 11:30 AM, sowmya s 
>> wrote:
>> >
>> > > Thank you yukon,
>> > >
>> > > I'm done with my coursework for this semester and have more time now
>> to
>> > > improve my proposal.
>> > > I'm looking at the BrokerController and MessageStore implementation
>> and
>> > > hooks to understand where the merge logic will best fit. So far I've
>> > looked
>> > > at the codebase from a Producer and Consumer perspective and looked at
>> > the
>> > > DefaultMQProducerImpl and  DefaultMQPushConsumerImpl for understanding
>> > the
>> > > link between how Producers send and Consumers receive messages.
>> > >
>> > > thanks,
>> > > Sowmya
>> > >
>> > > On Fri, Mar 16, 2018 at 8:07 PM, yukon  wrote:
>> > >
>> > > > Cool, let's focus on it and see whether is there anything can be
>> > > polished.
>> > > >
>> > > > Regards
>> > > >
>> > > > On Fri, Mar 16, 2018 at 11:54 PM, sowmya s 
>> > wrote:
>> > > >
>> > > > > Hey yukon,
>> > > > > I submitted my draft on the summer of code homepage a couple of
>> days
>> > > ago,
>> > > > > also attaching the link here for reference,
>> > > > >
>> > > > > https://docs.google.com/file/d/1nXktUO_TF9-
>> > rSHSnGj5z5QZoHzMhosxm/edit?
>> > > > > usp=docslist_api&filetype=msword
>> > > > >
>> > > > > Thanks,
>> > > > > Sowmya
>> > > > >
>> > > > > On Thu, Mar 15, 2018 at 2:08 AM yukon  wrote:
>> > > > >
>> > > > > > Hi,
>> > > > > >
>> > > > > > Of course, we can work together to finetune your design draft.
>> > > > > >
>> > > > > > Regards,
>> > > > > > yukon
>> > > > > >
>> > > > > > On Thu, Mar 15, 2018 at 5:33 AM, sowmya s > >
>> > > > wrote:
>> > > > > >
>> > > > > > > Hello, yukon and Von,
>> > > > > > >
>> > > > > > > I've shared my GSOC - 18 draft of the project. I'm looking
>> > forward
>> > > to
>> > > > > > > working with all of you to finetune the proposal. I will be
>> > > > allocating
>> > > > > 20
>> > > > > > > hours per week f

Re: [GSOC|ROCKETMQ-122] Support Global Ordered Messaging

[yukon]

Hi,

I added some comments, but it seems your google doc link isn't shared:

```
To resolve this issue, we need a global ordered messaging mechanism to have
a good scalability without hot-pot issue.

A practical proposal is that sending messages to distributed
queue/partition with an ordered key. Messages are ordered in the same
queue, then implement an OrderConsumer to pull messages from these queues
then perform merge sort, finally deliver these ordered messages to users.
```
As mentioned in ROCKETMQ-122, we should address the hot-pot issue.

That means the messages can be sent to multiple distributed
queues/partitions and still can be consumed in order manner.

A simple explanation of hot-pot issue:

Assume that all the generated messages in a hot shop of taobao/amazon
should be consumed in order, in the current state, these massive messages
should be sent to a fixed queue/partition, surely to a fixed server which
may cause high pressure in this server. You can easily infer that we can't
resolve this issue through extending the scale of broker cluster.

Could your arch resolve this hot-pot issue?

Regards,
yukon


On Thu, Mar 22, 2018 at 2:15 PM, sowmya s  wrote:

> It's going great.
> In the DefaultMessageStore, there is a class
> CommitLogDispatcherBuildConsumeQueue, I think that a variant of that class
> or another implementation which are merged from the commitlog,
> Since the DispachRequest has a store timestamp, the queue can be populated
> based on that time stamp.
>
> Also, I have some more questions,
> Does the fleet of broker slaves all store the various commit logs that are
> received by them or is the commit log store, replication managed by the
> broker controller?
>
> The other direction I was thinking about is adding/modifying the
> PullAPIWrapper, since the DefaultMQPushConsumerImpl uses the
> DefaultMQPullConsumerImpl and that uses the PullAPIWrapper, maybe adding
> this change there will solve the merge issue for the consumers.
>
> Do you think I am on the right track?
>
> thanks,
> Sowmya
>
>
>
> On Mon, Mar 19, 2018 at 8:15 PM, yukon  wrote:
>
> > Hi,
> >
> > Sorry for the late reply.
> >
> > As for:
> >
> > ```
> > I'm looking at the BrokerController and MessageStore implementation and
> > hooks to understand where the merge logic will best fit.
> > ```
> >
> > how is it going?
> >
> > Regards
> >
> > On Sat, Mar 17, 2018 at 11:30 AM, sowmya s  wrote:
> >
> > > Thank you yukon,
> > >
> > > I'm done with my coursework for this semester and have more time now to
> > > improve my proposal.
> > > I'm looking at the BrokerController and MessageStore implementation and
> > > hooks to understand where the merge logic will best fit. So far I've
> > looked
> > > at the codebase from a Producer and Consumer perspective and looked at
> > the
> > > DefaultMQProducerImpl and  DefaultMQPushConsumerImpl for understanding
> > the
> > > link between how Producers send and Consumers receive messages.
> > >
> > > thanks,
> > > Sowmya
> > >
> > > On Fri, Mar 16, 2018 at 8:07 PM, yukon  wrote:
> > >
> > > > Cool, let's focus on it and see whether is there anything can be
> > > polished.
> > > >
> > > > Regards
> > > >
> > > > On Fri, Mar 16, 2018 at 11:54 PM, sowmya s 
> > wrote:
> > > >
> > > > > Hey yukon,
> > > > > I submitted my draft on the summer of code homepage a couple of
> days
> > > ago,
> > > > > also attaching the link here for reference,
> > > > >
> > > > > https://docs.google.com/file/d/1nXktUO_TF9-
> > rSHSnGj5z5QZoHzMhosxm/edit?
> > > > > usp=docslist_api&filetype=msword
> > > > >
> > > > > Thanks,
> > > > > Sowmya
> > > > >
> > > > > On Thu, Mar 15, 2018 at 2:08 AM yukon  wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > Of course, we can work together to finetune your design draft.
> > > > > >
> > > > > > Regards,
> > > > > > yukon
> > > > > >
> > > > > > On Thu, Mar 15, 2018 at 5:33 AM, sowmya s 
> > > > wrote:
> > > > > >
> > > > > > > Hello, yukon and Von,
> > > > > > >
> > > > > > > I've shared my GSOC - 18 draft of the project. I'm looking
> > forward
> > > to
> > > > > > > working with all of you to finetune the proposal. I will be
> > > > allocating
> > > > > 20
> > > > > > > hours per week from now to the proposal acceptance phase to
> > address
> > > > > > > questions and dive deep into any suggestions that you provide.
> > > > > > > I am really looking forward to work on this project.
> > > > > > >
> > > > > > > thanks,
> > > > > > > Sowmya
> > > > > > >
> > > > > > > On Tue, Mar 13, 2018 at 11:47 PM, sowmya s <
> sowmya9...@gmail.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > Hello yukon,
> > > > > > > >
> > > > > > > > Thank you for the inputs. I was able to look at the ~/store
> and
> > > > kind
> > > > > of
> > > > > > > > understand the storage structure. I also looked at the
> > > > > > > > DefaultMQProducerImpl and DefaultMQPullConsumerImpl, used in
> > the
> > > > > > > examples.
> > > > > > > > Now I understand why you proposed a merge sort like app

Re: [GSOC|ROCKETMQ-122] Support Global Ordered Messaging

[sowmya s]

It's going great.
In the DefaultMessageStore, there is a class
CommitLogDispatcherBuildConsumeQueue, I think that a variant of that class
or another implementation which are merged from the commitlog,
Since the DispachRequest has a store timestamp, the queue can be populated
based on that time stamp.

Also, I have some more questions,
Does the fleet of broker slaves all store the various commit logs that are
received by them or is the commit log store, replication managed by the
broker controller?

The other direction I was thinking about is adding/modifying the
PullAPIWrapper, since the DefaultMQPushConsumerImpl uses the
DefaultMQPullConsumerImpl and that uses the PullAPIWrapper, maybe adding
this change there will solve the merge issue for the consumers.

Do you think I am on the right track?

thanks,
Sowmya



On Mon, Mar 19, 2018 at 8:15 PM, yukon  wrote:

> Hi,
>
> Sorry for the late reply.
>
> As for:
>
> ```
> I'm looking at the BrokerController and MessageStore implementation and
> hooks to understand where the merge logic will best fit.
> ```
>
> how is it going?
>
> Regards
>
> On Sat, Mar 17, 2018 at 11:30 AM, sowmya s  wrote:
>
> > Thank you yukon,
> >
> > I'm done with my coursework for this semester and have more time now to
> > improve my proposal.
> > I'm looking at the BrokerController and MessageStore implementation and
> > hooks to understand where the merge logic will best fit. So far I've
> looked
> > at the codebase from a Producer and Consumer perspective and looked at
> the
> > DefaultMQProducerImpl and  DefaultMQPushConsumerImpl for understanding
> the
> > link between how Producers send and Consumers receive messages.
> >
> > thanks,
> > Sowmya
> >
> > On Fri, Mar 16, 2018 at 8:07 PM, yukon  wrote:
> >
> > > Cool, let's focus on it and see whether is there anything can be
> > polished.
> > >
> > > Regards
> > >
> > > On Fri, Mar 16, 2018 at 11:54 PM, sowmya s 
> wrote:
> > >
> > > > Hey yukon,
> > > > I submitted my draft on the summer of code homepage a couple of days
> > ago,
> > > > also attaching the link here for reference,
> > > >
> > > > https://docs.google.com/file/d/1nXktUO_TF9-
> rSHSnGj5z5QZoHzMhosxm/edit?
> > > > usp=docslist_api&filetype=msword
> > > >
> > > > Thanks,
> > > > Sowmya
> > > >
> > > > On Thu, Mar 15, 2018 at 2:08 AM yukon  wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > Of course, we can work together to finetune your design draft.
> > > > >
> > > > > Regards,
> > > > > yukon
> > > > >
> > > > > On Thu, Mar 15, 2018 at 5:33 AM, sowmya s 
> > > wrote:
> > > > >
> > > > > > Hello, yukon and Von,
> > > > > >
> > > > > > I've shared my GSOC - 18 draft of the project. I'm looking
> forward
> > to
> > > > > > working with all of you to finetune the proposal. I will be
> > > allocating
> > > > 20
> > > > > > hours per week from now to the proposal acceptance phase to
> address
> > > > > > questions and dive deep into any suggestions that you provide.
> > > > > > I am really looking forward to work on this project.
> > > > > >
> > > > > > thanks,
> > > > > > Sowmya
> > > > > >
> > > > > > On Tue, Mar 13, 2018 at 11:47 PM, sowmya s  >
> > > > wrote:
> > > > > >
> > > > > > > Hello yukon,
> > > > > > >
> > > > > > > Thank you for the inputs. I was able to look at the ~/store and
> > > kind
> > > > of
> > > > > > > understand the storage structure. I also looked at the
> > > > > > > DefaultMQProducerImpl and DefaultMQPullConsumerImpl, used in
> the
> > > > > > examples.
> > > > > > > Now I understand why you proposed a merge sort like approach
> for
> > > > > > > performing global ordering. Since the proposals are open, I am
> > > > > finalizing
> > > > > > > my draft and will have it up for review very soon.
> > > > > > >
> > > > > > > thanks,
> > > > > > > Sowmya
> > > > > > >
> > > > > > > On Fri, Mar 9, 2018 at 7:03 AM, yukon 
> wrote:
> > > > > > >
> > > > > > >> Hi Sowmya,
> > > > > > >>
> > > > > > >> ```
> > > > > > >> Also, it would be great if you can at a high level, help me
> > > > understand
> > > > > > >> how the messages in the message queues are stored before the
> > > > consumer
> > > > > > reads
> > > > > > >> them.
> > > > > > >> ```
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> As shown in this figure, messages are sent to brokers by
> > producer
> > > > and
> > > > > > >> stored in commit log[1], then messages are dispatched to
> > > > ConsumeQueue
> > > > > by
> > > > > > >> topic, the consumer pulls messages from the queue.
> > > > > > >>
> > > > > > >> I recommend you run a broker and send/consume some messages,
> > then
> > > > > check
> > > > > > >> out the ~/store directory for details.
> > > > > > >>
> > > > > > >> Regards,
> > > > > > >> yukon
> > > > > > >>
> > > > > > >> On Thu, Mar 8, 2018 at 12:09 PM, sowmya s <
> sowmya9...@gmail.com
> > >
> > > > > wrote:
> > > > > > >>
> > > > > > >>> Hello yukon,
> > > > > > >>>
> > > > > > >>> Currently FIFO can be achieved with a producer sending to one
> > > > message
> > > > >

Re: [GSOC|ROCKETMQ-122] Support Global Ordered Messaging

[sowmya s]

Hey yukon, Von,

I'm checking to see if you have had a chance to look at the proposal. It
would be very nice to get some feedback and direction before the deadline
for the final proposals.

thanks,
Sowmya

On Fri, Mar 16, 2018 at 8:07 PM, yukon  wrote:

> Cool, let's focus on it and see whether is there anything can be polished.
>
> Regards
>
> On Fri, Mar 16, 2018 at 11:54 PM, sowmya s  wrote:
>
> > Hey yukon,
> > I submitted my draft on the summer of code homepage a couple of days ago,
> > also attaching the link here for reference,
> >
> > https://docs.google.com/file/d/1nXktUO_TF9-rSHSnGj5z5QZoHzMhosxm/edit?
> > usp=docslist_api&filetype=msword
> >
> > Thanks,
> > Sowmya
> >
> > On Thu, Mar 15, 2018 at 2:08 AM yukon  wrote:
> >
> > > Hi,
> > >
> > > Of course, we can work together to finetune your design draft.
> > >
> > > Regards,
> > > yukon
> > >
> > > On Thu, Mar 15, 2018 at 5:33 AM, sowmya s 
> wrote:
> > >
> > > > Hello, yukon and Von,
> > > >
> > > > I've shared my GSOC - 18 draft of the project. I'm looking forward to
> > > > working with all of you to finetune the proposal. I will be
> allocating
> > 20
> > > > hours per week from now to the proposal acceptance phase to address
> > > > questions and dive deep into any suggestions that you provide.
> > > > I am really looking forward to work on this project.
> > > >
> > > > thanks,
> > > > Sowmya
> > > >
> > > > On Tue, Mar 13, 2018 at 11:47 PM, sowmya s 
> > wrote:
> > > >
> > > > > Hello yukon,
> > > > >
> > > > > Thank you for the inputs. I was able to look at the ~/store and
> kind
> > of
> > > > > understand the storage structure. I also looked at the
> > > > > DefaultMQProducerImpl and DefaultMQPullConsumerImpl, used in the
> > > > examples.
> > > > > Now I understand why you proposed a merge sort like approach for
> > > > > performing global ordering. Since the proposals are open, I am
> > > finalizing
> > > > > my draft and will have it up for review very soon.
> > > > >
> > > > > thanks,
> > > > > Sowmya
> > > > >
> > > > > On Fri, Mar 9, 2018 at 7:03 AM, yukon  wrote:
> > > > >
> > > > >> Hi Sowmya,
> > > > >>
> > > > >> ```
> > > > >> Also, it would be great if you can at a high level, help me
> > understand
> > > > >> how the messages in the message queues are stored before the
> > consumer
> > > > reads
> > > > >> them.
> > > > >> ```
> > > > >>
> > > > >>
> > > > >>
> > > > >> As shown in this figure, messages are sent to brokers by producer
> > and
> > > > >> stored in commit log[1], then messages are dispatched to
> > ConsumeQueue
> > > by
> > > > >> topic, the consumer pulls messages from the queue.
> > > > >>
> > > > >> I recommend you run a broker and send/consume some messages, then
> > > check
> > > > >> out the ~/store directory for details.
> > > > >>
> > > > >> Regards,
> > > > >> yukon
> > > > >>
> > > > >> On Thu, Mar 8, 2018 at 12:09 PM, sowmya s 
> > > wrote:
> > > > >>
> > > > >>> Hello yukon,
> > > > >>>
> > > > >>> Currently FIFO can be achieved with a producer sending to one
> > message
> > > > >>> queue, and when global ordering is required, multiple producers
> > have
> > > to
> > > > >>> send to a single topic queue.
> > > > >>>
> > > > >>> We want to allow multiple producers to send messages on a topic
> to
> > > > >>> multiple message queues and still provide ordering guarantees to
> > the
> > > > >>> consumer, so that all consumers see the same order of data and
> also
> > > the
> > > > >>> data is delivered in an ordered fashion.
> > > > >>>
> > > > >>> 1) Your idea of using a merge sort with the assumption that the
> > first
> > > > >>> arriving message is treated as the first message to be delivered,
> > > > however,
> > > > >>> I want to propose an approach where when the producer sends a
> > message
> > > > to a
> > > > >>> message queue, it must be done in a synchronous fashion and the
> > > > response
> > > > >>> will be that the message is accepted, which means that the
> message
> > > > follows
> > > > >>> the convention that all messages delivered prior by that producer
> > > have
> > > > been
> > > > >>> stored across groups and if not the producer will need to resend
> > the
> > > > >>> message.
> > > > >>>
> > > > >>> We can use a variant of total causal ordering in the layer
> between
> > > the
> > > > >>> message queue and store.
> > > > >>>
> > > > >>> I have been busy with my class project so I couldn't make a lot
> of
> > > > >>> progress in detailing my approach. Also, it would be great if you
> > can
> > > > at a
> > > > >>> high level, help me understand how the messages in the message
> > queues
> > > > are
> > > > >>> stored before the consumer reads them.
> > > > >>>
> > > > >>> Does the consumer read directly from the message queue that the
> > > > producer
> > > > >>> sends data to? does the broker receive the queued producer
> > messages,
> > > > store
> > > > >>> them and then pushes them to another queue for the consumer to
> read
> > > > from?
> > > > >>>
> > > > >>> For refere

Re: [GSOC|ROCKETMQ-122] Support Global Ordered Messaging

[yukon]

Hi,

Sorry for the late reply.

As for:

```
I'm looking at the BrokerController and MessageStore implementation and
hooks to understand where the merge logic will best fit.
```

how is it going?

Regards

On Sat, Mar 17, 2018 at 11:30 AM, sowmya s  wrote:

> Thank you yukon,
>
> I'm done with my coursework for this semester and have more time now to
> improve my proposal.
> I'm looking at the BrokerController and MessageStore implementation and
> hooks to understand where the merge logic will best fit. So far I've looked
> at the codebase from a Producer and Consumer perspective and looked at the
> DefaultMQProducerImpl and  DefaultMQPushConsumerImpl for understanding the
> link between how Producers send and Consumers receive messages.
>
> thanks,
> Sowmya
>
> On Fri, Mar 16, 2018 at 8:07 PM, yukon  wrote:
>
> > Cool, let's focus on it and see whether is there anything can be
> polished.
> >
> > Regards
> >
> > On Fri, Mar 16, 2018 at 11:54 PM, sowmya s  wrote:
> >
> > > Hey yukon,
> > > I submitted my draft on the summer of code homepage a couple of days
> ago,
> > > also attaching the link here for reference,
> > >
> > > https://docs.google.com/file/d/1nXktUO_TF9-rSHSnGj5z5QZoHzMhosxm/edit?
> > > usp=docslist_api&filetype=msword
> > >
> > > Thanks,
> > > Sowmya
> > >
> > > On Thu, Mar 15, 2018 at 2:08 AM yukon  wrote:
> > >
> > > > Hi,
> > > >
> > > > Of course, we can work together to finetune your design draft.
> > > >
> > > > Regards,
> > > > yukon
> > > >
> > > > On Thu, Mar 15, 2018 at 5:33 AM, sowmya s 
> > wrote:
> > > >
> > > > > Hello, yukon and Von,
> > > > >
> > > > > I've shared my GSOC - 18 draft of the project. I'm looking forward
> to
> > > > > working with all of you to finetune the proposal. I will be
> > allocating
> > > 20
> > > > > hours per week from now to the proposal acceptance phase to address
> > > > > questions and dive deep into any suggestions that you provide.
> > > > > I am really looking forward to work on this project.
> > > > >
> > > > > thanks,
> > > > > Sowmya
> > > > >
> > > > > On Tue, Mar 13, 2018 at 11:47 PM, sowmya s 
> > > wrote:
> > > > >
> > > > > > Hello yukon,
> > > > > >
> > > > > > Thank you for the inputs. I was able to look at the ~/store and
> > kind
> > > of
> > > > > > understand the storage structure. I also looked at the
> > > > > > DefaultMQProducerImpl and DefaultMQPullConsumerImpl, used in the
> > > > > examples.
> > > > > > Now I understand why you proposed a merge sort like approach for
> > > > > > performing global ordering. Since the proposals are open, I am
> > > > finalizing
> > > > > > my draft and will have it up for review very soon.
> > > > > >
> > > > > > thanks,
> > > > > > Sowmya
> > > > > >
> > > > > > On Fri, Mar 9, 2018 at 7:03 AM, yukon  wrote:
> > > > > >
> > > > > >> Hi Sowmya,
> > > > > >>
> > > > > >> ```
> > > > > >> Also, it would be great if you can at a high level, help me
> > > understand
> > > > > >> how the messages in the message queues are stored before the
> > > consumer
> > > > > reads
> > > > > >> them.
> > > > > >> ```
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> As shown in this figure, messages are sent to brokers by
> producer
> > > and
> > > > > >> stored in commit log[1], then messages are dispatched to
> > > ConsumeQueue
> > > > by
> > > > > >> topic, the consumer pulls messages from the queue.
> > > > > >>
> > > > > >> I recommend you run a broker and send/consume some messages,
> then
> > > > check
> > > > > >> out the ~/store directory for details.
> > > > > >>
> > > > > >> Regards,
> > > > > >> yukon
> > > > > >>
> > > > > >> On Thu, Mar 8, 2018 at 12:09 PM, sowmya s  >
> > > > wrote:
> > > > > >>
> > > > > >>> Hello yukon,
> > > > > >>>
> > > > > >>> Currently FIFO can be achieved with a producer sending to one
> > > message
> > > > > >>> queue, and when global ordering is required, multiple producers
> > > have
> > > > to
> > > > > >>> send to a single topic queue.
> > > > > >>>
> > > > > >>> We want to allow multiple producers to send messages on a topic
> > to
> > > > > >>> multiple message queues and still provide ordering guarantees
> to
> > > the
> > > > > >>> consumer, so that all consumers see the same order of data and
> > also
> > > > the
> > > > > >>> data is delivered in an ordered fashion.
> > > > > >>>
> > > > > >>> 1) Your idea of using a merge sort with the assumption that the
> > > first
> > > > > >>> arriving message is treated as the first message to be
> delivered,
> > > > > however,
> > > > > >>> I want to propose an approach where when the producer sends a
> > > message
> > > > > to a
> > > > > >>> message queue, it must be done in a synchronous fashion and the
> > > > > response
> > > > > >>> will be that the message is accepted, which means that the
> > message
> > > > > follows
> > > > > >>> the convention that all messages delivered prior by that
> producer
> > > > have
> > > > > been
> > > > > >>> stored across groups and if not the producer will need to
> resend
> > 

Re: [GSOC|ROCKETMQ-122] Support Global Ordered Messaging

[sowmya s]

Thank you yukon,

I'm done with my coursework for this semester and have more time now to
improve my proposal.
I'm looking at the BrokerController and MessageStore implementation and
hooks to understand where the merge logic will best fit. So far I've looked
at the codebase from a Producer and Consumer perspective and looked at the
DefaultMQProducerImpl and  DefaultMQPushConsumerImpl for understanding the
link between how Producers send and Consumers receive messages.

thanks,
Sowmya

On Fri, Mar 16, 2018 at 8:07 PM, yukon  wrote:

> Cool, let's focus on it and see whether is there anything can be polished.
>
> Regards
>
> On Fri, Mar 16, 2018 at 11:54 PM, sowmya s  wrote:
>
> > Hey yukon,
> > I submitted my draft on the summer of code homepage a couple of days ago,
> > also attaching the link here for reference,
> >
> > https://docs.google.com/file/d/1nXktUO_TF9-rSHSnGj5z5QZoHzMhosxm/edit?
> > usp=docslist_api&filetype=msword
> >
> > Thanks,
> > Sowmya
> >
> > On Thu, Mar 15, 2018 at 2:08 AM yukon  wrote:
> >
> > > Hi,
> > >
> > > Of course, we can work together to finetune your design draft.
> > >
> > > Regards,
> > > yukon
> > >
> > > On Thu, Mar 15, 2018 at 5:33 AM, sowmya s 
> wrote:
> > >
> > > > Hello, yukon and Von,
> > > >
> > > > I've shared my GSOC - 18 draft of the project. I'm looking forward to
> > > > working with all of you to finetune the proposal. I will be
> allocating
> > 20
> > > > hours per week from now to the proposal acceptance phase to address
> > > > questions and dive deep into any suggestions that you provide.
> > > > I am really looking forward to work on this project.
> > > >
> > > > thanks,
> > > > Sowmya
> > > >
> > > > On Tue, Mar 13, 2018 at 11:47 PM, sowmya s 
> > wrote:
> > > >
> > > > > Hello yukon,
> > > > >
> > > > > Thank you for the inputs. I was able to look at the ~/store and
> kind
> > of
> > > > > understand the storage structure. I also looked at the
> > > > > DefaultMQProducerImpl and DefaultMQPullConsumerImpl, used in the
> > > > examples.
> > > > > Now I understand why you proposed a merge sort like approach for
> > > > > performing global ordering. Since the proposals are open, I am
> > > finalizing
> > > > > my draft and will have it up for review very soon.
> > > > >
> > > > > thanks,
> > > > > Sowmya
> > > > >
> > > > > On Fri, Mar 9, 2018 at 7:03 AM, yukon  wrote:
> > > > >
> > > > >> Hi Sowmya,
> > > > >>
> > > > >> ```
> > > > >> Also, it would be great if you can at a high level, help me
> > understand
> > > > >> how the messages in the message queues are stored before the
> > consumer
> > > > reads
> > > > >> them.
> > > > >> ```
> > > > >>
> > > > >>
> > > > >>
> > > > >> As shown in this figure, messages are sent to brokers by producer
> > and
> > > > >> stored in commit log[1], then messages are dispatched to
> > ConsumeQueue
> > > by
> > > > >> topic, the consumer pulls messages from the queue.
> > > > >>
> > > > >> I recommend you run a broker and send/consume some messages, then
> > > check
> > > > >> out the ~/store directory for details.
> > > > >>
> > > > >> Regards,
> > > > >> yukon
> > > > >>
> > > > >> On Thu, Mar 8, 2018 at 12:09 PM, sowmya s 
> > > wrote:
> > > > >>
> > > > >>> Hello yukon,
> > > > >>>
> > > > >>> Currently FIFO can be achieved with a producer sending to one
> > message
> > > > >>> queue, and when global ordering is required, multiple producers
> > have
> > > to
> > > > >>> send to a single topic queue.
> > > > >>>
> > > > >>> We want to allow multiple producers to send messages on a topic
> to
> > > > >>> multiple message queues and still provide ordering guarantees to
> > the
> > > > >>> consumer, so that all consumers see the same order of data and
> also
> > > the
> > > > >>> data is delivered in an ordered fashion.
> > > > >>>
> > > > >>> 1) Your idea of using a merge sort with the assumption that the
> > first
> > > > >>> arriving message is treated as the first message to be delivered,
> > > > however,
> > > > >>> I want to propose an approach where when the producer sends a
> > message
> > > > to a
> > > > >>> message queue, it must be done in a synchronous fashion and the
> > > > response
> > > > >>> will be that the message is accepted, which means that the
> message
> > > > follows
> > > > >>> the convention that all messages delivered prior by that producer
> > > have
> > > > been
> > > > >>> stored across groups and if not the producer will need to resend
> > the
> > > > >>> message.
> > > > >>>
> > > > >>> We can use a variant of total causal ordering in the layer
> between
> > > the
> > > > >>> message queue and store.
> > > > >>>
> > > > >>> I have been busy with my class project so I couldn't make a lot
> of
> > > > >>> progress in detailing my approach. Also, it would be great if you
> > can
> > > > at a
> > > > >>> high level, help me understand how the messages in the message
> > queues
> > > > are
> > > > >>> stored before the consumer reads them.
> > > > >>>
> > > > >>> Does the consumer read dir

Re: [GSOC|ROCKETMQ-122] Support Global Ordered Messaging

[yukon]

Cool, let's focus on it and see whether is there anything can be polished.

Regards

On Fri, Mar 16, 2018 at 11:54 PM, sowmya s  wrote:

> Hey yukon,
> I submitted my draft on the summer of code homepage a couple of days ago,
> also attaching the link here for reference,
>
> https://docs.google.com/file/d/1nXktUO_TF9-rSHSnGj5z5QZoHzMhosxm/edit?
> usp=docslist_api&filetype=msword
>
> Thanks,
> Sowmya
>
> On Thu, Mar 15, 2018 at 2:08 AM yukon  wrote:
>
> > Hi,
> >
> > Of course, we can work together to finetune your design draft.
> >
> > Regards,
> > yukon
> >
> > On Thu, Mar 15, 2018 at 5:33 AM, sowmya s  wrote:
> >
> > > Hello, yukon and Von,
> > >
> > > I've shared my GSOC - 18 draft of the project. I'm looking forward to
> > > working with all of you to finetune the proposal. I will be allocating
> 20
> > > hours per week from now to the proposal acceptance phase to address
> > > questions and dive deep into any suggestions that you provide.
> > > I am really looking forward to work on this project.
> > >
> > > thanks,
> > > Sowmya
> > >
> > > On Tue, Mar 13, 2018 at 11:47 PM, sowmya s 
> wrote:
> > >
> > > > Hello yukon,
> > > >
> > > > Thank you for the inputs. I was able to look at the ~/store and kind
> of
> > > > understand the storage structure. I also looked at the
> > > > DefaultMQProducerImpl and DefaultMQPullConsumerImpl, used in the
> > > examples.
> > > > Now I understand why you proposed a merge sort like approach for
> > > > performing global ordering. Since the proposals are open, I am
> > finalizing
> > > > my draft and will have it up for review very soon.
> > > >
> > > > thanks,
> > > > Sowmya
> > > >
> > > > On Fri, Mar 9, 2018 at 7:03 AM, yukon  wrote:
> > > >
> > > >> Hi Sowmya,
> > > >>
> > > >> ```
> > > >> Also, it would be great if you can at a high level, help me
> understand
> > > >> how the messages in the message queues are stored before the
> consumer
> > > reads
> > > >> them.
> > > >> ```
> > > >>
> > > >>
> > > >>
> > > >> As shown in this figure, messages are sent to brokers by producer
> and
> > > >> stored in commit log[1], then messages are dispatched to
> ConsumeQueue
> > by
> > > >> topic, the consumer pulls messages from the queue.
> > > >>
> > > >> I recommend you run a broker and send/consume some messages, then
> > check
> > > >> out the ~/store directory for details.
> > > >>
> > > >> Regards,
> > > >> yukon
> > > >>
> > > >> On Thu, Mar 8, 2018 at 12:09 PM, sowmya s 
> > wrote:
> > > >>
> > > >>> Hello yukon,
> > > >>>
> > > >>> Currently FIFO can be achieved with a producer sending to one
> message
> > > >>> queue, and when global ordering is required, multiple producers
> have
> > to
> > > >>> send to a single topic queue.
> > > >>>
> > > >>> We want to allow multiple producers to send messages on a topic to
> > > >>> multiple message queues and still provide ordering guarantees to
> the
> > > >>> consumer, so that all consumers see the same order of data and also
> > the
> > > >>> data is delivered in an ordered fashion.
> > > >>>
> > > >>> 1) Your idea of using a merge sort with the assumption that the
> first
> > > >>> arriving message is treated as the first message to be delivered,
> > > however,
> > > >>> I want to propose an approach where when the producer sends a
> message
> > > to a
> > > >>> message queue, it must be done in a synchronous fashion and the
> > > response
> > > >>> will be that the message is accepted, which means that the message
> > > follows
> > > >>> the convention that all messages delivered prior by that producer
> > have
> > > been
> > > >>> stored across groups and if not the producer will need to resend
> the
> > > >>> message.
> > > >>>
> > > >>> We can use a variant of total causal ordering in the layer between
> > the
> > > >>> message queue and store.
> > > >>>
> > > >>> I have been busy with my class project so I couldn't make a lot of
> > > >>> progress in detailing my approach. Also, it would be great if you
> can
> > > at a
> > > >>> high level, help me understand how the messages in the message
> queues
> > > are
> > > >>> stored before the consumer reads them.
> > > >>>
> > > >>> Does the consumer read directly from the message queue that the
> > > producer
> > > >>> sends data to? does the broker receive the queued producer
> messages,
> > > store
> > > >>> them and then pushes them to another queue for the consumer to read
> > > from?
> > > >>>
> > > >>> For reference on total causal ordering:
> > https://www.cl.cam.ac.uk/teach
> > > >>> ing/0910/ConcDistS/10b-ProcGp-order.pdf
> > > >>>
> > > >>> thanks,
> > > >>> Sowmya
> > > >>>
> > > >>> On Mon, Mar 5, 2018 at 4:26 AM, yukon  wrote:
> > > >>>
> > >  Hi Sowmya,
> > > 
> > >  Sorry for the late reply, do you have any update on this project?
> > > 
> > >  In RocketMQ, one message queue is a FIFO queue natively, so I
> > proposed
> > >  a simple solution that performs merge sort on multiple queues to
> > > improve
> > >  performance and

Re: [GSOC|ROCKETMQ-122] Support Global Ordered Messaging

[sowmya s]

Hey yukon,
I submitted my draft on the summer of code homepage a couple of days ago,
also attaching the link here for reference,

https://docs.google.com/file/d/1nXktUO_TF9-rSHSnGj5z5QZoHzMhosxm/edit?usp=docslist_api&filetype=msword

Thanks,
Sowmya

On Thu, Mar 15, 2018 at 2:08 AM yukon  wrote:

> Hi,
>
> Of course, we can work together to finetune your design draft.
>
> Regards,
> yukon
>
> On Thu, Mar 15, 2018 at 5:33 AM, sowmya s  wrote:
>
> > Hello, yukon and Von,
> >
> > I've shared my GSOC - 18 draft of the project. I'm looking forward to
> > working with all of you to finetune the proposal. I will be allocating 20
> > hours per week from now to the proposal acceptance phase to address
> > questions and dive deep into any suggestions that you provide.
> > I am really looking forward to work on this project.
> >
> > thanks,
> > Sowmya
> >
> > On Tue, Mar 13, 2018 at 11:47 PM, sowmya s  wrote:
> >
> > > Hello yukon,
> > >
> > > Thank you for the inputs. I was able to look at the ~/store and kind of
> > > understand the storage structure. I also looked at the
> > > DefaultMQProducerImpl and DefaultMQPullConsumerImpl, used in the
> > examples.
> > > Now I understand why you proposed a merge sort like approach for
> > > performing global ordering. Since the proposals are open, I am
> finalizing
> > > my draft and will have it up for review very soon.
> > >
> > > thanks,
> > > Sowmya
> > >
> > > On Fri, Mar 9, 2018 at 7:03 AM, yukon  wrote:
> > >
> > >> Hi Sowmya,
> > >>
> > >> ```
> > >> Also, it would be great if you can at a high level, help me understand
> > >> how the messages in the message queues are stored before the consumer
> > reads
> > >> them.
> > >> ```
> > >>
> > >>
> > >>
> > >> As shown in this figure, messages are sent to brokers by producer and
> > >> stored in commit log[1], then messages are dispatched to ConsumeQueue
> by
> > >> topic, the consumer pulls messages from the queue.
> > >>
> > >> I recommend you run a broker and send/consume some messages, then
> check
> > >> out the ~/store directory for details.
> > >>
> > >> Regards,
> > >> yukon
> > >>
> > >> On Thu, Mar 8, 2018 at 12:09 PM, sowmya s 
> wrote:
> > >>
> > >>> Hello yukon,
> > >>>
> > >>> Currently FIFO can be achieved with a producer sending to one message
> > >>> queue, and when global ordering is required, multiple producers have
> to
> > >>> send to a single topic queue.
> > >>>
> > >>> We want to allow multiple producers to send messages on a topic to
> > >>> multiple message queues and still provide ordering guarantees to the
> > >>> consumer, so that all consumers see the same order of data and also
> the
> > >>> data is delivered in an ordered fashion.
> > >>>
> > >>> 1) Your idea of using a merge sort with the assumption that the first
> > >>> arriving message is treated as the first message to be delivered,
> > however,
> > >>> I want to propose an approach where when the producer sends a message
> > to a
> > >>> message queue, it must be done in a synchronous fashion and the
> > response
> > >>> will be that the message is accepted, which means that the message
> > follows
> > >>> the convention that all messages delivered prior by that producer
> have
> > been
> > >>> stored across groups and if not the producer will need to resend the
> > >>> message.
> > >>>
> > >>> We can use a variant of total causal ordering in the layer between
> the
> > >>> message queue and store.
> > >>>
> > >>> I have been busy with my class project so I couldn't make a lot of
> > >>> progress in detailing my approach. Also, it would be great if you can
> > at a
> > >>> high level, help me understand how the messages in the message queues
> > are
> > >>> stored before the consumer reads them.
> > >>>
> > >>> Does the consumer read directly from the message queue that the
> > producer
> > >>> sends data to? does the broker receive the queued producer messages,
> > store
> > >>> them and then pushes them to another queue for the consumer to read
> > from?
> > >>>
> > >>> For reference on total causal ordering:
> https://www.cl.cam.ac.uk/teach
> > >>> ing/0910/ConcDistS/10b-ProcGp-order.pdf
> > >>>
> > >>> thanks,
> > >>> Sowmya
> > >>>
> > >>> On Mon, Mar 5, 2018 at 4:26 AM, yukon  wrote:
> > >>>
> >  Hi Sowmya,
> > 
> >  Sorry for the late reply, do you have any update on this project?
> > 
> >  In RocketMQ, one message queue is a FIFO queue natively, so I
> proposed
> >  a simple solution that performs merge sort on multiple queues to
> > improve
> >  performance and scalability.
> > 
> >  While the order issue across producers is difficult, we could assume
> >  that the message first arrives the broker should be consumed
> firstly.
> > 
> >  But it would be wonderful if you have a real design about the order
> >  issue across producers based on the RocketMQ design and the storage
> >  structure.
> > 
> >  Looking forward your design ~
> > 
> >  Regards,
> > >>

Re: [GSOC|ROCKETMQ-122] Support Global Ordered Messaging

[yukon]

Hi,

Of course, we can work together to finetune your design draft.

Regards,
yukon

On Thu, Mar 15, 2018 at 5:33 AM, sowmya s  wrote:

> Hello, yukon and Von,
>
> I've shared my GSOC - 18 draft of the project. I'm looking forward to
> working with all of you to finetune the proposal. I will be allocating 20
> hours per week from now to the proposal acceptance phase to address
> questions and dive deep into any suggestions that you provide.
> I am really looking forward to work on this project.
>
> thanks,
> Sowmya
>
> On Tue, Mar 13, 2018 at 11:47 PM, sowmya s  wrote:
>
> > Hello yukon,
> >
> > Thank you for the inputs. I was able to look at the ~/store and kind of
> > understand the storage structure. I also looked at the
> > DefaultMQProducerImpl and DefaultMQPullConsumerImpl, used in the
> examples.
> > Now I understand why you proposed a merge sort like approach for
> > performing global ordering. Since the proposals are open, I am finalizing
> > my draft and will have it up for review very soon.
> >
> > thanks,
> > Sowmya
> >
> > On Fri, Mar 9, 2018 at 7:03 AM, yukon  wrote:
> >
> >> Hi Sowmya,
> >>
> >> ```
> >> Also, it would be great if you can at a high level, help me understand
> >> how the messages in the message queues are stored before the consumer
> reads
> >> them.
> >> ```
> >>
> >>
> >>
> >> As shown in this figure, messages are sent to brokers by producer and
> >> stored in commit log[1], then messages are dispatched to ConsumeQueue by
> >> topic, the consumer pulls messages from the queue.
> >>
> >> I recommend you run a broker and send/consume some messages, then check
> >> out the ~/store directory for details.
> >>
> >> Regards,
> >> yukon
> >>
> >> On Thu, Mar 8, 2018 at 12:09 PM, sowmya s  wrote:
> >>
> >>> Hello yukon,
> >>>
> >>> Currently FIFO can be achieved with a producer sending to one message
> >>> queue, and when global ordering is required, multiple producers have to
> >>> send to a single topic queue.
> >>>
> >>> We want to allow multiple producers to send messages on a topic to
> >>> multiple message queues and still provide ordering guarantees to the
> >>> consumer, so that all consumers see the same order of data and also the
> >>> data is delivered in an ordered fashion.
> >>>
> >>> 1) Your idea of using a merge sort with the assumption that the first
> >>> arriving message is treated as the first message to be delivered,
> however,
> >>> I want to propose an approach where when the producer sends a message
> to a
> >>> message queue, it must be done in a synchronous fashion and the
> response
> >>> will be that the message is accepted, which means that the message
> follows
> >>> the convention that all messages delivered prior by that producer have
> been
> >>> stored across groups and if not the producer will need to resend the
> >>> message.
> >>>
> >>> We can use a variant of total causal ordering in the layer between the
> >>> message queue and store.
> >>>
> >>> I have been busy with my class project so I couldn't make a lot of
> >>> progress in detailing my approach. Also, it would be great if you can
> at a
> >>> high level, help me understand how the messages in the message queues
> are
> >>> stored before the consumer reads them.
> >>>
> >>> Does the consumer read directly from the message queue that the
> producer
> >>> sends data to? does the broker receive the queued producer messages,
> store
> >>> them and then pushes them to another queue for the consumer to read
> from?
> >>>
> >>> For reference on total causal ordering: https://www.cl.cam.ac.uk/teach
> >>> ing/0910/ConcDistS/10b-ProcGp-order.pdf
> >>>
> >>> thanks,
> >>> Sowmya
> >>>
> >>> On Mon, Mar 5, 2018 at 4:26 AM, yukon  wrote:
> >>>
>  Hi Sowmya,
> 
>  Sorry for the late reply, do you have any update on this project?
> 
>  In RocketMQ, one message queue is a FIFO queue natively, so I proposed
>  a simple solution that performs merge sort on multiple queues to
> improve
>  performance and scalability.
> 
>  While the order issue across producers is difficult, we could assume
>  that the message first arrives the broker should be consumed firstly.
> 
>  But it would be wonderful if you have a real design about the order
>  issue across producers based on the RocketMQ design and the storage
>  structure.
> 
>  Looking forward your design ~
> 
>  Regards,
>  yukon
> 
>  On Fri, Mar 2, 2018 at 2:50 AM, sowmya s 
> wrote:
> 
> > Hello all,
> >
> > Adding a few more thoughts on the problem of establishing an order of
> > messages across producers.
> >
> > Consider the Scenario
> >
> > Producer-1 produces messages a1, b1 and c1 into a queue Queue1
> > Producer-2 produces messages a2, b2 and c2 into queue Queue2.
> >
> > If we assume that time(a1) < time(b1) < time(c1) and similarly
> > time(a2) < time(b2) < time(c2)
> >
> > Are the following orders acceptab

Re: [GSOC|ROCKETMQ-122] Support Global Ordered Messaging

[sowmya s]

Hello, yukon and Von,

I've shared my GSOC - 18 draft of the project. I'm looking forward to
working with all of you to finetune the proposal. I will be allocating 20
hours per week from now to the proposal acceptance phase to address
questions and dive deep into any suggestions that you provide.
I am really looking forward to work on this project.

thanks,
Sowmya

On Tue, Mar 13, 2018 at 11:47 PM, sowmya s  wrote:

> Hello yukon,
>
> Thank you for the inputs. I was able to look at the ~/store and kind of
> understand the storage structure. I also looked at the
> DefaultMQProducerImpl and DefaultMQPullConsumerImpl, used in the examples.
> Now I understand why you proposed a merge sort like approach for
> performing global ordering. Since the proposals are open, I am finalizing
> my draft and will have it up for review very soon.
>
> thanks,
> Sowmya
>
> On Fri, Mar 9, 2018 at 7:03 AM, yukon  wrote:
>
>> Hi Sowmya,
>>
>> ```
>> Also, it would be great if you can at a high level, help me understand
>> how the messages in the message queues are stored before the consumer reads
>> them.
>> ```
>>
>>
>>
>> As shown in this figure, messages are sent to brokers by producer and
>> stored in commit log[1], then messages are dispatched to ConsumeQueue by
>> topic, the consumer pulls messages from the queue.
>>
>> I recommend you run a broker and send/consume some messages, then check
>> out the ~/store directory for details.
>>
>> Regards,
>> yukon
>>
>> On Thu, Mar 8, 2018 at 12:09 PM, sowmya s  wrote:
>>
>>> Hello yukon,
>>>
>>> Currently FIFO can be achieved with a producer sending to one message
>>> queue, and when global ordering is required, multiple producers have to
>>> send to a single topic queue.
>>>
>>> We want to allow multiple producers to send messages on a topic to
>>> multiple message queues and still provide ordering guarantees to the
>>> consumer, so that all consumers see the same order of data and also the
>>> data is delivered in an ordered fashion.
>>>
>>> 1) Your idea of using a merge sort with the assumption that the first
>>> arriving message is treated as the first message to be delivered, however,
>>> I want to propose an approach where when the producer sends a message to a
>>> message queue, it must be done in a synchronous fashion and the response
>>> will be that the message is accepted, which means that the message follows
>>> the convention that all messages delivered prior by that producer have been
>>> stored across groups and if not the producer will need to resend the
>>> message.
>>>
>>> We can use a variant of total causal ordering in the layer between the
>>> message queue and store.
>>>
>>> I have been busy with my class project so I couldn't make a lot of
>>> progress in detailing my approach. Also, it would be great if you can at a
>>> high level, help me understand how the messages in the message queues are
>>> stored before the consumer reads them.
>>>
>>> Does the consumer read directly from the message queue that the producer
>>> sends data to? does the broker receive the queued producer messages, store
>>> them and then pushes them to another queue for the consumer to read from?
>>>
>>> For reference on total causal ordering: https://www.cl.cam.ac.uk/teach
>>> ing/0910/ConcDistS/10b-ProcGp-order.pdf
>>>
>>> thanks,
>>> Sowmya
>>>
>>> On Mon, Mar 5, 2018 at 4:26 AM, yukon  wrote:
>>>
 Hi Sowmya,

 Sorry for the late reply, do you have any update on this project?

 In RocketMQ, one message queue is a FIFO queue natively, so I proposed
 a simple solution that performs merge sort on multiple queues to improve
 performance and scalability.

 While the order issue across producers is difficult, we could assume
 that the message first arrives the broker should be consumed firstly.

 But it would be wonderful if you have a real design about the order
 issue across producers based on the RocketMQ design and the storage
 structure.

 Looking forward your design ~

 Regards,
 yukon

 On Fri, Mar 2, 2018 at 2:50 AM, sowmya s  wrote:

> Hello all,
>
> Adding a few more thoughts on the problem of establishing an order of
> messages across producers.
>
> Consider the Scenario
>
> Producer-1 produces messages a1, b1 and c1 into a queue Queue1
> Producer-2 produces messages a2, b2 and c2 into queue Queue2.
>
> If we assume that time(a1) < time(b1) < time(c1) and similarly
> time(a2) < time(b2) < time(c2)
>
> Are the following orders acceptable to the consumer?
>
> > a1, a2, b1, b2, c1, c2
> > a1, b1, c1, a2, b2, c2
> > a1 b1, b2, a2, a3, b3
>
> and an order displayed at one consumer is consistent across all
> consumers.
>
> This can be achieved using Total Causal Ordering at the Producer or
> Queue level, using Leslie Lamport's clock and synchronization approach.
>
> For reference is the paper at

Re: [GSOC|ROCKETMQ-122] Support Global Ordered Messaging

[sowmya s]

Hello yukon,

Thank you for the inputs. I was able to look at the ~/store and kind of
understand the storage structure. I also looked at the
DefaultMQProducerImpl and DefaultMQPullConsumerImpl, used in the examples.
Now I understand why you proposed a merge sort like approach for performing
global ordering. Since the proposals are open, I am finalizing my draft and
will have it up for review very soon.

thanks,
Sowmya

On Fri, Mar 9, 2018 at 7:03 AM, yukon  wrote:

> Hi Sowmya,
>
> ```
> Also, it would be great if you can at a high level, help me understand how
> the messages in the message queues are stored before the consumer reads
> them.
> ```
>
>
>
> As shown in this figure, messages are sent to brokers by producer and
> stored in commit log[1], then messages are dispatched to ConsumeQueue by
> topic, the consumer pulls messages from the queue.
>
> I recommend you run a broker and send/consume some messages, then check
> out the ~/store directory for details.
>
> Regards,
> yukon
>
> On Thu, Mar 8, 2018 at 12:09 PM, sowmya s  wrote:
>
>> Hello yukon,
>>
>> Currently FIFO can be achieved with a producer sending to one message
>> queue, and when global ordering is required, multiple producers have to
>> send to a single topic queue.
>>
>> We want to allow multiple producers to send messages on a topic to
>> multiple message queues and still provide ordering guarantees to the
>> consumer, so that all consumers see the same order of data and also the
>> data is delivered in an ordered fashion.
>>
>> 1) Your idea of using a merge sort with the assumption that the first
>> arriving message is treated as the first message to be delivered, however,
>> I want to propose an approach where when the producer sends a message to a
>> message queue, it must be done in a synchronous fashion and the response
>> will be that the message is accepted, which means that the message follows
>> the convention that all messages delivered prior by that producer have been
>> stored across groups and if not the producer will need to resend the
>> message.
>>
>> We can use a variant of total causal ordering in the layer between the
>> message queue and store.
>>
>> I have been busy with my class project so I couldn't make a lot of
>> progress in detailing my approach. Also, it would be great if you can at a
>> high level, help me understand how the messages in the message queues are
>> stored before the consumer reads them.
>>
>> Does the consumer read directly from the message queue that the producer
>> sends data to? does the broker receive the queued producer messages, store
>> them and then pushes them to another queue for the consumer to read from?
>>
>> For reference on total causal ordering: https://www.cl.cam.ac.uk/teach
>> ing/0910/ConcDistS/10b-ProcGp-order.pdf
>>
>> thanks,
>> Sowmya
>>
>> On Mon, Mar 5, 2018 at 4:26 AM, yukon  wrote:
>>
>>> Hi Sowmya,
>>>
>>> Sorry for the late reply, do you have any update on this project?
>>>
>>> In RocketMQ, one message queue is a FIFO queue natively, so I proposed a
>>> simple solution that performs merge sort on multiple queues to improve
>>> performance and scalability.
>>>
>>> While the order issue across producers is difficult, we could assume
>>> that the message first arrives the broker should be consumed firstly.
>>>
>>> But it would be wonderful if you have a real design about the order
>>> issue across producers based on the RocketMQ design and the storage
>>> structure.
>>>
>>> Looking forward your design ~
>>>
>>> Regards,
>>> yukon
>>>
>>> On Fri, Mar 2, 2018 at 2:50 AM, sowmya s  wrote:
>>>
 Hello all,

 Adding a few more thoughts on the problem of establishing an order of
 messages across producers.

 Consider the Scenario

 Producer-1 produces messages a1, b1 and c1 into a queue Queue1
 Producer-2 produces messages a2, b2 and c2 into queue Queue2.

 If we assume that time(a1) < time(b1) < time(c1) and similarly time(a2)
 < time(b2) < time(c2)

 Are the following orders acceptable to the consumer?

 > a1, a2, b1, b2, c1, c2
 > a1, b1, c1, a2, b2, c2
 > a1 b1, b2, a2, a3, b3

 and an order displayed at one consumer is consistent across all
 consumers.

 This can be achieved using Total Causal Ordering at the Producer or
 Queue level, using Leslie Lamport's clock and synchronization approach.

 For reference is the paper attached, http://lamport.azurewebsites.n
 et/pubs/time-clocks.pdf


 --
 Regards,

 Sowmya

 On Sun, Feb 25, 2018 at 7:37 PM, sowmya s  wrote:

> Hello,
>
> I'm trying to work on the issue ROCKETMQ-122
>  as
> a part of Google Summer of Code 2018. I've been spending some time to
> understand the system, architecture and the existing Messaging Patterns.
> I still have a few questions and would like to clarify my assum

Re: [GSOC|ROCKETMQ-122] Support Global Ordered Messaging

[yukon]

Hi Sowmya,

```
Also, it would be great if you can at a high level, help me understand how
the messages in the message queues are stored before the consumer reads
them.
```



As shown in this figure, messages are sent to brokers by producer and
stored in commit log[1], then messages are dispatched to ConsumeQueue by
topic, the consumer pulls messages from the queue.

I recommend you run a broker and send/consume some messages, then check out
the ~/store directory for details.

Regards,
yukon

On Thu, Mar 8, 2018 at 12:09 PM, sowmya s  wrote:

> Hello yukon,
>
> Currently FIFO can be achieved with a producer sending to one message
> queue, and when global ordering is required, multiple producers have to
> send to a single topic queue.
>
> We want to allow multiple producers to send messages on a topic to
> multiple message queues and still provide ordering guarantees to the
> consumer, so that all consumers see the same order of data and also the
> data is delivered in an ordered fashion.
>
> 1) Your idea of using a merge sort with the assumption that the first
> arriving message is treated as the first message to be delivered, however,
> I want to propose an approach where when the producer sends a message to a
> message queue, it must be done in a synchronous fashion and the response
> will be that the message is accepted, which means that the message follows
> the convention that all messages delivered prior by that producer have been
> stored across groups and if not the producer will need to resend the
> message.
>
> We can use a variant of total causal ordering in the layer between the
> message queue and store.
>
> I have been busy with my class project so I couldn't make a lot of
> progress in detailing my approach. Also, it would be great if you can at a
> high level, help me understand how the messages in the message queues are
> stored before the consumer reads them.
>
> Does the consumer read directly from the message queue that the producer
> sends data to? does the broker receive the queued producer messages, store
> them and then pushes them to another queue for the consumer to read from?
>
> For reference on total causal ordering: https://www.cl.cam.ac.uk/
> teaching/0910/ConcDistS/10b-ProcGp-order.pdf
>
> thanks,
> Sowmya
>
> On Mon, Mar 5, 2018 at 4:26 AM, yukon  wrote:
>
>> Hi Sowmya,
>>
>> Sorry for the late reply, do you have any update on this project?
>>
>> In RocketMQ, one message queue is a FIFO queue natively, so I proposed a
>> simple solution that performs merge sort on multiple queues to improve
>> performance and scalability.
>>
>> While the order issue across producers is difficult, we could assume that
>> the message first arrives the broker should be consumed firstly.
>>
>> But it would be wonderful if you have a real design about the order
>> issue across producers based on the RocketMQ design and the storage
>> structure.
>>
>> Looking forward your design ~
>>
>> Regards,
>> yukon
>>
>> On Fri, Mar 2, 2018 at 2:50 AM, sowmya s  wrote:
>>
>>> Hello all,
>>>
>>> Adding a few more thoughts on the problem of establishing an order of
>>> messages across producers.
>>>
>>> Consider the Scenario
>>>
>>> Producer-1 produces messages a1, b1 and c1 into a queue Queue1
>>> Producer-2 produces messages a2, b2 and c2 into queue Queue2.
>>>
>>> If we assume that time(a1) < time(b1) < time(c1) and similarly time(a2)
>>> < time(b2) < time(c2)
>>>
>>> Are the following orders acceptable to the consumer?
>>>
>>> > a1, a2, b1, b2, c1, c2
>>> > a1, b1, c1, a2, b2, c2
>>> > a1 b1, b2, a2, a3, b3
>>>
>>> and an order displayed at one consumer is consistent across all
>>> consumers.
>>>
>>> This can be achieved using Total Causal Ordering at the Producer or
>>> Queue level, using Leslie Lamport's clock and synchronization approach.
>>>
>>> For reference is the paper attached, http://lamport.azurewebsites.n
>>> et/pubs/time-clocks.pdf
>>>
>>>
>>> --
>>> Regards,
>>>
>>> Sowmya
>>>
>>> On Sun, Feb 25, 2018 at 7:37 PM, sowmya s  wrote:
>>>
 Hello,

 I'm trying to work on the issue ROCKETMQ-122
  as
 a part of Google Summer of Code 2018. I've been spending some time to
 understand the system, architecture and the existing Messaging Patterns.
 I still have a few questions and would like to clarify my assumptions.

- Is the current FIFO order example limited to one message queue
per producer? Can the producer send the same message to multiple queues?
Will the consumers of the queues be able to read messages in Order?
- Can I assume that each producer will send messages to one queue?
- Global Order is to be identified across all GlobalOrderedProducer
(a new producer that is to be used for global order) instances that are
running.
- I think using a global clock can help establish the order between
2 or more producers, however using s

Re: [GSOC|ROCKETMQ-122] Support Global Ordered Messaging

[sowmya s]

Hello yukon,

Currently FIFO can be achieved with a producer sending to one message
queue, and when global ordering is required, multiple producers have to
send to a single topic queue.

We want to allow multiple producers to send messages on a topic to multiple
message queues and still provide ordering guarantees to the consumer, so
that all consumers see the same order of data and also the data is
delivered in an ordered fashion.

1) Your idea of using a merge sort with the assumption that the first
arriving message is treated as the first message to be delivered, however,
I want to propose an approach where when the producer sends a message to a
message queue, it must be done in a synchronous fashion and the response
will be that the message is accepted, which means that the message follows
the convention that all messages delivered prior by that producer have been
stored across groups and if not the producer will need to resend the
message.

We can use a variant of total causal ordering in the layer between the
message queue and store.

I have been busy with my class project so I couldn't make a lot of progress
in detailing my approach. Also, it would be great if you can at a high
level, help me understand how the messages in the message queues are stored
before the consumer reads them.

Does the consumer read directly from the message queue that the producer
sends data to? does the broker receive the queued producer messages, store
them and then pushes them to another queue for the consumer to read from?

For reference on total causal ordering:
https://www.cl.cam.ac.uk/teaching/0910/ConcDistS/10b-ProcGp-order.pdf

thanks,
Sowmya

On Mon, Mar 5, 2018 at 4:26 AM, yukon  wrote:

> Hi Sowmya,
>
> Sorry for the late reply, do you have any update on this project?
>
> In RocketMQ, one message queue is a FIFO queue natively, so I proposed a
> simple solution that performs merge sort on multiple queues to improve
> performance and scalability.
>
> While the order issue across producers is difficult, we could assume that
> the message first arrives the broker should be consumed firstly.
>
> But it would be wonderful if you have a real design about the order issue
> across producers based on the RocketMQ design and the storage structure.
>
> Looking forward your design ~
>
> Regards,
> yukon
>
> On Fri, Mar 2, 2018 at 2:50 AM, sowmya s  wrote:
>
>> Hello all,
>>
>> Adding a few more thoughts on the problem of establishing an order of
>> messages across producers.
>>
>> Consider the Scenario
>>
>> Producer-1 produces messages a1, b1 and c1 into a queue Queue1
>> Producer-2 produces messages a2, b2 and c2 into queue Queue2.
>>
>> If we assume that time(a1) < time(b1) < time(c1) and similarly time(a2) <
>> time(b2) < time(c2)
>>
>> Are the following orders acceptable to the consumer?
>>
>> > a1, a2, b1, b2, c1, c2
>> > a1, b1, c1, a2, b2, c2
>> > a1 b1, b2, a2, a3, b3
>>
>> and an order displayed at one consumer is consistent across all
>> consumers.
>>
>> This can be achieved using Total Causal Ordering at the Producer or Queue
>> level, using Leslie Lamport's clock and synchronization approach.
>>
>> For reference is the paper attached, http://lamport.azurewebsites.n
>> et/pubs/time-clocks.pdf
>>
>>
>> --
>> Regards,
>>
>> Sowmya
>>
>> On Sun, Feb 25, 2018 at 7:37 PM, sowmya s  wrote:
>>
>>> Hello,
>>>
>>> I'm trying to work on the issue ROCKETMQ-122
>>>  as
>>> a part of Google Summer of Code 2018. I've been spending some time to
>>> understand the system, architecture and the existing Messaging Patterns.
>>> I still have a few questions and would like to clarify my assumptions.
>>>
>>>- Is the current FIFO order example limited to one message queue per
>>>producer? Can the producer send the same message to multiple queues? Will
>>>the consumers of the queues be able to read messages in Order?
>>>- Can I assume that each producer will send messages to one queue?
>>>- Global Order is to be identified across all GlobalOrderedProducer
>>>(a new producer that is to be used for global order) instances that are
>>>running.
>>>- I think using a global clock can help establish the order between
>>>2 or more producers, however using some form of vector clock might
>>>also help identify the global order of messages between the producers.
>>>- A GlobalOrderedConsumer ( consumer that knows how to read globally
>>>ordered messages) can then compare messages across all message queues 
>>> from
>>>the corresponding producers and extract the messages. [ this is the
>>>approach recommended by yukon on the issue page ]
>>>- We can also potentially have another layer in the Message Queue
>>>which accumulates all messages sent from producers and provides one 
>>> ordered
>>>message queue for consumers to read from.
>>>
>>> Thank you for your patience and please let me know if my understanding
>>> o

Re: [GSOC|ROCKETMQ-122] Support Global Ordered Messaging

[yukon]

Hi Sowmya,

Sorry for the late reply, do you have any update on this project?

In RocketMQ, one message queue is a FIFO queue natively, so I proposed a
simple solution that performs merge sort on multiple queues to improve
performance and scalability.

While the order issue across producers is difficult, we could assume that
the message first arrives the broker should be consumed firstly.

But it would be wonderful if you have a real design about the order issue
across producers based on the RocketMQ design and the storage structure.

Looking forward your design ~

Regards,
yukon

On Fri, Mar 2, 2018 at 2:50 AM, sowmya s  wrote:

> Hello all,
>
> Adding a few more thoughts on the problem of establishing an order of
> messages across producers.
>
> Consider the Scenario
>
> Producer-1 produces messages a1, b1 and c1 into a queue Queue1
> Producer-2 produces messages a2, b2 and c2 into queue Queue2.
>
> If we assume that time(a1) < time(b1) < time(c1) and similarly time(a2) <
> time(b2) < time(c2)
>
> Are the following orders acceptable to the consumer?
>
> > a1, a2, b1, b2, c1, c2
> > a1, b1, c1, a2, b2, c2
> > a1 b1, b2, a2, a3, b3
>
> and an order displayed at one consumer is consistent across all consumers.
>
> This can be achieved using Total Causal Ordering at the Producer or Queue
> level, using Leslie Lamport's clock and synchronization approach.
>
> For reference is the paper attached, http://lamport.azurewebsites.
> net/pubs/time-clocks.pdf
>
>
> --
> Regards,
>
> Sowmya
>
> On Sun, Feb 25, 2018 at 7:37 PM, sowmya s  wrote:
>
>> Hello,
>>
>> I'm trying to work on the issue ROCKETMQ-122
>>  as
>> a part of Google Summer of Code 2018. I've been spending some time to
>> understand the system, architecture and the existing Messaging Patterns.
>> I still have a few questions and would like to clarify my assumptions.
>>
>>- Is the current FIFO order example limited to one message queue per
>>producer? Can the producer send the same message to multiple queues? Will
>>the consumers of the queues be able to read messages in Order?
>>- Can I assume that each producer will send messages to one queue?
>>- Global Order is to be identified across all GlobalOrderedProducer
>>(a new producer that is to be used for global order) instances that are
>>running.
>>- I think using a global clock can help establish the order between 2
>>or more producers, however using some form of vector clock might also
>>help identify the global order of messages between the producers.
>>- A GlobalOrderedConsumer ( consumer that knows how to read globally
>>ordered messages) can then compare messages across all message queues from
>>the corresponding producers and extract the messages. [ this is the
>>approach recommended by yukon on the issue page ]
>>- We can also potentially have another layer in the Message Queue
>>which accumulates all messages sent from producers and provides one 
>> ordered
>>message queue for consumers to read from.
>>
>> Thank you for your patience and please let me know if my understanding of
>> the problem and the assumptions are right.
>>
>> Best Regards,
>> Sowmya
>>
>>
>
>
> --
> Regards,
>
> Sowmya
>
>

Re: [GSOC|ROCKETMQ-122] Support Global Ordered Messaging

[sowmya s]

Hello all,

Adding a few more thoughts on the problem of establishing an order of
messages across producers.

Consider the Scenario

Producer-1 produces messages a1, b1 and c1 into a queue Queue1
Producer-2 produces messages a2, b2 and c2 into queue Queue2.

If we assume that time(a1) < time(b1) < time(c1) and similarly time(a2) <
time(b2) < time(c2)

Are the following orders acceptable to the consumer?

> a1, a2, b1, b2, c1, c2
> a1, b1, c1, a2, b2, c2
> a1 b1, b2, a2, a3, b3

and an order displayed at one consumer is consistent across all consumers.

This can be achieved using Total Causal Ordering at the Producer or Queue
level, using Leslie Lamport's clock and synchronization approach.

For reference is the paper attached,
http://lamport.azurewebsites.net/pubs/time-clocks.pdf


-- 
Regards,

Sowmya

On Sun, Feb 25, 2018 at 7:37 PM, sowmya s  wrote:

> Hello,
>
> I'm trying to work on the issue ROCKETMQ-122
>  as a
> part of Google Summer of Code 2018. I've been spending some time to
> understand the system, architecture and the existing Messaging Patterns.
> I still have a few questions and would like to clarify my assumptions.
>
>- Is the current FIFO order example limited to one message queue per
>producer? Can the producer send the same message to multiple queues? Will
>the consumers of the queues be able to read messages in Order?
>- Can I assume that each producer will send messages to one queue?
>- Global Order is to be identified across all GlobalOrderedProducer (a
>new producer that is to be used for global order) instances that are
>running.
>- I think using a global clock can help establish the order between 2
>or more producers, however using some form of vector clock might also
>help identify the global order of messages between the producers.
>- A GlobalOrderedConsumer ( consumer that knows how to read globally
>ordered messages) can then compare messages across all message queues from
>the corresponding producers and extract the messages. [ this is the
>approach recommended by yukon on the issue page ]
>- We can also potentially have another layer in the Message Queue
>which accumulates all messages sent from producers and provides one ordered
>message queue for consumers to read from.
>
> Thank you for your patience and please let me know if my understanding of
> the problem and the assumptions are right.
>
> Best Regards,
> Sowmya
>
>


-- 
Regards,

Sowmya