Thanks Penghui & Sijie for the comments, updated the wiki link. 1. we should add the new ordering_key, it is as Penghui mentioned, There may features that all enabled, but want to use key for different purposes. 2. regarding consumer listener in "failover", it is originally for failover/exclusive mode, which only let one consumer alive. In Key_Failover, it is more like shared mode, all alive consumer should be able to consume messages.
Best Regards. Jia Zhai Beijing, China Mobile: +86 15810491983 On Fri, Apr 5, 2019 at 1:04 AM Sijie Guo <guosi...@gmail.com> wrote: > Penguin, > > Thank you for your clarification! That makes sense to me then. > > Sijie > > On Thu, Apr 4, 2019 at 2:21 AM PengHui Li <codelipeng...@gmail.com> wrote: > > > @Sijie Guo <si...@apache.org> > > > > Sorry, I may not have described my point in the previous email. > > I try to describe it again. :) > > > > User can use key to achieve business logic(e.g. use user_id as message > key) > > At the same time, want to use another identifier as the basis for the > > order. > > > > As the message key has many uses, avoid duplicate messages, compaction > > topics. > > I prefer provide a mechanism to overwrite key based sorting behavior. > > Of course this is not mandatory. > > > > If the basis for sorting is exactly the same as the real purpose of the > > message key, > > just use the message key. Otherwise user provide the order key to > > overwrite. > > > > Best Regards. > > Penghui > > > > Sijie Guo <guosi...@gmail.com> 于2019年4月4日周四 下午3:36写道: > > > >> Jia thank you for raising this up. > >> > >> A couple of comments here > >> > >> 1. technically you don't need a consistent hashing mechanism for > dividing > >> hash ranges for consumers. Any hash mechanism that can map a key to a > >> consumer should work here. The only advantage of consistent hashing > >> mechanism is to reduce the disruptions when a consumer joins and leaves. > >> So > >> I would prefer making the hashing mechanism pluggable. > >> > >> 2. regarding using key or introducing a new ordering key. I would prefer > >> using `key` here. having two `keys` actually makes people confused. The > >> `key` was used for compaction but the compaction doesn't mutate the > >> original data. it just creates a new compacted ledger. in most of the > >> time, > >> when people use `key_failover`, they are probably accessing the original > >> data. Event they are trying to subscribe to compacted topic using > >> `key_failover`, the ordering of messages by key will still be true. so I > >> wouldn't worry about this part that much. > >> > >> 3. in `failover` there is a consumer listener that listens on the > >> partition > >> assignment changes. Is there any requirements or plans on adding similar > >> listeners for `key_failover`? > >> > >> Thanks, > >> Sijie > >> > >> On Wed, Apr 3, 2019 at 2:09 AM Jia Zhai <zhai...@apache.org> wrote: > >> > >> > Hi all, > >> > > >> > I would like to init a discuss of "PIP 34: Add new subscribe type -- > >> > Key_Failover" here. > >> > > >> > At present, there are 3 kinds of subscription modes: exclusive, > shared, > >> and > >> > failover. Shared mode subscription is used a lot, because consumers > >> could > >> > effectively parallel consume messages of same partition. But using > >> shared > >> > mode will not keep the message order of same key, It would be good to > >> > leverage share mode, while also keeping the ordering of messages. > >> > > >> > This Proposal trying to introduce a new subscribe type Key_Failover to > >> > extend shared type. By Key_Failover, one partition could have several > >> > consumers to parallel consume messages, while all messages with the > same > >> > key will be dispatched to only one consumer. > >> > > >> > Here is the link contains more information of this PIP: > >> > > >> > > >> > https://github.com/apache/pulsar/wiki/PIP-34%3A-Add-new-subscribe-type-Key_Failover > >> > > >> > Thanks. > >> > > >> > > >