Pop is not a appropriate terminology, I would suggest a more general term. such as lpull(lite pull)? The previous pull model will be retained. About this, I'd like to hear more voices from the community.
ayanamist <[email protected]> 于2021年1月20日周三 下午5:09写道： > I will supplement a work flow later. > Internally, it adds "POP"+"ACK" requests to broker+client interaction which > "POP" fetches messages and "ACK" acknowledges these messages' offsets, then > all flow can be moved from client to broker side. There will be some > changes in push consumer so that it keeps client usage as same as before, > and broker controls whether this consumer group use POP or PULL > > On Wed, Jan 20, 2021 at 3:32 PM tiger lee <[email protected]> wrote: > > > [ ] +1 approve > > it would make Consumer Client more simple to use. and do u have a work > > flow to show how POP mode work ? > > > > heng du <[email protected]> 于2021年1月18日周一 下午2:37写道： > > > > > Hi RocketMQ Community, > > > > > > This is the vote for the kickoff of RIP-19 RocketMQ Pop Consuming. > > > > > > In order to better implement a lightweight client, @ayanamist proposes > a > > > new consumption model, and at the same time transfers the load > balancing > > > logic of the original client to the broker, which not only solves the > > > original queue occupancy problem but also It can also avoid the > > consumption > > > delay caused by a certain consumer hangs. > > > > > > The vote will be open for at least 72 hours or until a necessary number > > of > > > votes are reached. > > > > > > Please vote accordingly: > > > > > > [ ] +1 approve > > > [ ] +0 no opinion > > > [ ] -1 disapprove with the reason > > > > > > > > > > > > > > > Best Regards! > > > Henry > > > > > > ayanamist <[email protected]> 于2021年1月8日周五 上午11:25写道： > > > > > > > # RIP-19 RocketMQ Pop Consuming > > > > > > > > # Status > > > > > > > > - Current State: Proposed > > > > - Authors: [ayanamist]([ > > > > https://github.com/ayanamist/](https://github.com/ayanamist/)) > > > > - Shepherds: [duhengforever]([ > > > > https://github.com/duhenglucky/](https://github.com/duhengforever/)) > > > > - Mailing List discussion: [email protected]; > > > > [email protected] > > > > - Pull Request: RIP-19 > > > > - Released: - > > > > > > > > # Background & Motivation > > > > > > > > ### What do we need to do > > > > > > > > - Will we add a new module? > > > > > > > > No. > > > > > > > > - Will we add new APIs? > > > > > > > > Yes. > > > > > > > > - Will we add new feature? > > > > > > > > Yes. > > > > > > > > ### Why should we do that > > > > > > > > - Are there any problems of our current project? > > > > > > > > The current subscription load balancing strategy is based on the > > > > dimension of message queue. All behaviors are owned by the client > side. > > > > There are three main steps: > > > > > > > > 1. Each consumer regularly obtains the total number of topic > > message > > > > queues and all consumers. > > > > 2. Using a general algorithm to sort the queues by consumer ip > and > > > > queue index to calculate which message queue is allocated to which > > > > consumer. > > > > 3. Each consumer pulls messages using allocated orders described > > > above. > > > > > > > > According to this allocation method, if an abnormality occurs in > a > > > > consumer (the application itself is abnormal, or a broker is > upgrading) > > > so > > > > that it causes slow subscription, messages will be accumulated, but > > this > > > > queue will not be re-allocated to another consumer, so the > accumulation > > > > will become more and more serious. > > > > > > > > > > > > Chinese version: > > > > > > > > 当前的消费负载均衡策略是以队列的维度来进行，所有行为全部是由客户端主动来完成，主要分为三步： > > > > > > > > 1. 每个consumer定时去获取消费的topic的队列总数，以及consumer总数 > > > > 2. 将队列按编号、consumer按ip排序，用统一的分配算法计算该consumer分配哪些消费队列 > > > > 3. 每个consumer去根据算法分配出来的队列，拉取消息消费 > > > > > > > > > > > > > > > > > > > > > > 按照这个分配方式，如果有一个队列有异常（应用自身异常，或某个broker在升级）导致消费较慢或者停止，该队列会出现堆积现象，因为队列不会被分配给其他机器，因此如果长时间不处理，队列的堆积会越来越严重。 > > > > > > > > - What can we benefit proposed changes? > > > > > > > > The accumulated messages will be subscribed by other consumers if > > one > > > > consumer behaves abnormally. > > > > > > > > Chinese version: > > > > > > > > 在某个队列消费异常的情况下，可以快速的由其它消费者接手进行消费，缓解堆积状态。 > > > > > > > > # Goals > > > > > > > > - What problem is this proposal designed to solve? > > > > > > > > The accumulated messages will be subscribed by other consumers if > > one > > > > consumer behaves abnormally. > > > > > > > > Chinese version: > > > > > > > > 在某个队列消费异常的情况下，可以快速的由其它消费者接手进行消费，缓解堆积状态。 > > > > > > > > - To what degree should we solve the problem? > > > > > > > > This RIP must guarantee below point: > > > > > > > > 1. High availablity: Subscription of one message queue will not > be > > > > affected by single consumer failure. > > > > 2. High performance: This implementation affects latency and > > > throughput > > > > less than 10%. > > > > > > > > > > > > Chinese version: > > > > > > > > 新方案需要保证两点： > > > > > > > > 1. 高可用：单一队列的消费能力不受某个消费客户端异常的影响 > > > > 2. 高性能：POP订阅对消息消费的延迟和吞吐的影响在10%以内 > > > > > > > > # Non-Goals > > > > > > > > - What problem is this proposal NOT designed to solve? > > > > > > > > Improve client-side load balancing. > > > > > > > > - Are there any limits of this proposal? > > > > > > > > Nothing specific. > > > > > > > > # Changes > > > > > > > > ## Architecture > > > > > > > > Current "Pull mode": > > > >  > > > > > > > > Proposed "Pop mode": > > > >  > > > > > > > > Move inter-queue balance of one topic from client side to server > side. > > > > Clients make pull request without specified queues to broker, and > > broker > > > > fetch messages from queues internally and returns, which ensures one > > > queue > > > > will be consumed by multiple clients. The whole behavior is like a > > queue > > > > pop process. > > > > > > > > It will add a new request command querying queue assignments in > broker, > > > and > > > > add pop-feature-support flag to pull request which makes broker use > pop > > > > mode. > > > > > > > > ## Interface Design/Change > > > > > > > > - Method signature changes > > > > > > > > Nothing specific. > > > > > > > > - Method behavior changes > > > > > > > > Nothing specific. > > > > > > > > - CLI command changes > > > > > > > > Add `setConsumeMode` for admin to switch between old pull mode > and > > > new > > > > pop mode for one subscription. > > > > > > > > - Log format or content changes > > > > > > > > Nothing specific. > > > > > > > > ## Compatibility, Deprecation, and Migration Plan > > > > > > > > - Are backward and forward compatibility taken into consideration? > > > > > > > > New RequestCode between client and broker are added, so there > are 2 > > > > compatibility situations: > > > > > > > > 1. old client+new broker: old clients won't make request with > > > > pop-feature-support flag, so broker will not enable pop mode, which > > keep > > > > all things as before. > > > > 2. new client+old broker: new clients will detect whether broker > > > > support the new request command querying queue assignments, if not, > it > > > will > > > > fallback to use old pull mode. > > > > > > > > - Are there deprecated APIs? > > > > > > > > Nothing specific. > > > > > > > > - How do we do migration? > > > > > > > > Nothing specific. > > > > > > > > ## Implementation Outline > > > > > > > > We will implement the proposed changes by **2** phases. > > > > > > > > ## Phase 1 > > > > > > > > 1. Implement server-side balance capability in broker > > > > 2. Implement client-side request using new pop-mode > > > > > > > > ## Phase 2 > > > > > > > > 1. Implement new sdk compatibility with old broker. > > > > 2. Implement feature detection in broker and client. > > > > > > > > # Rejected Alternatives > > > > > > > > ## How does alternatives solve the issue you proposed? > > > > > > > > Improve client rebalance logic? I don't get a quite good idea. > > > > > > > > ## Pros and Cons of alternatives > > > > > > > > Client rebalance logic will become quite complicated. > > > > > > > > ## Why should we reject above alternatives > > > > > > > > > > -- Nothing is impossible
