+1, This is a correct and great step for the cloud-native infra, especially for RocketMQ community. Looking forward to seeing a more practical and concrete proposal.
panwei zhang <[email protected]> 于2021年1月20日周三 下午9:59写道： > +1 > > tiger lee <[email protected]> 于2021年1月20日周三 下午3:32写道： > >> [ ] +1 approve >> it would make Consumer Client more simple to use. and do u have a work >> flow to show how POP mode work ? >> >> heng du <[email protected]> 于2021年1月18日周一 下午2:37写道： >> >>> Hi RocketMQ Community, >>> >>> This is the vote for the kickoff of RIP-19 RocketMQ Pop Consuming. >>> >>> In order to better implement a lightweight client, @ayanamist proposes a >>> new consumption model, and at the same time transfers the load balancing >>> logic of the original client to the broker, which not only solves the >>> original queue occupancy problem but also It can also avoid the >>> consumption >>> delay caused by a certain consumer hangs. >>> >>> The vote will be open for at least 72 hours or until a necessary number >>> of >>> votes are reached. >>> >>> Please vote accordingly: >>> >>> [ ] +1 approve >>> [ ] +0 no opinion >>> [ ] -1 disapprove with the reason >>> >>> >>> >>> >>> Best Regards! >>> Henry >>> >>> ayanamist <[email protected]> 于2021年1月8日周五 上午11:25写道： >>> >>> > # RIP-19 RocketMQ Pop Consuming >>> > >>> > # Status >>> > >>> > - Current State: Proposed >>> > - Authors: [ayanamist]([ >>> > https://github.com/ayanamist/](https://github.com/ayanamist/)) >>> > - Shepherds: [duhengforever]([ >>> > https://github.com/duhenglucky/](https://github.com/duhengforever/)) >>> > - Mailing List discussion: [email protected]; >>> > [email protected] >>> > - Pull Request: RIP-19 >>> > - Released: - >>> > >>> > # Background & Motivation >>> > >>> > ### What do we need to do >>> > >>> > - Will we add a new module? >>> > >>> > No. >>> > >>> > - Will we add new APIs? >>> > >>> > Yes. >>> > >>> > - Will we add new feature? >>> > >>> > Yes. >>> > >>> > ### Why should we do that >>> > >>> > - Are there any problems of our current project? >>> > >>> > The current subscription load balancing strategy is based on the >>> > dimension of message queue. All behaviors are owned by the client side. >>> > There are three main steps: >>> > >>> > 1. Each consumer regularly obtains the total number of topic >>> message >>> > queues and all consumers. >>> > 2. Using a general algorithm to sort the queues by consumer ip and >>> > queue index to calculate which message queue is allocated to which >>> > consumer. >>> > 3. Each consumer pulls messages using allocated orders described >>> above. >>> > >>> > According to this allocation method, if an abnormality occurs in a >>> > consumer (the application itself is abnormal, or a broker is >>> upgrading) so >>> > that it causes slow subscription, messages will be accumulated, but >>> this >>> > queue will not be re-allocated to another consumer, so the accumulation >>> > will become more and more serious. >>> > >>> > >>> > Chinese version: >>> > >>> > 当前的消费负载均衡策略是以队列的维度来进行，所有行为全部是由客户端主动来完成，主要分为三步： >>> > >>> > 1. 每个consumer定时去获取消费的topic的队列总数，以及consumer总数 >>> > 2. 将队列按编号、consumer按ip排序，用统一的分配算法计算该consumer分配哪些消费队列 >>> > 3. 每个consumer去根据算法分配出来的队列，拉取消息消费 >>> > >>> > >>> > >>> > >>> 按照这个分配方式，如果有一个队列有异常（应用自身异常，或某个broker在升级）导致消费较慢或者停止，该队列会出现堆积现象，因为队列不会被分配给其他机器，因此如果长时间不处理，队列的堆积会越来越严重。 >>> > >>> > - What can we benefit proposed changes? >>> > >>> > The accumulated messages will be subscribed by other consumers if >>> one >>> > consumer behaves abnormally. >>> > >>> > Chinese version: >>> > >>> > 在某个队列消费异常的情况下，可以快速的由其它消费者接手进行消费，缓解堆积状态。 >>> > >>> > # Goals >>> > >>> > - What problem is this proposal designed to solve? >>> > >>> > The accumulated messages will be subscribed by other consumers if >>> one >>> > consumer behaves abnormally. >>> > >>> > Chinese version: >>> > >>> > 在某个队列消费异常的情况下，可以快速的由其它消费者接手进行消费，缓解堆积状态。 >>> > >>> > - To what degree should we solve the problem? >>> > >>> > This RIP must guarantee below point: >>> > >>> > 1. High availablity: Subscription of one message queue will not be >>> > affected by single consumer failure. >>> > 2. High performance: This implementation affects latency and >>> throughput >>> > less than 10%. >>> > >>> > >>> > Chinese version: >>> > >>> > 新方案需要保证两点： >>> > >>> > 1. 高可用：单一队列的消费能力不受某个消费客户端异常的影响 >>> > 2. 高性能：POP订阅对消息消费的延迟和吞吐的影响在10%以内 >>> > >>> > # Non-Goals >>> > >>> > - What problem is this proposal NOT designed to solve? >>> > >>> > Improve client-side load balancing. >>> > >>> > - Are there any limits of this proposal? >>> > >>> > Nothing specific. >>> > >>> > # Changes >>> > >>> > ## Architecture >>> > >>> > Current "Pull mode": >>> >  >>> > >>> > Proposed "Pop mode": >>> >  >>> > >>> > Move inter-queue balance of one topic from client side to server side. >>> > Clients make pull request without specified queues to broker, and >>> broker >>> > fetch messages from queues internally and returns, which ensures one >>> queue >>> > will be consumed by multiple clients. The whole behavior is like a >>> queue >>> > pop process. >>> > >>> > It will add a new request command querying queue assignments in >>> broker, and >>> > add pop-feature-support flag to pull request which makes broker use pop >>> > mode. >>> > >>> > ## Interface Design/Change >>> > >>> > - Method signature changes >>> > >>> > Nothing specific. >>> > >>> > - Method behavior changes >>> > >>> > Nothing specific. >>> > >>> > - CLI command changes >>> > >>> > Add `setConsumeMode` for admin to switch between old pull mode and >>> new >>> > pop mode for one subscription. >>> > >>> > - Log format or content changes >>> > >>> > Nothing specific. >>> > >>> > ## Compatibility, Deprecation, and Migration Plan >>> > >>> > - Are backward and forward compatibility taken into consideration? >>> > >>> > New RequestCode between client and broker are added, so there are 2 >>> > compatibility situations: >>> > >>> > 1. old client+new broker: old clients won't make request with >>> > pop-feature-support flag, so broker will not enable pop mode, which >>> keep >>> > all things as before. >>> > 2. new client+old broker: new clients will detect whether broker >>> > support the new request command querying queue assignments, if not, it >>> will >>> > fallback to use old pull mode. >>> > >>> > - Are there deprecated APIs? >>> > >>> > Nothing specific. >>> > >>> > - How do we do migration? >>> > >>> > Nothing specific. >>> > >>> > ## Implementation Outline >>> > >>> > We will implement the proposed changes by **2** phases. >>> > >>> > ## Phase 1 >>> > >>> > 1. Implement server-side balance capability in broker >>> > 2. Implement client-side request using new pop-mode >>> > >>> > ## Phase 2 >>> > >>> > 1. Implement new sdk compatibility with old broker. >>> > 2. Implement feature detection in broker and client. >>> > >>> > # Rejected Alternatives >>> > >>> > ## How does alternatives solve the issue you proposed? >>> > >>> > Improve client rebalance logic? I don't get a quite good idea. >>> > >>> > ## Pros and Cons of alternatives >>> > >>> > Client rebalance logic will become quite complicated. >>> > >>> > ## Why should we reject above alternatives >>> > >>> >> -- Nothing is impossible
