+1
"heng du" <[email protected]>写道： > Hi RocketMQ Community, > > This is the vote for the kickoff of RIP-21 RocketMQ logic queue. > > In order to better meet the needs of stream computing and scaling, the > concept of logical queues is proposed in this RIP, which enables RocketMQ > to have the ability to scale in seconds without changing the number of > queues. > > The vote will be open for at least 72 hours or until a necessary number > of votes are reached. > > Please vote accordingly: > > [ ] +1 approve > [ ] +0 no opinion > [ ] -1 disapprove with the reason > > > > ayanamist <[email protected]> 于2021年5月18日周二 上午10:14写道： > > > Below is Markdown text with some GFM syntax. > > > > # Status > > - Current State: Proposed > > - Authors: [ayanamist](https://github.com/ayanamist) > > - Shepherds: [email protected] > > - Mailing List discussion: [email protected] > > - Pull Request: #PR_NUMBER > > - Released: <relased_version> > > # Background & Motivation > > What do we need to do > > - Will we add a new module? > > No. > > - Will we add new APIs? > > It will add a new constant to mock broker names for logical queues. > > - Will we add a new feature? > > Yes. > > > > Why should we do that > > - Are there any problems with our current project? > > Currently, the MessageQueue of RocketMQ is coupled with broker name, which > > results that the queue number will change if broker number increases or > > decreases, which causes all queues to rebalance, which may cause service > > disruption like flink job restarts in minutes. > > - What can we benefit from proposed changes? > > The number of logical queues is not related with the number of brokers: We > > can increase broker number without changing logical queue number, moreover, > > we can increase logical queue number without deploying a new broker. > > # Goals > > - What problem is this proposal designed to solve? > > Provide an abstraction, logical queue, to decouple between queue number and > > broker number. > > - To what degree should we solve the problem? > > We should not hurt availability or performance in the implementation. > > # Non-Goals > > - What problem is this proposal NOT designed to solve? > > We will not improve the mechanism of queues rebalance. > > - Are there any limits of this proposal? > > Only newer clients with changes in this proposal will benefit. > > # Changes > > ## Architecture > > > > We use one or more MessageQueue to make one LogicalQueue; One LogicalQueue > > is unique in one topic, and one MessageQueue belongs to one and only one > > LogicalQueue. > > > > | brokerName | MessageQueue | LogicalQueue | > > |------------|--------------|--------------| > > | broker1 | 0 | 0 | > > | broker1 | 1 | 1 | > > | broker2 | 0 | 2 | > > | broker2 | 1 | 3 | > > > > After one LogicalQueue migrated from broker1 to broker2, there will be two > > MessageQueues for one LogicalQueue. We only migrate mapping but not actual > > data, so broker1 is still serving for old data consuming but not data > > producing. > > > > | brokerName | MessageQueue | LogicalQueue | QueueStatus | > > |------------|--------------|--------------|--------------------| > > | **broker1** | **0** | **0(0-100)** | **ReadOnly** > > | > > | broker1 | 1 | 1 | Normal | > > | broker2 | 0 | 2 | Normal | > > | broker2 | 1 | 3 | Normal | > > | **broker2** | **2** | **0(101-)** | **Normal** > > | > > > > After broker1 cleans all data from the commit log and consume queue, > > QueueStatus becomes Expired(neither readable nor writable). > > > > | brokerName | MessageQueue | LogicalQueue | QueueStatus | > > |------------|--------------|--------------|-------------| > > | **broker1** | **0** | **0(-)** | **Expired** | > > | broker1 | 1 | 1 | Normal | > > | broker2 | 0 | 2 | Normal | > > | broker2 | 1 | 3 | Normal | > > | broker2 | 2 | 0(101-) | Normal | > > > > If this LogicalQueue is migrated back to broker1, it will reuse this > > expired MessageQueue > > > > | brokerName | MessageQueue | LogicalQueue | QueueStatus | > > |------------|--------------|--------------|-------------| > > | **broker1** | **0** | **0(201-)** | **Normal** | > > | broker1 | 1 | 1 | Normal | > > | broker2 | 0 | 2 | Normal | > > | broker2 | 1 | 3 | Normal | > > | **broker2** | **2** | **0(101-200)** | **ReadOnly** | > > > > If this LogicalQueue is migrated back to broker1 while MessageQueue not > > expired, it will create a new MessageQueue > > > > | brokerName | MessageQueue | LogicalQueue | QueueStatus | > > |------------|--------------|--------------|-------------| > > | **broker1** | **0** | **0(0-100)** | **ReadOnly** | > > | broker1 | 1 | 1 | Normal | > > | **broker1** | **2** | **0(201-)** | **Normal** | > > | broker2 | 0 | 2 | Normal | > > | broker2 | 1 | 3 | Normal | > > | **broker2** | **2** | **0(101-200)** | **ReadOnly** | > > > > If broker2 is offlined, all LogicalQueue in this broker should be migrated > > away. > > > > | brokerName | MessageQueue | LogicalQueue | QueueStatus | > > |------------|--------------|--------------|-------------| > > | broker1 | 0 | 0 | Normal | > > | broker1 | 1 | 1 | Normal | > > | **broker1** | **2** | **2(101-)** | **Normal** | > > | **broker1** | **3** | **3(101-)** | **Normal** | > > | **broker2** | **0** | **2(0-100)** | **ReadOnly** | > > | **broker2** | **1** | **3(0-100)** | **ReadOnly** | > > > > When all data including commit log and consume queue in broker2 are > > cleaned, broker2 can be removed. > > > > | brokerName | MessageQueue | LogicalQueue | QueueStatus | > > |------------|--------------|--------------|-------------| > > | broker1 | 0 | 0 | Normal | > > | broker1 | 1 | 1 | Normal | > > | **broker1** | **2** | **2(101-)** | **Normal** | > > | **broker1** | **3** | **3(101-)** | **Normal** | > > > > When a new broker is deployed, we can migrate some LogicalQueues to this > > broker to spare some producing traffic. > > > > | brokerName | MessageQueue | LogicalQueue | QueueStatus | > > |------------|--------------|--------------|-------------| > > | broker1 | 0 | 0 | Normal | > > | broker1 | 1 | 1 | Normal | > > | **broker1** | **2** | **2(101-200)** | **ReadOnly** | > > | **broker1** | **3** | **3(101-200)** | **ReadOnly** | > > | **broker3** | **0** | **2(201-)** | **Normal** | > > | **broker3** | **1** | **3(201-)** | **Normal** | > > > > All mapping data are stored separately in brokers, and via registerBroker > > to be gathered in namesrv; Each broker only stores its own logical queue > > mapping but not other one's. All management operations should be invoked by > > CLI or direct rpc command, automatic operation support is missing now, and > > require [RIP-18]( > > > > https://github.com/apache/rocketmq/wiki/RIP-18-Metadata-management-architecture-upgrade > > ) > > to be implemented first. > > > > ## Interface Design/Change > > - Method signature changes > > No. > > - Method behavior changes > > When a topic is enabled LogicalQueue support, broker name of MessageQueue > > result returned by some methods like `fetchSubscribeMessageQueues` or > > `fetchPublishMessageQueues` will be a fake one, since LogicalQueue does not > > have broker name concept. > > - CLI command changes > > Add some operation command for LogicalQueue, like > > `UpdateLogicalQueueMapping` `DeleteLogicalQueueMapping` > > `QueryLogicalQueueMapping` `UpdateLogicalQueueNum` `MigrateLogicalQueue`. > > Also `updateTopic` adds a new parameter to enable LogicalQueue support > > immediately after the topic is created. > > - Log format or content changes > > No. > > ## Compatibility, Deprecation, and Migration Plan > > - Are backward and forward compatibility taken into consideration? > > Yes. > > - Everything will work well if no topic is enabled LogicalQueue support, > > whether on old/new broker/namesrv/client. > > - LogicalQueue support will work only under new broker+namesrv+client, > > otherwise, everything will work like no LogicalQueue support. > > - Are there deprecated APIs? > > No. > > - How do we do migration? > > No need to migrate, this is a feature which needs to be enabled manually. > > ## Implementation Outline > > We will implement the proposed changes by 2 phases. > > ### Phase 1 basic function > > 1. Implement LogicalQueue routes query and merge mechanism in namesrv. > > 2. Implement LogicalQueue mapping report from broker to namesrv. > > 3. Implement LogicalQueue support in the client. > > 4. Implement LogicalQueue migration in broker. > > 5. Implement LogicalQueue admin processor and CLI. > > ### Phase 2 compatibility > > 1. Implement proxy request+response for old client in broker. > > 2. Implement intermediate state(support is partly enabled in some brokers > > but not all) protection in namesrv. > > # Rejected Alternatives > > - How does alternatives solve the issue you proposed? > > Implement a whole new Logical Queue architecture from scratch, this > > absolutely will solve the problem. > > - Pros and Cons of alternatives > > Pros: from-scratch way does not bring anything good. > > Cons: from-scratch way will break existent concept, add much more > > complexity to code and not user-friendly. > > - Why should we reject the above alternatives > > It does no good. > >
