Git-Yang opened a new issue #840: URL: https://github.com/apache/rocketmq-externals/issues/840
Based on the existing foundation of rocketmq-replicator, the optimization of data synchronization and offset synchronization mainly includes the following points: 1. Supports direct data synchronization, avoiding intermediate storage 1.1 Add WokerDirectTask, responsible for managing RmqSourceTask and RmqSinkTask at the same time. (Read the message through RmqSourceTask, and then write the message directly to the target cluster through RmqSinkTask, omitting intermediate storage.) (done) 1.2 Add synchronous and asynchronous data transmission. (doing) 1.3 Add a retry mechanism for sending failures. (doing) 2. Offset synchronization: directly synchronize the consumption offset to the target cluster 2.1 Based on WokerDirectTask, add MetaSinkTask. Implementation logic is the same as 1.1 (done) 2.2 Increase the check mechanism to ensure that the synchronized offset matches the actual location as much as possible (done) 3. Other 3.1 Support ACL (done) 3.2 Support automatic creation of subscription group (done) 3.3 Instance creation optimization: (done) - DefaultMQPullConsumer uses a fixed group, avoids instance conflicts in the same process by creating a unique instance name, and avoids the flood of subscription groups - DefaultMQPullConsumer uses prefix+wokerId to ensure the reuse of the same instance subscription group. There are two main benefits: one is to avoid the flooding of subscription group data, and the other is to avoid the problem of slow startup caused by consumption from scratch when the service restarts (slow startup) It is due to the addition of configuration verification logic). 基于rocketmq-replicator现有基础上，对数据同步及offset同步进行优化，主要包含一下几点： 1. 支持数据直接同步，避免中间存储 1.1 增加WokerDirectTask，负责同时管理RmqSourceTask和RmqSinkTask。（通过RmqSourceTask读取消息，再通过RmqSinkTask将消息直接写入目标集群，省略中间存储。） (done) 1.2 增加同步和异步数据发送 （doing） 1.3 增加发送失败重试机制 （doing） 2. offset同步：直接将消费offset同步到目标集群 2.1 基于WokerDirectTask，增加MetaSinkTask。实现逻辑同 1.1 （done） 2.2 增加check机制，尽可能保证同步的offset与实际位置匹配 (done) 3. 其他 3.1 支持ACL（done） 3.2 支持自动创建订阅组（done） 3.3 实例创建优化：（done） - DefaultMQPullConsumer 使用固定group，通过创建唯一实例名避免同进程下实例冲突，避免订阅组数量泛滥 - DefaultMQPullConsumer 使用前缀+wokerId的方式，保证同一个实例订阅组复用，主要有两点好处：一是避免订阅组数据泛滥，二是避免服务重启时从头开始消费造成的启动慢的问题（启动慢是由于增加了配置检验逻辑）。 3.4 部分bug修复： - topic构建路由逻辑bug（done） 3. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have). Are you currently using any workarounds to address this issue? - blocker 4. If there are some sub-tasks using -[] for each subtask and create a corresponding issue to map to the sub task: - [1](https://github.com/apache/rocketmq-externals/issues/824): 1.1 Support WorkerDirectTask - [2](https://github.com/apache/rocketmq-externals/issues/826): 1.1 Add RmqSinTask - [3](https://github.com/apache/rocketmq-externals/issues/821): 3.2 Auto create subGroup - [4](https://github.com/apache/rocketmq-externals/issues/833): 3.4 Topic build routing logic error - [5](xxxxx) Not yet submitted - ... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
