+1 for this proposal. Obviously, supporting gRPC could make it easier for RocketMQ contributors to write multi-Language SDKs. Looking forward to more details of this proposal.
Regards, yukon On Wed, Jun 9, 2021 at 11:10 AM Zhanhui Li <lizhan...@apache.org> wrote: > Hi, > This proposal, in general, is in the right direction that helps RocketMQ > provide full-fledged SDK for popular languages and platforms. Taking full > advantage of gRPC does save a lot of effort in terms of serialization and > RPC tiers. Obviously, this proposal also brings complexities and potential > compatibility issues. > > One of the potential issues is that gRPC does not expose Channel in the > implementation while RocketMQ processors make heavy use of it, even if both > of them are built on top of Netty 4.x. Will this an issue when reuse > existing code? > > Zhanhui Li > > On Tue, Jun 8, 2021 at 8:28 PM i yangkun <yangku...@outlook.com> wrote: > > > Background & Motivation > > What do we need to do > > > > > > * Will we add a new module? > > maybe. > > * Will we add new APIs? > > Yes. > > > > * Will we add new feature? > > Yes. > > > > > > Why should we do that > > > > > > * Are there any problems of our current project? > > > > a. Remoting module is too complicated to maintain, gRPC makes it easier > to > > establish a robust communication layer, the current remoting module would > > be simplified radically. > > > > b. gRPC has been the de-facto standard in CloudNative, service mesh would > > be easily applied if gRPC is enabled. > > > > c. The private protocol of RocketMQ depends on the FastJson, it is > > difficult to adapt for other language. > > > > On the other side, since the pop consumer has been merged, we could > > implement new SDK based on gRPC and pop, which is easier to develop and > > maintain. > > > > Chinese Version: > > > > a. Remoting 模块对于长期的维护而言过于复杂了，我们可以使用 gRPC 更轻松地建立起一个健壮的通信层，这会使得现有的 remoting > > 模块从根本上得到简化。 > > > > b. gRPC 目前已经是云原生时代的事实标准，使用 gRPC 可以使得我们天然获取一些云原生的能力，比如 Service Mesh。 > > > > c. 目前 RocketMQ 的私有协议强烈依赖 FastJson，多语言的适配将会变得困难。 > > > > > > 从另外一个角度来说，鉴于 pop 消费者已经被合并，我们可以基于 gRPC 和 pop 实现新的 SDK，新的 SDK 将会更加易于开发和维护。 > > > > Goals > > > > > > * What problem is this proposal designed to solve? > > > > Support gRPC's protocol, simplify current communication layer oof > > RocketMQ, make it easier to adapt for other language, which is not > limited > > to CPP/GO/C#/GO。 > > > > Chinese Version: > > > > 支持 gRPC 协议，简化 RocketMQ 现有的通信层，复用 gRPC 的能力，简化多语言适配成本，不限于 CPP/GO/C#/GO。 > > > > * To what degree should we solve the problem? > > This RIP must guarantee below point: > > > > 1. Compatibility: Both of gRPC and RemotingCommand should be > supported. > > 2. High performance: This implementation does not affects latency and > > throughput. > > > > > > Chinese Version: > > > > 新方案需要保证两点： > > > > 1. 兼容性：同时支持 gRPC 和 RemotingCommand 协议，不影响现有功能。 > > 2. 高性能：基于 gRPC 的实现不影响整理的延时和吞吐量。 > > > > > > Non-Goals > > > > > > * What problem is this proposal NOT designed to solve? > > Nothing specific. > > * Are there any limits of this proposal? > > Nothing specific. > > > > > > Changes > > Architecture > > > > > > Current broker processor and client. > > > > [ > > > https://intranetproxy.alipay.com/skylark/lark/0/2021/png/200096/1623142547507-128b85f5-98f4-4568-85f8-28ef32982b7c.png > > ] > > > > Proposed gRPC processor and client. > > > > [ > > > https://intranetproxy.alipay.com/skylark/lark/0/2021/png/200096/1623142552491-a7f58ac0-cd7d-4ddd-936e-fb296b667196.png > > ] > > > > Broker would provide a protocol negotiate procedure to distinguish > > RemotingCommand from gRPC protocol. two kinds or processor in broker > would > > re-use the same port to serve for RPC from different SDK. > > > > > > Chinese Version: > > > > broker 本身提供协议协商机制用于区分 RemotingCommnad 和 gRPC 协议，broker 针对 gRPC 和 > > RemotingCommand 提供不同的 processor 为各自的 SDK 服务。 > > > > Interface Design/Change > > > > > > * Method signature changes > > Nothing specific. > > * Method behavior changes > > Nothing specific. > > > > * CLI command changes > > Nothing specific. > > * Log format or content changes > > Nothing specific. > > > > > > Compatibility, Deprecation, and Migration Plan > > > > > > * Are backward and forward compatibility taken into consideration? > > > > Broker support processor of RemotingCommand and gRPC simultaneously, so > > there are one compatibility situations: > > > > If user migrates from original SDK to gRPC SDK in push mode, the > > re-balance policy should make sure that it would not cause repeated > > consumption for a lot of messages. > > > > * Are there deprecated APIs? > > Nothing specific. > > * How do we do migration? > > Nothing specific. > > > > > > Implementation Outline > > > > > > We will implement the proposed changes by 4 phases. > > > > > > Phase 1 > > > > 1. Provides gRPC protocol definition(IDL) > > > > Phase 2 > > > > 1. Implement gRPC processor of broker. > > 2. Implement protocol negotiation of two kinds of protocol(gRPC and > > RemotingCommand) > > > > Phase 3 > > > > 1. Implement new JAVA and CPP native SDK based on gRPC > > > > Phase 4 > > > > 1. Implement native SDK base on gRPC for other language. > > > > > > Rejected Alternatives > > > > > > How does alternatives solve the issue you proposed? > > > > > > Thrift? not so much impact as gRPC in community. > > > > > > Pros and Cons of alternatives > > > > > > Nothing specific. > > > > Why should we reject above alternatives > > > > >
