Thanks for your reply. The KV storage feature will be load according to the demands of users, just like a plugin, if we don't need this, nothing additional will be loaded at runtime.
vintagewang <[email protected]> 于2021年9月23日周四 下午2:02写道： > RocketMQ Broker's features and source line of code are already very large. > In the future, if some capabilities can be split into other components, or > new components, so that they can be developed like microservices, otherwise > Broker will become more and more like a monolithic application, which will > eventually lead to the complexity is high enough to evolve quickly. > > Xiaorui Wang > Apache RocketMQ PMC Chair > > On Thu, Sep 23, 2021 at 10:11 AM Amber Liu <[email protected]> wrote: > > > # RIP-22 Support KV semantic storage > > > > ## Status > > - Current Status: Draft > > - Authors: [ltamber](https://github.com/ltamber) > > - Shepherds: [duhengforever](mailto:[email protected]) > > - Mailing List discussion: <[email protected]> > > - Pull Request: #PR_NUMBER > > - Released: <released_version> > > ## Background & Motivation > > ### what do we need to do > > - will we add a new module? **no**. > > - will we add new APIs? **yes**. > > - will we add new feature? **yes**. > > ### Why should we do that > > - Are there any problems of our current project? > > Currently, we can't get/put key-value from/into rocketmq, so if we use > > [connector](https://github.com/apache/rocketmq-externals), like > > FileSource, > > BinlogSource, we can't persist current read position/dump position to > > rocketmq rather than an external meta store like zookeeper/mysql, this > will > > bring more operator risk by introduce another component. this issue was > > also in [streaming](https://github.com/apache/rocketmq-streams) > scenarios > > when developer want to persist meta info like checkpoint. > > - What can we benefit proposed changes? > > rocketmq would not rely on external componet such as zookeeper/etcd to > > support meta data storage. > > ### Goals > > - What problem is this proposal designed to solve? > > Design a distribution persistent key-value store, application can put > > key-value into broker, and then get the value after a while, in the same > > time, it can also have the ability like compareAndSet, prefix get and so > > on. > > - To what degree should we solve the problem? > > This RIP must guarantee below point: > > 1. High availablity: if one broker in the broker group is down, > > application can put/get key-value through other broker, the availablity > is > > same with the message of rocketmq. > > 2. High capacity: the amount of key-value may very large, so the > > key-value can not store in memory, we must store the key-value in disk > > device. > > ### Non-Goals > > - What problem is this proposal NOT designed to solve? > > Nothing specific. > > - Are there any limits of this proposal? > > Nothing specific. > > ## Changes > > ### Architecture > >  > > We will introduce [rocksdb](https://github.com/facebook/rocksdb) to > > persist > > key-value data, to say it more accurately, we use rocksdb to compact the > > value with the same key, we will not enable WAL in rocksdb to decrease > > write amplification (most case), instead we can recover the rocksdb state > > and consistency by redo rocketmq commitlog. so the put/get flow showed on > > the above figure is: > > put: the key-value message will put into commitlog first, and then > through > > the `reputService` redo commitlog, the key-value will put to rocksdb > > asynchronous, until this reput finished broker will not response to > client. > > get: application will get key-value from rocksdb thought broker directly. > > In addition, if we don't want introduce [rocksdb]( > > https://github.com/facebook/rocksdb) and the meta data content will not > > occupy too many memory, we can also use a key-value store base on memory > > map, there will a periodic serialization and persistence thread to > > guarantee data won't loss if broker restart or system abnormal shutdown, > > and the memory state consistency will also guaranteed by redo rocketmq > > commitlog. > > ### Interface Design/Change > > - Method signature changes. **No** > > - Method behavior changes. **No** > > - CLI command changes. **No** > > - Log format or content changes. > > the properties of the message will add two flag, `kv_opType` indicate > > the request type is put key-value or get key-value, and `key` indicate > the > > request key both in put or get operation. In order to pass the key > through > > the network in the request header, we will encode/decode the key(byte > array > > format) use [base64]( > > https://docs.oracle.com/javase/8/docs/api/java/util/Base64.html) > encoding > > method. > >  > > ### Compatibility, Deprecation, and Migration Plan > > - Are backward and forward compatibility taken into consideration? > > New RequestCode between client and broker are added, so there are 2 > > compatibility situations: > > 1. old client+new broker: old clients won't make request with > key-value > > flag, so broker will not receive key-value request, which keep all things > > as before. > > 2. new client+old broker: new clients will send key-value request, > but > > the broker don't recognize the request code, and will return error msg. > so > > we should upgrade broker first to support this feature. > > - Are there deprecated APIs? > > Nothing specific. > > - How do we do migration? > > Nothing specific. > > ### Implementation Outline > > We will implement the proposed changes by two phases. > > #### Phase 1 > > 1. Implement reput logic from commitlog to rocksdb. > > 2. Implement broker support key-value request and response. > > 3. Implement client support key-value request and response. > > 4. Implement key-value store use memory map. > > 5. Implement key-value store use rocksdb. > > #### Phase 2 > > 1. Implement prefix get semantics. > > 2. Implement compareAndSet semantics. > > 3. Implement rocksdb snapshot export/import. > > >
