sorry for that. because of the latest RIP from the [wiki]( https://github.com/apache/rocketmq/wiki/RocketMQ-Improvement-Proposal) is RIP-21, I used RIP-22.
vongosling <[email protected]> 于2021年9月23日周四 下午8:35写道： > Thanks for your clarify. I have been confused by RIP 22, It seems we have > occupied 22, right[1]? > > > [1] https://github.com/apache/rocketmq/issues/2937 > > Amber Liu <[email protected]> 于2021年9月23日周四 下午3:29写道： > > > Sorry about the format problem, below is the correct one. > > RIP-22 Support KV semantic storageStatus > > > > - Current Status: Draft > > - Authors: ltamber <https://github.com/ltamber> > > > > > > - Shepherds: duhengforever <[email protected]> > > - Mailing List discussion: [email protected] > > > > > > - Pull Request: #PR_NUMBER > > - Released: <released_version> > > > > Background & Motivationwhat do we need to do > > > > - will we add a new module? *no*. > > - will we add new APIs? *yes*. > > > > > > - will we add new feature? *yes*. > > > > Why should we do that > > > > - Are there any problems of our current project? > > Currently, we can't get/put key-value from/into rocketmq, so if we > use > > connector <https://github.com/apache/rocketmq-externals>, like > > FileSource, BinlogSource, we can't persist current read position/dump > > position to rocketmq rather than an external meta store like > > zookeeper/mysql, this will bring more operator risk by introduce > another > > component. this issue was also in streaming > > <https://github.com/apache/rocketmq-streams> scenarios when developer > > want to persist meta info like checkpoint. > > - What can we benefit proposed changes? > > rocketmq would not rely on external componet such as zookeeper/etcd > > to support meta data storage. > > > > Goals > > > > - What problem is this proposal designed to solve? > > Design a distribution persistent key-value store, application can > > put key-value into broker, and then get the value after a while, in > the > > same time, it can also have the ability like compareAndSet, prefix get > > and > > so on. > > - To what degree should we solve the problem? > > This RIP must guarantee below point: > > 1. High availablity: if one broker in the broker group is down, > > application can put/get key-value through other broker, the > availablity > > is > > same with the message of rocketmq. > > 2. High capacity: the amount of key-value may very large, so the > > key-value can not store in memory, we must store the key-value in > disk > > device. > > > > Non-Goals > > > > - What problem is this proposal NOT designed to solve? > > Nothing specific. > > - Are there any limits of this proposal? > > Nothing specific. > > > > ChangesArchitecture > > > > > > > > We will introduce rocksdb <https://github.com/facebook/rocksdb> to > persist > > key-value data, to say it more accurately, we use rocksdb to compact the > > value with the same key, we will not enable WAL in rocksdb to decrease > > write amplification (most case), instead we can recover the rocksdb state > > and consistency by redo rocketmq commitlog. so the put/get flow showed on > > the above figure is: > > put: the key-value message will put into commitlog first, and then > through > > the reputService redo commitlog, the key-value will put to rocksdb > > asynchronous, until this reput finished broker will not response to > client. > > get: application will get key-value from rocksdb thought broker directly. > > In addition, if we don't want introduce rocksdb > > <https://github.com/facebook/rocksdb> and the meta data content will not > > occupy too many memory, we can also use a key-value store base on memory > > map, there will a periodic serialization and persistence thread to > > guarantee data won't loss if broker restart or system abnormal shutdown, > > and the memory state consistency will also guaranteed by redo rocketmq > > commitlog. > > Interface Design/Change > > > > - Method signature changes. *No* > > - Method behavior changes. *No* > > > > > > - CLI command changes. *No* > > - Log format or content changes. > > the properties of the message will add two flag, kv_opType indicate > > the request type is put key-value or get key-value, and key indicate > the > > request key both in put or get operation. In order to pass the key > > through > > the network in the request header, we will encode/decode the key(byte > > array > > format) use base64 > > <https://docs.oracle.com/javase/8/docs/api/java/util/Base64.html> > > encoding method. > > > > > > Compatibility, Deprecation, and Migration Plan > > > > - Are backward and forward compatibility taken into consideration? > > New RequestCode between client and broker are added, so there are 2 > > compatibility situations: > > 1. old client+new broker: old clients won't make request with > > key-value flag, so broker will not receive key-value request, which > keep > > all things as before. > > 2. new client+old broker: new clients will send key-value request, > > but the broker don't recognize the request code, and will return error > > msg. > > so we should upgrade broker first to support this feature. > > - Are there deprecated APIs? > > Nothing specific. > > > > > > - How do we do migration? > > Nothing specific. > > > > Implementation Outline > > > > We will implement the proposed changes by two phases. > > Phase 1 > > > > 1. Implement reput logic from commitlog to rocksdb. > > 2. Implement broker support key-value request and response. > > > > > > 1. Implement client support key-value request and response. > > 2. Implement key-value store use memory map. > > > > > > 1. Implement key-value store use rocksdb. > > > > Phase 2 > > > > 1. Implement prefix get semantics. > > 2. Implement compareAndSet semantics. > > > > > > 1. Implement rocksdb snapshot export/import. > > > > > > Amber Liu <[email protected]> 于2021年9月23日周四 上午10:10写道： > > > > > # RIP-22 Support KV semantic storage > > > > > > ## Status > > > - Current Status: Draft > > > - Authors: [ltamber](https://github.com/ltamber) > > > - Shepherds: [duhengforever](mailto:[email protected]) > > > - Mailing List discussion: <[email protected]> > > > - Pull Request: #PR_NUMBER > > > - Released: <released_version> > > > ## Background & Motivation > > > ### what do we need to do > > > - will we add a new module? **no**. > > > - will we add new APIs? **yes**. > > > - will we add new feature? **yes**. > > > ### Why should we do that > > > - Are there any problems of our current project? > > > Currently, we can't get/put key-value from/into rocketmq, so if we > use > > > [connector](https://github.com/apache/rocketmq-externals), like > > > FileSource, BinlogSource, we can't persist current read position/dump > > > position to rocketmq rather than an external meta store like > > > zookeeper/mysql, this will bring more operator risk by introduce > another > > > component. this issue was also in [streaming]( > > > https://github.com/apache/rocketmq-streams) scenarios when developer > > want > > > to persist meta info like checkpoint. > > > - What can we benefit proposed changes? > > > rocketmq would not rely on external componet such as zookeeper/etcd > to > > > support meta data storage. > > > ### Goals > > > - What problem is this proposal designed to solve? > > > Design a distribution persistent key-value store, application can > put > > > key-value into broker, and then get the value after a while, in the > same > > > time, it can also have the ability like compareAndSet, prefix get and > so > > on. > > > - To what degree should we solve the problem? > > > This RIP must guarantee below point: > > > 1. High availablity: if one broker in the broker group is down, > > > application can put/get key-value through other broker, the availablity > > is > > > same with the message of rocketmq. > > > 2. High capacity: the amount of key-value may very large, so the > > > key-value can not store in memory, we must store the key-value in disk > > > device. > > > ### Non-Goals > > > - What problem is this proposal NOT designed to solve? > > > Nothing specific. > > > - Are there any limits of this proposal? > > > Nothing specific. > > > ## Changes > > > ### Architecture > > >  > > > We will introduce [rocksdb](https://github.com/facebook/rocksdb) to > > > persist key-value data, to say it more accurately, we use rocksdb to > > > compact the value with the same key, we will not enable WAL in rocksdb > to > > > decrease write amplification (most case), instead we can recover the > > > rocksdb state and consistency by redo rocketmq commitlog. so the > put/get > > > flow showed on the above figure is: > > > put: the key-value message will put into commitlog first, and then > > through > > > the `reputService` redo commitlog, the key-value will put to rocksdb > > > asynchronous, until this reput finished broker will not response to > > client. > > > get: application will get key-value from rocksdb thought broker > directly. > > > In addition, if we don't want introduce [rocksdb]( > > > https://github.com/facebook/rocksdb) and the meta data content will > not > > > occupy too many memory, we can also use a key-value store base on > memory > > > map, there will a periodic serialization and persistence thread to > > > guarantee data won't loss if broker restart or system abnormal > shutdown, > > > and the memory state consistency will also guaranteed by redo rocketmq > > > commitlog. > > > ### Interface Design/Change > > > - Method signature changes. **No** > > > - Method behavior changes. **No** > > > - CLI command changes. **No** > > > - Log format or content changes. > > > the properties of the message will add two flag, `kv_opType` > indicate > > > the request type is put key-value or get key-value, and `key` indicate > > the > > > request key both in put or get operation. In order to pass the key > > through > > > the network in the request header, we will encode/decode the key(byte > > array > > > format) use [base64]( > > > https://docs.oracle.com/javase/8/docs/api/java/util/Base64.html) > > > encoding method. > > >  > > > ### Compatibility, Deprecation, and Migration Plan > > > - Are backward and forward compatibility taken into consideration? > > > New RequestCode between client and broker are added, so there are 2 > > > compatibility situations: > > > 1. old client+new broker: old clients won't make request with > > > key-value flag, so broker will not receive key-value request, which > keep > > > all things as before. > > > 2. new client+old broker: new clients will send key-value request, > > but > > > the broker don't recognize the request code, and will return error msg. > > so > > > we should upgrade broker first to support this feature. > > > - Are there deprecated APIs? > > > Nothing specific. > > > - How do we do migration? > > > Nothing specific. > > > ### Implementation Outline > > > We will implement the proposed changes by two phases. > > > #### Phase 1 > > > 1. Implement reput logic from commitlog to rocksdb. > > > 2. Implement broker support key-value request and response. > > > 3. Implement client support key-value request and response. > > > 4. Implement key-value store use memory map. > > > 5. Implement key-value store use rocksdb. > > > #### Phase 2 > > > 1. Implement prefix get semantics. > > > 2. Implement compareAndSet semantics. > > > 3. Implement rocksdb snapshot export/import. > > > > > > > > -- > Best Regards :-) >
