Thanks all for your positive feedback! Two mentioned on-going jiras are also merged today. https://issues.apache.org/jira/browse/HDDS-6790 Add document https://issues.apache.org/jira/browse/HDDS-6791 Add smoke test
I will start a formal vote thread based on this discussion result. Bests, Sammi On Mon, 23 May 2022 at 20:48, Sammi Chen <sammic...@apache.org> wrote: > Dear Ozone Devs, > > I 'm starting this discussion thread for the proposal to merge the Ozone > Datanode RocksDB merge feature branch (HDDS-3630 > https://issues.apache.org/jira/browse/HDDS-3630) to the master branch. > > Currently there will be one RocksDB for each Container on datanode, which > leads to hundreds of thousands of RocksDB instances on one datanode. It's > very challenging to manage this amount of RocksDB instances in one JVM. > Please refer to the "problem statement" section of the design document[1] > for challenge details. Unlike the current approach, Datanode RocksDB merge > feature will use only one RocksDB for each data volume. With far fewer > RocksDB instances to manage, the write path performance and DN stability > are improved, Refer to the Micro Benchmark Data section of the design > document[1]. > > The feature has been developed in the last 8 months. And it has also been > deployed in Tencent production for 5 months. Now we have reviewed all core > patches, and merged them into the feature branch, including new disk > layout, new container schema, container replication, container deletion, > block deletion, container scanner, non-rolling upgrade. There are no > blocking JIRAs left. > There are two on-progress JIRAs[2], one is about the feature document md, > another is add acceptance test case. Hopefully, we will resolve them this > week. > > For the sake of master branch feature stability, we provide a > configuration to enable/disable this feature. Currently by default, this > feature is disabled. > > For more information, please check out the Merge RocksDB in datanode > feature wiki page here: > > https://cwiki.apache.org/confluence/display/OZONE/Merge+Rocksdb+in+Datanode+%28HDDS-3630%29+Merge+Checklist > > Great thanks to Mark Gui<guim...@126.com>, majority feature > investigation, design and core development is contributed by him. Thanks a > lot to Runzhi Wang, for first finding current schema V2 layout challenges > and initiating the proposal of merge RocksDB, to Nanda, Mukul, Wei-Chiu > and Stephen O’Donnell for the help in design review, design discussion and > code review. If I miss anyone here, sorry for that, but your efforts are > well appreciated. > > > If there are no objections for the merge, I will start the official vote > later. > > [1] Merge RocksDB in Datanode Design Doc > <https://docs.google.com/document/d/1oPCRUVqTJDbYITbwvoVCOZ2-k108FwOXVhq9ukDKHbQ/edit?usp=sharing> > [2] https://issues.apache.org/jira/browse/HDDS-6790 Add document > https://issues.apache.org/jira/browse/HDDS-6791 Add smoke test > > > Thanks, > Sammi > > >
