Besides Arpit's questions, will there be any backward compatibility issues if we enable this by default?ThanksKaijie ---- On 星期一, 23 五月 2022 23:51:16 +0800 aagar...@cloudera.com.INVALID wrote ----+1 for proceeding with the merge.
This work is crucial for improving the DataNode stability. Thanks to the Tencent team for driving this effort and Sammi for starting the discussion. Two questions: 1. Will the feature flag disable all the new code paths by default? 2. Are the Tencent engineers comfortable enabling this by default with the testing already done? Thanks, Arpit > On May 23, 2022, at 5:48 AM, Sammi Chen <sammic...@apache.org> wrote: > > Dear Ozone Devs, > > I 'm starting this discussion thread for the proposal to merge the Ozone > Datanode RocksDB merge feature branch (HDDS-3630 > https://issues.apache.org/jira/browse/HDDS-3630) to the master branch. > > Currently there will be one RocksDB for each Container on datanode, which > leads to hundreds of thousands of RocksDB instances on one datanode. It's > very challenging to manage this amount of RocksDB instances in one JVM. > Please refer to the "problem statement" section of the design document[1] > for challenge details. Unlike the current approach, Datanode RocksDB merge > feature will use only one RocksDB for each data volume. With far fewer > RocksDB instances to manage, the write path performance and DN stability > are improved, Refer to the Micro Benchmark Data section of the design > document[1]. > > The feature has been developed in the last 8 months. And it has also been > deployed in Tencent production for 5 months. Now we have reviewed all core > patches, and merged them into the feature branch, including new disk > layout, new container schema, container replication, container deletion, > block deletion, container scanner, non-rolling upgrade. There are no > blocking JIRAs left. > There are two on-progress JIRAs[2], one is about the feature document md, > another is add acceptance test case. Hopefully, we will resolve them this > week. > > For the sake of master branch feature stability, we provide a configuration > to enable/disable this feature. Currently by default, this feature is > disabled. > > For more information, please check out the Merge RocksDB in datanode > feature wiki page here: > > https://cwiki.apache.org/confluence/display/OZONE/Merge+Rocksdb+in+Datanode+%28HDDS-3630%29+Merge+Checklist > > Great thanks to Mark Gui<guim...@126.com>, majority feature investigation, > design and core development is contributed by him. Thanks a lot to Runzhi > Wang, for first finding current schema V2 layout challenges and initiating > the proposal of merge RocksDB, to Nanda, Mukul, Wei-Chiu and Stephen > O’Donnell for the help in design review, design discussion and code > review. If I miss anyone here, sorry for that, but your efforts are well > appreciated. > > > If there are no objections for the merge, I will start the official vote > later. > > [1] Merge RocksDB in Datanode Design Doc > <https://docs.google.com/document/d/1oPCRUVqTJDbYITbwvoVCOZ2-k108FwOXVhq9ukDKHbQ/edit?usp=sharing> > [2] https://issues.apache.org/jira/browse/HDDS-6790 Add document > https://issues.apache.org/jira/browse/HDDS-6791 Add smoke test > > > Thanks, > Sammi --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@ozone.apache.org For additional commands, e-mail: dev-h...@ozone.apache.org
