Hi All, I would like to propose prefix based FileSystem Optimizations (FSO) work HDDS-2939 <https://issues.apache.org/jira/browse/HDDS-2939> merging into Ozone master branch. This optimization is intended to support atomic rename and delete operations in Ozone namespace.
Presently, a rename/delete operation can become prohibitively expensive for such directories which have large sub-trees/sub-paths. This optimization allows to perform rename, delete of any directory in a deterministic/constant time atomically. The main functionality has been implemented and supports o3fs, ofs and ObjectStore APIs, for details please refer to sub-tasks of HDDS-2939 which we have been actively working in the feature branch <https://github.com/apache/ozone/tree/HDDS-2939> for the last several months. There is a configuration to turn this feature ON and OFF while merging back to master, it will be turned OFF by default for now. Merging back to master will greatly help in stabilizing/testing the feature more rigorously while ensuring the ongoing work on the master branch is not negatively impacted by the feature as it is turned OFF by default. Following graph compares the performance between master(V0) and new optimized code(V1) for the delete op in an unsecure cluster. Rename op also has similar results. Please refer to the attached test report <https://issues.apache.org/jira/secure/attachment/13023395/Performance+Comparison+Between++Master+and+HDDS-2939+branch-Report-001.pdf>in jira for more details. *Ozone wiki page:* https://cwiki.apache.org/confluence/display/OZONE/FileSystem+Optimizations+-+HDDS-2939 Number of JIRAs Resolved: 27 Patch Available/WIP: 3 jira There are few ongoing sub-tasks, IMHO they are blockers for merge. We will be actively working on these sub-tasks. Detailed design document is available in JIRA : https://issues.apache.org/jira/secure/attachment/12991926/Ozone%20FS%20Namespace%20Proposal%20v1.0.docx https://issues.apache.org/jira/secure/attachment/13023399/OzoneFS%20Optimizations_DesignOverview_%20HDDS-2939.pdf Quick stats on combined HDDS-2939 Patch : Commits 8b8a7e3ec84ad14e7128ff5fb3ded542ea9080d7..6d76676d2956346895de630a96c182188f9f3e38 157 files changed, 16289 insertions(+), 776 deletions(-) Added/modified test cases= ~160 Thanks a lot to all contributors/reviewers. So, I feel this branch is ready to merge into master branch. Could you please provide your feedback. If there are no objections, I will proceed for merge voting. Thanks, Rakesh
