Thanks for this meaningful proposal. Some nit comments: a. Snapshot, Symbolic link and reserved feature are not mentioned at the design doc, should it be considered or different to this core design? b. For the benchmark result, what Read/Write request ratio? And do you meet any GC issues when reaching `108K QPS`? If true, would you mind sharing STW time cost? c. Is this deployed in your internal cluster now? If true, any performance benefit differences compare to the benchmark? d. This is one huge feature IMO, If discussion passes, suggest creating a single branch to develop and follow-up works.
Thanks again for this meaningful proposal. Best Regards, - He Xiaoqiao On Tue, Feb 20, 2024 at 5:38 PM Yuanbo Liu <liuyuanb...@gmail.com> wrote: > Nice to see this feature brought up. We've implemented this feature > internally and gained significant performance improvement. I'll be glad to > work on some jiras if necessary. > > > On Tue, Feb 20, 2024 at 4:41 PM ZanderXu <zande...@apache.org> wrote: > > > Thank you everyone for reviewing this ticket. > > > > I think if there are no problems with the goal and the overall solution, > we > > are ready to push this ticket forward and I will create some detailed > > sub-tasks for this ticket. > > > > I will split this project into three milestones to make this project > > cleaner for review and merge. > > Milestone 1: Replacing the current global lock with two locks, global FS > > lock and global BM lock. End-user can choose which locking mode to use > > through configuration. > > Milestone 2: Replacing the global FS write lock with directory tree-based > > fine-grained lock. > > Milestone 3: Replacing the global BM lock with directory tree-based > > fine-grained lock. > > > > Each milestone can be merged into the trunk branch in time, which has > > multiple benefits: > > 1. Phased performance improvements can be quickly used by everyone > > 2. All developers can better understand the implementation ideas of the > > fine-grained locking mechanism as soon as possible > > 3. Each milestone is developed based on the latest trunk branch to reduce > > conflicts > > > > If you have any concerns, please feel free to discuss them together. > > I hope you can join us to push this project forward together, thanks. > > > > > > On Mon, 5 Feb 2024 at 11:33, haiyang hu <haiyang87...@gmail.com> wrote: > > > > > Thank you for raising the issue of this long-standing bottleneck, this > > > will be a very important improvement! > > > > > > Hopefully can participate and push forward together. > > > > > > Best Regards~ > > > > > > Brahma Reddy Battula <bra...@apache.org> 于2024年2月3日周六 00:40写道： > > > > > >> Thanks for bringing this and considering all the history around this. > > >> One of the outstanding bottleneck(global lock) from a long time. > > >> > > >> Hopefully we can push forward this time. > > >> > > >> > > >> On Fri, Feb 2, 2024 at 12:23 PM Hui Fei <feihui.u...@gmail.com> > wrote: > > >> > > >> > Thanks for driving this. It's very meaningful. The performance > > >> improvement > > >> > looks very good. > > >> > > > >> > Many users are facing the write performance issue. As far as I know, > > >> some > > >> > companies already implemented the similar idea on their internal > > >> branches. > > >> > But the internal branch is very different from the community one. So > > >> it's > > >> > very hard to be in sync with the community. If this improvement can > be > > >> > involved in the community, that would be great to both end-user and > > the > > >> > community. > > >> > > > >> > It is very worth doing. > > >> > > > >> > Zengqiang XU <zande...@apache.org> 于2024年2月2日周五 11:07写道： > > >> > > > >> > > Hi everyone > > >> > > > > >> > > I have started a discussion about NameNode Fine-grained Locking to > > >> > improve > > >> > > performance of write operations in NameNode. > > >> > > > > >> > > I started this discussion again for serval main reasons: > > >> > > 1. We have implemented it and gained nearly 7x performance > > >> improvement in > > >> > > our prod environment > > >> > > 2. Many other companies made similar improvements based on their > > >> internal > > >> > > branch. > > >> > > 3. This topic has been discussed for a long time, but still > without > > >> any > > >> > > results. > > >> > > > > >> > > I hope we can push this important improvement in the community so > > that > > >> > all > > >> > > end-users can enjoy this significant improvement. > > >> > > > > >> > > I'd really appreciate you can join in and work with me to push > this > > >> > feature > > >> > > forward. > > >> > > > > >> > > Thanks very much. > > >> > > > > >> > > Ticket: HDFS-17366 < > > https://issues.apache.org/jira/browse/HDFS-17366> > > >> > > Design: NameNode Fine-grained locking based on directory tree > > >> > > < > > >> > > > > >> > > > >> > > > https://docs.google.com/document/d/1bVBQcI4jfzS0UrczB7UhsrQTXmrERGvBV-a9W3HCCjk/edit?usp=sharing > > >> > > > > > >> > > > > >> > > > >> > > > > > >
