Nice to see this feature brought up. We've implemented this feature internally and gained significant performance improvement. I'll be glad to work on some jiras if necessary.
On Tue, Feb 20, 2024 at 4:41 PM ZanderXu <zande...@apache.org> wrote: > Thank you everyone for reviewing this ticket. > > I think if there are no problems with the goal and the overall solution, we > are ready to push this ticket forward and I will create some detailed > sub-tasks for this ticket. > > I will split this project into three milestones to make this project > cleaner for review and merge. > Milestone 1: Replacing the current global lock with two locks, global FS > lock and global BM lock. End-user can choose which locking mode to use > through configuration. > Milestone 2: Replacing the global FS write lock with directory tree-based > fine-grained lock. > Milestone 3: Replacing the global BM lock with directory tree-based > fine-grained lock. > > Each milestone can be merged into the trunk branch in time, which has > multiple benefits: > 1. Phased performance improvements can be quickly used by everyone > 2. All developers can better understand the implementation ideas of the > fine-grained locking mechanism as soon as possible > 3. Each milestone is developed based on the latest trunk branch to reduce > conflicts > > If you have any concerns, please feel free to discuss them together. > I hope you can join us to push this project forward together, thanks. > > > On Mon, 5 Feb 2024 at 11:33, haiyang hu <haiyang87...@gmail.com> wrote: > > > Thank you for raising the issue of this long-standing bottleneck, this > > will be a very important improvement! > > > > Hopefully can participate and push forward together. > > > > Best Regards~ > > > > Brahma Reddy Battula <bra...@apache.org> 于2024年2月3日周六 00:40写道： > > > >> Thanks for bringing this and considering all the history around this. > >> One of the outstanding bottleneck(global lock) from a long time. > >> > >> Hopefully we can push forward this time. > >> > >> > >> On Fri, Feb 2, 2024 at 12:23 PM Hui Fei <feihui.u...@gmail.com> wrote: > >> > >> > Thanks for driving this. It's very meaningful. The performance > >> improvement > >> > looks very good. > >> > > >> > Many users are facing the write performance issue. As far as I know, > >> some > >> > companies already implemented the similar idea on their internal > >> branches. > >> > But the internal branch is very different from the community one. So > >> it's > >> > very hard to be in sync with the community. If this improvement can be > >> > involved in the community, that would be great to both end-user and > the > >> > community. > >> > > >> > It is very worth doing. > >> > > >> > Zengqiang XU <zande...@apache.org> 于2024年2月2日周五 11:07写道： > >> > > >> > > Hi everyone > >> > > > >> > > I have started a discussion about NameNode Fine-grained Locking to > >> > improve > >> > > performance of write operations in NameNode. > >> > > > >> > > I started this discussion again for serval main reasons: > >> > > 1. We have implemented it and gained nearly 7x performance > >> improvement in > >> > > our prod environment > >> > > 2. Many other companies made similar improvements based on their > >> internal > >> > > branch. > >> > > 3. This topic has been discussed for a long time, but still without > >> any > >> > > results. > >> > > > >> > > I hope we can push this important improvement in the community so > that > >> > all > >> > > end-users can enjoy this significant improvement. > >> > > > >> > > I'd really appreciate you can join in and work with me to push this > >> > feature > >> > > forward. > >> > > > >> > > Thanks very much. > >> > > > >> > > Ticket: HDFS-17366 < > https://issues.apache.org/jira/browse/HDFS-17366> > >> > > Design: NameNode Fine-grained locking based on directory tree > >> > > < > >> > > > >> > > >> > https://docs.google.com/document/d/1bVBQcI4jfzS0UrczB7UhsrQTXmrERGvBV-a9W3HCCjk/edit?usp=sharing > >> > > > > >> > > > >> > > >> > > >
