Thanks for suggestions. Actually Started working on this improvement. And cut the development branch :) >From the proposal doc and the current reviewing work, seems that it doesn't touch the existing logic codes too much. It keeps the original logic there.
@Yuanbo @Zengqiang XU <zande...@apache.org> Could you share any internal improvement info Xiaoqiao mentioned above? Xiaoqiao He <hexiaoq...@apache.org> 于2024年2月26日周一 19:50写道： > Thanks for this meaningful proposal. Some nit comments: > a. Snapshot, Symbolic link and reserved feature are not mentioned at the > design doc, should it be considered > or different to this core design? > b. For the benchmark result, what Read/Write request ratio? And do you meet > any GC issues when reaching > `108K QPS`? If true, would you mind sharing STW time cost? > c. Is this deployed in your internal cluster now? If true, any performance > benefit differences compare to the > benchmark? > d. This is one huge feature IMO, If discussion passes, suggest creating a > single branch to develop and follow-up > works. > > Thanks again for this meaningful proposal. > > Best Regards, > - He Xiaoqiao > > > On Tue, Feb 20, 2024 at 5:38 PM Yuanbo Liu <liuyuanb...@gmail.com> wrote: > > > Nice to see this feature brought up. We've implemented this feature > > internally and gained significant performance improvement. I'll be glad > to > > work on some jiras if necessary. > > > > > > On Tue, Feb 20, 2024 at 4:41 PM ZanderXu <zande...@apache.org> wrote: > > > > > Thank you everyone for reviewing this ticket. > > > > > > I think if there are no problems with the goal and the overall > solution, > > we > > > are ready to push this ticket forward and I will create some detailed > > > sub-tasks for this ticket. > > > > > > I will split this project into three milestones to make this project > > > cleaner for review and merge. > > > Milestone 1: Replacing the current global lock with two locks, global > FS > > > lock and global BM lock. End-user can choose which locking mode to use > > > through configuration. > > > Milestone 2: Replacing the global FS write lock with directory > tree-based > > > fine-grained lock. > > > Milestone 3: Replacing the global BM lock with directory tree-based > > > fine-grained lock. > > > > > > Each milestone can be merged into the trunk branch in time, which has > > > multiple benefits: > > > 1. Phased performance improvements can be quickly used by everyone > > > 2. All developers can better understand the implementation ideas of the > > > fine-grained locking mechanism as soon as possible > > > 3. Each milestone is developed based on the latest trunk branch to > reduce > > > conflicts > > > > > > If you have any concerns, please feel free to discuss them together. > > > I hope you can join us to push this project forward together, thanks. > > > > > > > > > On Mon, 5 Feb 2024 at 11:33, haiyang hu <haiyang87...@gmail.com> > wrote: > > > > > > > Thank you for raising the issue of this long-standing bottleneck, > this > > > > will be a very important improvement! > > > > > > > > Hopefully can participate and push forward together. > > > > > > > > Best Regards~ > > > > > > > > Brahma Reddy Battula <bra...@apache.org> 于2024年2月3日周六 00:40写道： > > > > > > > >> Thanks for bringing this and considering all the history around > this. > > > >> One of the outstanding bottleneck(global lock) from a long time. > > > >> > > > >> Hopefully we can push forward this time. > > > >> > > > >> > > > >> On Fri, Feb 2, 2024 at 12:23 PM Hui Fei <feihui.u...@gmail.com> > > wrote: > > > >> > > > >> > Thanks for driving this. It's very meaningful. The performance > > > >> improvement > > > >> > looks very good. > > > >> > > > > >> > Many users are facing the write performance issue. As far as I > know, > > > >> some > > > >> > companies already implemented the similar idea on their internal > > > >> branches. > > > >> > But the internal branch is very different from the community one. > So > > > >> it's > > > >> > very hard to be in sync with the community. If this improvement > can > > be > > > >> > involved in the community, that would be great to both end-user > and > > > the > > > >> > community. > > > >> > > > > >> > It is very worth doing. > > > >> > > > > >> > Zengqiang XU <zande...@apache.org> 于2024年2月2日周五 11:07写道： > > > >> > > > > >> > > Hi everyone > > > >> > > > > > >> > > I have started a discussion about NameNode Fine-grained Locking > to > > > >> > improve > > > >> > > performance of write operations in NameNode. > > > >> > > > > > >> > > I started this discussion again for serval main reasons: > > > >> > > 1. We have implemented it and gained nearly 7x performance > > > >> improvement in > > > >> > > our prod environment > > > >> > > 2. Many other companies made similar improvements based on their > > > >> internal > > > >> > > branch. > > > >> > > 3. This topic has been discussed for a long time, but still > > without > > > >> any > > > >> > > results. > > > >> > > > > > >> > > I hope we can push this important improvement in the community > so > > > that > > > >> > all > > > >> > > end-users can enjoy this significant improvement. > > > >> > > > > > >> > > I'd really appreciate you can join in and work with me to push > > this > > > >> > feature > > > >> > > forward. > > > >> > > > > > >> > > Thanks very much. > > > >> > > > > > >> > > Ticket: HDFS-17366 < > > > https://issues.apache.org/jira/browse/HDFS-17366> > > > >> > > Design: NameNode Fine-grained locking based on directory tree > > > >> > > < > > > >> > > > > > >> > > > > >> > > > > > > https://docs.google.com/document/d/1bVBQcI4jfzS0UrczB7UhsrQTXmrERGvBV-a9W3HCCjk/edit?usp=sharing > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > > > > > > >
