Thank you for your hard work! This is a very meaningful improvement, and from the design document, we can see a significant increase in HDFS read/write throughput.
I am happy to see the progress made on HDFS-17384. However, I still have some concerns, which roughly involve the following aspects: 1. While ZanderXu and Hui Fei have deep expertise in HDFS and are familiar with related development details, we still need more community member to review the code to ensure that the relevant upgrades meet expectations. 2. We need more details on benchmarks to ensure that test results can be reproduced and to allow more community member to participate in the testing process. Looking forward to everything going smoothly in the future. Best Regards, - Shilun Fan. On Wed, Apr 24, 2024 at 3:51 PM Xiaoqiao He <hexiaoq...@apache.org> wrote: > cc private@h.a.o. > > On Wed, Apr 24, 2024 at 3:35 PM ZanderXu <zande...@apache.org> wrote: > > > > Here are some summaries about the first phase: > > 1. There are no big changes in this phase > > 2. This phase just uses FS lock and BM lock to replace the original > global > > lock > > 3. It's useful to improve the performance, since some operations just > need > > to hold FS lock or BM lock instead of the global lock > > 4. This feature is turned off by default, you can enable it by setting > > dfs.namenode.lock.model.provider.class to > > org.apache.hadoop.hdfs.server.namenode.fgl.FineGrainedFSNamesystemLock > > 5. This phase is very import for the ongoing development of the entire > FGL > > > > Here I would like to express my special thanks to @kokonguyen191 and > > @yuanboliu for their contributions. And you are also welcome to join us > > and complete it together. > > > > > > On Wed, 24 Apr 2024 at 14:54, ZanderXu <zande...@apache.org> wrote: > > > > > Hi everyone > > > > > > All subtasks of the first phase of the FGL have been completed and I > plan > > > to merge them into the trunk and start the second phase based on the > trunk. > > > > > > Here is the PR that used to merge the first phases into trunk: > > > https://github.com/apache/hadoop/pull/6762 > > > Here is the ticket: https://issues.apache.org/jira/browse/HDFS-17384 > > > > > > I hope you can help to review this PR when you are available and give > some > > > ideas. > > > > > > > > > HDFS-17385 <https://issues.apache.org/jira/browse/HDFS-17385> is used > for > > > the second phase and I have created some subtasks to describe > solutions for > > > some problems, such as: snapshot, getListing, quota. > > > You are welcome to join us to complete it together. > > > > > > > > > ---------- Forwarded message --------- > > > From: Zengqiang XU <zande...@apache.org> > > > Date: Fri, 2 Feb 2024 at 11:07 > > > Subject: Discussion about NameNode Fine-grained locking > > > To: <hdfs-dev@hadoop.apache.org> > > > Cc: Zengqiang XU <xuzengqiang5...@gmail.com> > > > > > > > > > Hi everyone > > > > > > I have started a discussion about NameNode Fine-grained Locking to > improve > > > performance of write operations in NameNode. > > > > > > I started this discussion again for serval main reasons: > > > 1. We have implemented it and gained nearly 7x performance improvement > in > > > our prod environment > > > 2. Many other companies made similar improvements based on their > internal > > > branch. > > > 3. This topic has been discussed for a long time, but still without any > > > results. > > > > > > I hope we can push this important improvement in the community so that > all > > > end-users can enjoy this significant improvement. > > > > > > I'd really appreciate you can join in and work with me to push this > > > feature forward. > > > > > > Thanks very much. > > > > > > Ticket: HDFS-17366 <https://issues.apache.org/jira/browse/HDFS-17366> > > > Design: NameNode Fine-grained locking based on directory tree > > > < > https://docs.google.com/document/d/1X499gHxT0WSU1fj8uo4RuF3GqKxWkWXznXx4tspTBLY/edit?usp=sharing > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: private-unsubscr...@hadoop.apache.org > For additional commands, e-mail: private-h...@hadoop.apache.org > >
