[ https://issues.apache.org/jira/browse/HDFS-17577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17866240#comment-17866240 ]
ASF GitHub Bot commented on HDFS-17577: --------------------------------------- liangyu-1 commented on PR #6935: URL: https://github.com/apache/hadoop/pull/6935#issuecomment-2230086391 > @liangyu-1 Can we add a corresponding unit test for this? We need to fix the checkstyle issue. hi, @slfan1989, I have just add an Unit Test on the result of function DFSClient.getConf().getNoLocalWrite(), is it OK now? Thanks > Add Support for CreateFlag.NO_LOCAL_WRITE in File Creation to Manage Disk > Space and Network Load in Labeled YARN Nodes > ---------------------------------------------------------------------------------------------------------------------- > > Key: HDFS-17577 > URL: https://issues.apache.org/jira/browse/HDFS-17577 > Project: Hadoop HDFS > Issue Type: New Feature > Components: dfsclient > Reporter: liang yu > Priority: Major > Labels: pull-request-available > Attachments: 6D939050-0BC4-4B17-A6A3-A1EBBD60338D.png > > > {*}Description{*}: I am currently using Apache Flink to write files into > Hadoop. The Flink application runs on a labeled YARN queue. During operation, > it has been observed that the local disks on these labeled nodes get filled > up quickly, and the network load is significantly high. This issue arises > because Hadoop prioritizes writing files to the local node first, and the > number of these labeled nodes is quite limited. > > {*}Problem{*}: The current behavior leads to inefficient disk space > utilization and high network traffic on these few labeled nodes, which could > potentially affect the performance and reliability of the application. As > shown in the picture, the host I circled have a average net_bytes_sent speed > 1.2GB/s while the others are just 50MB/s, this imbalance in network and disk > space nearly destroyed the whole cluster. > !6D939050-0BC4-4B17-A6A3-A1EBBD60338D.png|width=901,height=257! > > {*}Implementation{*}: The implementation would involve adding an > configuration _dfs.client.write.no_local_write_ to support the > {{CreateFlag.NO_LOCAL_WRITE}} during the file creation process in Hadoop's > file system APIs. This will provide flexibility to applications like Flink > running in labeled queues to opt for non-local writes when necessary. > -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org