[ 
https://issues.apache.org/jira/browse/HDFS-17577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17866240#comment-17866240
 ] 

ASF GitHub Bot commented on HDFS-17577:
---------------------------------------

liangyu-1 commented on PR #6935:
URL: https://github.com/apache/hadoop/pull/6935#issuecomment-2230086391

   > @liangyu-1 Can we add a corresponding unit test for this? We need to fix 
the checkstyle issue.
   
   hi, @slfan1989, I have just add an Unit Test on the result of function 
DFSClient.getConf().getNoLocalWrite(), is it OK now?
   
   Thanks




> Add Support for CreateFlag.NO_LOCAL_WRITE in File Creation to Manage Disk 
> Space and Network Load in Labeled YARN Nodes
> ----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-17577
>                 URL: https://issues.apache.org/jira/browse/HDFS-17577
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: dfsclient
>            Reporter: liang yu
>            Priority: Major
>              Labels: pull-request-available
>         Attachments: 6D939050-0BC4-4B17-A6A3-A1EBBD60338D.png
>
>
> {*}Description{*}: I am currently using Apache Flink to write files into 
> Hadoop. The Flink application runs on a labeled YARN queue. During operation, 
> it has been observed that the local disks on these labeled nodes get filled 
> up quickly, and the network load is significantly high. This issue arises 
> because Hadoop prioritizes writing files to the local node first, and the 
> number of these labeled nodes is quite limited.
>  
> {*}Problem{*}: The current behavior leads to inefficient disk space 
> utilization and high network traffic on these few labeled nodes, which could 
> potentially affect the performance and reliability of the application. As 
> shown in the picture, the host I circled have a average net_bytes_sent speed 
> 1.2GB/s while the others are just 50MB/s, this imbalance in network and disk 
> space nearly destroyed the whole cluster. 
> !6D939050-0BC4-4B17-A6A3-A1EBBD60338D.png|width=901,height=257!
>  
> {*}Implementation{*}: The implementation would involve adding an 
> configuration _dfs.client.write.no_local_write_ to support the 
> {{CreateFlag.NO_LOCAL_WRITE}} during the file creation process in Hadoop's 
> file system APIs. This will provide flexibility to applications like Flink 
> running in labeled queues to opt for non-local writes when necessary.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

Reply via email to