[
https://issues.apache.org/jira/browse/HDFS-17577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17866592#comment-17866592
]
ASF GitHub Bot commented on HDFS-17577:
---------------------------------------
liangyu-1 commented on PR #6935:
URL: https://github.com/apache/hadoop/pull/6935#issuecomment-2232224843
> Sorry for late response. And I didn't know why & what this PR want to do.
When I said 'What about to invoke the following interface and set flag to
CreateFlag.IGNORE_CLIENT_LOCALITY.', I means that the current implement also
support to skip localization when write data to HDFS. Please check again.
Thanks.
@Hexiaoqiao Thanks for your reply, I think I have understood your
suggestion. You mean that I can implement that interface and add the Flag in
that function.
But in my scenario, I am using flink's fileSystem API, and I have read the
source code of the flink API that it used `FileSystem.create(Path f) `, which
means that if I want to use the interface in hadoop, I have to change the
source code of flink filesystem API and rebuild the whole flink project. I
think this will also happens if someone's using Spark API or something else.
But in my pr, I can solve the problem by just adding the hadoop
configuration, this is more convenient.
> Add Support for CreateFlag.NO_LOCAL_WRITE in File Creation to Manage Disk
> Space and Network Load in Labeled YARN Nodes
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-17577
> URL: https://issues.apache.org/jira/browse/HDFS-17577
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: dfsclient
> Reporter: liang yu
> Priority: Major
> Labels: pull-request-available
> Attachments: 6D939050-0BC4-4B17-A6A3-A1EBBD60338D.png
>
>
> {*}Description{*}: I am currently using Apache Flink to write files into
> Hadoop. The Flink application runs on a labeled YARN queue. During operation,
> it has been observed that the local disks on these labeled nodes get filled
> up quickly, and the network load is significantly high. This issue arises
> because Hadoop prioritizes writing files to the local node first, and the
> number of these labeled nodes is quite limited.
>
> {*}Problem{*}: The current behavior leads to inefficient disk space
> utilization and high network traffic on these few labeled nodes, which could
> potentially affect the performance and reliability of the application. As
> shown in the picture, the host I circled have a average net_bytes_sent speed
> 1.2GB/s while the others are just 50MB/s, this imbalance in network and disk
> space nearly destroyed the whole cluster.
> !6D939050-0BC4-4B17-A6A3-A1EBBD60338D.png|width=901,height=257!
>
> {*}Implementation{*}: The implementation would involve adding an
> configuration _dfs.client.write.no_local_write_ to support the
> {{CreateFlag.NO_LOCAL_WRITE}} during the file creation process in Hadoop's
> file system APIs. This will provide flexibility to applications like Flink
> running in labeled queues to opt for non-local writes when necessary.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
