[
https://issues.apache.org/jira/browse/FLINK-36112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xintong Song updated FLINK-36112:
---------------------------------
Component/s: Connectors / FileSystem
> Add Support for CreateFlag.NO_LOCAL_WRITE in FLINK on YARN's File Creation to
> Manage Disk Space and Network Load in Labeled YARN Nodes
> --------------------------------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-36112
> URL: https://issues.apache.org/jira/browse/FLINK-36112
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / FileSystem
> Reporter: liang yu
> Assignee: liang yu
> Priority: Major
> Labels: pull-request-available
> Fix For: 2.0-preview
>
> Attachments: image-2024-08-20-18-51-11-864.png
>
>
> {*}Description{*}: I am currently using Apache Flink to write files into
> Hadoop. The Flink application runs on a labeled YARN queue. During operation,
> it has been observed that the local disks on these labeled nodes get filled
> up quickly, and the network load is significantly high. This issue arises
> because Hadoop prioritizes writing files to the local node first, and the
> number of these labeled nodes is quite limited.
>
> {*}Problem{*}: The current behavior leads to inefficient disk space
> utilization and high network traffic on these few labeled nodes, which could
> potentially affect the performance and reliability of the application. As
> shown in the picture, the host I circled have a average net_bytes_sent speed
> 1.2GB/s while the others are just 50MB/s, this imbalance in network and disk
> space nearly destroyed the whole cluster.
>
> !image-2024-08-20-18-51-11-864.png|width=1056,height=301!
>
> {*}Implementation{*}: The implementation would involve adding a method of
> FileSystem.class to support the {{CreateFlag.NO_LOCAL_WRITE}} when we try to
> create a new file through HadoopFileSystem.create() API. What's more, I
> modify the code of FileSink class so that we can choose to enable
> no_local_write or disable this feature. This will provide flexibility to
> Flink running in labeled Yarn queues to opt for non-local writes when
> necessary.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
