liangyu-1 opened a new pull request, #6935:
URL: https://github.com/apache/hadoop/pull/6935
… to Manage Disk Space and Network Load in Labeled YARN Nodes
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines:
https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g.,
'HADOOP-17799. Your PR title ...'.
-->
### Description of PR
As described in
[HDFS-17577](https://issues.apache.org/jira/browse/HDFS-17577)
I am currently using Apache Flink to write files into Hadoop. The Flink
application runs on a labeled YARN queue. During operation, it has been
observed that the local disks on these labeled nodes get filled up quickly, and
the network load is significantly high. This issue arises because Hadoop
prioritizes writing files to the local node first, and the number of these
labeled nodes is quite limited.
The current behavior leads to inefficient disk space utilization and high
network traffic on these few labeled nodes, which could potentially affect the
performance and reliability of the application. As shown in the picture, the
host I circled have a average net_bytes_sent speed 1.2GB/s while the others are
just 50MB/s, this imbalance in network and disk space nearly destroyed the
whole cluster.
**Implementation:**
I add an configuration `dfs.client.write.no_local_write` to support the
`CreateFlag.NO_LOCAL_WRITE` during the file creation process in Hadoop's file
system APIs. This will provide flexibility to applications like Flink running
in labeled queues to opt for non-local writes when necessary.
### How was this patch tested?
I have rebuilt the whole hadoop-hdfs-client module, and then test them using
flink on the labeled YARN queue, the distribution of disk storage across the
nodes in the cluster is more even, and the network load has also improved.
### For code changes:
- [ ] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the
endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
