[
https://issues.apache.org/jira/browse/HDFS-17577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17864517#comment-17864517
]
ASF GitHub Bot commented on HDFS-17577:
---------------------------------------
hadoop-yetus commented on PR #6935:
URL: https://github.com/apache/hadoop/pull/6935#issuecomment-2219786846
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 20s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
|||| _ trunk Compile Tests _ |
| -1 :x: | mvninstall | 2m 13s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6935/1/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| -1 :x: | compile | 0m 21s |
[/branch-compile-hadoop-hdfs-project_hadoop-hdfs-client-jdkUbuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6935/1/artifact/out/branch-compile-hadoop-hdfs-project_hadoop-hdfs-client-jdkUbuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2.txt)
| hadoop-hdfs-client in trunk failed with JDK
Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2. |
| -1 :x: | compile | 0m 23s |
[/branch-compile-hadoop-hdfs-project_hadoop-hdfs-client-jdkPrivateBuild-1.8.0_412-8u412-ga-1~20.04.1-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6935/1/artifact/out/branch-compile-hadoop-hdfs-project_hadoop-hdfs-client-jdkPrivateBuild-1.8.0_412-8u412-ga-1~20.04.1-b08.txt)
| hadoop-hdfs-client in trunk failed with JDK Private
Build-1.8.0_412-8u412-ga-1~20.04.1-b08. |
| -0 :warning: | checkstyle | 0m 21s |
[/buildtool-branch-checkstyle-hadoop-hdfs-project_hadoop-hdfs-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6935/1/artifact/out/buildtool-branch-checkstyle-hadoop-hdfs-project_hadoop-hdfs-client.txt)
| The patch fails to run checkstyle in hadoop-hdfs-client |
| -1 :x: | mvnsite | 2m 13s |
[/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6935/1/artifact/out/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs-client.txt)
| hadoop-hdfs-client in trunk failed. |
| -1 :x: | javadoc | 0m 33s |
[/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-client-jdkUbuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6935/1/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-client-jdkUbuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2.txt)
| hadoop-hdfs-client in trunk failed with JDK
Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2. |
| -1 :x: | javadoc | 0m 22s |
[/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-client-jdkPrivateBuild-1.8.0_412-8u412-ga-1~20.04.1-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6935/1/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-client-jdkPrivateBuild-1.8.0_412-8u412-ga-1~20.04.1-b08.txt)
| hadoop-hdfs-client in trunk failed with JDK Private
Build-1.8.0_412-8u412-ga-1~20.04.1-b08. |
| -1 :x: | spotbugs | 0m 22s |
[/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6935/1/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client.txt)
| hadoop-hdfs-client in trunk failed. |
| +1 :green_heart: | shadedclient | 4m 36s | | branch has no errors
when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 22s |
[/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6935/1/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs-client.txt)
| hadoop-hdfs-client in the patch failed. |
| -1 :x: | compile | 0m 22s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-client-jdkUbuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6935/1/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-client-jdkUbuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2.txt)
| hadoop-hdfs-client in the patch failed with JDK
Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2. |
| -1 :x: | javac | 0m 22s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-client-jdkUbuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6935/1/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-client-jdkUbuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2.txt)
| hadoop-hdfs-client in the patch failed with JDK
Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2. |
| -1 :x: | compile | 0m 22s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-client-jdkPrivateBuild-1.8.0_412-8u412-ga-1~20.04.1-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6935/1/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-client-jdkPrivateBuild-1.8.0_412-8u412-ga-1~20.04.1-b08.txt)
| hadoop-hdfs-client in the patch failed with JDK Private
Build-1.8.0_412-8u412-ga-1~20.04.1-b08. |
| -1 :x: | javac | 0m 22s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-client-jdkPrivateBuild-1.8.0_412-8u412-ga-1~20.04.1-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6935/1/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-client-jdkPrivateBuild-1.8.0_412-8u412-ga-1~20.04.1-b08.txt)
| hadoop-hdfs-client in the patch failed with JDK Private
Build-1.8.0_412-8u412-ga-1~20.04.1-b08. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 19s |
[/buildtool-patch-checkstyle-hadoop-hdfs-project_hadoop-hdfs-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6935/1/artifact/out/buildtool-patch-checkstyle-hadoop-hdfs-project_hadoop-hdfs-client.txt)
| The patch fails to run checkstyle in hadoop-hdfs-client |
| -1 :x: | mvnsite | 0m 22s |
[/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6935/1/artifact/out/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs-client.txt)
| hadoop-hdfs-client in the patch failed. |
| -1 :x: | javadoc | 0m 12s |
[/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-client-jdkUbuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6935/1/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-client-jdkUbuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2.txt)
| hadoop-hdfs-client in the patch failed with JDK
Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2. |
| -1 :x: | javadoc | 0m 23s |
[/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-client-jdkPrivateBuild-1.8.0_412-8u412-ga-1~20.04.1-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6935/1/artifact/out/patch-javadoc-hadoop-hdfs-project_hadoop-hdfs-client-jdkPrivateBuild-1.8.0_412-8u412-ga-1~20.04.1-b08.txt)
| hadoop-hdfs-client in the patch failed with JDK Private
Build-1.8.0_412-8u412-ga-1~20.04.1-b08. |
| -1 :x: | spotbugs | 0m 22s |
[/patch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6935/1/artifact/out/patch-spotbugs-hadoop-hdfs-project_hadoop-hdfs-client.txt)
| hadoop-hdfs-client in the patch failed. |
| +1 :green_heart: | shadedclient | 3m 33s | | patch has no errors
when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 0m 22s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6935/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-client.txt)
| hadoop-hdfs-client in the patch failed. |
| +0 :ok: | asflicense | 0m 18s | | ASF License check generated no
output? |
| | | 14m 27s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.46 ServerAPI=1.46 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6935/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6935 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux a459709db3d4 5.15.0-106-generic #116-Ubuntu SMP Wed Apr 17
09:17:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / de67a948d21a90ad3fc30fc845c2db76293f467c |
| Default Java | Private Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.23+9-post-Ubuntu-1ubuntu120.04.2
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_412-8u412-ga-1~20.04.1-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6935/1/testReport/ |
| Max. process+thread count | 51 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client U:
hadoop-hdfs-project/hadoop-hdfs-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6935/1/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Add Support for CreateFlag.NO_LOCAL_WRITE in File Creation to Manage Disk
> Space and Network Load in Labeled YARN Nodes
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-17577
> URL: https://issues.apache.org/jira/browse/HDFS-17577
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: dfsclient
> Reporter: liang yu
> Priority: Major
> Labels: pull-request-available
> Attachments: 6D939050-0BC4-4B17-A6A3-A1EBBD60338D.png
>
>
> {*}Description{*}: I am currently using Apache Flink to write files into
> Hadoop. The Flink application runs on a labeled YARN queue. During operation,
> it has been observed that the local disks on these labeled nodes get filled
> up quickly, and the network load is significantly high. This issue arises
> because Hadoop prioritizes writing files to the local node first, and the
> number of these labeled nodes is quite limited.
>
> {*}Problem{*}: The current behavior leads to inefficient disk space
> utilization and high network traffic on these few labeled nodes, which could
> potentially affect the performance and reliability of the application. As
> shown in the picture, the host I circled have a average net_bytes_sent speed
> 1.2GB/s while the others are just 50MB/s, this imbalance in network and disk
> space nearly destroyed the whole cluster.
> !6D939050-0BC4-4B17-A6A3-A1EBBD60338D.png|width=901,height=257!
>
> {*}Implementation{*}: The implementation would involve adding an
> configuration _dfs.client.write.no_local_write_ to support the
> {{CreateFlag.NO_LOCAL_WRITE}} during the file creation process in Hadoop's
> file system APIs. This will provide flexibility to applications like Flink
> running in labeled queues to opt for non-local writes when necessary.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
