[
https://issues.apache.org/jira/browse/HDFS-17868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18048730#comment-18048730
]
ASF GitHub Bot commented on HDFS-17868:
---------------------------------------
hadoop-yetus commented on PR #8159:
URL: https://github.com/apache/hadoop/pull/8159#issuecomment-3705205323
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 41s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 22m 8s | | trunk passed |
| +1 :green_heart: | compile | 0m 44s | | trunk passed with JDK
Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 50s | | trunk passed with JDK
Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | checkstyle | 0m 31s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 50s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 39s | | trunk passed with JDK
Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 39s | | trunk passed with JDK
Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | spotbugs | 1m 57s | | trunk passed |
| +1 :green_heart: | shadedclient | 15m 48s | | branch has no errors
when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 46s | | the patch passed |
| +1 :green_heart: | compile | 0m 41s | | the patch passed with JDK
Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 41s | | the patch passed |
| +1 :green_heart: | compile | 0m 43s | | the patch passed with JDK
Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 43s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 24s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8159/1/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 41 new + 271 unchanged
- 0 fixed = 312 total (was 271) |
| +1 :green_heart: | mvnsite | 0m 46s | | the patch passed |
| -1 :x: | javadoc | 0m 32s |
[/results-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8159/1/artifact/out/results-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04.txt)
| hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-21.0.7+6-Ubuntu-0ubuntu120.04
with JDK Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04 generated 18 new + 9982 unchanged
- 18 fixed = 10000 total (was 10000) |
| -1 :x: | javadoc | 0m 33s |
[/results-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8159/1/artifact/out/results-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04.txt)
| hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-17.0.15+6-Ubuntu-0ubuntu120.04
with JDK Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 generated 16 new + 9676
unchanged - 0 fixed = 9692 total (was 9676) |
| +1 :green_heart: | spotbugs | 1m 58s | | the patch passed |
| +1 :green_heart: | shadedclient | 15m 23s | | patch has no errors
when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 173m 22s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8159/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 27s | | The patch does not
generate ASF License warnings. |
| | | 240m 11s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.tools.TestDFSAdmin |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.52 ServerAPI=1.52 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8159/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/8159 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle |
| uname | Linux 6ad28467dbab 5.15.0-164-generic #174-Ubuntu SMP Fri Nov 14
20:25:16 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 5987cdaea07d1a4b6ff7c9ddd64afeacb54300ce |
| Default Java | Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Multi-JDK versions |
/usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.7+6-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.15+6-Ubuntu-0ubuntu120.04 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8159/1/testReport/ |
| Max. process+thread count | 4373 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8159/1/console |
| versions | git=2.25.1 maven=3.9.11 spotbugs=4.9.7 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> introduce BlockPlacementPolicyCrossDC for multi datacenter stretched hdfs
> cluster
> ---------------------------------------------------------------------------------
>
> Key: HDFS-17868
> URL: https://issues.apache.org/jira/browse/HDFS-17868
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: block placement
> Reporter: YUBI LEE
> Priority: Major
> Labels: pull-request-available
>
> I was inspired by the ideas presented in
> https://dan.naver.com/25/sessions/692 and
> https://www.youtube.com/watch?v=1h4k_Dbt0t8 and implemented a new block
> placement policy called BlockPlacementPolicyCrossDC.
> Thanks to [~acedia28] for the original ideas. It would be great if
> [~acedia28] could share an improved or more mature version of this block
> placement policy.
> This implementation introduces the following configuration options
> (default values are shown in parentheses):
> {code}
> dfs.block.replicator.cross.dc.async.enabled (false)
> dfs.block.replicator.cross.dc.preferred.datacenter
> dfs.block.replicator.cross.dc.bandwidth.limit.mb (5120)
> dfs.block.replicator.cross.dc.bandwidth.refill.period.sec (1)
> dfs.block.replicator.cross.dc.sync.paths
> dfs.block.replicator.cross.dc.limited.sync.paths
> {code}
> BlockPlacementPolicyCrossDC assumes that DataNodes are registered with
> two-level rack topology information, such as /dc1/rack1 or /dc2/rack2.
> Based on the ideas from the session mentioned above, this policy supports
> three different HDFS block write modes:
> 1. Synchronous write
> The standard HDFS behavior, where block replicas are synchronously written to
> all target DataNodes.
> 2. Limited synchronous write
> Uses bucket4j to control cross-datacenter traffic.
> Writes below the configured threshold are performed synchronously, while
> writes exceeding the threshold fall back to asynchronous replication.
> 3. Asynchronous write
> The client is initially given only DataNode candidates located in the same
> datacenter.
> With asynchronous write, the hdfs client app will not suffer from write
> latency caused by cross-datacenter communication.
> Any under-replicas are created later through background HDFS replication to
> other datacenters.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
