[
https://issues.apache.org/jira/browse/HDFS-17157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17755305#comment-17755305
]
ASF GitHub Bot commented on HDFS-17157:
---------------------------------------
hadoop-yetus commented on PR #5954:
URL: https://github.com/apache/hadoop/pull/5954#issuecomment-1681453746
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 0m 32s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 35m 11s | | trunk passed |
| +1 :green_heart: | compile | 0m 55s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 0m 48s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 44s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 56s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 52s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 7s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 2m 4s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 1s | | branch has no errors
when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 46s | | the patch passed |
| +1 :green_heart: | compile | 0m 47s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 0m 47s | | the patch passed |
| +1 :green_heart: | compile | 0m 43s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 0m 43s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 35s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5954/1/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 199 unchanged
- 0 fixed = 201 total (was 199) |
| +1 :green_heart: | mvnsite | 0m 45s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 38s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 1s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 56s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 16s | | patch has no errors
when building and testing our client artifacts. |
|||| _ Other Tests _ |
| -1 :x: | unit | 188m 15s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5954/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 39s | | The patch does not
generate ASF License warnings. |
| | | 286m 22s | | |
| Reason | Tests |
|-------:|:------|
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestObserverNode |
| | hadoop.tools.TestHdfsConfigFields |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5954/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5954 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 43ce060b5c99 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f5e36c6ef1fb896a617233c4ccd4084c9551bfd0 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5954/1/testReport/ |
| Max. process+thread count | 3808 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5954/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Transient network failure in lease recovery could be mitigated to ensure
> better consistency
> -------------------------------------------------------------------------------------------
>
> Key: HDFS-17157
> URL: https://issues.apache.org/jira/browse/HDFS-17157
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
> Affects Versions: 2.0.0-alpha, 3.3.6
> Reporter: Haoze Wu
> Priority: Major
> Labels: pull-request-available
>
> This case is related to HDFS-12070.
> In HDFS-12070, we saw how a faulty drive at a certain datanode could lead to
> permanent block recovery failure and leaves the file open indefinitely. In
> the patch, instead of failing the whole lease recovery process when the
> second stage of block recovery is failed at one datanode, the whole lease
> recovery process is failed if only these are failed for all the datanodes.
> Attached is the code snippet for the second stage of the block recovery, in
> BlockRecoveryWorker#syncBlock:
> {code:java}
> ...
> final List<BlockRecord> successList = new ArrayList<>();
> for (BlockRecord r : participatingList) {
> try {
> r.updateReplicaUnderRecovery(bpid, recoveryId, blockId,
> newBlock.getNumBytes());
> successList.add(r);
> } catch (IOException e) {
> ...{code}
> However, because of transient network failure, the RPC in
> updateReplicaUnderRecovery initiated from the primary datanode to another
> datanode could return an EOFException while the other side does not process
> the RPC at all or throw an IOException when reading from the socket.
> {code:java}
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:824)
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:788)
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1495)
> at org.apache.hadoop.ipc.Client.call(Client.java:1437)
> at org.apache.hadoop.ipc.Client.call(Client.java:1347)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:228)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> at com.sun.proxy.$Proxy29.updateReplicaUnderRecovery(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.InterDatanodeProtocolTranslatorPB.updateReplicaUnderRecovery(InterDatanodeProtocolTranslatorPB.java:112)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$BlockRecord.updateReplicaUnderRecovery(BlockRecoveryWorker.java:88)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$BlockRecord.access$700(BlockRecoveryWorker.java:71)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$RecoveryTaskContiguous.syncBlock(BlockRecoveryWorker.java:300)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$RecoveryTaskContiguous.recover(BlockRecoveryWorker.java:188)
> at
> org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1.run(BlockRecoveryWorker.java:606)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.io.EOFException
> at java.io.DataInputStream.readInt(DataInputStream.java:392)
> at
> org.apache.hadoop.ipc.Client$IpcStreams.readResponse(Client.java:1796)
> at
> org.apache.hadoop.ipc.Client$Connection.receiveRpcResponse(Client.java:1165)
> at org.apache.hadoop.ipc.Client$Connection.run(Client.java:1061)
> {code}
> Then if there is any other datanode in which the second stage of block
> recovery success, the lease recovery would be successful and close the file.
> However, the last block failed to be synced to that failed datanode and this
> inconsistency could potentially last for a very long time.
> To fix the issue, I propose adding a configurable retry of
> updateReplicaUnderRecovery RPC so that transient network failure could be
> mitigated.
> Any comments and suggestions would be appreciated.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
