[jira] [Commented] (HDFS-13522) RBF: Support observer node from Router-Based Federation
[
https://issues.apache.org/jira/browse/HDFS-13522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17522059#comment-17522059
]
Simbarashe Dzinamarira commented on HDFS-13522:
---
[~Song Jiacheng] you don't need to change clients. However, that will mean
observer reads cannot be disabled.
> RBF: Support observer node from Router-Based Federation
> ---
>
> Key: HDFS-13522
> URL: https://issues.apache.org/jira/browse/HDFS-13522
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: federation, namenode
>Reporter: Erik Krogen
>Assignee: Simbarashe Dzinamarira
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-13522.001.patch, HDFS-13522.002.patch,
> HDFS-13522_WIP.patch, RBF_ Observer support.pdf, Router+Observer RPC
> clogging.png, ShortTerm-Routers+Observer.png
>
> Time Spent: 6.5h
> Remaining Estimate: 0h
>
> Changes will need to occur to the router to support the new observer node.
> One such change will be to make the router understand the observer state,
> e.g. {{FederationNamenodeServiceState}}.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16540) Data locality is lost when DataNode pod restarts in kubernetes
[
https://issues.apache.org/jira/browse/HDFS-16540?focusedWorklogId=756827&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-756827
]
ASF GitHub Bot logged work on HDFS-16540:
-
Author: ASF GitHub Bot
Created on: 14/Apr/22 03:46
Start Date: 14/Apr/22 03:46
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on PR #4170:
URL: https://github.com/apache/hadoop/pull/4170#issuecomment-1098680455
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 17m 3s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 35s | | trunk passed |
| +1 :green_heart: | compile | 1m 31s | | trunk passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 20s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 5s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 31s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 7s | | trunk passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 31s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 40s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 14s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 18s | | the patch passed |
| +1 :green_heart: | compile | 1m 25s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 1m 25s | | the patch passed |
| +1 :green_heart: | compile | 1m 15s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 15s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 53s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 20s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 54s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 26s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 32s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 6s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 341m 40s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 41s | | The patch does not
generate ASF License warnings. |
| | | 474m 5s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4170/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4170 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 98cf76256475 4.15.0-162-generic #170-Ubuntu SMP Mon Oct 18
11:38:05 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1316ff0eada1e29dec8ca56ab266c9bcbe60051c |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4170/1/testReport/ |
| Max. process+thread count | 2175 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4170/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This
[jira] [Work logged] (HDFS-16538) EC decoding failed due to not enough valid inputs
[
https://issues.apache.org/jira/browse/HDFS-16538?focusedWorklogId=756826&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-756826
]
ASF GitHub Bot logged work on HDFS-16538:
-
Author: ASF GitHub Bot
Created on: 14/Apr/22 03:40
Start Date: 14/Apr/22 03:40
Worklog Time Spent: 10m
Work Description: liubingxing commented on PR #4167:
URL: https://github.com/apache/hadoop/pull/4167#issuecomment-1098678251
I will add a UT later
Issue Time Tracking
---
Worklog Id: (was: 756826)
Time Spent: 40m (was: 0.5h)
> EC decoding failed due to not enough valid inputs
> --
>
> Key: HDFS-16538
> URL: https://issues.apache.org/jira/browse/HDFS-16538
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: qinyuren
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Currently, we found this error if the #StripeReader.readStripe() have more
> than one block read failed.
> We use the EC policy ec(6+3) in our cluster.
> {code:java}
> Caused by: org.apache.hadoop.HadoopIllegalArgumentException: No enough valid
> inputs are provided, not recoverable
> at
> org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.checkInputBuffers(ByteBufferDecodingState.java:119)
> at
> org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.(ByteBufferDecodingState.java:47)
> at
> org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:86)
> at
> org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:170)
> at
> org.apache.hadoop.hdfs.StripeReader.decodeAndFillBuffer(StripeReader.java:462)
> at
> org.apache.hadoop.hdfs.StatefulStripeReader.decode(StatefulStripeReader.java:94)
> at
> org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:406)
> at
> org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:327)
> at
> org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:420)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:892)
> at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
> at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
> {code}
>
>
> {code:java}
> while (!futures.isEmpty()) {
> try {
> StripingChunkReadResult r = StripedBlockUtil
> .getNextCompletedStripedRead(service, futures, 0);
> dfsStripedInputStream.updateReadStats(r.getReadStats());
> DFSClient.LOG.debug("Read task returned: {}, for stripe {}",
> r, alignedStripe);
> StripingChunk returnedChunk = alignedStripe.chunks[r.index];
> Preconditions.checkNotNull(returnedChunk);
> Preconditions.checkState(returnedChunk.state == StripingChunk.PENDING);
> if (r.state == StripingChunkReadResult.SUCCESSFUL) {
> returnedChunk.state = StripingChunk.FETCHED;
> alignedStripe.fetchedChunksNum++;
> updateState4SuccessRead(r);
> if (alignedStripe.fetchedChunksNum == dataBlkNum) {
> clearFutures();
> break;
> }
> } else {
> returnedChunk.state = StripingChunk.MISSING;
> // close the corresponding reader
> dfsStripedInputStream.closeReader(readerInfos[r.index]);
> final int missing = alignedStripe.missingChunksNum;
> alignedStripe.missingChunksNum++;
> checkMissingBlocks();
> readDataForDecoding();
> readParityChunks(alignedStripe.missingChunksNum - missing);
> } {code}
> If there are two blocks read failed, the #readDataForDecoding() will be
> called twice;
> The *decodeInputs array* will be initialized twice, and the *parity* *data*
> in decodeInputs array which filled by #readParityChunks will be set to null.
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16509) Fix decommission UnsupportedOperationException: Remove unsupported
[ https://issues.apache.org/jira/browse/HDFS-16509?focusedWorklogId=756825&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-756825 ] ASF GitHub Bot logged work on HDFS-16509: - Author: ASF GitHub Bot Created on: 14/Apr/22 03:38 Start Date: 14/Apr/22 03:38 Worklog Time Spent: 10m Work Description: cndaimin commented on PR #4077: URL: https://github.com/apache/hadoop/pull/4077#issuecomment-1098677337 @Hexiaoqiao I think it would be better to backport. Thanks @Hexiaoqiao @jojochuang @tomscut Issue Time Tracking --- Worklog Id: (was: 756825) Time Spent: 3h (was: 2h 50m) > Fix decommission UnsupportedOperationException: Remove unsupported > -- > > Key: HDFS-16509 > URL: https://issues.apache.org/jira/browse/HDFS-16509 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 3.3.1, 3.3.2 >Reporter: daimin >Assignee: daimin >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 3h > Remaining Estimate: 0h > > We encountered an "UnsupportedOperationException: Remove unsupported" error > when some datanodes were in decommission. The reason of the exception is that > datanode.getBlockIterator() returns an Iterator does not support remove, > however DatanodeAdminDefaultMonitor#processBlocksInternal invokes it.remove() > when a block not found, e.g, the file containing the block is deleted. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16535) SlotReleaser should reuse the domain socket based on socket paths
[ https://issues.apache.org/jira/browse/HDFS-16535?focusedWorklogId=756822&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-756822 ] ASF GitHub Bot logged work on HDFS-16535: - Author: ASF GitHub Bot Created on: 14/Apr/22 03:27 Start Date: 14/Apr/22 03:27 Worklog Time Spent: 10m Work Description: leosunli commented on PR #4158: URL: https://github.com/apache/hadoop/pull/4158#issuecomment-1098672455 @jojochuang thank for your reminder, I will review this PR. Issue Time Tracking --- Worklog Id: (was: 756822) Time Spent: 40m (was: 0.5h) > SlotReleaser should reuse the domain socket based on socket paths > - > > Key: HDFS-16535 > URL: https://issues.apache.org/jira/browse/HDFS-16535 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 3.3.1, 3.4.0 >Reporter: Quanlong Huang >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > HDFS-13639 improves the performance of short-circuit shm slot releasing by > reusing the domain socket that the client previously used to send release > request to the DataNode. > This is good when there are only one DataNode locates with the client (truth > in most of the production environment). However, if we launch multiple > DataNodes on a machine (usually for testing, e.g. Impala's end-to-end tests), > the request could be sent to the wrong DataNode. See an example in > IMPALA-11234. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16509) Fix decommission UnsupportedOperationException: Remove unsupported
[ https://issues.apache.org/jira/browse/HDFS-16509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoqiao He resolved HDFS-16509. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed Committed to trunk. Thanks [~cndaimin] for your contributions. > Fix decommission UnsupportedOperationException: Remove unsupported > -- > > Key: HDFS-16509 > URL: https://issues.apache.org/jira/browse/HDFS-16509 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 3.3.1, 3.3.2 >Reporter: daimin >Assignee: daimin >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 2h 50m > Remaining Estimate: 0h > > We encountered an "UnsupportedOperationException: Remove unsupported" error > when some datanodes were in decommission. The reason of the exception is that > datanode.getBlockIterator() returns an Iterator does not support remove, > however DatanodeAdminDefaultMonitor#processBlocksInternal invokes it.remove() > when a block not found, e.g, the file containing the block is deleted. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-13522) RBF: Support observer node from Router-Based Federation
[
https://issues.apache.org/jira/browse/HDFS-13522?focusedWorklogId=756818&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-756818
]
ASF GitHub Bot logged work on HDFS-13522:
-
Author: ASF GitHub Bot
Created on: 14/Apr/22 03:10
Start Date: 14/Apr/22 03:10
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on PR #4127:
URL: https://github.com/apache/hadoop/pull/4127#issuecomment-1098665361
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 50s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 12 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 16m 14s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 26s | | trunk passed |
| +1 :green_heart: | compile | 24m 50s | | trunk passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 21m 14s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 4s | | trunk passed |
| +1 :green_heart: | mvnsite | 5m 12s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 56s | | trunk passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 5m 9s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 10m 20s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 23s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 40s | | the patch passed |
| +1 :green_heart: | compile | 24m 14s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 24m 14s | | the patch passed |
| +1 :green_heart: | compile | 26m 20s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 26m 20s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 4m 2s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4127/6/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 6 new + 340 unchanged - 1 fixed = 346 total (was
341) |
| +1 :green_heart: | mvnsite | 5m 22s | | the patch passed |
| +1 :green_heart: | xml | 0m 3s | | The patch has no ill-formed XML
file. |
| +1 :green_heart: | javadoc | 3m 52s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 5m 12s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 10m 55s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 39s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 17m 46s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 2m 29s | | hadoop-hdfs-client in the patch
passed. |
| +1 :green_heart: | unit | 333m 37s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | unit | 32m 58s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 1s | | The patch does not
generate ASF License warnings. |
| | | 643m 58s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4127/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4127 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell xml |
| uname | Linux 13de7b65785b 4.15.0-163-generic #171-Ubuntu SMP Fri Nov 5
11:55:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / fdea891212f0c2d57b950c87695b62964123d219 |
| Default Java | Private Build-1.8.0_312
[jira] [Work logged] (HDFS-16509) Fix decommission UnsupportedOperationException: Remove unsupported
[ https://issues.apache.org/jira/browse/HDFS-16509?focusedWorklogId=756817&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-756817 ] ASF GitHub Bot logged work on HDFS-16509: - Author: ASF GitHub Bot Created on: 14/Apr/22 03:10 Start Date: 14/Apr/22 03:10 Worklog Time Spent: 10m Work Description: Hexiaoqiao commented on PR #4077: URL: https://github.com/apache/hadoop/pull/4077#issuecomment-1098665168 Committed to trunk. Thanks @cndaimin for your contributions. Thanks @jojochuang @tomscut for your reviews. BTW, @cndaimin would you mind to check if we need to backport to branch-3.3/branch-3.2, Thanks. Issue Time Tracking --- Worklog Id: (was: 756817) Time Spent: 2h 50m (was: 2h 40m) > Fix decommission UnsupportedOperationException: Remove unsupported > -- > > Key: HDFS-16509 > URL: https://issues.apache.org/jira/browse/HDFS-16509 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 3.3.1, 3.3.2 >Reporter: daimin >Assignee: daimin >Priority: Major > Labels: pull-request-available > Time Spent: 2h 50m > Remaining Estimate: 0h > > We encountered an "UnsupportedOperationException: Remove unsupported" error > when some datanodes were in decommission. The reason of the exception is that > datanode.getBlockIterator() returns an Iterator does not support remove, > however DatanodeAdminDefaultMonitor#processBlocksInternal invokes it.remove() > when a block not found, e.g, the file containing the block is deleted. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-16479) EC: NameNode should not send a reconstruction work when the source datanodes are insufficient
[
https://issues.apache.org/jira/browse/HDFS-16479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takanobu Asanuma resolved HDFS-16479.
-
Fix Version/s: 3.4.0
3.2.4
3.3.4
Assignee: Takanobu Asanuma
Resolution: Fixed
Resolved. Thanks for reporting the issue, [~yuanbo].
> EC: NameNode should not send a reconstruction work when the source datanodes
> are insufficient
> -
>
> Key: HDFS-16479
> URL: https://issues.apache.org/jira/browse/HDFS-16479
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ec, erasure-coding
>Reporter: Yuanbo Liu
>Assignee: Takanobu Asanuma
>Priority: Critical
> Labels: pull-request-available
> Fix For: 3.4.0, 3.2.4, 3.3.4
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> We got this exception from DataNodes
> {color:#707070}java.lang.IllegalArgumentException: No enough live striped
> blocks.{color}
> {color:#707070} at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:141){color}
> {color:#707070} at
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedReader.(StripedReader.java:128){color}
> {color:#707070} at
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedReconstructor.(StripedReconstructor.java:135){color}
> {color:#707070} at
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedBlockReconstructor.(StripedBlockReconstructor.java:41){color}
> {color:#707070} at
> org.apache.hadoop.hdfs.server.datanode.erasurecode.ErasureCodingWorker.processErasureCodingTasks(ErasureCodingWorker.java:133){color}
> {color:#707070} at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActive(BPOfferService.java:796){color}
> {color:#707070} at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:680){color}
> {color:#707070} at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor$CommandProcessingThread.processCommand(BPServiceActor.java:1314){color}
> {color:#707070} at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor$CommandProcessingThread.lambda$enqueue$2(BPServiceActor.java:1360){color}
> {color:#707070} at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor$CommandProcessingThread.processQueue(BPServiceActor.java:1287){color}
> After going through the code of ErasureCodingWork.java, we found
> {code:java}
> targets[0].getDatanodeDescriptor().addBlockToBeErasureCoded( new
> ExtendedBlock(blockPoolId, stripedBlk), getSrcNodes(), targets,
> getLiveBlockIndicies(), stripedBlk.getErasureCodingPolicy());
> {code}
>
> the liveBusyBlockIndicies is not considered as liveBlockIndicies, hence
> erasure coding reconstruction sometimes will fail as 'No enough live striped
> blocks'.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16509) Fix decommission UnsupportedOperationException: Remove unsupported
[ https://issues.apache.org/jira/browse/HDFS-16509?focusedWorklogId=756816&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-756816 ] ASF GitHub Bot logged work on HDFS-16509: - Author: ASF GitHub Bot Created on: 14/Apr/22 03:07 Start Date: 14/Apr/22 03:07 Worklog Time Spent: 10m Work Description: Hexiaoqiao merged PR #4077: URL: https://github.com/apache/hadoop/pull/4077 Issue Time Tracking --- Worklog Id: (was: 756816) Time Spent: 2h 40m (was: 2.5h) > Fix decommission UnsupportedOperationException: Remove unsupported > -- > > Key: HDFS-16509 > URL: https://issues.apache.org/jira/browse/HDFS-16509 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 3.3.1, 3.3.2 >Reporter: daimin >Assignee: daimin >Priority: Major > Labels: pull-request-available > Time Spent: 2h 40m > Remaining Estimate: 0h > > We encountered an "UnsupportedOperationException: Remove unsupported" error > when some datanodes were in decommission. The reason of the exception is that > datanode.getBlockIterator() returns an Iterator does not support remove, > however DatanodeAdminDefaultMonitor#processBlocksInternal invokes it.remove() > when a block not found, e.g, the file containing the block is deleted. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16525) System.err should be used when error occurs in multiple methods in DFSAdmin class
[
https://issues.apache.org/jira/browse/HDFS-16525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17522032#comment-17522032
]
yanbin.zhang commented on HDFS-16525:
-
[~weichiu] [~ferhui] Could you please give some comments or suggestions.
> System.err should be used when error occurs in multiple methods in DFSAdmin
> class
> -
>
> Key: HDFS-16525
> URL: https://issues.apache.org/jira/browse/HDFS-16525
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: dfsadmin
>Affects Versions: 3.3.2
>Reporter: yanbin.zhang
>Assignee: yanbin.zhang
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> System.err should be used when error occurs in multiple methods in DFSAdmin
> class,as follows:
> {code:java}
> //DFSAdmin#refreshCallQueue
> ...
> try{
> proxy.getProxy().refreshCallQueue();
> System.out.println("Refresh call queue successful for "
> + proxy.getAddress());
> }catch (IOException ioe){
> System.out.println("Refresh call queue failed for "
> + proxy.getAddress());
> exceptions.add(ioe);
> }
> ...
> {code}
> The test method closed first in TestDFSAdminWithHA also needs to be
> modified,otherwise an error will be reported,similar to the following:
> {code:java}
> [ERROR] Failures:
> [ERROR]
> TestDFSAdminWithHA.testRefreshCallQueueNN1DownNN2Up:726->assertOutputMatches:77
> Expected output to match 'Refresh call queue failed for.*
> Refresh call queue successful for.*
> ' but err_output was:
> Refresh call queue failed for localhost/127.0.0.1:12876
> refreshCallQueue: Call From h110/10.1.234.110 to localhost:12876 failed on
> connection exception: java.net.ConnectException: Connection refused; For more
> details see: http://wiki.apache.org/hadoop/ConnectionRefused and output was:
> Refresh call queue successful for localhost/127.0.0.1:12878{code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16450) Give priority to releasing DNs with less free space
[
https://issues.apache.org/jira/browse/HDFS-16450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17522029#comment-17522029
]
yanbin.zhang commented on HDFS-16450:
-
OK, I'll try to do it.
> Give priority to releasing DNs with less free space
> ---
>
> Key: HDFS-16450
> URL: https://issues.apache.org/jira/browse/HDFS-16450
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.3.0
>Reporter: yanbin.zhang
>Assignee: yanbin.zhang
>Priority: Major
> Attachments: HDFS-16450.001.patch
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> When deleting redundant replicas, the one with the least free space should be
> prioritized.
> {code:java}
> //BlockPlacementPolicyDefault#chooseReplicaToDelete
> final DatanodeStorageInfo storage;
> if (oldestHeartbeatStorage != null) {
> storage = oldestHeartbeatStorage;
> } else if (minSpaceStorage != null) {
> storage = minSpaceStorage;
> } else {
> return null;
> }
> excessTypes.remove(storage.getStorageType());
> return storage; {code}
> Change the above logic to the following:
> {code:java}
> //BlockPlacementPolicyDefault#chooseReplicaToDelete
> final DatanodeStorageInfo storage;
> if (minSpaceStorage != null) {
> storage = minSpaceStorage;
> } else if (oldestHeartbeatStorage != null) {
> storage = oldestHeartbeatStorage;
> } else {
> return null;
> } {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16479) EC: NameNode should not send a reconstruction work when the source datanodes are insufficient
[
https://issues.apache.org/jira/browse/HDFS-16479?focusedWorklogId=756809&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-756809
]
ASF GitHub Bot logged work on HDFS-16479:
-
Author: ASF GitHub Bot
Created on: 14/Apr/22 02:23
Start Date: 14/Apr/22 02:23
Worklog Time Spent: 10m
Work Description: tasanuma merged PR #4138:
URL: https://github.com/apache/hadoop/pull/4138
Issue Time Tracking
---
Worklog Id: (was: 756809)
Time Spent: 2h (was: 1h 50m)
> EC: NameNode should not send a reconstruction work when the source datanodes
> are insufficient
> -
>
> Key: HDFS-16479
> URL: https://issues.apache.org/jira/browse/HDFS-16479
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ec, erasure-coding
>Reporter: Yuanbo Liu
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> We got this exception from DataNodes
> {color:#707070}java.lang.IllegalArgumentException: No enough live striped
> blocks.{color}
> {color:#707070} at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:141){color}
> {color:#707070} at
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedReader.(StripedReader.java:128){color}
> {color:#707070} at
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedReconstructor.(StripedReconstructor.java:135){color}
> {color:#707070} at
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedBlockReconstructor.(StripedBlockReconstructor.java:41){color}
> {color:#707070} at
> org.apache.hadoop.hdfs.server.datanode.erasurecode.ErasureCodingWorker.processErasureCodingTasks(ErasureCodingWorker.java:133){color}
> {color:#707070} at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActive(BPOfferService.java:796){color}
> {color:#707070} at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:680){color}
> {color:#707070} at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor$CommandProcessingThread.processCommand(BPServiceActor.java:1314){color}
> {color:#707070} at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor$CommandProcessingThread.lambda$enqueue$2(BPServiceActor.java:1360){color}
> {color:#707070} at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor$CommandProcessingThread.processQueue(BPServiceActor.java:1287){color}
> After going through the code of ErasureCodingWork.java, we found
> {code:java}
> targets[0].getDatanodeDescriptor().addBlockToBeErasureCoded( new
> ExtendedBlock(blockPoolId, stripedBlk), getSrcNodes(), targets,
> getLiveBlockIndicies(), stripedBlk.getErasureCodingPolicy());
> {code}
>
> the liveBusyBlockIndicies is not considered as liveBlockIndicies, hence
> erasure coding reconstruction sometimes will fail as 'No enough live striped
> blocks'.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16479) EC: NameNode should not send a reconstruction work when the source datanodes are insufficient
[
https://issues.apache.org/jira/browse/HDFS-16479?focusedWorklogId=756808&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-756808
]
ASF GitHub Bot logged work on HDFS-16479:
-
Author: ASF GitHub Bot
Created on: 14/Apr/22 02:23
Start Date: 14/Apr/22 02:23
Worklog Time Spent: 10m
Work Description: tasanuma commented on PR #4138:
URL: https://github.com/apache/hadoop/pull/4138#issuecomment-1098645791
@ayushtkn Thanks for your review! I'll merge it.
Issue Time Tracking
---
Worklog Id: (was: 756808)
Time Spent: 1h 50m (was: 1h 40m)
> EC: NameNode should not send a reconstruction work when the source datanodes
> are insufficient
> -
>
> Key: HDFS-16479
> URL: https://issues.apache.org/jira/browse/HDFS-16479
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ec, erasure-coding
>Reporter: Yuanbo Liu
>Priority: Critical
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> We got this exception from DataNodes
> {color:#707070}java.lang.IllegalArgumentException: No enough live striped
> blocks.{color}
> {color:#707070} at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:141){color}
> {color:#707070} at
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedReader.(StripedReader.java:128){color}
> {color:#707070} at
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedReconstructor.(StripedReconstructor.java:135){color}
> {color:#707070} at
> org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedBlockReconstructor.(StripedBlockReconstructor.java:41){color}
> {color:#707070} at
> org.apache.hadoop.hdfs.server.datanode.erasurecode.ErasureCodingWorker.processErasureCodingTasks(ErasureCodingWorker.java:133){color}
> {color:#707070} at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActive(BPOfferService.java:796){color}
> {color:#707070} at
> org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:680){color}
> {color:#707070} at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor$CommandProcessingThread.processCommand(BPServiceActor.java:1314){color}
> {color:#707070} at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor$CommandProcessingThread.lambda$enqueue$2(BPServiceActor.java:1360){color}
> {color:#707070} at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor$CommandProcessingThread.processQueue(BPServiceActor.java:1287){color}
> After going through the code of ErasureCodingWork.java, we found
> {code:java}
> targets[0].getDatanodeDescriptor().addBlockToBeErasureCoded( new
> ExtendedBlock(blockPoolId, stripedBlk), getSrcNodes(), targets,
> getLiveBlockIndicies(), stripedBlk.getErasureCodingPolicy());
> {code}
>
> the liveBusyBlockIndicies is not considered as liveBlockIndicies, hence
> erasure coding reconstruction sometimes will fail as 'No enough live striped
> blocks'.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16535) SlotReleaser should reuse the domain socket based on socket paths
[ https://issues.apache.org/jira/browse/HDFS-16535?focusedWorklogId=756807&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-756807 ] ASF GitHub Bot logged work on HDFS-16535: - Author: ASF GitHub Bot Created on: 14/Apr/22 02:20 Start Date: 14/Apr/22 02:20 Worklog Time Spent: 10m Work Description: jojochuang commented on PR #4158: URL: https://github.com/apache/hadoop/pull/4158#issuecomment-1098644389 @leosunli is this something you'd be interested in reviewing? Issue Time Tracking --- Worklog Id: (was: 756807) Time Spent: 0.5h (was: 20m) > SlotReleaser should reuse the domain socket based on socket paths > - > > Key: HDFS-16535 > URL: https://issues.apache.org/jira/browse/HDFS-16535 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs-client >Affects Versions: 3.3.1, 3.4.0 >Reporter: Quanlong Huang >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > HDFS-13639 improves the performance of short-circuit shm slot releasing by > reusing the domain socket that the client previously used to send release > request to the DataNode. > This is good when there are only one DataNode locates with the client (truth > in most of the production environment). However, if we launch multiple > DataNodes on a machine (usually for testing, e.g. Impala's end-to-end tests), > the request could be sent to the wrong DataNode. See an example in > IMPALA-11234. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16513) [SBN read] Observer Namenode should not trigger the edits rolling of active Namenode
[ https://issues.apache.org/jira/browse/HDFS-16513?focusedWorklogId=756802&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-756802 ] ASF GitHub Bot logged work on HDFS-16513: - Author: ASF GitHub Bot Created on: 14/Apr/22 01:57 Start Date: 14/Apr/22 01:57 Worklog Time Spent: 10m Work Description: tomscut commented on PR #4087: URL: https://github.com/apache/hadoop/pull/4087#issuecomment-1098634304 In summary, at this stage, should we disable OBN triggerActiveLogRoll first, or disable all SNN triggerActiveLogRoll directly? @xkrogen @sunchao I look forward to your discussion. Thanks a lot. Issue Time Tracking --- Worklog Id: (was: 756802) Time Spent: 3h (was: 2h 50m) > [SBN read] Observer Namenode should not trigger the edits rolling of active > Namenode > > > Key: HDFS-16513 > URL: https://issues.apache.org/jira/browse/HDFS-16513 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Time Spent: 3h > Remaining Estimate: 0h > > To avoid frequent edtis rolling, we should disable OBN from triggering the > edits rolling of active Namenode. > It is sufficient to retain only the triggering of SNN and the auto rolling of > ANN. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16513) [SBN read] Observer Namenode should not trigger the edits rolling of active Namenode
[ https://issues.apache.org/jira/browse/HDFS-16513?focusedWorklogId=756801&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-756801 ] ASF GitHub Bot logged work on HDFS-16513: - Author: ASF GitHub Bot Created on: 14/Apr/22 01:48 Start Date: 14/Apr/22 01:48 Worklog Time Spent: 10m Work Description: tomscut commented on PR #4087: URL: https://github.com/apache/hadoop/pull/4087#issuecomment-1098630345 Thank you @xkrogen for your detailed explanation. I left out some information. You are right. I thought it was ANN automatic rolledits feature first, then discuss whether to let SNN trigger ANN to rolledits. I got the order of the two wrong. And I thought that "if the active NN is not rolling its logs periodically" meant that the configuration cycle is very large, or that EditLogTailerThread exits because of some UnknowException. As a result, ANN cannot normally roll its logs. Let SNN trigger ANN to roll edits, just to add another layer of assurance. I made a mistake here. Issue Time Tracking --- Worklog Id: (was: 756801) Time Spent: 2h 50m (was: 2h 40m) > [SBN read] Observer Namenode should not trigger the edits rolling of active > Namenode > > > Key: HDFS-16513 > URL: https://issues.apache.org/jira/browse/HDFS-16513 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Time Spent: 2h 50m > Remaining Estimate: 0h > > To avoid frequent edtis rolling, we should disable OBN from triggering the > edits rolling of active Namenode. > It is sufficient to retain only the triggering of SNN and the auto rolling of > ANN. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-16507) [SBN read] Avoid purging edit log which is in progress
[
https://issues.apache.org/jira/browse/HDFS-16507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17521995#comment-17521995
]
tomscut edited comment on HDFS-16507 at 4/14/22 1:10 AM:
-
Thanks [~xkrogen] for your comments.
If the situation arises that ` minTxIdToKeep > curSegmentTxId `,
Preconditions.CheckArgument` while fail, then throw an IllegalArgumentException.
This will cause ImageServlet#doPut to fail, and then cause the SNN checkpoint
to fail, and maybe the SNN will retry several times until ANN rolls the edits
log itself. But ANN avoids purging the inprogress edit log, so it will not
crash. We can see the stack as follows.
Please point out if my description is incorrect. Thank you.
The stack of purgeLogsOlderThan:
{code:java}
java.lang.Thread.getStackTrace(Thread.java:1552)
org.apache.hadoop.util.StringUtils.getStackTrace(StringUtils.java:1032)
org.apache.hadoop.hdfs.server.namenode.FileJournalManager.purgeLogsOlderThan(FileJournalManager.java:185)
org.apache.hadoop.hdfs.server.namenode.JournalSet$5.apply(JournalSet.java:623)
org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:388)
org.apache.hadoop.hdfs.server.namenode.JournalSet.purgeLogsOlderThan(JournalSet.java:620)
org.apache.hadoop.hdfs.server.namenode.FSEditLog.purgeLogsOlderThan(FSEditLog.java:1512)
org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager.purgeOldStorage(NNStorageRetentionManager.java:177)
org.apache.hadoop.hdfs.server.namenode.FSImage.purgeOldStorage(FSImage.java:1249)
org.apache.hadoop.hdfs.server.namenode.ImageServlet$2.run(ImageServlet.java:617)
org.apache.hadoop.hdfs.server.namenode.ImageServlet$2.run(ImageServlet.java:516)
java.security.AccessController.doPrivileged(Native Method)
javax.security.auth.Subject.doAs(Subject.java:422)
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
org.apache.hadoop.hdfs.server.namenode.ImageServlet.doPut(ImageServlet.java:515)
javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1604)
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
org.eclipse.jetty.server.Server.handle(Server.java:539)
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
java.lang.Thread.run(Thread.java:745) {code}
was (Author: tomscut):
Thanks [~xkrogen] for your comments.
if the situation arises that ` minTxIdToKeep > curSegmentTxId `,
Preconditions.CheckArgument` while fail, then throw an IllegalArgumentException.
This will cause ImageServlet#doPut to fail, and then ca
[jira] [Commented] (HDFS-16507) [SBN read] Avoid purging edit log which is in progress
[
https://issues.apache.org/jira/browse/HDFS-16507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17521995#comment-17521995
]
tomscut commented on HDFS-16507:
Thanks [~xkrogen] for your comments.
if the situation arises that ` minTxIdToKeep > curSegmentTxId `,
Preconditions.CheckArgument` while fail, then throw an IllegalArgumentException.
This will cause ImageServlet#doPut to fail, and then cause the SNN checkpoint
to fail, and maybe the SNN will retry several times until ANN rolls the edits
log itself. But ANN avoids purging the inprogress edit log, so it will not
crash. We can see the stack as follows.
Please point out if my description is incorrect. Thank you.
The stack of purgeLogsOlderThan:
{code:java}
java.lang.Thread.getStackTrace(Thread.java:1552)
org.apache.hadoop.util.StringUtils.getStackTrace(StringUtils.java:1032)
org.apache.hadoop.hdfs.server.namenode.FileJournalManager.purgeLogsOlderThan(FileJournalManager.java:185)
org.apache.hadoop.hdfs.server.namenode.JournalSet$5.apply(JournalSet.java:623)
org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:388)
org.apache.hadoop.hdfs.server.namenode.JournalSet.purgeLogsOlderThan(JournalSet.java:620)
org.apache.hadoop.hdfs.server.namenode.FSEditLog.purgeLogsOlderThan(FSEditLog.java:1512)
org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager.purgeOldStorage(NNStorageRetentionManager.java:177)
org.apache.hadoop.hdfs.server.namenode.FSImage.purgeOldStorage(FSImage.java:1249)
org.apache.hadoop.hdfs.server.namenode.ImageServlet$2.run(ImageServlet.java:617)
org.apache.hadoop.hdfs.server.namenode.ImageServlet$2.run(ImageServlet.java:516)
java.security.AccessController.doPrivileged(Native Method)
javax.security.auth.Subject.doAs(Subject.java:422)
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
org.apache.hadoop.hdfs.server.namenode.ImageServlet.doPut(ImageServlet.java:515)
javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1604)
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
org.eclipse.jetty.server.Server.handle(Server.java:539)
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
java.lang.Thread.run(Thread.java:745) {code}
> [SBN read] Avoid purging edit log which is in progress
> --
>
> Key: HDFS-16507
> URL: https://issues.apache.org/jira/browse/HDFS-16507
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.1.0
>
[jira] [Work logged] (HDFS-13522) RBF: Support observer node from Router-Based Federation
[
https://issues.apache.org/jira/browse/HDFS-13522?focusedWorklogId=756746&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-756746
]
ASF GitHub Bot logged work on HDFS-13522:
-
Author: ASF GitHub Bot
Created on: 13/Apr/22 22:13
Start Date: 13/Apr/22 22:13
Worklog Time Spent: 10m
Work Description: simbadzina commented on code in PR #4127:
URL: https://github.com/apache/hadoop/pull/4127#discussion_r849678932
##
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/NameNodeProxiesClient.java:
##
@@ -349,6 +349,18 @@ public static ClientProtocol
createProxyWithAlignmentContext(
boolean withRetries, AtomicBoolean fallbackToSimpleAuth,
AlignmentContext alignmentContext)
throws IOException {
+if (!conf.getBoolean(HdfsClientConfigKeys.DFS_OBSERVER_READ_ENABLE,
Review Comment:
Hi @goiri I'm going to be away for a few weeks so I can't do the split soon.
If it's a must have, I can do it when I'm back. Or if anybody has bandwidth
they can help out.
Issue Time Tracking
---
Worklog Id: (was: 756746)
Time Spent: 6h 20m (was: 6h 10m)
> RBF: Support observer node from Router-Based Federation
> ---
>
> Key: HDFS-13522
> URL: https://issues.apache.org/jira/browse/HDFS-13522
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: federation, namenode
>Reporter: Erik Krogen
>Assignee: Simbarashe Dzinamarira
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-13522.001.patch, HDFS-13522.002.patch,
> HDFS-13522_WIP.patch, RBF_ Observer support.pdf, Router+Observer RPC
> clogging.png, ShortTerm-Routers+Observer.png
>
> Time Spent: 6h 20m
> Remaining Estimate: 0h
>
> Changes will need to occur to the router to support the new observer node.
> One such change will be to make the router understand the observer state,
> e.g. {{FederationNamenodeServiceState}}.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14786) A new block placement policy tolerating availability zone failure
[
https://issues.apache.org/jira/browse/HDFS-14786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17521952#comment-17521952
]
Mingliang Liu commented on HDFS-14786:
--
[~panlijie] I did not have chance to work on this so I did not assign to me.
Feel free to pick it up if you are interested. It became more interesting when
EKS supports placement group. My previous projects later explore S3 as long
term storage tier for that highly critical use case (HBase/Phoenix).
> A new block placement policy tolerating availability zone failure
> -
>
> Key: HDFS-14786
> URL: https://issues.apache.org/jira/browse/HDFS-14786
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: block placement
>Reporter: Mingliang Liu
>Priority: Major
>
> {{NetworkTopology}} assumes "/datacenter/rack/host" 3 layer topology. Default
> block placement policies are rack awareness for better fault tolerance. Newer
> block placement policy like {{BlockPlacementPolicyRackFaultTolerant}} tries
> its best to place the replicas to most racks, which further tolerates more
> racks failing. HADOOP-8470 brought {{NetworkTopologyWithNodeGroup}} to add
> another layer under rack, i.e. "/datacenter/rack/host/nodegroup" 4 layer
> topology. With that, replicas within a rack can be placed in different node
> groups for better isolation.
> Existing block placement policies tolerate one rack failure since at least
> two racks are chosen in those cases. Chances are all replicas could be placed
> in the same datacenter, though there are multiple data centers in the same
> cluster topology. In other words, fault of higher layers beyond rack is not
> well tolerated.
> However, more deployments in public cloud are leveraging multiple available
> zones (AZ) for high-availability since the inter-AZ latency seems affordable
> in many cases. In a single AZ, some cloud providers like AWS support
> [partitioned placement
> groups|https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placement-groups.html#placement-groups-partition]
> which basically are different racks. A simple network topology mapped to
> HDFS is "/availabilityzone/rack/host" 3 layers.
> To achieve high availability tolerating zone failure, this JIRA proposes a
> new data placement policy which tries its best to place replicas in most AZs,
> most racks, and most evenly distributed.
> Examples with 3 replicas, we choose racks as following:
> - 1AZ: fall back to {{BlockPlacementPolicyRackFaultTolerant}} to place among
> most racks
> - 2AZ: randomly choose one rack in one AZ and randomly choose two racks in
> the other AZ
> - 3AZ: randomly choose one rack in every AZ
> - 4AZ: randomly choose three AZs and randomly choose one rack in every AZ
> After racks are picked, hosts are chosen randomly within racks honoring local
> storage, favorite nodes, excluded nodes, storage types etc. Data may become
> imbalance if topology is very uneven in AZs. This seems not a problem as in
> public cloud, infrastructure provisioning is more flexible than 1P.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16540) Data locality is lost when DataNode pod restarts in kubernetes
[ https://issues.apache.org/jira/browse/HDFS-16540?focusedWorklogId=756643&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-756643 ] ASF GitHub Bot logged work on HDFS-16540: - Author: ASF GitHub Bot Created on: 13/Apr/22 19:51 Start Date: 13/Apr/22 19:51 Worklog Time Spent: 10m Work Description: huaxiangsun opened a new pull request, #4170: URL: https://github.com/apache/hadoop/pull/4170 …etes ### Description of PR When Dn with the same uuid is registered with a different ip, host2DatanodeMap needs to be updated accordingly. ### How was this patch tested? Tested 3.3.2 with the patch on a eks cluster, restarted the pod hosting DataNode and HBase region server. After that, doing a major compaction of Hbase region, made sure that locality is kept. There is also a new unittest case added. ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? Issue Time Tracking --- Worklog Id: (was: 756643) Remaining Estimate: 0h Time Spent: 10m > Data locality is lost when DataNode pod restarts in kubernetes > --- > > Key: HDFS-16540 > URL: https://issues.apache.org/jira/browse/HDFS-16540 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 3.3.2 >Reporter: Huaxiang Sun >Assignee: Huaxiang Sun >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > We have HBase RegionServer and Hdfs DataNode running in one pod. When the pod > restarts, we found that data locality is lost after we do a major compaction > of hbase regions. After some debugging, we found that upon pod restarts, its > ip changes. In DatanodeManager, maps like networktopology are updated with > the new info. host2DatanodeMap is not updated accordingly. When hdfs client > with the new ip tries to find a local DataNode, it fails. > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16540) Data locality is lost when DataNode pod restarts in kubernetes
[ https://issues.apache.org/jira/browse/HDFS-16540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-16540: -- Labels: pull-request-available (was: ) > Data locality is lost when DataNode pod restarts in kubernetes > --- > > Key: HDFS-16540 > URL: https://issues.apache.org/jira/browse/HDFS-16540 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 3.3.2 >Reporter: Huaxiang Sun >Assignee: Huaxiang Sun >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > We have HBase RegionServer and Hdfs DataNode running in one pod. When the pod > restarts, we found that data locality is lost after we do a major compaction > of hbase regions. After some debugging, we found that upon pod restarts, its > ip changes. In DatanodeManager, maps like networktopology are updated with > the new info. host2DatanodeMap is not updated accordingly. When hdfs client > with the new ip tries to find a local DataNode, it fails. > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-16540) Data locality is lost when DataNode pod restarts in kubernetes
[ https://issues.apache.org/jira/browse/HDFS-16540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack reassigned HDFS-16540: Assignee: Huaxiang Sun > Data locality is lost when DataNode pod restarts in kubernetes > --- > > Key: HDFS-16540 > URL: https://issues.apache.org/jira/browse/HDFS-16540 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 3.3.2 >Reporter: Huaxiang Sun >Assignee: Huaxiang Sun >Priority: Major > > We have HBase RegionServer and Hdfs DataNode running in one pod. When the pod > restarts, we found that data locality is lost after we do a major compaction > of hbase regions. After some debugging, we found that upon pod restarts, its > ip changes. In DatanodeManager, maps like networktopology are updated with > the new info. host2DatanodeMap is not updated accordingly. When hdfs client > with the new ip tries to find a local DataNode, it fails. > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16540) Data locality is lost when DataNode pod restarts in kubernetes
[ https://issues.apache.org/jira/browse/HDFS-16540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17521857#comment-17521857 ] Huaxiang Sun commented on HDFS-16540: - Will upload a patch. > Data locality is lost when DataNode pod restarts in kubernetes > --- > > Key: HDFS-16540 > URL: https://issues.apache.org/jira/browse/HDFS-16540 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 3.3.2 >Reporter: Huaxiang Sun >Priority: Major > > We have HBase RegionServer and Hdfs DataNode running in one pod. When the pod > restarts, we found that data locality is lost after we do a major compaction > of hbase regions. After some debugging, we found that upon pod restarts, its > ip changes. In DatanodeManager, maps like networktopology are updated with > the new info. host2DatanodeMap is not updated accordingly. When hdfs client > with the new ip tries to find a local DataNode, it fails. > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16540) Data locality is lost when DataNode pod restarts in kubernetes
Huaxiang Sun created HDFS-16540: --- Summary: Data locality is lost when DataNode pod restarts in kubernetes Key: HDFS-16540 URL: https://issues.apache.org/jira/browse/HDFS-16540 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 3.3.2 Reporter: Huaxiang Sun We have HBase RegionServer and Hdfs DataNode running in one pod. When the pod restarts, we found that data locality is lost after we do a major compaction of hbase regions. After some debugging, we found that upon pod restarts, its ip changes. In DatanodeManager, maps like networktopology are updated with the new info. host2DatanodeMap is not updated accordingly. When hdfs client with the new ip tries to find a local DataNode, it fails. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16526) Add metrics for slow DataNode
[ https://issues.apache.org/jira/browse/HDFS-16526?focusedWorklogId=756569&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-756569 ] ASF GitHub Bot logged work on HDFS-16526: - Author: ASF GitHub Bot Created on: 13/Apr/22 17:37 Start Date: 13/Apr/22 17:37 Worklog Time Spent: 10m Work Description: prasad-acit commented on PR #4162: URL: https://github.com/apache/hadoop/pull/4162#issuecomment-1098314022 UT failures are not related to the code changes. @hemanthboyina / @Hexiaoqiao can you plz review the MR? Issue Time Tracking --- Worklog Id: (was: 756569) Time Spent: 1h (was: 50m) > Add metrics for slow DataNode > - > > Key: HDFS-16526 > URL: https://issues.apache.org/jira/browse/HDFS-16526 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Renukaprasad C >Assignee: Renukaprasad C >Priority: Major > Labels: pull-request-available > Time Spent: 1h > Remaining Estimate: 0h > > Add some more metrics for slow datanode operations - FlushOrSync, > PacketResponder send ACK. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-13522) RBF: Support observer node from Router-Based Federation
[
https://issues.apache.org/jira/browse/HDFS-13522?focusedWorklogId=756523&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-756523
]
ASF GitHub Bot logged work on HDFS-13522:
-
Author: ASF GitHub Bot
Created on: 13/Apr/22 16:45
Start Date: 13/Apr/22 16:45
Worklog Time Spent: 10m
Work Description: simbadzina commented on PR #4127:
URL: https://github.com/apache/hadoop/pull/4127#issuecomment-1098268812
Hi @fengnanli could you please take a look and add folks in your team.
Issue Time Tracking
---
Worklog Id: (was: 756523)
Time Spent: 6h 10m (was: 6h)
> RBF: Support observer node from Router-Based Federation
> ---
>
> Key: HDFS-13522
> URL: https://issues.apache.org/jira/browse/HDFS-13522
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: federation, namenode
>Reporter: Erik Krogen
>Assignee: Simbarashe Dzinamarira
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-13522.001.patch, HDFS-13522.002.patch,
> HDFS-13522_WIP.patch, RBF_ Observer support.pdf, Router+Observer RPC
> clogging.png, ShortTerm-Routers+Observer.png
>
> Time Spent: 6h 10m
> Remaining Estimate: 0h
>
> Changes will need to occur to the router to support the new observer node.
> One such change will be to make the router understand the observer state,
> e.g. {{FederationNamenodeServiceState}}.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-13522) RBF: Support observer node from Router-Based Federation
[
https://issues.apache.org/jira/browse/HDFS-13522?focusedWorklogId=756513&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-756513
]
ASF GitHub Bot logged work on HDFS-13522:
-
Author: ASF GitHub Bot
Created on: 13/Apr/22 16:29
Start Date: 13/Apr/22 16:29
Worklog Time Spent: 10m
Work Description: simbadzina commented on code in PR #4127:
URL: https://github.com/apache/hadoop/pull/4127#discussion_r849678932
##
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/NameNodeProxiesClient.java:
##
@@ -349,6 +349,18 @@ public static ClientProtocol
createProxyWithAlignmentContext(
boolean withRetries, AtomicBoolean fallbackToSimpleAuth,
AlignmentContext alignmentContext)
throws IOException {
+if (!conf.getBoolean(HdfsClientConfigKeys.DFS_OBSERVER_READ_ENABLE,
Review Comment:
Hi @goiri I'm going to be away for a few weeks to do the split. If it's a
must have, I can do it when I'm back. Or if anybody has bandwidth they can help
out.
Issue Time Tracking
---
Worklog Id: (was: 756513)
Time Spent: 6h (was: 5h 50m)
> RBF: Support observer node from Router-Based Federation
> ---
>
> Key: HDFS-13522
> URL: https://issues.apache.org/jira/browse/HDFS-13522
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: federation, namenode
>Reporter: Erik Krogen
>Assignee: Simbarashe Dzinamarira
>Priority: Major
> Labels: pull-request-available
> Attachments: HDFS-13522.001.patch, HDFS-13522.002.patch,
> HDFS-13522_WIP.patch, RBF_ Observer support.pdf, Router+Observer RPC
> clogging.png, ShortTerm-Routers+Observer.png
>
> Time Spent: 6h
> Remaining Estimate: 0h
>
> Changes will need to occur to the router to support the new observer node.
> One such change will be to make the router understand the observer state,
> e.g. {{FederationNamenodeServiceState}}.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16513) [SBN read] Observer Namenode should not trigger the edits rolling of active Namenode
[ https://issues.apache.org/jira/browse/HDFS-16513?focusedWorklogId=756503&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-756503 ] ASF GitHub Bot logged work on HDFS-16513: - Author: ASF GitHub Bot Created on: 13/Apr/22 16:16 Start Date: 13/Apr/22 16:16 Worklog Time Spent: 10m Work Description: xkrogen commented on PR #4087: URL: https://github.com/apache/hadoop/pull/4087#issuecomment-1098241563 > The pendingDatanodeMessage issue mentioned here strikes me as a bit risky. ... I'm not following. The issue described from HDFS-2737 says that "if the active NN is not rolling its logs periodically ... many datanode messages \[will\] be queued up in the PendingDatanodeMessage structure". Certainly it is bad if we don't have a way to ensure that the logs are rolled regularly. But HDFS-14378 just proposes making the ANN roll its own edit logs, instead of relying on the SbNN to roll them. I don't see the risk Issue Time Tracking --- Worklog Id: (was: 756503) Time Spent: 2h 40m (was: 2.5h) > [SBN read] Observer Namenode should not trigger the edits rolling of active > Namenode > > > Key: HDFS-16513 > URL: https://issues.apache.org/jira/browse/HDFS-16513 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: tomscut >Assignee: tomscut >Priority: Major > Labels: pull-request-available > Time Spent: 2h 40m > Remaining Estimate: 0h > > To avoid frequent edtis rolling, we should disable OBN from triggering the > edits rolling of active Namenode. > It is sufficient to retain only the triggering of SNN and the auto rolling of > ANN. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16507) [SBN read] Avoid purging edit log which is in progress
[
https://issues.apache.org/jira/browse/HDFS-16507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17521809#comment-17521809
]
Erik Krogen commented on HDFS-16507:
{quote}
Do you mean, if the situation arises that ` minTxIdToKeep > curSegmentTxId `,
ANN will crash because `Preconditions.CheckArgument` failure?
{quote}
Yeah, that is my concern.
> [SBN read] Avoid purging edit log which is in progress
> --
>
> Key: HDFS-16507
> URL: https://issues.apache.org/jira/browse/HDFS-16507
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: tomscut
>Assignee: tomscut
>Priority: Critical
> Labels: pull-request-available
> Fix For: 3.4.0, 3.2.4, 3.3.4
>
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
> We introduced [Standby Read] feature in branch-3.1.0, but found a FATAL
> exception. It looks like it's purging edit logs which is in process.
> According to the analysis, I suspect that the editlog which is in progress to
> be purged(after SNN checkpoint) does not finalize(See HDFS-14317) before ANN
> rolls edit its self.
> The stack:
> {code:java}
> java.lang.Thread.getStackTrace(Thread.java:1552)
> org.apache.hadoop.util.StringUtils.getStackTrace(StringUtils.java:1032)
>
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager.purgeLogsOlderThan(FileJournalManager.java:185)
>
> org.apache.hadoop.hdfs.server.namenode.JournalSet$5.apply(JournalSet.java:623)
>
> org.apache.hadoop.hdfs.server.namenode.JournalSet.mapJournalsAndReportErrors(JournalSet.java:388)
>
> org.apache.hadoop.hdfs.server.namenode.JournalSet.purgeLogsOlderThan(JournalSet.java:620)
>
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.purgeLogsOlderThan(FSEditLog.java:1512)
> org.apache.hadoop.hdfs.server.namenode.NNStorageRetentionManager.purgeOldStorage(NNStorageRetentionManager.java:177)
>
> org.apache.hadoop.hdfs.server.namenode.FSImage.purgeOldStorage(FSImage.java:1249)
>
> org.apache.hadoop.hdfs.server.namenode.ImageServlet$2.run(ImageServlet.java:617)
>
> org.apache.hadoop.hdfs.server.namenode.ImageServlet$2.run(ImageServlet.java:516)
> java.security.AccessController.doPrivileged(Native Method)
> javax.security.auth.Subject.doAs(Subject.java:422)
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
>
> org.apache.hadoop.hdfs.server.namenode.ImageServlet.doPut(ImageServlet.java:515)
> javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
> javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
>
> org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1604)
>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
>
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
>
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
>
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
>
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
>
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
>
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
>
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> org.eclipse.jetty.server.Server.handle(Server.java:539)
> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)
>
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
>
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
> org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
>
> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
>
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
>
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.pro
[jira] [Work logged] (HDFS-16509) Fix decommission UnsupportedOperationException: Remove unsupported
[
https://issues.apache.org/jira/browse/HDFS-16509?focusedWorklogId=756460&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-756460
]
ASF GitHub Bot logged work on HDFS-16509:
-
Author: ASF GitHub Bot
Created on: 13/Apr/22 15:13
Start Date: 13/Apr/22 15:13
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on PR #4077:
URL: https://github.com/apache/hadoop/pull/4077#issuecomment-1098174309
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 44s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 38m 41s | | trunk passed |
| +1 :green_heart: | compile | 1m 28s | | trunk passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 20s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 5s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 30s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 9s | | trunk passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 31s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 30s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 9s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 18s | | the patch passed |
| +1 :green_heart: | compile | 1m 20s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 1m 20s | | the patch passed |
| +1 :green_heart: | compile | 1m 15s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 15s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 53s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 20s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 51s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 21s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 17s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 30s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 228m 12s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 50s | | The patch does not
generate ASF License warnings. |
| | | 334m 37s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4077/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4077 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 921b06966301 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b516ffebac3baa6a41c75bb215a154aeb22dbdc3 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4077/6/testReport/ |
| Max. process+thread count | 3312 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4077/6/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This mes
[jira] [Work logged] (HDFS-14478) Add libhdfs APIs for openFile
[
https://issues.apache.org/jira/browse/HDFS-14478?focusedWorklogId=756451&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-756451
]
ASF GitHub Bot logged work on HDFS-14478:
-
Author: ASF GitHub Bot
Created on: 13/Apr/22 14:58
Start Date: 13/Apr/22 14:58
Worklog Time Spent: 10m
Work Description: steveloughran commented on PR #4166:
URL: https://github.com/apache/hadoop/pull/4166#issuecomment-1098157004
failures on branch-3.3 are the same,
[exec] The following tests FAILED:
[exec] 14 - memcheck_rpc_engine (Failed)
[exec] 34 - memcheck_hdfs_config_connect_bugs (Failed)
[exec] 38 -
memcheck_libhdfs_mini_stress_valgrind_hdfspp_test_static (Failed)
Issue Time Tracking
---
Worklog Id: (was: 756451)
Time Spent: 2h 20m (was: 2h 10m)
> Add libhdfs APIs for openFile
> -
>
> Key: HDFS-14478
> URL: https://issues.apache.org/jira/browse/HDFS-14478
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client, libhdfs, native
>Reporter: Sahil Takiar
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> HADOOP-15229 added a "FileSystem builder-based openFile() API" that allows
> specifying configuration values for opening files (similar to HADOOP-14365).
> Support for {{openFile}} will be a little tricky as it is asynchronous and
> {{FutureDataInputStreamBuilder#build}} returns a {{CompletableFuture}}.
> At a high level, the API for {{openFile}} could look something like this:
> {code:java}
> hdfsFile hdfsOpenFile(hdfsFS fs, const char* path, int flags,
> int bufferSize, short replication, tSize blocksize);
> hdfsOpenFileBuilder *hdfsOpenFileBuilderAlloc(hdfsFS fs,
> const char *path);
> hdfsOpenFileBuilder *hdfsOpenFileBuilderMust(hdfsOpenFileBuilder *builder,
> const char *key, const char *value);
> hdfsOpenFileBuilder *hdfsOpenFileBuilderOpt(hdfsOpenFileBuilder *builder,
> const char *key, const char *value);
> hdfsOpenFileFuture *hdfsOpenFileBuilderBuild(hdfsOpenFileBuilder *builder);
> void hdfsOpenFileBuilderFree(hdfsOpenFileBuilder *builder);
> hdfsFile hdfsOpenFileFutureGet(hdfsOpenFileFuture *future);
> hdfsFile hdfsOpenFileFutureGetWithTimeout(hdfsOpenFileFuture *future,
> int64_t timeout, javaConcurrentTimeUnit timeUnit);
> int hdfsOpenFileFutureCancel(hdfsOpenFileFuture *future,
> int mayInterruptIfRunning);
> void hdfsOpenFileFutureFree(hdfsOpenFileFuture *future);
> {code}
> Instead of exposing all the functionality of {{CompleteableFuture}} libhdfs
> would just expose the functionality of {{Future}}.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16534) Split datanode block pool locks to volume grain.
[ https://issues.apache.org/jira/browse/HDFS-16534?focusedWorklogId=756414&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-756414 ] ASF GitHub Bot logged work on HDFS-16534: - Author: ASF GitHub Bot Created on: 13/Apr/22 13:56 Start Date: 13/Apr/22 13:56 Worklog Time Spent: 10m Work Description: Hexiaoqiao commented on PR #4141: URL: https://github.com/apache/hadoop/pull/4141#issuecomment-1098081845 LGTM +1 from my side. I would like to check into trunk if no furthermore feedback while 5 work days later. > 1、some method is not good change to volume lock, and if split to volume lock it have to get locks and release locks sequence(like acquire lock1, lock2, lock3, release lock 3 lock2 lock1).So just acquire block pool lock is enough. > 2、some method like contains() is no need to acquire volume lock.Now it acquire block pool lock read lock, so it no need to acquire block pool lock read lock and volume lock. I would like to clarify this explanation. This improvement only change part of methods from block pool level lock to volume level lock rather all. Because, A. No Improvement. Such as for add/remove Volume method, Lock level 0 (block pool level lock) is enough and safe to execute logic. Some others is similar. or B. No Necessary. For some query request, acquire block pool level read lock has safe-guarded, so it is not necessary to acquire volume level read lock again. Thanks @MingXiangLi for your works. Issue Time Tracking --- Worklog Id: (was: 756414) Time Spent: 1h 20m (was: 1h 10m) > Split datanode block pool locks to volume grain. > > > Key: HDFS-16534 > URL: https://issues.apache.org/jira/browse/HDFS-16534 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Mingxiang Li >Assignee: Mingxiang Li >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > This is sub task of HDFS-15382. > https://issues.apache.org/jira/browse/HDFS-15180 have split lock to block > pool grain and do some prepare.This pr is the last part of volume lock. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14478) Add libhdfs APIs for openFile
[
https://issues.apache.org/jira/browse/HDFS-14478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17521689#comment-17521689
]
Steve Loughran commented on HDFS-14478:
---
in 3.4; backporting to 3.3
> Add libhdfs APIs for openFile
> -
>
> Key: HDFS-14478
> URL: https://issues.apache.org/jira/browse/HDFS-14478
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client, libhdfs, native
>Reporter: Sahil Takiar
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> HADOOP-15229 added a "FileSystem builder-based openFile() API" that allows
> specifying configuration values for opening files (similar to HADOOP-14365).
> Support for {{openFile}} will be a little tricky as it is asynchronous and
> {{FutureDataInputStreamBuilder#build}} returns a {{CompletableFuture}}.
> At a high level, the API for {{openFile}} could look something like this:
> {code:java}
> hdfsFile hdfsOpenFile(hdfsFS fs, const char* path, int flags,
> int bufferSize, short replication, tSize blocksize);
> hdfsOpenFileBuilder *hdfsOpenFileBuilderAlloc(hdfsFS fs,
> const char *path);
> hdfsOpenFileBuilder *hdfsOpenFileBuilderMust(hdfsOpenFileBuilder *builder,
> const char *key, const char *value);
> hdfsOpenFileBuilder *hdfsOpenFileBuilderOpt(hdfsOpenFileBuilder *builder,
> const char *key, const char *value);
> hdfsOpenFileFuture *hdfsOpenFileBuilderBuild(hdfsOpenFileBuilder *builder);
> void hdfsOpenFileBuilderFree(hdfsOpenFileBuilder *builder);
> hdfsFile hdfsOpenFileFutureGet(hdfsOpenFileFuture *future);
> hdfsFile hdfsOpenFileFutureGetWithTimeout(hdfsOpenFileFuture *future,
> int64_t timeout, javaConcurrentTimeUnit timeUnit);
> int hdfsOpenFileFutureCancel(hdfsOpenFileFuture *future,
> int mayInterruptIfRunning);
> void hdfsOpenFileFutureFree(hdfsOpenFileFuture *future);
> {code}
> Instead of exposing all the functionality of {{CompleteableFuture}} libhdfs
> would just expose the functionality of {{Future}}.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-14478) Add libhdfs APIs for openFile
[
https://issues.apache.org/jira/browse/HDFS-14478?focusedWorklogId=756374&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-756374
]
ASF GitHub Bot logged work on HDFS-14478:
-
Author: ASF GitHub Bot
Created on: 13/Apr/22 13:15
Start Date: 13/Apr/22 13:15
Worklog Time Spent: 10m
Work Description: steveloughran merged PR #4166:
URL: https://github.com/apache/hadoop/pull/4166
Issue Time Tracking
---
Worklog Id: (was: 756374)
Time Spent: 2h 10m (was: 2h)
> Add libhdfs APIs for openFile
> -
>
> Key: HDFS-14478
> URL: https://issues.apache.org/jira/browse/HDFS-14478
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client, libhdfs, native
>Reporter: Sahil Takiar
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> HADOOP-15229 added a "FileSystem builder-based openFile() API" that allows
> specifying configuration values for opening files (similar to HADOOP-14365).
> Support for {{openFile}} will be a little tricky as it is asynchronous and
> {{FutureDataInputStreamBuilder#build}} returns a {{CompletableFuture}}.
> At a high level, the API for {{openFile}} could look something like this:
> {code:java}
> hdfsFile hdfsOpenFile(hdfsFS fs, const char* path, int flags,
> int bufferSize, short replication, tSize blocksize);
> hdfsOpenFileBuilder *hdfsOpenFileBuilderAlloc(hdfsFS fs,
> const char *path);
> hdfsOpenFileBuilder *hdfsOpenFileBuilderMust(hdfsOpenFileBuilder *builder,
> const char *key, const char *value);
> hdfsOpenFileBuilder *hdfsOpenFileBuilderOpt(hdfsOpenFileBuilder *builder,
> const char *key, const char *value);
> hdfsOpenFileFuture *hdfsOpenFileBuilderBuild(hdfsOpenFileBuilder *builder);
> void hdfsOpenFileBuilderFree(hdfsOpenFileBuilder *builder);
> hdfsFile hdfsOpenFileFutureGet(hdfsOpenFileFuture *future);
> hdfsFile hdfsOpenFileFutureGetWithTimeout(hdfsOpenFileFuture *future,
> int64_t timeout, javaConcurrentTimeUnit timeUnit);
> int hdfsOpenFileFutureCancel(hdfsOpenFileFuture *future,
> int mayInterruptIfRunning);
> void hdfsOpenFileFutureFree(hdfsOpenFileFuture *future);
> {code}
> Instead of exposing all the functionality of {{CompleteableFuture}} libhdfs
> would just expose the functionality of {{Future}}.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-14478) Add libhdfs APIs for openFile
[
https://issues.apache.org/jira/browse/HDFS-14478?focusedWorklogId=756373&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-756373
]
ASF GitHub Bot logged work on HDFS-14478:
-
Author: ASF GitHub Bot
Created on: 13/Apr/22 13:14
Start Date: 13/Apr/22 13:14
Worklog Time Spent: 10m
Work Description: steveloughran commented on PR #4166:
URL: https://github.com/apache/hadoop/pull/4166#issuecomment-1098036077
The same tests fail on my arm64 docker vm too; so not related.
+1 for sahil's patch.
I'd like to followup with some tests of failure conditions, especially once
#2584 is in, but not here
Issue Time Tracking
---
Worklog Id: (was: 756373)
Time Spent: 2h (was: 1h 50m)
> Add libhdfs APIs for openFile
> -
>
> Key: HDFS-14478
> URL: https://issues.apache.org/jira/browse/HDFS-14478
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client, libhdfs, native
>Reporter: Sahil Takiar
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> HADOOP-15229 added a "FileSystem builder-based openFile() API" that allows
> specifying configuration values for opening files (similar to HADOOP-14365).
> Support for {{openFile}} will be a little tricky as it is asynchronous and
> {{FutureDataInputStreamBuilder#build}} returns a {{CompletableFuture}}.
> At a high level, the API for {{openFile}} could look something like this:
> {code:java}
> hdfsFile hdfsOpenFile(hdfsFS fs, const char* path, int flags,
> int bufferSize, short replication, tSize blocksize);
> hdfsOpenFileBuilder *hdfsOpenFileBuilderAlloc(hdfsFS fs,
> const char *path);
> hdfsOpenFileBuilder *hdfsOpenFileBuilderMust(hdfsOpenFileBuilder *builder,
> const char *key, const char *value);
> hdfsOpenFileBuilder *hdfsOpenFileBuilderOpt(hdfsOpenFileBuilder *builder,
> const char *key, const char *value);
> hdfsOpenFileFuture *hdfsOpenFileBuilderBuild(hdfsOpenFileBuilder *builder);
> void hdfsOpenFileBuilderFree(hdfsOpenFileBuilder *builder);
> hdfsFile hdfsOpenFileFutureGet(hdfsOpenFileFuture *future);
> hdfsFile hdfsOpenFileFutureGetWithTimeout(hdfsOpenFileFuture *future,
> int64_t timeout, javaConcurrentTimeUnit timeUnit);
> int hdfsOpenFileFutureCancel(hdfsOpenFileFuture *future,
> int mayInterruptIfRunning);
> void hdfsOpenFileFutureFree(hdfsOpenFileFuture *future);
> {code}
> Instead of exposing all the functionality of {{CompleteableFuture}} libhdfs
> would just expose the functionality of {{Future}}.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16513) [SBN read] Observer Namenode should not trigger the edits rolling of active Namenode
[
https://issues.apache.org/jira/browse/HDFS-16513?focusedWorklogId=756365&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-756365
]
ASF GitHub Bot logged work on HDFS-16513:
-
Author: ASF GitHub Bot
Created on: 13/Apr/22 12:58
Start Date: 13/Apr/22 12:58
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on PR #4087:
URL: https://github.com/apache/hadoop/pull/4087#issuecomment-1098019249
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 44s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 43m 23s | | trunk passed |
| +1 :green_heart: | compile | 1m 42s | | trunk passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 33s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 13s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 37s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 9s | | trunk passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 37s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 57s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 30s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 24s | | the patch passed |
| +1 :green_heart: | compile | 1m 30s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 1m 30s | | the patch passed |
| +1 :green_heart: | compile | 1m 21s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 21s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 0s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 30s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 55s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 33s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 52s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 3s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 240m 42s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 49s | | The patch does not
generate ASF License warnings. |
| | | 360m 33s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4087/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4087 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux ec3b877e8bf4 4.15.0-169-generic #177-Ubuntu SMP Thu Feb 3
10:50:38 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 0b1946bd7012b025bdc7ef89e679edba15f68242 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4087/2/testReport/ |
| Max. process+thread count | 3012 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4087/2/console |
| versions | git=2
[jira] [Work logged] (HDFS-16539) RBF: Support refreshing/changing router fairness policy controller without rebooting router
[
https://issues.apache.org/jira/browse/HDFS-16539?focusedWorklogId=756361&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-756361
]
ASF GitHub Bot logged work on HDFS-16539:
-
Author: ASF GitHub Bot
Created on: 13/Apr/22 12:52
Start Date: 13/Apr/22 12:52
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on PR #4168:
URL: https://github.com/apache/hadoop/pull/4168#issuecomment-1098013964
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 39s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 58s | | trunk passed |
| +1 :green_heart: | compile | 0m 47s | | trunk passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 30s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 47s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 48s | | trunk passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 55s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 39s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 14s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 0m 36s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 0m 32s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 32s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 18s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4168/1/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 2 new + 1
unchanged - 0 fixed = 3 total (was 1) |
| +1 :green_heart: | mvnsite | 0m 35s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 35s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 53s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| -1 :x: | spotbugs | 1m 35s |
[/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs-rbf.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4168/1/artifact/out/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs-rbf.html)
| hadoop-hdfs-project/hadoop-hdfs-rbf generated 1 new + 0 unchanged - 0 fixed
= 1 total (was 0) |
| +1 :green_heart: | shadedclient | 24m 3s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 22m 6s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 39s | | The patch does not
generate ASF License warnings. |
| | | 126m 20s | | |
| Reason | Tests |
|---:|:--|
| SpotBugs | module:hadoop-hdfs-project/hadoop-hdfs-rbf |
| | Inconsistent synchronization of
org.apache.hadoop.hdfs.server.federation.router.RouterRpcClient.routerRpcFairnessPolicyController;
locked 50% of time Unsynchronized access at RouterRpcClient.java:50% of time
Unsynchronized access at RouterRpcClient.java:[line 1611] |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4168/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4168 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 243f27ee10f9 4.15.0-169-generic #177-Ubuntu SMP Thu Feb 3
10:50:38 UTC 202
[jira] [Work logged] (HDFS-16538) EC decoding failed due to not enough valid inputs
[
https://issues.apache.org/jira/browse/HDFS-16538?focusedWorklogId=756362&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-756362
]
ASF GitHub Bot logged work on HDFS-16538:
-
Author: ASF GitHub Bot
Created on: 13/Apr/22 12:52
Start Date: 13/Apr/22 12:52
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on PR #4167:
URL: https://github.com/apache/hadoop/pull/4167#issuecomment-1098014110
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 45s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 39m 1s | | trunk passed |
| +1 :green_heart: | compile | 1m 2s | | trunk passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 0m 55s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 35s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 1s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 50s | | trunk passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 40s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 47s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 45s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 50s | | the patch passed |
| +1 :green_heart: | compile | 0m 53s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 0m 53s | | the patch passed |
| +1 :green_heart: | compile | 0m 45s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 45s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 19s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 48s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 34s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 30s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 49s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 23s | | hadoop-hdfs-client in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not
generate ASF License warnings. |
| | | 101m 35s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4167/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4167 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 6a5ad419e32a 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 0507620c617b7868361a484773d3f74f0a1dd8dc |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4167/2/testReport/ |
| Max. process+thread count | 548 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client U:
hadoop-hdfs-project/hadoop-hdfs-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4167/2/console |
[jira] [Work logged] (HDFS-16538) EC decoding failed due to not enough valid inputs
[
https://issues.apache.org/jira/browse/HDFS-16538?focusedWorklogId=756317&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-756317
]
ASF GitHub Bot logged work on HDFS-16538:
-
Author: ASF GitHub Bot
Created on: 13/Apr/22 10:55
Start Date: 13/Apr/22 10:55
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on PR #4167:
URL: https://github.com/apache/hadoop/pull/4167#issuecomment-1097910585
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 42s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 38m 25s | | trunk passed |
| +1 :green_heart: | compile | 1m 2s | | trunk passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 0m 54s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 34s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 0s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 50s | | trunk passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 40s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 46s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 25s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 49s | | the patch passed |
| +1 :green_heart: | compile | 0m 54s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 0m 54s | | the patch passed |
| +1 :green_heart: | compile | 0m 47s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 47s | | the patch passed |
| +1 :green_heart: | blanks | 0m 1s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 19s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 50s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 34s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 28s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 47s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 22s | | hadoop-hdfs-client in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not
generate ASF License warnings. |
| | | 100m 29s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4167/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4167 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux f5521b8832b5 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 8846746dd03b9b54a7db1d7d79f2835eb1c6adb6 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4167/1/testReport/ |
| Max. process+thread count | 543 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-client U:
hadoop-hdfs-project/hadoop-hdfs-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4167/1/console |
[jira] [Work logged] (HDFS-16514) Reduce the failover sleep time if multiple namenode are configured
[
https://issues.apache.org/jira/browse/HDFS-16514?focusedWorklogId=756312&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-756312
]
ASF GitHub Bot logged work on HDFS-16514:
-
Author: ASF GitHub Bot
Created on: 13/Apr/22 10:44
Start Date: 13/Apr/22 10:44
Worklog Time Spent: 10m
Work Description: cndaimin commented on code in PR #4088:
URL: https://github.com/apache/hadoop/pull/4088#discussion_r849339424
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/retry/RetryPolicies.java:
##
@@ -639,19 +647,24 @@ public FailoverOnNetworkExceptionRetry(RetryPolicy
fallbackPolicy,
public FailoverOnNetworkExceptionRetry(RetryPolicy fallbackPolicy,
int maxFailovers, int maxRetries, long delayMillis, long maxDelayBase)
{
+ this(fallbackPolicy, maxFailovers, maxRetries, delayMillis,
maxDelayBase, 2);
+}
+public FailoverOnNetworkExceptionRetry(RetryPolicy fallbackPolicy,
+int maxFailovers, int maxRetries, long delayMillis, long maxDelayBase,
int nnSize) {
this.fallbackPolicy = fallbackPolicy;
this.maxFailovers = maxFailovers;
this.maxRetries = maxRetries;
this.delayMillis = delayMillis;
this.maxDelayBase = maxDelayBase;
+ this.nnSize = nnSize;
}
/**
* @return 0 if this is our first failover/retry (i.e., retry immediately),
Review Comment:
The comments here looks need to be updated too.
Issue Time Tracking
---
Worklog Id: (was: 756312)
Time Spent: 1h (was: 50m)
> Reduce the failover sleep time if multiple namenode are configured
> --
>
> Key: HDFS-16514
> URL: https://issues.apache.org/jira/browse/HDFS-16514
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: qinyuren
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2022-03-21-18-11-37-191.png
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Recently, we used the [Standby Read] feature in our test cluster, and
> deployed 4 namenode as follow:
> node1 -> active nn
> node2 -> standby nn
> node3 -> observer nn
> node3 -> observer nn
> If we set ’dfs.client.failover.random.order=true‘, the client may failover
> twice and wait a long time to send msync to active namenode.
> !image-2022-03-21-18-11-37-191.png|width=698,height=169!
> I think we can reduce the sleep time of the first several failover based on
> the number of namenode
> For example, if 4 namenode are configured, the sleep time of first three
> failover operations is set to zero.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16539) RBF: Support refreshing/changing router fairness policy controller without rebooting router
[ https://issues.apache.org/jira/browse/HDFS-16539?focusedWorklogId=756311&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-756311 ] ASF GitHub Bot logged work on HDFS-16539: - Author: ASF GitHub Bot Created on: 13/Apr/22 10:44 Start Date: 13/Apr/22 10:44 Worklog Time Spent: 10m Work Description: kokonguyen191 opened a new pull request, #4168: URL: https://github.com/apache/hadoop/pull/4168 ### Description of PR Add support for refreshing/changing router fairness policy controller without the need to shutdown and boot a router. This patch makes use of the generic refresh feature on RouterAdmin. Usage: `hdfs dfsrouteradmin -refreshRouterArgs ROUTER_ADDR RefreshFairnessPolicyController` ### How was this patch tested? Unit test and local deployment. ### For code changes: - [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? Issue Time Tracking --- Worklog Id: (was: 756311) Remaining Estimate: 0h Time Spent: 10m > RBF: Support refreshing/changing router fairness policy controller without > rebooting router > --- > > Key: HDFS-16539 > URL: https://issues.apache.org/jira/browse/HDFS-16539 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Felix N >Assignee: Felix N >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > Add support for refreshing/changing router fairness policy controller without > the need to reboot a router. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16539) RBF: Support refreshing/changing router fairness policy controller without rebooting router
[ https://issues.apache.org/jira/browse/HDFS-16539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-16539: -- Labels: pull-request-available (was: ) > RBF: Support refreshing/changing router fairness policy controller without > rebooting router > --- > > Key: HDFS-16539 > URL: https://issues.apache.org/jira/browse/HDFS-16539 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Felix N >Assignee: Felix N >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Add support for refreshing/changing router fairness policy controller without > the need to reboot a router. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16539) RBF: Support refreshing/changing router fairness policy controller without rebooting router
[ https://issues.apache.org/jira/browse/HDFS-16539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix N updated HDFS-16539: --- Description: Add support for refreshing/changing router fairness policy controller without the need to reboot a router. > RBF: Support refreshing/changing router fairness policy controller without > rebooting router > --- > > Key: HDFS-16539 > URL: https://issues.apache.org/jira/browse/HDFS-16539 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Felix N >Assignee: Felix N >Priority: Minor > > Add support for refreshing/changing router fairness policy controller without > the need to reboot a router. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-16539) RBF: Support refreshing/changing router fairness policy controller without rebooting router
Felix N created HDFS-16539: -- Summary: RBF: Support refreshing/changing router fairness policy controller without rebooting router Key: HDFS-16539 URL: https://issues.apache.org/jira/browse/HDFS-16539 Project: Hadoop HDFS Issue Type: Improvement Reporter: Felix N Assignee: Felix N -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16509) Fix decommission UnsupportedOperationException: Remove unsupported
[
https://issues.apache.org/jira/browse/HDFS-16509?focusedWorklogId=756281&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-756281
]
ASF GitHub Bot logged work on HDFS-16509:
-
Author: ASF GitHub Bot
Created on: 13/Apr/22 09:39
Start Date: 13/Apr/22 09:39
Worklog Time Spent: 10m
Work Description: cndaimin commented on code in PR #4077:
URL: https://github.com/apache/hadoop/pull/4077#discussion_r849286228
##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java:
##
@@ -673,6 +675,36 @@ public void testDecommissionWithOpenfile()
fdos.close();
}
+ @Test(timeout = 2)
+ public void testDecommissionWithUnknownBlock() throws IOException {
+startCluster(1, 3);
+
+FSNamesystem ns = getCluster().getNamesystem(0);
+DatanodeManager datanodeManager =
ns.getBlockManager().getDatanodeManager();
+
+BlockInfo blk = new BlockInfoContiguous(new Block(1L), (short) 1);
+DatanodeDescriptor dn = datanodeManager.getDatanodes().iterator().next();
+dn.getStorageInfos()[0].addBlock(blk, blk);
+
+datanodeManager.getDatanodeAdminManager().startDecommission(dn);
+waitNodeDecommissioned(dn);
+ }
+
+ private void waitNodeDecommissioned(DatanodeInfo node) {
Review Comment:
Yes, we can use this method, updated. Thanks a lot! @Hexiaoqiao
Issue Time Tracking
---
Worklog Id: (was: 756281)
Time Spent: 2h 20m (was: 2h 10m)
> Fix decommission UnsupportedOperationException: Remove unsupported
> --
>
> Key: HDFS-16509
> URL: https://issues.apache.org/jira/browse/HDFS-16509
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.3.1, 3.3.2
>Reporter: daimin
>Assignee: daimin
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> We encountered an "UnsupportedOperationException: Remove unsupported" error
> when some datanodes were in decommission. The reason of the exception is that
> datanode.getBlockIterator() returns an Iterator does not support remove,
> however DatanodeAdminDefaultMonitor#processBlocksInternal invokes it.remove()
> when a block not found, e.g, the file containing the block is deleted.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16509) Fix decommission UnsupportedOperationException: Remove unsupported
[
https://issues.apache.org/jira/browse/HDFS-16509?focusedWorklogId=756274&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-756274
]
ASF GitHub Bot logged work on HDFS-16509:
-
Author: ASF GitHub Bot
Created on: 13/Apr/22 09:26
Start Date: 13/Apr/22 09:26
Worklog Time Spent: 10m
Work Description: Hexiaoqiao commented on code in PR #4077:
URL: https://github.com/apache/hadoop/pull/4077#discussion_r849274680
##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java:
##
@@ -673,6 +675,36 @@ public void testDecommissionWithOpenfile()
fdos.close();
}
+ @Test(timeout = 2)
+ public void testDecommissionWithUnknownBlock() throws IOException {
+startCluster(1, 3);
+
+FSNamesystem ns = getCluster().getNamesystem(0);
+DatanodeManager datanodeManager =
ns.getBlockManager().getDatanodeManager();
+
+BlockInfo blk = new BlockInfoContiguous(new Block(1L), (short) 1);
+DatanodeDescriptor dn = datanodeManager.getDatanodes().iterator().next();
+dn.getStorageInfos()[0].addBlock(blk, blk);
+
+datanodeManager.getDatanodeAdminManager().startDecommission(dn);
+waitNodeDecommissioned(dn);
+ }
+
+ private void waitNodeDecommissioned(DatanodeInfo node) {
Review Comment:
Hi @cndaimin, I think we could invoke this method rather than add new one,
what do you think
about?https://github.com/apache/hadoop/blob/45394433a112334e48087bd60674538af739922a/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/AdminStatesBaseTest.java#L332
Issue Time Tracking
---
Worklog Id: (was: 756274)
Time Spent: 2h 10m (was: 2h)
> Fix decommission UnsupportedOperationException: Remove unsupported
> --
>
> Key: HDFS-16509
> URL: https://issues.apache.org/jira/browse/HDFS-16509
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 3.3.1, 3.3.2
>Reporter: daimin
>Assignee: daimin
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> We encountered an "UnsupportedOperationException: Remove unsupported" error
> when some datanodes were in decommission. The reason of the exception is that
> datanode.getBlockIterator() returns an Iterator does not support remove,
> however DatanodeAdminDefaultMonitor#processBlocksInternal invokes it.remove()
> when a block not found, e.g, the file containing the block is deleted.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16538) EC decoding failed due to not enough valid inputs
[
https://issues.apache.org/jira/browse/HDFS-16538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
qinyuren updated HDFS-16538:
Description:
Currently, we found this error if the #StripeReader.readStripe() have more than
one block read failed.
We use the EC policy ec(6+3) in our cluster.
{code:java}
Caused by: org.apache.hadoop.HadoopIllegalArgumentException: No enough valid
inputs are provided, not recoverable
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.checkInputBuffers(ByteBufferDecodingState.java:119)
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.(ByteBufferDecodingState.java:47)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:86)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:170)
at
org.apache.hadoop.hdfs.StripeReader.decodeAndFillBuffer(StripeReader.java:462)
at
org.apache.hadoop.hdfs.StatefulStripeReader.decode(StatefulStripeReader.java:94)
at org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:406)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:327)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:420)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:892)
at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
{code}
{code:java}
while (!futures.isEmpty()) {
try {
StripingChunkReadResult r = StripedBlockUtil
.getNextCompletedStripedRead(service, futures, 0);
dfsStripedInputStream.updateReadStats(r.getReadStats());
DFSClient.LOG.debug("Read task returned: {}, for stripe {}",
r, alignedStripe);
StripingChunk returnedChunk = alignedStripe.chunks[r.index];
Preconditions.checkNotNull(returnedChunk);
Preconditions.checkState(returnedChunk.state == StripingChunk.PENDING);
if (r.state == StripingChunkReadResult.SUCCESSFUL) {
returnedChunk.state = StripingChunk.FETCHED;
alignedStripe.fetchedChunksNum++;
updateState4SuccessRead(r);
if (alignedStripe.fetchedChunksNum == dataBlkNum) {
clearFutures();
break;
}
} else {
returnedChunk.state = StripingChunk.MISSING;
// close the corresponding reader
dfsStripedInputStream.closeReader(readerInfos[r.index]);
final int missing = alignedStripe.missingChunksNum;
alignedStripe.missingChunksNum++;
checkMissingBlocks();
readDataForDecoding();
readParityChunks(alignedStripe.missingChunksNum - missing);
} {code}
If there are two blocks read failed, the #readDataForDecoding() will be called
twice;
The *decodeInputs array* will be initialized twice, and the *parity* *data* in
decodeInputs array which filled by #readParityChunks will be set to null.
was:
We found this error if the #StripeReader.readStripe() have more than one block
read failed.
{code:java}
Caused by: org.apache.hadoop.HadoopIllegalArgumentException: No enough valid
inputs are provided, not recoverable
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.checkInputBuffers(ByteBufferDecodingState.java:119)
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.(ByteBufferDecodingState.java:47)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:86)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:170)
at
org.apache.hadoop.hdfs.StripeReader.decodeAndFillBuffer(StripeReader.java:462)
at
org.apache.hadoop.hdfs.StatefulStripeReader.decode(StatefulStripeReader.java:94)
at org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:406)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:327)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:420)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:892)
at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
{code}
{code:java}
while (!futures.isEmpty()) {
try {
StripingChunkReadResult r = StripedBlockUtil
.getNextCompletedStripedRead(service, futures, 0);
dfsStripedInputStream.updateReadStats(r.getReadStats());
DFSClient.LOG.debug("Read task returned: {}, for stripe {}",
r, alignedStripe);
StripingChunk returnedChunk = alignedStripe.chunks[r.index];
Preconditions.checkNotNull(returnedChunk);
Preconditions.checkState(returnedChunk.state == StripingChunk.PENDING);
[jira] [Work logged] (HDFS-16538) EC decoding failed due to not enough valid inputs
[
https://issues.apache.org/jira/browse/HDFS-16538?focusedWorklogId=756267&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-756267
]
ASF GitHub Bot logged work on HDFS-16538:
-
Author: ASF GitHub Bot
Created on: 13/Apr/22 09:13
Start Date: 13/Apr/22 09:13
Worklog Time Spent: 10m
Work Description: liubingxing opened a new pull request, #4167:
URL: https://github.com/apache/hadoop/pull/4167
We found this error if the #StripeReader.readStripe() have more than one
block read failed.
```
Caused by: org.apache.hadoop.HadoopIllegalArgumentException: No enough valid
inputs are provided, not recoverable
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.checkInputBuffers(ByteBufferDecodingState.java:119)
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.(ByteBufferDecodingState.java:47)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:86)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:170)
at
org.apache.hadoop.hdfs.StripeReader.decodeAndFillBuffer(StripeReader.java:462)
at
org.apache.hadoop.hdfs.StatefulStripeReader.decode(StatefulStripeReader.java:94)
at
org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:406)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:327)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:420)
at
org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:892)
at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
```
```java
while (!futures.isEmpty()) {
try {
StripingChunkReadResult r = StripedBlockUtil
.getNextCompletedStripedRead(service, futures, 0);
dfsStripedInputStream.updateReadStats(r.getReadStats());
DFSClient.LOG.debug("Read task returned: {}, for stripe {}",
r, alignedStripe);
StripingChunk returnedChunk = alignedStripe.chunks[r.index];
Preconditions.checkNotNull(returnedChunk);
Preconditions.checkState(returnedChunk.state == StripingChunk.PENDING);
if (r.state == StripingChunkReadResult.SUCCESSFUL) {
returnedChunk.state = StripingChunk.FETCHED;
alignedStripe.fetchedChunksNum++;
updateState4SuccessRead(r);
if (alignedStripe.fetchedChunksNum == dataBlkNum) {
clearFutures();
break;
}
} else {
returnedChunk.state = StripingChunk.MISSING;
// close the corresponding reader
dfsStripedInputStream.closeReader(readerInfos[r.index]);
final int missing = alignedStripe.missingChunksNum;
alignedStripe.missingChunksNum++;
checkMissingBlocks();
readDataForDecoding();
readParityChunks(alignedStripe.missingChunksNum - missing);
}
```
If there are two blocks read failed, the #readDataForDecoding() will be
called twice;
The **decodeInputs array** will be initialized twice, and the **parity
data** in decodeInputs array which filled by #readParityChunks will be set to
null.
Issue Time Tracking
---
Worklog Id: (was: 756267)
Remaining Estimate: 0h
Time Spent: 10m
> EC decoding failed due to not enough valid inputs
> --
>
> Key: HDFS-16538
> URL: https://issues.apache.org/jira/browse/HDFS-16538
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: qinyuren
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We found this error if the #StripeReader.readStripe() have more than one
> block read failed.
>
> {code:java}
> Caused by: org.apache.hadoop.HadoopIllegalArgumentException: No enough valid
> inputs are provided, not recoverable
> at
> org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.checkInputBuffers(ByteBufferDecodingState.java:119)
> at
> org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.(ByteBufferDecodingState.java:47)
> at
> org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:86)
> at
> org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:170)
> at
> org.apache.hadoop.hdfs.StripeReader.decodeAndFillBuffer(StripeReader.java:462)
> at
> org.apache.hadoop.hdfs.StatefulStripeReader.decode(StatefulStripeReader.java:94)
> at
> org.apache.hadoop.hdfs.StripeReader.readStripe(Strip
[jira] [Updated] (HDFS-16538) EC decoding failed due to not enough valid inputs
[
https://issues.apache.org/jira/browse/HDFS-16538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDFS-16538:
--
Labels: pull-request-available (was: )
> EC decoding failed due to not enough valid inputs
> --
>
> Key: HDFS-16538
> URL: https://issues.apache.org/jira/browse/HDFS-16538
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: qinyuren
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We found this error if the #StripeReader.readStripe() have more than one
> block read failed.
>
> {code:java}
> Caused by: org.apache.hadoop.HadoopIllegalArgumentException: No enough valid
> inputs are provided, not recoverable
> at
> org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.checkInputBuffers(ByteBufferDecodingState.java:119)
> at
> org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.(ByteBufferDecodingState.java:47)
> at
> org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:86)
> at
> org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:170)
> at
> org.apache.hadoop.hdfs.StripeReader.decodeAndFillBuffer(StripeReader.java:462)
> at
> org.apache.hadoop.hdfs.StatefulStripeReader.decode(StatefulStripeReader.java:94)
> at
> org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:406)
> at
> org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:327)
> at
> org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:420)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:892)
> at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
> at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
> {code}
>
>
> {code:java}
> while (!futures.isEmpty()) {
> try {
> StripingChunkReadResult r = StripedBlockUtil
> .getNextCompletedStripedRead(service, futures, 0);
> dfsStripedInputStream.updateReadStats(r.getReadStats());
> DFSClient.LOG.debug("Read task returned: {}, for stripe {}",
> r, alignedStripe);
> StripingChunk returnedChunk = alignedStripe.chunks[r.index];
> Preconditions.checkNotNull(returnedChunk);
> Preconditions.checkState(returnedChunk.state == StripingChunk.PENDING);
> if (r.state == StripingChunkReadResult.SUCCESSFUL) {
> returnedChunk.state = StripingChunk.FETCHED;
> alignedStripe.fetchedChunksNum++;
> updateState4SuccessRead(r);
> if (alignedStripe.fetchedChunksNum == dataBlkNum) {
> clearFutures();
> break;
> }
> } else {
> returnedChunk.state = StripingChunk.MISSING;
> // close the corresponding reader
> dfsStripedInputStream.closeReader(readerInfos[r.index]);
> final int missing = alignedStripe.missingChunksNum;
> alignedStripe.missingChunksNum++;
> checkMissingBlocks();
> readDataForDecoding();
> readParityChunks(alignedStripe.missingChunksNum - missing);
> } {code}
> If there are two blocks read failed, the #readDataForDecoding() will be
> called twice;
> The *decodeInputs array* will be initialized twice, and the *parity* *data*
> in decodeInputs array which filled by #readParityChunks will be set to null.
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-16538) EC decoding failed due to not enough valid inputs
[
https://issues.apache.org/jira/browse/HDFS-16538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
qinyuren updated HDFS-16538:
Description:
We found this error if the #StripeReader.readStripe() have more than one block
read failed.
{code:java}
Caused by: org.apache.hadoop.HadoopIllegalArgumentException: No enough valid
inputs are provided, not recoverable
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.checkInputBuffers(ByteBufferDecodingState.java:119)
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.(ByteBufferDecodingState.java:47)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:86)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:170)
at
org.apache.hadoop.hdfs.StripeReader.decodeAndFillBuffer(StripeReader.java:462)
at
org.apache.hadoop.hdfs.StatefulStripeReader.decode(StatefulStripeReader.java:94)
at org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:406)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:327)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:420)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:892)
at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
{code}
{code:java}
while (!futures.isEmpty()) {
try {
StripingChunkReadResult r = StripedBlockUtil
.getNextCompletedStripedRead(service, futures, 0);
dfsStripedInputStream.updateReadStats(r.getReadStats());
DFSClient.LOG.debug("Read task returned: {}, for stripe {}",
r, alignedStripe);
StripingChunk returnedChunk = alignedStripe.chunks[r.index];
Preconditions.checkNotNull(returnedChunk);
Preconditions.checkState(returnedChunk.state == StripingChunk.PENDING);
if (r.state == StripingChunkReadResult.SUCCESSFUL) {
returnedChunk.state = StripingChunk.FETCHED;
alignedStripe.fetchedChunksNum++;
updateState4SuccessRead(r);
if (alignedStripe.fetchedChunksNum == dataBlkNum) {
clearFutures();
break;
}
} else {
returnedChunk.state = StripingChunk.MISSING;
// close the corresponding reader
dfsStripedInputStream.closeReader(readerInfos[r.index]);
final int missing = alignedStripe.missingChunksNum;
alignedStripe.missingChunksNum++;
checkMissingBlocks();
readDataForDecoding();
readParityChunks(alignedStripe.missingChunksNum - missing);
} {code}
If there are two blocks read failed, the #readDataForDecoding() will be called
twice;
The *decodeInputs array* will be initialized twice, and the *parity* *data* in
decodeInputs array which filled by #readParityChunks will be set to null.
was:
We found this error if the #StripeReader.readStripe() have more than one block
read failed.
{code:java}
Caused by: org.apache.hadoop.HadoopIllegalArgumentException: No enough valid
inputs are provided, not recoverable
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.checkInputBuffers(ByteBufferDecodingState.java:119)
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.(ByteBufferDecodingState.java:47)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:86)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:170)
at
org.apache.hadoop.hdfs.StripeReader.decodeAndFillBuffer(StripeReader.java:462)
at
org.apache.hadoop.hdfs.StatefulStripeReader.decode(StatefulStripeReader.java:94)
at org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:406)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:327)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:420)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:892)
at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
{code}
{code:java}
while (!futures.isEmpty()) {
try {
StripingChunkReadResult r = StripedBlockUtil
.getNextCompletedStripedRead(service, futures, 0);
dfsStripedInputStream.updateReadStats(r.getReadStats());
DFSClient.LOG.debug("Read task returned: {}, for stripe {}",
r, alignedStripe);
StripingChunk returnedChunk = alignedStripe.chunks[r.index];
Preconditions.checkNotNull(returnedChunk);
Preconditions.checkState(returnedChunk.state == StripingChunk.PENDING);
if (r.state == StripingChunkReadResult.SUCCESSFUL)
[jira] [Updated] (HDFS-16538) EC decoding failed due to not enough valid inputs
[
https://issues.apache.org/jira/browse/HDFS-16538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
qinyuren updated HDFS-16538:
Description:
We found this error if the #StripeReader.readStripe() have more than one block
read failed.
{code:java}
Caused by: org.apache.hadoop.HadoopIllegalArgumentException: No enough valid
inputs are provided, not recoverable
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.checkInputBuffers(ByteBufferDecodingState.java:119)
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.(ByteBufferDecodingState.java:47)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:86)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:170)
at
org.apache.hadoop.hdfs.StripeReader.decodeAndFillBuffer(StripeReader.java:462)
at
org.apache.hadoop.hdfs.StatefulStripeReader.decode(StatefulStripeReader.java:94)
at org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:406)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:327)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:420)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:892)
at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
{code}
{code:java}
while (!futures.isEmpty()) {
try {
StripingChunkReadResult r = StripedBlockUtil
.getNextCompletedStripedRead(service, futures, 0);
dfsStripedInputStream.updateReadStats(r.getReadStats());
DFSClient.LOG.debug("Read task returned: {}, for stripe {}",
r, alignedStripe);
StripingChunk returnedChunk = alignedStripe.chunks[r.index];
Preconditions.checkNotNull(returnedChunk);
Preconditions.checkState(returnedChunk.state == StripingChunk.PENDING);
if (r.state == StripingChunkReadResult.SUCCESSFUL) {
returnedChunk.state = StripingChunk.FETCHED;
alignedStripe.fetchedChunksNum++;
updateState4SuccessRead(r);
if (alignedStripe.fetchedChunksNum == dataBlkNum) {
clearFutures();
break;
}
} else {
returnedChunk.state = StripingChunk.MISSING;
// close the corresponding reader
dfsStripedInputStream.closeReader(readerInfos[r.index]);
final int missing = alignedStripe.missingChunksNum;
alignedStripe.missingChunksNum++;
checkMissingBlocks();
readDataForDecoding();
readParityChunks(alignedStripe.missingChunksNum - missing);
} {code}
If there are two blocks read failed, the #readDataForDecoding() will be called
twice;
The decodeInputs[] will be initialized twice, the *parity* data in
decodeInputs which filled by #readParityChunks will be set to null.
was:
We found this error if the #StripeReader.readStripe() have more than one block
read failed.
{code:java}
Caused by: org.apache.hadoop.HadoopIllegalArgumentException: No enough valid
inputs are provided, not recoverable
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.checkInputBuffers(ByteBufferDecodingState.java:119)
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.(ByteBufferDecodingState.java:47)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:86)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:170)
at
org.apache.hadoop.hdfs.StripeReader.decodeAndFillBuffer(StripeReader.java:462)
at
org.apache.hadoop.hdfs.StatefulStripeReader.decode(StatefulStripeReader.java:94)
at org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:406)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:327)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:420)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:892)
at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
{code}
{code:java}
while (!futures.isEmpty()) {
try {
StripingChunkReadResult r = StripedBlockUtil
.getNextCompletedStripedRead(service, futures, 0);
dfsStripedInputStream.updateReadStats(r.getReadStats());
DFSClient.LOG.debug("Read task returned: {}, for stripe {}",
r, alignedStripe);
StripingChunk returnedChunk = alignedStripe.chunks[r.index];
Preconditions.checkNotNull(returnedChunk);
Preconditions.checkState(returnedChunk.state == StripingChunk.PENDING);
if (r.state == StripingChunkReadResult.SUCCESSFUL) {
returnedC
[jira] [Updated] (HDFS-16538) EC decoding failed due to not enough valid inputs
[
https://issues.apache.org/jira/browse/HDFS-16538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
qinyuren updated HDFS-16538:
Description:
We found this error if the #StripeReader.readStripe() have more than one block
read failed.
{code:java}
Caused by: org.apache.hadoop.HadoopIllegalArgumentException: No enough valid
inputs are provided, not recoverable
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.checkInputBuffers(ByteBufferDecodingState.java:119)
at
org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.(ByteBufferDecodingState.java:47)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:86)
at
org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:170)
at
org.apache.hadoop.hdfs.StripeReader.decodeAndFillBuffer(StripeReader.java:462)
at
org.apache.hadoop.hdfs.StatefulStripeReader.decode(StatefulStripeReader.java:94)
at org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:406)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:327)
at
org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:420)
at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:892)
at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
{code}
{code:java}
while (!futures.isEmpty()) {
try {
StripingChunkReadResult r = StripedBlockUtil
.getNextCompletedStripedRead(service, futures, 0);
dfsStripedInputStream.updateReadStats(r.getReadStats());
DFSClient.LOG.debug("Read task returned: {}, for stripe {}",
r, alignedStripe);
StripingChunk returnedChunk = alignedStripe.chunks[r.index];
Preconditions.checkNotNull(returnedChunk);
Preconditions.checkState(returnedChunk.state == StripingChunk.PENDING);
if (r.state == StripingChunkReadResult.SUCCESSFUL) {
returnedChunk.state = StripingChunk.FETCHED;
alignedStripe.fetchedChunksNum++;
updateState4SuccessRead(r);
if (alignedStripe.fetchedChunksNum == dataBlkNum) {
clearFutures();
break;
}
} else {
returnedChunk.state = StripingChunk.MISSING;
// close the corresponding reader
dfsStripedInputStream.closeReader(readerInfos[r.index]);
final int missing = alignedStripe.missingChunksNum;
alignedStripe.missingChunksNum++;
checkMissingBlocks();
readDataForDecoding();
readParityChunks(alignedStripe.missingChunksNum - missing);
} {code}
If there are two blocks read failed, the #readDataForDecoding() will be called
twice;
The decodeInputs[] will be initialized twice, this cause the *parity* data in
decodeInputs which fill by #readParityChunks miss.
> EC decoding failed due to not enough valid inputs
> --
>
> Key: HDFS-16538
> URL: https://issues.apache.org/jira/browse/HDFS-16538
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: qinyuren
>Priority: Major
>
> We found this error if the #StripeReader.readStripe() have more than one
> block read failed.
>
> {code:java}
> Caused by: org.apache.hadoop.HadoopIllegalArgumentException: No enough valid
> inputs are provided, not recoverable
> at
> org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.checkInputBuffers(ByteBufferDecodingState.java:119)
> at
> org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.(ByteBufferDecodingState.java:47)
> at
> org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:86)
> at
> org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode(RawErasureDecoder.java:170)
> at
> org.apache.hadoop.hdfs.StripeReader.decodeAndFillBuffer(StripeReader.java:462)
> at
> org.apache.hadoop.hdfs.StatefulStripeReader.decode(StatefulStripeReader.java:94)
> at
> org.apache.hadoop.hdfs.StripeReader.readStripe(StripeReader.java:406)
> at
> org.apache.hadoop.hdfs.DFSStripedInputStream.readOneStripe(DFSStripedInputStream.java:327)
> at
> org.apache.hadoop.hdfs.DFSStripedInputStream.readWithStrategy(DFSStripedInputStream.java:420)
> at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:892)
> at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
> at java.base/java.io.DataInputStream.read(DataInputStream.java:149)
> {code}
>
>
> {code:java}
> while (!futures.isEmpty()) {
> try {
> StripingChunkReadResult r = StripedBlockUtil
> .getNextCompletedStripedRead(service, futures, 0);
> dfsStripedInpu
[jira] [Created] (HDFS-16538) EC decoding failed due to not enough valid inputs
qinyuren created HDFS-16538: --- Summary: EC decoding failed due to not enough valid inputs Key: HDFS-16538 URL: https://issues.apache.org/jira/browse/HDFS-16538 Project: Hadoop HDFS Issue Type: Bug Reporter: qinyuren -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDFS-16509) Fix decommission UnsupportedOperationException: Remove unsupported
[
https://issues.apache.org/jira/browse/HDFS-16509?focusedWorklogId=756246&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-756246
]
ASF GitHub Bot logged work on HDFS-16509:
-
Author: ASF GitHub Bot
Created on: 13/Apr/22 08:33
Start Date: 13/Apr/22 08:33
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on PR #4077:
URL: https://github.com/apache/hadoop/pull/4077#issuecomment-1097711076
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 44s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 38m 48s | | trunk passed |
| +1 :green_heart: | compile | 1m 29s | | trunk passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 1m 19s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 6s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 31s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 8s | | trunk passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 33s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 30s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 48s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 14s | | the patch passed |
| +1 :green_heart: | compile | 1m 21s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javac | 1m 21s | | the patch passed |
| +1 :green_heart: | compile | 1m 15s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 15s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 53s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 19s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 51s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | javadoc | 1m 27s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 14s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 28s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 226m 14s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 49s | | The patch does not
generate ASF License warnings. |
| | | 332m 44s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4077/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4077 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux d25f1674353c 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / a243b760e8acd1d223ba0b9ae144dee4b1993e14 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4077/5/testReport/ |
| Max. process+thread count | 3572 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4077/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This mes
[jira] [Work logged] (HDFS-14478) Add libhdfs APIs for openFile
[
https://issues.apache.org/jira/browse/HDFS-14478?focusedWorklogId=756227&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-756227
]
ASF GitHub Bot logged work on HDFS-14478:
-
Author: ASF GitHub Bot
Created on: 13/Apr/22 08:00
Start Date: 13/Apr/22 08:00
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on PR #4166:
URL: https://github.com/apache/hadoop/pull/4166#issuecomment-1097679558
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 54s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 24m 26s | | trunk passed |
| +1 :green_heart: | compile | 4m 11s | | trunk passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | compile | 4m 9s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | mvnsite | 0m 22s | | trunk passed |
| +1 :green_heart: | shadedclient | 56m 23s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 14s | | the patch passed |
| +1 :green_heart: | compile | 3m 57s | | the patch passed with JDK
Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04 |
| +1 :green_heart: | cc | 3m 57s | | the patch passed |
| +1 :green_heart: | golang | 3m 57s | | the patch passed |
| +1 :green_heart: | javac | 3m 57s | | the patch passed |
| +1 :green_heart: | compile | 4m 1s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 4m 1s | | the patch passed |
| +1 :green_heart: | golang | 4m 1s | | the patch passed |
| +1 :green_heart: | javac | 4m 1s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 16s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 52s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 117m 41s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 30s | | The patch does not
generate ASF License warnings. |
| | | 209m 9s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4166/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4166 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit
codespell golang |
| uname | Linux e0ace121df62 4.15.0-153-generic #160-Ubuntu SMP Thu Jul 29
06:54:29 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 0d383e5698f449c0fa1fd6c4abeb3b2ad136eb38 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.14.1+1-Ubuntu-0ubuntu1.20.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4166/1/testReport/ |
| Max. process+thread count | 612 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4166/1/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
Issue Time Tracking
---
Worklog Id: (was: 756227)
Time Spent: 1h 50m (was: 1h 40m)
> Add libhdfs APIs for openFile
> -
>
> Key: HDFS-14478
> URL: https://issues.apache.org/jira/browse/HDFS-14478
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client, libhdfs, native
>Reporter: Sahil Takiar
>Assignee: Steve Loughran
>
