[jira] [Commented] (HDDS-959) KeyOutputStream should handle retry failures
[ https://issues.apache.org/jira/browse/HDDS-959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16746609#comment-16746609 ] Hudson commented on HDDS-959: - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15789 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/15789/]) HDDS-959. KeyOutputStream should handle retry failures. Contributed by (msingh: rev 4ac0404fe01c2266068b6fc54588e3a4bcec3e12) * (edit) hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/KeyInputStream.java * (edit) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/TestContainerStateMachineIdempotency.java * (edit) hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/client/ContainerOperationClient.java * (edit) hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientManager.java * (edit) hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockOutputStream.java * (edit) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestXceiverClientManager.java * (edit) hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockInputStream.java * (edit) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientAbstract.java * (edit) hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/KeyOutputStream.java * (edit) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestContainerSmallFile.java * (edit) hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/scm/TestGetCommittedBlockLengthAndPutKey.java > KeyOutputStream should handle retry failures > > > Key: HDDS-959 > URL: https://issues.apache.org/jira/browse/HDDS-959 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Client >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Fix For: 0.4.0 > > Attachments: HDDS-959.001.patch, HDDS-959.002.patch, > HDDS-959.003.patch, HDDS-959.004.patch, HDDS-959.005.patch > > > With ratis version updated to 0.4.0-a8c4ca0-SNAPSHOT, retry failures are > fatal for a raft client. If an operation in raft client does not succeed > after maximum number of retries(RaftRetryFailureException) all subsequent > operations are failed with AlreadyClosedException. This jira aims to handle > such exceptions. Since we maintain a cache for clients in > XceiverClientManager, the corresponding client needs to be invalidated in the > cache. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-959) KeyOutputStream should handle retry failures
[
https://issues.apache.org/jira/browse/HDDS-959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16746109#comment-16746109
]
Hadoop QA commented on HDDS-959:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
31s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 36m 4s{color}
| {color:red} hadoop-ozone in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 3m 22s{color}
| {color:red} hadoop-hdds in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 50m 43s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.client.rpc.TestFailureHandlingByClient |
| | hadoop.ozone.container.TestContainerReplication |
| | hadoop.ozone.container.common.impl.TestContainerDataYaml |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDDS-959 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12955367/HDDS-959.005.patch |
| Optional Tests | asflicense javac javadoc unit findbugs checkstyle |
| uname | Linux 569f11e61642 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HDDS-Build/ozone.sh |
| git revision | trunk / 96a84b6 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| unit |
https://builds.apache.org/job/PreCommit-HDDS-Build/2063/artifact/out/patch-unit-hadoop-ozone.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDDS-Build/2063/artifact/out/patch-unit-hadoop-hdds.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDDS-Build/2063/testReport/ |
| Max. process+thread count | 1083 (vs. ulimit of 1) |
| modules | C: hadoop-hdds/client hadoop-ozone/client
hadoop-ozone/integration-test U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HDDS-Build/2063/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> KeyOutputStream should handle retry failures
>
>
> Key: HDDS-959
> URL: https://issues.apache.org/jira/browse/HDDS-959
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Client
>Reporter: Lokesh Jain
>Assignee:
[jira] [Commented] (HDDS-959) KeyOutputStream should handle retry failures
[ https://issues.apache.org/jira/browse/HDDS-959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16746047#comment-16746047 ] Lokesh Jain commented on HDDS-959: -- Uploaded rebased v5 patch. > KeyOutputStream should handle retry failures > > > Key: HDDS-959 > URL: https://issues.apache.org/jira/browse/HDDS-959 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Client >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Fix For: 0.4.0 > > Attachments: HDDS-959.001.patch, HDDS-959.002.patch, > HDDS-959.003.patch, HDDS-959.004.patch, HDDS-959.005.patch > > > With ratis version updated to 0.4.0-a8c4ca0-SNAPSHOT, retry failures are > fatal for a raft client. If an operation in raft client does not succeed > after maximum number of retries(RaftRetryFailureException) all subsequent > operations are failed with AlreadyClosedException. This jira aims to handle > such exceptions. Since we maintain a cache for clients in > XceiverClientManager, the corresponding client needs to be invalidated in the > cache. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-959) KeyOutputStream should handle retry failures
[ https://issues.apache.org/jira/browse/HDDS-959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743692#comment-16743692 ] Mukul Kumar Singh commented on HDDS-959: Thanks for updating the patch [~ljain]. +1, the v4 patch looks good to me. I will commit this shortly. > KeyOutputStream should handle retry failures > > > Key: HDDS-959 > URL: https://issues.apache.org/jira/browse/HDDS-959 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Client >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Fix For: 0.4.0 > > Attachments: HDDS-959.001.patch, HDDS-959.002.patch, > HDDS-959.003.patch, HDDS-959.004.patch > > > With ratis version updated to 0.4.0-a8c4ca0-SNAPSHOT, retry failures are > fatal for a raft client. If an operation in raft client does not succeed > after maximum number of retries(RaftRetryFailureException) all subsequent > operations are failed with AlreadyClosedException. This jira aims to handle > such exceptions. Since we maintain a cache for clients in > XceiverClientManager, the corresponding client needs to be invalidated in the > cache. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-959) KeyOutputStream should handle retry failures
[
https://issues.apache.org/jira/browse/HDDS-959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740083#comment-16740083
]
Hadoop QA commented on HDDS-959:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
28s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 34m 11s{color}
| {color:red} hadoop-ozone in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 5m
17s{color} | {color:green} hadoop-hdds in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 50m 28s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.ozone.container.common.statemachine.commandhandler.TestBlockDeletion |
| | hadoop.ozone.container.TestContainerReplication |
| | hadoop.ozone.web.client.TestKeys |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDDS-959 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12954527/HDDS-959.004.patch |
| Optional Tests | asflicense javac javadoc unit findbugs checkstyle |
| uname | Linux f8e2a75733ac 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HDDS-Build/ozone.sh |
| git revision | trunk / d4ca907 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| unit |
https://builds.apache.org/job/PreCommit-HDDS-Build/2042/artifact/out/patch-unit-hadoop-ozone.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDDS-Build/2042/testReport/ |
| Max. process+thread count | 1067 (vs. ulimit of 1) |
| modules | C: hadoop-hdds/client hadoop-ozone/client
hadoop-ozone/integration-test U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HDDS-Build/2042/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> KeyOutputStream should handle retry failures
>
>
> Key: HDDS-959
> URL: https://issues.apache.org/jira/browse/HDDS-959
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Client
>Reporter: Lokesh Jain
>Assignee: Lokesh Jain
>Priority: Major
> Fix For: 0.4.0
>
> Attachments:
[jira] [Commented] (HDDS-959) KeyOutputStream should handle retry failures
[ https://issues.apache.org/jira/browse/HDDS-959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740054#comment-16740054 ] Lokesh Jain commented on HDDS-959: -- [~shashikant] Uploaded rebased v4 patch. > KeyOutputStream should handle retry failures > > > Key: HDDS-959 > URL: https://issues.apache.org/jira/browse/HDDS-959 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Client >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Fix For: 0.4.0 > > Attachments: HDDS-959.001.patch, HDDS-959.002.patch, > HDDS-959.003.patch, HDDS-959.004.patch > > > With ratis version updated to 0.4.0-a8c4ca0-SNAPSHOT, retry failures are > fatal for a raft client. If an operation in raft client does not succeed > after maximum number of retries(RaftRetryFailureException) all subsequent > operations are failed with AlreadyClosedException. This jira aims to handle > such exceptions. Since we maintain a cache for clients in > XceiverClientManager, the corresponding client needs to be invalidated in the > cache. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-959) KeyOutputStream should handle retry failures
[ https://issues.apache.org/jira/browse/HDDS-959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16739279#comment-16739279 ] Shashikant Banerjee commented on HDDS-959: -- Thanks [~ljain]. The patch does not apply to trunk anymore. Can you please rename? > KeyOutputStream should handle retry failures > > > Key: HDDS-959 > URL: https://issues.apache.org/jira/browse/HDDS-959 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Client >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Fix For: 0.4.0 > > Attachments: HDDS-959.001.patch, HDDS-959.002.patch, > HDDS-959.003.patch > > > With ratis version updated to 0.4.0-a8c4ca0-SNAPSHOT, retry failures are > fatal for a raft client. If an operation in raft client does not succeed > after maximum number of retries(RaftRetryFailureException) all subsequent > operations are failed with AlreadyClosedException. This jira aims to handle > such exceptions. Since we maintain a cache for clients in > XceiverClientManager, the corresponding client needs to be invalidated in the > cache. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-959) KeyOutputStream should handle retry failures
[ https://issues.apache.org/jira/browse/HDDS-959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16738014#comment-16738014 ] Lokesh Jain commented on HDDS-959: -- The test failures do not seem to be related. The failure in TestKeys#testPutAndGetKeyWithDnRestart is not related because the test uses cluster with a single datanode. Therefore there is no issue of a leader election. The failure in TestOzoneRpcClient is related to XceiverClientGrpc. > KeyOutputStream should handle retry failures > > > Key: HDDS-959 > URL: https://issues.apache.org/jira/browse/HDDS-959 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: Ozone Client >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Fix For: 0.4.0 > > Attachments: HDDS-959.001.patch, HDDS-959.002.patch, > HDDS-959.003.patch > > > With ratis version updated to 0.4.0-a8c4ca0-SNAPSHOT, retry failures are > fatal for a raft client. If an operation in raft client does not succeed > after maximum number of retries(RaftRetryFailureException) all subsequent > operations are failed with AlreadyClosedException. This jira aims to handle > such exceptions. Since we maintain a cache for clients in > XceiverClientManager, the corresponding client needs to be invalidated in the > cache. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-959) KeyOutputStream should handle retry failures
[
https://issues.apache.org/jira/browse/HDDS-959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737222#comment-16737222
]
Hadoop QA commented on HDDS-959:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
26s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 25m 30s{color}
| {color:red} hadoop-ozone in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
31s{color} | {color:green} hadoop-hdds in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 40m 31s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.client.rpc.TestOzoneRpcClient |
| | hadoop.ozone.web.client.TestKeys |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDDS-959 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12954146/HDDS-959.003.patch |
| Optional Tests | asflicense javac javadoc unit findbugs checkstyle |
| uname | Linux 15d6cda13fca 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HDDS-Build/ozone.sh |
| git revision | trunk / 695e93c |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| unit |
https://builds.apache.org/job/PreCommit-HDDS-Build/2019/artifact/out/patch-unit-hadoop-ozone.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDDS-Build/2019/testReport/ |
| Max. process+thread count | 1097 (vs. ulimit of 1) |
| modules | C: hadoop-hdds/client hadoop-ozone/client
hadoop-ozone/integration-test U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HDDS-Build/2019/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> KeyOutputStream should handle retry failures
>
>
> Key: HDDS-959
> URL: https://issues.apache.org/jira/browse/HDDS-959
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Client
>Reporter: Lokesh Jain
>Assignee: Lokesh Jain
>Priority: Major
> Fix For: 0.4.0
>
> Attachments: HDDS-959.001.patch, HDDS-959.002.patch,
> HDDS-959.003.patch
>
>
> With ratis version updated to
[jira] [Commented] (HDDS-959) KeyOutputStream should handle retry failures
[ https://issues.apache.org/jira/browse/HDDS-959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16737009#comment-16737009 ] Lokesh Jain commented on HDDS-959: -- [~shashikant] Thanks for reviewing the patch! v3 patch addresses your comments. |In case of preallocation of blocks, there is a possibility that multiple blocks write on the same pipeline using the same XceiverClientSpi object.| The problem is that currently the xceiverClient is initialized during preallocation itself. v3 patch changes logic to initialize client only during data write. |we can maintain a static array/list of Exception Classes and use it | v3 patch introduces new function KeyOutputStream#checkForRetryFailure for checking the specific exceptions. > KeyOutputStream should handle retry failures > > > Key: HDDS-959 > URL: https://issues.apache.org/jira/browse/HDDS-959 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Attachments: HDDS-959.001.patch, HDDS-959.002.patch, > HDDS-959.003.patch > > > With ratis version updated to 0.4.0-a8c4ca0-SNAPSHOT, retry failures are > fatal for a raft client. If an operation in raft client does not succeed > after maximum number of retries(RaftRetryFailureException) all subsequent > operations are failed with AlreadyClosedException. This jira aims to handle > such exceptions. Since we maintain a cache for clients in > XceiverClientManager, the corresponding client needs to be invalidated in the > cache. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-959) KeyOutputStream should handle retry failures
[ https://issues.apache.org/jira/browse/HDDS-959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16735586#comment-16735586 ] Shashikant Banerjee commented on HDDS-959: -- Thanks [~ljain] for the patch. The patch looks good overall to me. Some comments: # In case of preallocation of blocks, there is a possibility that multiple blocks write on the same pipeline using the same XceiverClientSpi object. In case, one block write fails with AlreadyClosedException or RaftRetryFailureException, right now fail one and move on to the next bock which will still fail since it still has the reference to the old XceiverClientSpi object. In such cases, i think what we can do is refresh the XceiverClientSpi instance by doing release of the old one and reacquiring a client instance back. # KeyOutputStream#checkForException : > instead of passing a variable no of exception classes to be checked from mutiple functions, we can maintain a static array/list of Exception Classes and use it . # Adding some more comments on why and how AlreadyClosedException is handled will be insightful. > KeyOutputStream should handle retry failures > > > Key: HDDS-959 > URL: https://issues.apache.org/jira/browse/HDDS-959 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Attachments: HDDS-959.001.patch, HDDS-959.002.patch > > > With ratis version updated to 0.4.0-a8c4ca0-SNAPSHOT, retry failures are > fatal for a raft client. If an operation in raft client does not succeed > after maximum number of retries(RaftRetryFailureException) all subsequent > operations are failed with AlreadyClosedException. This jira aims to handle > such exceptions. Since we maintain a cache for clients in > XceiverClientManager, the corresponding client needs to be invalidated in the > cache. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-959) KeyOutputStream should handle retry failures
[
https://issues.apache.org/jira/browse/HDDS-959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16733070#comment-16733070
]
Hadoop QA commented on HDDS-959:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
26s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 30s{color}
| {color:red} hadoop-ozone in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
43s{color} | {color:green} hadoop-hdds in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 40s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.s3.endpoint.TestRootList |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDDS-959 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12953634/HDDS-959.002.patch |
| Optional Tests | asflicense javac javadoc unit findbugs checkstyle |
| uname | Linux 09988dcfdd51 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HDDS-Build/ozone.sh |
| git revision | trunk / 040a202 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| unit |
https://builds.apache.org/job/PreCommit-HDDS-Build/1991/artifact/out/patch-unit-hadoop-ozone.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDDS-Build/1991/testReport/ |
| Max. process+thread count | 199 (vs. ulimit of 1) |
| modules | C: hadoop-hdds/client hadoop-ozone/client
hadoop-ozone/integration-test U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HDDS-Build/1991/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> KeyOutputStream should handle retry failures
>
>
> Key: HDDS-959
> URL: https://issues.apache.org/jira/browse/HDDS-959
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Lokesh Jain
>Assignee: Lokesh Jain
>Priority: Major
> Attachments: HDDS-959.001.patch, HDDS-959.002.patch
>
>
> With ratis version updated to 0.4.0-a8c4ca0-SNAPSHOT, retry failures are
> fatal for a raft client. If an operation in raft client does not succeed
> after maximum number of
[jira] [Commented] (HDDS-959) KeyOutputStream should handle retry failures
[ https://issues.apache.org/jira/browse/HDDS-959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16733046#comment-16733046 ] Lokesh Jain commented on HDDS-959: -- v2 patch fixes failure in TestFailureHandlingByClient. > KeyOutputStream should handle retry failures > > > Key: HDDS-959 > URL: https://issues.apache.org/jira/browse/HDDS-959 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Attachments: HDDS-959.001.patch, HDDS-959.002.patch > > > With ratis version updated to 0.4.0-a8c4ca0-SNAPSHOT, retry failures are > fatal for a raft client. If an operation in raft client does not succeed > after maximum number of retries(RaftRetryFailureException) all subsequent > operations are failed with AlreadyClosedException. This jira aims to handle > such exceptions. Since we maintain a cache for clients in > XceiverClientManager, the corresponding client needs to be invalidated in the > cache. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-959) KeyOutputStream should handle retry failures
[
https://issues.apache.org/jira/browse/HDDS-959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16733029#comment-16733029
]
Hadoop QA commented on HDDS-959:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 8m
30s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
1s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
26s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source: . {color}
|
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 26m 0s{color}
| {color:red} hadoop-ozone in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m
41s{color} | {color:green} hadoop-hdds in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 49m 34s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.ozone.web.client.TestKeys |
| | hadoop.ozone.client.rpc.TestFailureHandlingByClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDDS-959 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12953620/HDDS-959.001.patch |
| Optional Tests | asflicense javac javadoc unit findbugs checkstyle |
| uname | Linux 64aafedc7d50 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HDDS-Build/ozone.sh |
| git revision | trunk / cb26f15 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| unit |
https://builds.apache.org/job/PreCommit-HDDS-Build/1990/artifact/out/patch-unit-hadoop-ozone.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDDS-Build/1990/testReport/ |
| Max. process+thread count | 1094 (vs. ulimit of 1) |
| modules | C: hadoop-hdds/client hadoop-ozone/client
hadoop-ozone/integration-test U: . |
| Console output |
https://builds.apache.org/job/PreCommit-HDDS-Build/1990/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> KeyOutputStream should handle retry failures
>
>
> Key: HDDS-959
> URL: https://issues.apache.org/jira/browse/HDDS-959
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Lokesh Jain
>Assignee: Lokesh Jain
>Priority: Major
> Attachments: HDDS-959.001.patch
>
>
> With ratis version updated to 0.4.0-a8c4ca0-SNAPSHOT, retry failures are
> fatal for a raft client. If an operation in raft client does
[jira] [Commented] (HDDS-959) KeyOutputStream should handle retry failures
[ https://issues.apache.org/jira/browse/HDDS-959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16732981#comment-16732981 ] Lokesh Jain commented on HDDS-959: -- [~shashikant] Can you please take a look? > KeyOutputStream should handle retry failures > > > Key: HDDS-959 > URL: https://issues.apache.org/jira/browse/HDDS-959 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Lokesh Jain >Assignee: Lokesh Jain >Priority: Major > Attachments: HDDS-959.001.patch > > > With ratis version updated to 0.4.0-a8c4ca0-SNAPSHOT, retry failures are > fatal for a raft client. If an operation in raft client does not succeed > after maximum number of retries(RaftRetryFailureException) all subsequent > operations are failed with AlreadyClosedException. This jira aims to handle > such exceptions. Since we maintain a cache for clients in > XceiverClientManager, the corresponding client needs to be invalidated in the > cache. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
