[jira] [Commented] (RATIS-569) StatusRuntimeException on the datanode because clients do not shutdown the observer cleanly.
[ https://issues.apache.org/jira/browse/RATIS-569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16925902#comment-16925902 ] Lokesh Jain commented on RATIS-569: --- [~swagle] Thanks for working on this! The changes look good to me. +1. > StatusRuntimeException on the datanode because clients do not shutdown the > observer cleanly. > > > Key: RATIS-569 > URL: https://issues.apache.org/jira/browse/RATIS-569 > Project: Ratis > Issue Type: Bug > Components: server >Affects Versions: 0.3.0 >Reporter: Mukul Kumar Singh >Assignee: Siddharth Wagle >Priority: Blocker > Labels: ozone > Attachments: RATIS-569.01.patch, RATIS-569.02.patch, > RATIS-569.03.patch > > > Running TestDataValidate in Ozone leads to StatusRuntimeException on the > datanode frequently. > This causes an unclean shutdown on the stream on the datanode. > In GrpcClientProtocolClient, shutdownNow should be followed by a > awaitTermination to wait for a clean shutdown. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (RATIS-569) StatusRuntimeException on the datanode because clients do not shutdown the observer cleanly.
[
https://issues.apache.org/jira/browse/RATIS-569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923934#comment-16923934
]
Hadoop QA commented on RATIS-569:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 18m 39s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 33s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | ratis.examples.filestore.TestFileStoreWithNetty |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/ratis:date2019-09-06 |
| JIRA Issue | RATIS-569 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12979606/RATIS-569.03.patch |
| Optional Tests | dupname asflicense javac javadoc unit findbugs
checkstyle compile |
| uname | Linux 399d8cfd51e9 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24
10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-RATIS-Build/yetus-personality.sh
|
| git revision | master / 741d8bd |
| maven | version: Apache Maven 3.6.0
(97c98ec64a1fdfee7767ce5ffb20918da4f719f3; 2018-10-24T18:41:47Z) |
| Default Java | 1.8.0_222 |
| unit |
https://builds.apache.org/job/PreCommit-RATIS-Build/958/artifact/out/patch-unit-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-RATIS-Build/958/testReport/ |
| Max. process+thread count | 2013 (vs. ulimit of 5000) |
| modules | C: ratis-grpc U: ratis-grpc |
| Console output |
https://builds.apache.org/job/PreCommit-RATIS-Build/958/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> StatusRuntimeException on the datanode because clients do not shutdown the
> observer cleanly.
>
>
> Key: RATIS-569
> URL: https://issues.apache.org/jira/browse/RATIS-569
> Project: Ratis
> Issue Type: Bug
> Components: server
>Affects Versions: 0.3.0
>Reporter: Mukul Kumar Singh
>Assignee: Siddharth Wagle
>Priority: Blocker
> Labels: ozone
> Attachments: RATIS-569.01.patch, RATIS-569.02.patch,
>
[jira] [Commented] (RATIS-569) StatusRuntimeException on the datanode because clients do not shutdown the observer cleanly.
[ https://issues.apache.org/jira/browse/RATIS-569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16923923#comment-16923923 ] Siddharth Wagle commented on RATIS-569: --- Verified the changes by manually running MiniOzoneChaosCluster which is able to reproduce the _StatusRuntimeException_ 100% of the times without the patch. > StatusRuntimeException on the datanode because clients do not shutdown the > observer cleanly. > > > Key: RATIS-569 > URL: https://issues.apache.org/jira/browse/RATIS-569 > Project: Ratis > Issue Type: Bug > Components: server >Affects Versions: 0.3.0 >Reporter: Mukul Kumar Singh >Assignee: Siddharth Wagle >Priority: Blocker > Labels: ozone > Attachments: RATIS-569.01.patch, RATIS-569.02.patch, > RATIS-569.03.patch > > > Running TestDataValidate in Ozone leads to StatusRuntimeException on the > datanode frequently. > This causes an unclean shutdown on the stream on the datanode. > In GrpcClientProtocolClient, shutdownNow should be followed by a > awaitTermination to wait for a clean shutdown. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (RATIS-569) StatusRuntimeException on the datanode because clients do not shutdown the observer cleanly.
[
https://issues.apache.org/jira/browse/RATIS-569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922951#comment-16922951
]
Hadoop QA commented on RATIS-569:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 16m 30s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 22s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
ratis.server.simulation.TestRaftReconfigurationWithSimulatedRpc |
| | ratis.netty.TestRaftReconfigurationWithNetty |
| | ratis.grpc.TestRaftAsyncWithGrpc |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/ratis:date2019-09-04 |
| JIRA Issue | RATIS-569 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12979480/RATIS-569.02.patch |
| Optional Tests | dupname asflicense javac javadoc unit findbugs
checkstyle compile |
| uname | Linux 1f0e42ea1f5c 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24
10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-RATIS-Build/yetus-personality.sh
|
| git revision | master / 78e95b9 |
| maven | version: Apache Maven 3.6.0
(97c98ec64a1fdfee7767ce5ffb20918da4f719f3; 2018-10-24T18:41:47Z) |
| Default Java | 1.8.0_222 |
| unit |
https://builds.apache.org/job/PreCommit-RATIS-Build/955/artifact/out/patch-unit-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-RATIS-Build/955/testReport/ |
| Max. process+thread count | 2047 (vs. ulimit of 5000) |
| modules | C: ratis-grpc U: ratis-grpc |
| Console output |
https://builds.apache.org/job/PreCommit-RATIS-Build/955/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> StatusRuntimeException on the datanode because clients do not shutdown the
> observer cleanly.
>
>
> Key: RATIS-569
> URL: https://issues.apache.org/jira/browse/RATIS-569
> Project: Ratis
> Issue Type: Bug
> Components: server
>Affects Versions: 0.3.0
>Reporter: Mukul Kumar Singh
>Assignee: Siddharth Wagle
>Priority:
[jira] [Commented] (RATIS-569) StatusRuntimeException on the datanode because clients do not shutdown the observer cleanly.
[ https://issues.apache.org/jira/browse/RATIS-569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916150#comment-16916150 ] Tsz Wo Nicholas Sze commented on RATIS-569: --- > ... Working on a unit test. Are you going to add a test? > StatusRuntimeException on the datanode because clients do not shutdown the > observer cleanly. > > > Key: RATIS-569 > URL: https://issues.apache.org/jira/browse/RATIS-569 > Project: Ratis > Issue Type: Bug > Components: server >Affects Versions: 0.3.0 >Reporter: Mukul Kumar Singh >Assignee: Siddharth Wagle >Priority: Blocker > Labels: ozone > Attachments: RATIS-569.01.patch > > > Running TestDataValidate in Ozone leads to StatusRuntimeException on the > datanode frequently. > This causes an unclean shutdown on the stream on the datanode. > In GrpcClientProtocolClient, shutdownNow should be followed by a > awaitTermination to wait for a clean shutdown. -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Commented] (RATIS-569) StatusRuntimeException on the datanode because clients do not shutdown the observer cleanly.
[
https://issues.apache.org/jira/browse/RATIS-569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916148#comment-16916148
]
Tsz Wo Nicholas Sze commented on RATIS-569:
---
Thanks [~swagle].
- The [gRPC
example|https://github.com/grpc/grpc-java/blob/master/examples/src/main/java/io/grpc/examples/routeguide/RouteGuideClient.java#L63]
uses 5s timeout. Let's do the same?
- Let's re-throw the example, if there is any.
{code}
public void close() throws InterruptedIOException {
try {
channel.shutdownNow().awaitTermination(5, TimeUnit.SECONDS);
} catch (InterruptedException e) {
throw IOUtils.toInterruptedIOException("Failed to close.", e);
}
}
{code}
> StatusRuntimeException on the datanode because clients do not shutdown the
> observer cleanly.
>
>
> Key: RATIS-569
> URL: https://issues.apache.org/jira/browse/RATIS-569
> Project: Ratis
> Issue Type: Bug
> Components: server
>Affects Versions: 0.3.0
>Reporter: Mukul Kumar Singh
>Assignee: Siddharth Wagle
>Priority: Blocker
> Labels: ozone
> Attachments: RATIS-569.01.patch
>
>
> Running TestDataValidate in Ozone leads to StatusRuntimeException on the
> datanode frequently.
> This causes an unclean shutdown on the stream on the datanode.
> In GrpcClientProtocolClient, shutdownNow should be followed by a
> awaitTermination to wait for a clean shutdown.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (RATIS-569) StatusRuntimeException on the datanode because clients do not shutdown the observer cleanly.
[
https://issues.apache.org/jira/browse/RATIS-569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16906612#comment-16906612
]
Hadoop QA commented on RATIS-569:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 5m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
47s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 42m 10s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 58m 52s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | ratis.examples.filestore.TestFileStoreWithNetty |
| | ratis.examples.filestore.TestFileStoreWithGrpc |
| | ratis.logservice.TestLogServiceWithGrpc |
| | ratis.logservice.server.TestMetaServer |
| | ratis.logservice.TestLogServiceWithNetty |
| | ratis.netty.TestRaftReconfigurationWithNetty |
| | ratis.server.simulation.TestRetryCacheWithSimulatedRpc |
| | ratis.grpc.TestRaftSnapshotWithGrpc |
| | ratis.server.raftlog.TestRaftLogMetrics |
| | ratis.grpc.TestWatchRequestWithGrpc |
| | ratis.server.simulation.TestServerRestartWithSimulatedRpc |
| | ratis.grpc.TestRaftAsyncWithGrpc |
| | ratis.server.simulation.TestRaftWithSimulatedRpc |
| | ratis.grpc.TestStateMachineShutdownWithGrpc |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/ratis:date2019-08-13 |
| JIRA Issue | RATIS-569 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12977534/RATIS-569.01.patch |
| Optional Tests | dupname asflicense javac javadoc unit findbugs
checkstyle compile |
| uname | Linux 757b73b7151f 4.15.0-52-generic #56-Ubuntu SMP Tue Jun 4
22:49:08 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-RATIS-Build/yetus-personality.sh
|
| git revision | master / 2739b4e |
| maven | version: Apache Maven 3.6.0
(97c98ec64a1fdfee7767ce5ffb20918da4f719f3; 2018-10-24T18:41:47Z) |
| Default Java | 1.8.0_222 |
| unit |
https://builds.apache.org/job/PreCommit-RATIS-Build/931/artifact/out/patch-unit-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-RATIS-Build/931/testReport/ |
| Max. process+thread count | 1733 (vs. ulimit of 5000) |
| modules | C: ratis-grpc U: ratis-grpc |
| Console output |
https://builds.apache.org/job/PreCommit-RATIS-Build/931/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This
[jira] [Commented] (RATIS-569) StatusRuntimeException on the datanode because clients do not shutdown the observer cleanly.
[ https://issues.apache.org/jira/browse/RATIS-569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16906555#comment-16906555 ] Siddharth Wagle commented on RATIS-569: --- cc: [~msingh] Working on a unit test. > StatusRuntimeException on the datanode because clients do not shutdown the > observer cleanly. > > > Key: RATIS-569 > URL: https://issues.apache.org/jira/browse/RATIS-569 > Project: Ratis > Issue Type: Bug > Components: server >Affects Versions: 0.3.0 >Reporter: Mukul Kumar Singh >Assignee: Siddharth Wagle >Priority: Blocker > Labels: ozone > Attachments: RATIS-569.01.patch > > > Running TestDataValidate in Ozone leads to StatusRuntimeException on the > datanode frequently. > This causes an unclean shutdown on the stream on the datanode. > In GrpcClientProtocolClient, shutdownNow should be followed by a > awaitTermination to wait for a clean shutdown. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
