[jira] [Commented] (RATIS-263) Deadlock in ratis because of leader election
[
https://issues.apache.org/jira/browse/RATIS-263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551136#comment-16551136
]
Hadoop QA commented on RATIS-263:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
9s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 18s{color} | {color:orange} root: The patch generated 2 new + 317 unchanged
- 2 fixed = 319 total (was 319) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 23m
43s{color} | {color:green} root in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
9s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 32m 13s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/ratis:date2018-07-20
|
| JIRA Issue | RATIS-263 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12932397/RATIS-263.patch |
| Optional Tests | asflicense javac javadoc unit findbugs checkstyle
compile |
| uname | Linux f8c25767ffcd 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-RATIS-Build/yetus-personality.sh
|
| git revision | master / 67bb9fc |
| Default Java | 1.8.0_171 |
| checkstyle |
https://builds.apache.org/job/PreCommit-RATIS-Build/248/artifact/out/diff-checkstyle-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-RATIS-Build/248/testReport/ |
| modules | C: ratis-server U: ratis-server |
| Console output |
https://builds.apache.org/job/PreCommit-RATIS-Build/248/console |
| Powered by | Apache Yetus 0.5.0 http://yetus.apache.org |
This message was automatically generated.
> Deadlock in ratis because of leader election
>
>
> Key: RATIS-263
> URL: https://issues.apache.org/jira/browse/RATIS-263
> Project: Ratis
> Issue Type: Bug
> Components: server
>Affects Versions: 0.2.0
>Reporter: Mukul Kumar Singh
>Assignee: Rajeshbabu Chintaguntla
>Priority: Critical
> Attachments: RATIS-263.patch
>
>
> There is a deadlock in Ratis between appendEntries and segmented raft log
> eviction.
> {code}
> "grpc-default-executor-6":
> at
> org.apache.ratis.server.impl.LogAppender.notifyAppend(LogAppender.java:466)
> - waiting to lock <0x83a1d398> (a
> org.apache.ratis.grpc.server.GRpcLogAppender)
>
[jira] [Commented] (RATIS-263) Deadlock in ratis because of leader election
[
https://issues.apache.org/jira/browse/RATIS-263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550596#comment-16550596
]
Rajeshbabu Chintaguntla commented on RATIS-263:
---
[~szetszwo] Yes the synchronized can be removed because the synchronization
required only when there is the leader state change. Uploaded the simple patch.
Please review.
> Deadlock in ratis because of leader election
>
>
> Key: RATIS-263
> URL: https://issues.apache.org/jira/browse/RATIS-263
> Project: Ratis
> Issue Type: Bug
> Components: server
>Affects Versions: 0.2.0
>Reporter: Mukul Kumar Singh
>Assignee: Rajeshbabu Chintaguntla
>Priority: Critical
> Attachments: RATIS-263.patch
>
>
> There is a deadlock in Ratis between appendEntries and segmented raft log
> eviction.
> {code}
> "grpc-default-executor-6":
> at
> org.apache.ratis.server.impl.LogAppender.notifyAppend(LogAppender.java:466)
> - waiting to lock <0x83a1d398> (a
> org.apache.ratis.grpc.server.GRpcLogAppender)
> at
> org.apache.ratis.server.impl.LeaderState$$Lambda$222/1001509199.accept(Unknown
> Source)
> at
> java.util.concurrent.CopyOnWriteArrayList.forEach(CopyOnWriteArrayList.java:890)
> at
> org.apache.ratis.server.impl.LeaderState$SenderList.forEach(LeaderState.java:93)
> at
> org.apache.ratis.server.impl.LeaderState.notifySenders(LeaderState.java:192)
> at
> org.apache.ratis.server.impl.RaftServerImpl.appendTransaction(RaftServerImpl.java:476)
> - locked <0x8084fac8> (a
> org.apache.ratis.server.impl.RaftServerImpl)
> at
> org.apache.ratis.server.impl.RaftServerImpl.submitClientRequestAsync(RaftServerImpl.java:524)
> at
> org.apache.ratis.server.impl.RaftServerProxy.submitClientRequestAsync(RaftServerProxy.java:153)
> at
> org.apache.ratis.grpc.client.RaftClientProtocolService$AppendRequestStreamObserver.processClientRequestAsync(RaftClientProtocolService.java:125)
> at
> org.apache.ratis.grpc.client.RaftClientProtocolService$AppendRequestStreamObserver$$Lambda$148/1250401784.accept(Unknown
> Source)
> at
> org.apache.ratis.util.SlidingWindow$Server.processRequestsFromHead(SlidingWindow.java:351)
> at
> org.apache.ratis.util.SlidingWindow$Server.receivedRequest(SlidingWindow.java:343)
> - locked <0xc0f7a060> (a
> org.apache.ratis.util.SlidingWindow$Server)
> at
> org.apache.ratis.grpc.client.RaftClientProtocolService$AppendRequestStreamObserver.onNext(RaftClientProtocolService.java:145)
> at
> org.apache.ratis.grpc.client.RaftClientProtocolService$AppendRequestStreamObserver.onNext(RaftClientProtocolService.java:109)
> at
> org.apache.ratis.shaded.io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:248)
> at
> org.apache.ratis.shaded.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:252)
> at
> org.apache.ratis.shaded.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:629)
> at
> org.apache.ratis.shaded.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> at
> org.apache.ratis.shaded.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> Thread evicting Segmented Raft Log
> {code}
> "Thread-72":
> at
> org.apache.ratis.server.impl.RaftServerImpl.getFollowerNextIndices(RaftServerImpl.java:1032)
> - waiting to lock <0x8084fac8> (a
> org.apache.ratis.server.impl.RaftServerImpl)
> at
> org.apache.ratis.server.storage.SegmentedRaftLog.checkAndEvictCache(SegmentedRaftLog.java:199)
> at
> org.apache.ratis.server.storage.SegmentedRaftLog.get(SegmentedRaftLog.java:190)
> at
> org.apache.ratis.server.impl.LogAppender.createRequest(LogAppender.java:178)
> at
> org.apache.ratis.grpc.server.GRpcLogAppender.appendLog(GRpcLogAppender.java:153)
> - locked <0x83a1d398> (a
> org.apache.ratis.grpc.server.GRpcLogAppender)
> at
> org.apache.ratis.grpc.server.GRpcLogAppender.run(GRpcLogAppender.java:93)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-263) Deadlock in ratis because of leader election
[
https://issues.apache.org/jira/browse/RATIS-263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16536653#comment-16536653
]
Tsz Wo Nicholas Sze commented on RATIS-263:
---
BTW, this seems to duplicate RATIS-159.
> Deadlock in ratis because of leader election
>
>
> Key: RATIS-263
> URL: https://issues.apache.org/jira/browse/RATIS-263
> Project: Ratis
> Issue Type: Bug
>Affects Versions: 0.2.0-alpha
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
>Priority: Critical
> Fix For: 0.2.0-alpha
>
>
> There is a deadlock in Ratis between appendEntries and segmented raft log
> eviction.
> {code}
> "grpc-default-executor-6":
> at
> org.apache.ratis.server.impl.LogAppender.notifyAppend(LogAppender.java:466)
> - waiting to lock <0x83a1d398> (a
> org.apache.ratis.grpc.server.GRpcLogAppender)
> at
> org.apache.ratis.server.impl.LeaderState$$Lambda$222/1001509199.accept(Unknown
> Source)
> at
> java.util.concurrent.CopyOnWriteArrayList.forEach(CopyOnWriteArrayList.java:890)
> at
> org.apache.ratis.server.impl.LeaderState$SenderList.forEach(LeaderState.java:93)
> at
> org.apache.ratis.server.impl.LeaderState.notifySenders(LeaderState.java:192)
> at
> org.apache.ratis.server.impl.RaftServerImpl.appendTransaction(RaftServerImpl.java:476)
> - locked <0x8084fac8> (a
> org.apache.ratis.server.impl.RaftServerImpl)
> at
> org.apache.ratis.server.impl.RaftServerImpl.submitClientRequestAsync(RaftServerImpl.java:524)
> at
> org.apache.ratis.server.impl.RaftServerProxy.submitClientRequestAsync(RaftServerProxy.java:153)
> at
> org.apache.ratis.grpc.client.RaftClientProtocolService$AppendRequestStreamObserver.processClientRequestAsync(RaftClientProtocolService.java:125)
> at
> org.apache.ratis.grpc.client.RaftClientProtocolService$AppendRequestStreamObserver$$Lambda$148/1250401784.accept(Unknown
> Source)
> at
> org.apache.ratis.util.SlidingWindow$Server.processRequestsFromHead(SlidingWindow.java:351)
> at
> org.apache.ratis.util.SlidingWindow$Server.receivedRequest(SlidingWindow.java:343)
> - locked <0xc0f7a060> (a
> org.apache.ratis.util.SlidingWindow$Server)
> at
> org.apache.ratis.grpc.client.RaftClientProtocolService$AppendRequestStreamObserver.onNext(RaftClientProtocolService.java:145)
> at
> org.apache.ratis.grpc.client.RaftClientProtocolService$AppendRequestStreamObserver.onNext(RaftClientProtocolService.java:109)
> at
> org.apache.ratis.shaded.io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:248)
> at
> org.apache.ratis.shaded.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:252)
> at
> org.apache.ratis.shaded.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:629)
> at
> org.apache.ratis.shaded.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> at
> org.apache.ratis.shaded.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> Thread evicting Segmented Raft Log
> {code}
> "Thread-72":
> at
> org.apache.ratis.server.impl.RaftServerImpl.getFollowerNextIndices(RaftServerImpl.java:1032)
> - waiting to lock <0x8084fac8> (a
> org.apache.ratis.server.impl.RaftServerImpl)
> at
> org.apache.ratis.server.storage.SegmentedRaftLog.checkAndEvictCache(SegmentedRaftLog.java:199)
> at
> org.apache.ratis.server.storage.SegmentedRaftLog.get(SegmentedRaftLog.java:190)
> at
> org.apache.ratis.server.impl.LogAppender.createRequest(LogAppender.java:178)
> at
> org.apache.ratis.grpc.server.GRpcLogAppender.appendLog(GRpcLogAppender.java:153)
> - locked <0x83a1d398> (a
> org.apache.ratis.grpc.server.GRpcLogAppender)
> at
> org.apache.ratis.grpc.server.GRpcLogAppender.run(GRpcLogAppender.java:93)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (RATIS-263) Deadlock in ratis because of leader election
[
https://issues.apache.org/jira/browse/RATIS-263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16536652#comment-16536652
]
Tsz Wo Nicholas Sze commented on RATIS-263:
---
It seems that we can safely remove "synchronized" from
RaftServerImpl.getFollowerNextIndices in order to fix the bug.
> Deadlock in ratis because of leader election
>
>
> Key: RATIS-263
> URL: https://issues.apache.org/jira/browse/RATIS-263
> Project: Ratis
> Issue Type: Bug
>Affects Versions: 0.2.0-alpha
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
>Priority: Critical
> Fix For: 0.2.0-alpha
>
>
> There is a deadlock in Ratis between appendEntries and segmented raft log
> eviction.
> {code}
> "grpc-default-executor-6":
> at
> org.apache.ratis.server.impl.LogAppender.notifyAppend(LogAppender.java:466)
> - waiting to lock <0x83a1d398> (a
> org.apache.ratis.grpc.server.GRpcLogAppender)
> at
> org.apache.ratis.server.impl.LeaderState$$Lambda$222/1001509199.accept(Unknown
> Source)
> at
> java.util.concurrent.CopyOnWriteArrayList.forEach(CopyOnWriteArrayList.java:890)
> at
> org.apache.ratis.server.impl.LeaderState$SenderList.forEach(LeaderState.java:93)
> at
> org.apache.ratis.server.impl.LeaderState.notifySenders(LeaderState.java:192)
> at
> org.apache.ratis.server.impl.RaftServerImpl.appendTransaction(RaftServerImpl.java:476)
> - locked <0x8084fac8> (a
> org.apache.ratis.server.impl.RaftServerImpl)
> at
> org.apache.ratis.server.impl.RaftServerImpl.submitClientRequestAsync(RaftServerImpl.java:524)
> at
> org.apache.ratis.server.impl.RaftServerProxy.submitClientRequestAsync(RaftServerProxy.java:153)
> at
> org.apache.ratis.grpc.client.RaftClientProtocolService$AppendRequestStreamObserver.processClientRequestAsync(RaftClientProtocolService.java:125)
> at
> org.apache.ratis.grpc.client.RaftClientProtocolService$AppendRequestStreamObserver$$Lambda$148/1250401784.accept(Unknown
> Source)
> at
> org.apache.ratis.util.SlidingWindow$Server.processRequestsFromHead(SlidingWindow.java:351)
> at
> org.apache.ratis.util.SlidingWindow$Server.receivedRequest(SlidingWindow.java:343)
> - locked <0xc0f7a060> (a
> org.apache.ratis.util.SlidingWindow$Server)
> at
> org.apache.ratis.grpc.client.RaftClientProtocolService$AppendRequestStreamObserver.onNext(RaftClientProtocolService.java:145)
> at
> org.apache.ratis.grpc.client.RaftClientProtocolService$AppendRequestStreamObserver.onNext(RaftClientProtocolService.java:109)
> at
> org.apache.ratis.shaded.io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:248)
> at
> org.apache.ratis.shaded.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:252)
> at
> org.apache.ratis.shaded.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:629)
> at
> org.apache.ratis.shaded.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> at
> org.apache.ratis.shaded.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> Thread evicting Segmented Raft Log
> {code}
> "Thread-72":
> at
> org.apache.ratis.server.impl.RaftServerImpl.getFollowerNextIndices(RaftServerImpl.java:1032)
> - waiting to lock <0x8084fac8> (a
> org.apache.ratis.server.impl.RaftServerImpl)
> at
> org.apache.ratis.server.storage.SegmentedRaftLog.checkAndEvictCache(SegmentedRaftLog.java:199)
> at
> org.apache.ratis.server.storage.SegmentedRaftLog.get(SegmentedRaftLog.java:190)
> at
> org.apache.ratis.server.impl.LogAppender.createRequest(LogAppender.java:178)
> at
> org.apache.ratis.grpc.server.GRpcLogAppender.appendLog(GRpcLogAppender.java:153)
> - locked <0x83a1d398> (a
> org.apache.ratis.grpc.server.GRpcLogAppender)
> at
> org.apache.ratis.grpc.server.GRpcLogAppender.run(GRpcLogAppender.java:93)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
