[
https://issues.apache.org/jira/browse/RATIS-483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang resolved RATIS-483.
-----------------------------------
Resolution: Won't Fix
I think the issue is not within Ozone itself but that something caused the
socket timeout. Usually it's thinks like JVM GC pause. With proper
configuration this shouldn't not be an issue and Ozone just respond to the
timeout.
> java.util.concurrent.TimeoutException exceptions seen in datanode logs
> -----------------------------------------------------------------------
>
> Key: RATIS-483
> URL: https://issues.apache.org/jira/browse/RATIS-483
> Project: Ratis
> Issue Type: Bug
> Reporter: Nilotpal Nandi
> Priority: Major
>
> steps taken :
> --------------------
> # created 12 datanode cluster.
> # started put key operation with size 100GB.
>
> Seeing following exceptions frequently in datanode logs
> --------------------------------------------------------------------------------
> {noformat}
> 2019-02-11 09:43:54,759 INFO org.apache.ratis.server.storage.RaftLogWorker:
> 541f579c-790a-44b0-9989-80415e119cf4-RaftLogWorker: created new log segment
> /data/disk1/ozone/meta/ratis/d0c1bf83-bfe5-4646-af49-d735ee312392/current/log_inprogress_3286
> 2019-02-11 09:44:17,443 WARN org.apache.ratis.server.impl.LogAppender:
> GrpcLogAppender(541f579c-790a-44b0-9989-80415e119cf4 ->
> 64b7e2c8-4663-42a4-a145-1bafefbccd4c): Failed get (t:1, i:3288),
> STATEMACHINELOGENTRY, client-3D1F07BD7204, cid=6942 in 11999532ns
> java.util.concurrent.TimeoutException
> at
> java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1771)
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
> at
> org.apache.ratis.server.storage.RaftLog$EntryWithData.getEntry(RaftLog.java:433)
> at org.apache.ratis.util.DataQueue.pollList(DataQueue.java:133)
> at
> org.apache.ratis.server.impl.LogAppender.createRequest(LogAppender.java:171)
> at
> org.apache.ratis.grpc.server.GrpcLogAppender.appendLog(GrpcLogAppender.java:152)
> at
> org.apache.ratis.grpc.server.GrpcLogAppender.runAppenderImpl(GrpcLogAppender.java:96)
> at org.apache.ratis.server.impl.LogAppender.runAppender(LogAppender.java:101)
> at java.lang.Thread.run(Thread.java:748)
> 2019-02-11 09:44:18,073 INFO org.apache.ratis.server.storage.RaftLogWorker:
> 541f579c-790a-44b0-9989-80415e119cf4-RaftLogWorker: Rolling segment
> log-3286_3295 to index:3295{noformat}
>
> {noformat}
> 2019-02-11 09:42:58,533 INFO org.apache.ratis.server.storage.RaftLogWorker:
> 541f579c-790a-44b0-9989-80415e119cf4-RaftLogWorker: created new log segment
> /data/disk1/ozone/meta/ratis/d0c1bf83-bfe5-4646-af49-d735ee312392/current/log_inprogress_3190
> 2019-02-11 09:43:20,620 WARN org.apache.ratis.server.impl.LogAppender:
> GrpcLogAppender(541f579c-790a-44b0-9989-80415e119cf4 ->
> 4e4626c5-f542-410a-a117-96386cd4e3b5): Failed get (t:1, i:3192),
> STATEMACHINELOGENTRY, client-AA54864AC41C, cid=6742 in 999326ns
> java.util.concurrent.TimeoutException
> at
> java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1771)
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
> at
> org.apache.ratis.server.storage.RaftLog$EntryWithData.getEntry(RaftLog.java:433)
> at org.apache.ratis.util.DataQueue.pollList(DataQueue.java:133)
> at
> org.apache.ratis.server.impl.LogAppender.createRequest(LogAppender.java:171)
> at
> org.apache.ratis.grpc.server.GrpcLogAppender.appendLog(GrpcLogAppender.java:152)
> at
> org.apache.ratis.grpc.server.GrpcLogAppender.runAppenderImpl(GrpcLogAppender.java:96)
> at org.apache.ratis.server.impl.LogAppender.runAppender(LogAppender.java:101)
> at java.lang.Thread.run(Thread.java:748)
> 2019-02-11 09:43:21,505 INFO org.apache.ratis.server.storage.RaftLogWorker:
> 541f579c-790a-44b0-9989-80415e119cf4-RaftLogWorker: Rolling segment
> log-3190_3199 to index:3199{noformat}
>
>
> {noformat}
> 2019-02-11 09:45:13,622 INFO org.apache.ratis.server.storage.RaftLogWorker:
> 6233367a-34c2-495b-ade4-6f7e11f5eb9e-RaftLogWorker: created new log segment
> /data/disk1/ozone/meta/ratis/390e6f99-716f-4708-8425-c1cdb4ef4cde/current/log_inprogress_3397
> 2019-02-11 09:45:13,637 WARN org.apache.ratis.server.impl.LogAppender:
> GrpcLogAppender(6233367a-34c2-495b-ade4-6f7e11f5eb9e ->
> 4c976a84-5271-474c-b297-ec9119617da9): Failed get (t:1, i:3397),
> STATEMACHINELOGENTRY, client-311AFF4454D6, cid=7135 in 74999474ns
> java.util.concurrent.TimeoutException
> at
> java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1771)
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
> at
> org.apache.ratis.server.storage.RaftLog$EntryWithData.getEntry(RaftLog.java:433)
> at org.apache.ratis.util.DataQueue.pollList(DataQueue.java:133)
> at
> org.apache.ratis.server.impl.LogAppender.createRequest(LogAppender.java:171)
> at
> org.apache.ratis.grpc.server.GrpcLogAppender.appendLog(GrpcLogAppender.java:152)
> at
> org.apache.ratis.grpc.server.GrpcLogAppender.runAppenderImpl(GrpcLogAppender.java:96)
> at org.apache.ratis.server.impl.LogAppender.runAppender(LogAppender.java:101)
> at java.lang.Thread.run(Thread.java:748)
> 2019-02-11 09:45:13,637 WARN org.apache.ratis.server.impl.LogAppender:
> GrpcLogAppender(6233367a-34c2-495b-ade4-6f7e11f5eb9e ->
> e7825b0a-b777-49cb-91e9-8a0b230a44ce): Failed get (t:1, i:3397),
> STATEMACHINELOGENTRY, client-311AFF4454D6, cid=7135 in 74999312ns
> java.util.concurrent.TimeoutException
> at
> java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1771)
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1915)
> at
> org.apache.ratis.server.storage.RaftLog$EntryWithData.getEntry(RaftLog.java:433)
> at org.apache.ratis.util.DataQueue.pollList(DataQueue.java:133)
> at
> org.apache.ratis.server.impl.LogAppender.createRequest(LogAppender.java:171)
> at
> org.apache.ratis.grpc.server.GrpcLogAppender.appendLog(GrpcLogAppender.java:152)
> at
> org.apache.ratis.grpc.server.GrpcLogAppender.runAppenderImpl(GrpcLogAppender.java:96)
> at org.apache.ratis.server.impl.LogAppender.runAppender(LogAppender.java:101)
> at java.lang.Thread.run(Thread.java:748){noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
