[
https://issues.apache.org/jira/browse/RATIS-797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17018838#comment-17018838
]
Sammi Chen commented on RATIS-797:
----------------------------------
Hi [~msingh] and [~shashikant], I found this issue in my test cluster too. I
feel like it's not really the segment file is corrupted. It's kind of like the
'end' and 'start' parameter is not correctly set when passed to
LogSegment#loadSegment.
Following is the logs ,
2020-01-19 15:46:31,446 INFO org.apache.ratis.server.impl.RaftServerImpl:
ed90869c-317e-4303-8922-9fa83a3983cb@group-4F125BF42C14: set configuration 0:
[ed90869c-317e-4303-8922-9fa83a3983cb:10.120.113.172:9858,
b65b0b6c-b0bb-429f-a23d-467c72d4b85c:10.120.139.111:9858,
1da74a1d-f64d-4ad4-b04c-85f26687e683:10.121.124.44:9858], old=null at 0
2020-01-19 15:46:31,446 INFO
org.apache.ratis.server.raftlog.segmented.LogSegment: Successfully read 1
entries from segment file
/data7/db-meta/d3653de7-811c-460d-9072-4f125bf42c14/current/log_0-0
2020-01-19 15:46:31,449 INFO org.apache.ratis.server.impl.RaftServerImpl:
ed90869c-317e-4303-8922-9fa83a3983cb@group-4F125BF42C14: set configuration 1:
[ed90869c-317e-4303-8922-9fa83a3983cb:10.120.113.172:9858,
b65b0b6c-b0bb-429f-a23d-467c72d4b85c:10.120.139.111:9858,
1da74a1d-f64d-4ad4-b04c-85f26687e683:10.121.124.44:9858], old=null at 1
2020-01-19 15:46:31,465 INFO
org.apache.ratis.server.raftlog.segmented.LogSegment: Successfully read 311
entries from segment file
/data7/db-meta/d3653de7-811c-460d-9072-4f125bf42c14/current/log_1-311
2020-01-19 15:46:31,466 INFO org.apache.ratis.server.impl.RaftServerImpl:
ed90869c-317e-4303-8922-9fa83a3983cb@group-4F125BF42C14: set configuration 312:
[ed90869c-317e-4303-8922-9fa83a3983cb:10.120.113.172:9858,
b65b0b6c-b0bb-429f-a23d-467c72d4b85c:10.120.139.111:9858,
1da74a1d-f64d-4ad4-b04c-85f26687e683:10.121.124.44:9858], old=null at 312
2020-01-19 15:46:31,466 INFO
org.apache.ratis.server.raftlog.segmented.LogSegment: Successfully read 2
entries from segment file
/data7/db-meta/d3653de7-811c-460d-9072-4f125bf42c14/current/log_312-313
2020-01-19 15:46:31,466 INFO org.apache.ratis.server.impl.RaftServerImpl:
ed90869c-317e-4303-8922-9fa83a3983cb@group-4F125BF42C14: set configuration 314:
[ed90869c-317e-4303-8922-9fa83a3983cb:10.120.113.172:9858,
b65b0b6c-b0bb-429f-a23d-467c72d4b85c:10.120.139.111:9858,
1da74a1d-f64d-4ad4-b04c-85f26687e683:10.121.124.44:9858], old=null at 314
2020-01-19 15:46:31,475 INFO
org.apache.ratis.server.raftlog.segmented.LogSegment: Successfully read 299
entries from segment file
/data7/db-meta/d3653de7-811c-460d-9072-4f125bf42c14/current/log_314-612
2020-01-19 15:46:31,475 INFO org.apache.ratis.server.impl.RaftServerImpl:
ed90869c-317e-4303-8922-9fa83a3983cb@group-4F125BF42C14: set configuration 613:
[ed90869c-317e-4303-8922-9fa83a3983cb:10.120.113.172:9858,
b65b0b6c-b0bb-429f-a23d-467c72d4b85c:10.120.139.111:9858,
1da74a1d-f64d-4ad4-b04c-85f26687e683:10.121.124.44:9858], old=null at 613
2020-01-19 15:46:31,475 INFO
org.apache.ratis.server.raftlog.segmented.LogSegment: Successfully read 2
entries from segment file
/data7/db-meta/d3653de7-811c-460d-9072-4f125bf42c14/current/log_613-614
2020-01-19 15:46:31,475 INFO org.apache.ratis.server.impl.RaftServerImpl:
ed90869c-317e-4303-8922-9fa83a3983cb@group-4F125BF42C14: set configuration 615:
[ed90869c-317e-4303-8922-9fa83a3983cb:10.120.113.172:9858,
b65b0b6c-b0bb-429f-a23d-467c72d4b85c:10.120.139.111:9858,
1da74a1d-f64d-4ad4-b04c-85f26687e683:10.121.124.44:9858], old=null at 615
2020-01-19 15:46:31,475 INFO
org.apache.ratis.server.raftlog.segmented.LogSegment: Successfully read 2
entries from segment file
/data7/db-meta/d3653de7-811c-460d-9072-4f125bf42c14/current/log_615-616
2020-01-19 15:46:31,476 INFO org.apache.ratis.server.impl.RaftServerImpl:
ed90869c-317e-4303-8922-9fa83a3983cb@group-4F125BF42C14: set configuration 617:
[ed90869c-317e-4303-8922-9fa83a3983cb:10.120.113.172:9858,
b65b0b6c-b0bb-429f-a23d-467c72d4b85c:10.120.139.111:9858,
1da74a1d-f64d-4ad4-b04c-85f26687e683:10.121.124.44:9858], old=null at 617
2020-01-19 15:46:31,476 INFO
org.apache.ratis.server.raftlog.segmented.LogSegment: Successfully read 2
entries from segment file
/data7/db-meta/d3653de7-811c-460d-9072-4f125bf42c14/current/log_617-618
2020-01-19 15:46:31,476 INFO org.apache.ratis.server.impl.RaftServerImpl:
ed90869c-317e-4303-8922-9fa83a3983cb@group-4F125BF42C14: set configuration 619:
[ed90869c-317e-4303-8922-9fa83a3983cb:10.120.113.172:9858,
b65b0b6c-b0bb-429f-a23d-467c72d4b85c:10.120.139.111:9858,
1da74a1d-f64d-4ad4-b04c-85f26687e683:10.121.124.44:9858], old=null at 619
2020-01-19 15:46:31,477 INFO
org.apache.ratis.server.raftlog.segmented.LogSegment: Successfully read 2
entries from segment file
/data7/db-meta/d3653de7-811c-460d-9072-4f125bf42c14/current/log_inprogress_619
2020-01-19 15:46:31,477 WARN
org.apache.ratis.server.raftlog.segmented.LogSegment: Segment file is
corrupted: expected to have -619 entries but only 2 entries read successfully
> Ratis segment file corruption after server restart
> --------------------------------------------------
>
> Key: RATIS-797
> URL: https://issues.apache.org/jira/browse/RATIS-797
> Project: Ratis
> Issue Type: Bug
> Components: server
> Reporter: Shashikant Banerjee
> Assignee: Shashikant Banerjee
> Priority: Blocker
> Fix For: 0.5.0
>
>
> While testing ozone, it was observed that ratis segment show corruptions
> after a server restart
> {code:java}
> 2020-01-08 02:06:46,576 INFO
> org.apache.ratis.server.raftlog.segmented.LogSegment: Successfully read 1
> entries from segment file
> /metadata/hadoop-ozone/datanode/ratis/data/5e26b460-ca4e-4791-bf70-1fd535056988/current/log_inprogress_0
> 2020-01-08 02:06:46,576 WARN
> org.apache.ratis.server.raftlog.segmented.LogSegment: Segment file is
> corrupted: expected to have 0 entries but only 1 entries read successfully
> 2020-01-08 02:06:46,580 INFO
> org.apache.ratis.server.raftlog.segmented.SegmentedRaftLogWorker:
> 2d422fc8-f7c2-4e41-a59b-abbf76330dfe@group-1FD535056988-SegmentedRaftLogWorker:
> flushIndex: setUnconditionally 0 -> 0
> 2020-01-08 02:06:46,618 INFO org.eclipse.jetty.util.log: Logging initialized
> @1978ms
> 2020-01-08 02:06:46,738 INFO org.apache.ratis.server.RaftServerConfigKeys:
> raft.server.snaps
> 2020-01-16 07:51:12,268 WARN
> org.apache.ratis.server.raftlog.segmented.LogSegment: Segment file is
> corrupted: expected to have -3668 entries but only 3500 entries read
> successfully
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
