[jira] [Commented] (RATIS-619) Ratis server on restart loads all the entries for the group
[
https://issues.apache.org/jira/browse/RATIS-619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16921633#comment-16921633
]
Mukul Kumar Singh commented on RATIS-619:
-
[~szetszwo], the current patch will help in avoiding memory/ cache being
populated with entries which will be evicted pretty soon. So this patch will
help with better memory utilization during restart.
> Ratis server on restart loads all the entries for the group
> ---
>
> Key: RATIS-619
> URL: https://issues.apache.org/jira/browse/RATIS-619
> Project: Ratis
> Issue Type: Bug
> Components: server
>Affects Versions: 0.3.0
>Reporter: Mukul Kumar Singh
>Assignee: Siddharth Wagle
>Priority: Blocker
> Labels: ozone
> Attachments: RATIS-619.01.patch, RATIS-619.02.patch
>
>
> Even after taking a snapshot, the raft log loads all the segment in the log
> {code}
> 2019-07-01 23:22:47,481 [pool-18-thread-1] INFO - Setting the last
> applied index to (t:2, i:15237039)
> {code}
> {code}
> 2019-07-01 23:22:47,516 INFO org.apache.ratis.server.RaftServerConfigKeys:
> raft.server.log.statemachine.data.caching.enabled = true (custom)
> 2019-07-01 23:22:47,531 INFO org.apache.ratis.server.impl.RaftServerImpl:
> 62941ca3-f244-4298-8497-f4c0bd57430a:group-4D230AB58084 set configuration 0:
> [1f3d7936-cb4e-4b68-86ed-578070472dea:1
> 0.17.213.36:9858, 62941ca3-f244-4298-8497-f4c0bd57430a:10.17.213.35:9858,
> f07c1f87-b377-40d9-8c56-4f1440c4fa77:10.17.213.37:9858], old=null at 0
> 2019-07-01 23:22:47,578 INFO org.apache.hadoop.http.HttpServer2: Jetty bound
> to port 9882
> 2019-07-01 23:22:47,579 INFO org.eclipse.jetty.server.Server:
> jetty-9.3.24.v20180605, build timestamp: 2018-06-05T10:11:56-07:00, git hash:
> 84205aa28f11a4f31f2a3b86d1bba2cc8ab69827
> 2019-07-01 23:22:47,601 INFO
> org.apache.ratis.server.raftlog.segmented.LogSegment: Successfully read 7461
> entries from segment file
> /data/1/ozone-0701/ratis/log/f7ddda32-45e0-4bec-a3e7-4d230
> ab58084/current/log_0-7460
> 2019-07-01 23:22:47,608 INFO org.eclipse.jetty.server.handler.ContextHandler:
> Started
> o.e.j.s.ServletContextHandler@6ce90bc5{/logs,file:///var/log/ozone/,AVAILABLE}
> 2019-07-01 23:22:47,608 INFO org.eclipse.jetty.server.handler.ContextHandler:
> Started
> o.e.j.s.ServletContextHandler@4b1c0397{/static,jar:file:/var/lib/hadoop-ozone/ozone-0.5.0-SNAPSHOT/share
> /ozone/lib/hadoop-hdds-container-service-0.5.0-SNAPSHOT.jar!/webapps/static,AVAILABLE}
> 2019-07-01 23:22:47,635 INFO
> org.apache.ratis.server.raftlog.segmented.LogSegment: Successfully read 7386
> entries from segment file
> /data/1/ozone-0701/ratis/log/f7ddda32-45e0-4bec-a3e7-4d230
> ab58084/current/log_7461-14846
> 2019-07-01 23:22:47,663 INFO
> org.apache.ratis.server.raftlog.segmented.LogSegment: Successfully read 7440
> entries from segment file
> /data/1/ozone-0701/ratis/log/f7ddda32-45e0-4bec-a3e7-4d230
> ab58084/current/log_14847-22286
> 2019-07-01 23:22:47,664 INFO org.eclipse.jetty.server.handler.ContextHandler:
> Started
> o.e.j.w.WebAppContext@8a62297{/,file:///tmp/jetty-0.0.0.0-9882-hddsDatanode-_-any-7539213566265642568.di
> r/webapp/,AVAILABLE}{/hddsDatanode}
> 2019-07-01 23:22:47,681 INFO
> org.apache.ratis.server.raftlog.segmented.LogSegment: Successfully read 7353
> entries from segment file
> /data/1/ozone-0701/ratis/log/f7ddda32-45e0-4bec-a3e7-4d230
> ab58084/current/log_22287-29639
> 2019-07-01 23:22:47,695 INFO org.eclipse.jetty.server.AbstractConnector:
> Started ServerConnector@5116ac09{HTTP/1.1,[http/1.1]}{0.0.0.0:9882}
> 2019-07-01 23:22:47,695 INFO
> org.apache.ratis.server.raftlog.segmented.LogSegment: Successfully read 7291
> entries from segment file
> /data/1/ozone-0701/ratis/log/f7ddda32-45e0-4bec-a3e7-4d230
> ab58084/current/log_29640-36930
> 2019-07-01 23:22:47,695 INFO org.eclipse.jetty.server.Server: Started @56648ms
> 2019-07-01 23:22:47,695 INFO org.apache.hadoop.hdds.server.BaseHttpServer:
> HTTP server of HDDSDATANODE is listening at http://0.0.0.0:9882
> 2019-07-01 23:22:47,709 INFO
> org.apache.ratis.server.raftlog.segmented.LogSegment: Successfully read 7049
> entries from segment file
> /data/1/ozone-0701/ratis/log/f7ddda32-45e0-4bec-a3e7-4d230
> ab58084/current/log_36931-43979
> 2019-07-01 23:22:47,732 INFO
> org.apache.ratis.server.raftlog.segmented.LogSegment: Successfully read 7141
> entries from segment file
> /data/1/ozone-0701/ratis/log/f7ddda32-45e0-4bec-a3e7-4d230
> ab58084/current/log_43980-51120
> 2019-07-01 23:22:47,747 INFO
> org.apache.ratis.server.raftlog.segmented.LogSegment: Successfully read 7321
> entries from segment file
> /data/1/ozone-0701/ratis/log/f7ddda32-45e0-4bec-a3e7-4d230ab58084/current/log_51121-58441
> 2019-07-01 23:22:47,768 INFO
>
[jira] [Commented] (RATIS-619) Ratis server on restart loads all the entries for the group
[
https://issues.apache.org/jira/browse/RATIS-619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916182#comment-16916182
]
Siddharth Wagle commented on RATIS-619:
---
[~szetszwo] This change was based on discussion with [~msingh] that we still
wanted the segment and index loaded. Individual entries will not be loaded in
the cache, so hoping to get some improvement.
> Ratis server on restart loads all the entries for the group
> ---
>
> Key: RATIS-619
> URL: https://issues.apache.org/jira/browse/RATIS-619
> Project: Ratis
> Issue Type: Bug
> Components: server
>Affects Versions: 0.3.0
>Reporter: Mukul Kumar Singh
>Assignee: Siddharth Wagle
>Priority: Blocker
> Labels: ozone
> Attachments: RATIS-619.01.patch, RATIS-619.02.patch
>
>
> Even after taking a snapshot, the raft log loads all the segment in the log
> {code}
> 2019-07-01 23:22:47,481 [pool-18-thread-1] INFO - Setting the last
> applied index to (t:2, i:15237039)
> {code}
> {code}
> 2019-07-01 23:22:47,516 INFO org.apache.ratis.server.RaftServerConfigKeys:
> raft.server.log.statemachine.data.caching.enabled = true (custom)
> 2019-07-01 23:22:47,531 INFO org.apache.ratis.server.impl.RaftServerImpl:
> 62941ca3-f244-4298-8497-f4c0bd57430a:group-4D230AB58084 set configuration 0:
> [1f3d7936-cb4e-4b68-86ed-578070472dea:1
> 0.17.213.36:9858, 62941ca3-f244-4298-8497-f4c0bd57430a:10.17.213.35:9858,
> f07c1f87-b377-40d9-8c56-4f1440c4fa77:10.17.213.37:9858], old=null at 0
> 2019-07-01 23:22:47,578 INFO org.apache.hadoop.http.HttpServer2: Jetty bound
> to port 9882
> 2019-07-01 23:22:47,579 INFO org.eclipse.jetty.server.Server:
> jetty-9.3.24.v20180605, build timestamp: 2018-06-05T10:11:56-07:00, git hash:
> 84205aa28f11a4f31f2a3b86d1bba2cc8ab69827
> 2019-07-01 23:22:47,601 INFO
> org.apache.ratis.server.raftlog.segmented.LogSegment: Successfully read 7461
> entries from segment file
> /data/1/ozone-0701/ratis/log/f7ddda32-45e0-4bec-a3e7-4d230
> ab58084/current/log_0-7460
> 2019-07-01 23:22:47,608 INFO org.eclipse.jetty.server.handler.ContextHandler:
> Started
> o.e.j.s.ServletContextHandler@6ce90bc5{/logs,file:///var/log/ozone/,AVAILABLE}
> 2019-07-01 23:22:47,608 INFO org.eclipse.jetty.server.handler.ContextHandler:
> Started
> o.e.j.s.ServletContextHandler@4b1c0397{/static,jar:file:/var/lib/hadoop-ozone/ozone-0.5.0-SNAPSHOT/share
> /ozone/lib/hadoop-hdds-container-service-0.5.0-SNAPSHOT.jar!/webapps/static,AVAILABLE}
> 2019-07-01 23:22:47,635 INFO
> org.apache.ratis.server.raftlog.segmented.LogSegment: Successfully read 7386
> entries from segment file
> /data/1/ozone-0701/ratis/log/f7ddda32-45e0-4bec-a3e7-4d230
> ab58084/current/log_7461-14846
> 2019-07-01 23:22:47,663 INFO
> org.apache.ratis.server.raftlog.segmented.LogSegment: Successfully read 7440
> entries from segment file
> /data/1/ozone-0701/ratis/log/f7ddda32-45e0-4bec-a3e7-4d230
> ab58084/current/log_14847-22286
> 2019-07-01 23:22:47,664 INFO org.eclipse.jetty.server.handler.ContextHandler:
> Started
> o.e.j.w.WebAppContext@8a62297{/,file:///tmp/jetty-0.0.0.0-9882-hddsDatanode-_-any-7539213566265642568.di
> r/webapp/,AVAILABLE}{/hddsDatanode}
> 2019-07-01 23:22:47,681 INFO
> org.apache.ratis.server.raftlog.segmented.LogSegment: Successfully read 7353
> entries from segment file
> /data/1/ozone-0701/ratis/log/f7ddda32-45e0-4bec-a3e7-4d230
> ab58084/current/log_22287-29639
> 2019-07-01 23:22:47,695 INFO org.eclipse.jetty.server.AbstractConnector:
> Started ServerConnector@5116ac09{HTTP/1.1,[http/1.1]}{0.0.0.0:9882}
> 2019-07-01 23:22:47,695 INFO
> org.apache.ratis.server.raftlog.segmented.LogSegment: Successfully read 7291
> entries from segment file
> /data/1/ozone-0701/ratis/log/f7ddda32-45e0-4bec-a3e7-4d230
> ab58084/current/log_29640-36930
> 2019-07-01 23:22:47,695 INFO org.eclipse.jetty.server.Server: Started @56648ms
> 2019-07-01 23:22:47,695 INFO org.apache.hadoop.hdds.server.BaseHttpServer:
> HTTP server of HDDSDATANODE is listening at http://0.0.0.0:9882
> 2019-07-01 23:22:47,709 INFO
> org.apache.ratis.server.raftlog.segmented.LogSegment: Successfully read 7049
> entries from segment file
> /data/1/ozone-0701/ratis/log/f7ddda32-45e0-4bec-a3e7-4d230
> ab58084/current/log_36931-43979
> 2019-07-01 23:22:47,732 INFO
> org.apache.ratis.server.raftlog.segmented.LogSegment: Successfully read 7141
> entries from segment file
> /data/1/ozone-0701/ratis/log/f7ddda32-45e0-4bec-a3e7-4d230
> ab58084/current/log_43980-51120
> 2019-07-01 23:22:47,747 INFO
> org.apache.ratis.server.raftlog.segmented.LogSegment: Successfully read 7321
> entries from segment file
> /data/1/ozone-0701/ratis/log/f7ddda32-45e0-4bec-a3e7-4d230ab58084/current/log_51121-58441
> 2019-07-01 23:22:47,768 INFO
>
[jira] [Commented] (RATIS-619) Ratis server on restart loads all the entries for the group
[
https://issues.apache.org/jira/browse/RATIS-619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916158#comment-16916158
]
Tsz Wo Nicholas Sze commented on RATIS-619:
---
The patch seems not fixing the problem: It still loads all the entries for the
group. It just does not keep the pre-snapshot entries in the cache.
> Ratis server on restart loads all the entries for the group
> ---
>
> Key: RATIS-619
> URL: https://issues.apache.org/jira/browse/RATIS-619
> Project: Ratis
> Issue Type: Bug
> Components: server
>Affects Versions: 0.3.0
>Reporter: Mukul Kumar Singh
>Assignee: Siddharth Wagle
>Priority: Blocker
> Labels: ozone
> Attachments: RATIS-619.01.patch, RATIS-619.02.patch
>
>
> Even after taking a snapshot, the raft log loads all the segment in the log
> {code}
> 2019-07-01 23:22:47,481 [pool-18-thread-1] INFO - Setting the last
> applied index to (t:2, i:15237039)
> {code}
> {code}
> 2019-07-01 23:22:47,516 INFO org.apache.ratis.server.RaftServerConfigKeys:
> raft.server.log.statemachine.data.caching.enabled = true (custom)
> 2019-07-01 23:22:47,531 INFO org.apache.ratis.server.impl.RaftServerImpl:
> 62941ca3-f244-4298-8497-f4c0bd57430a:group-4D230AB58084 set configuration 0:
> [1f3d7936-cb4e-4b68-86ed-578070472dea:1
> 0.17.213.36:9858, 62941ca3-f244-4298-8497-f4c0bd57430a:10.17.213.35:9858,
> f07c1f87-b377-40d9-8c56-4f1440c4fa77:10.17.213.37:9858], old=null at 0
> 2019-07-01 23:22:47,578 INFO org.apache.hadoop.http.HttpServer2: Jetty bound
> to port 9882
> 2019-07-01 23:22:47,579 INFO org.eclipse.jetty.server.Server:
> jetty-9.3.24.v20180605, build timestamp: 2018-06-05T10:11:56-07:00, git hash:
> 84205aa28f11a4f31f2a3b86d1bba2cc8ab69827
> 2019-07-01 23:22:47,601 INFO
> org.apache.ratis.server.raftlog.segmented.LogSegment: Successfully read 7461
> entries from segment file
> /data/1/ozone-0701/ratis/log/f7ddda32-45e0-4bec-a3e7-4d230
> ab58084/current/log_0-7460
> 2019-07-01 23:22:47,608 INFO org.eclipse.jetty.server.handler.ContextHandler:
> Started
> o.e.j.s.ServletContextHandler@6ce90bc5{/logs,file:///var/log/ozone/,AVAILABLE}
> 2019-07-01 23:22:47,608 INFO org.eclipse.jetty.server.handler.ContextHandler:
> Started
> o.e.j.s.ServletContextHandler@4b1c0397{/static,jar:file:/var/lib/hadoop-ozone/ozone-0.5.0-SNAPSHOT/share
> /ozone/lib/hadoop-hdds-container-service-0.5.0-SNAPSHOT.jar!/webapps/static,AVAILABLE}
> 2019-07-01 23:22:47,635 INFO
> org.apache.ratis.server.raftlog.segmented.LogSegment: Successfully read 7386
> entries from segment file
> /data/1/ozone-0701/ratis/log/f7ddda32-45e0-4bec-a3e7-4d230
> ab58084/current/log_7461-14846
> 2019-07-01 23:22:47,663 INFO
> org.apache.ratis.server.raftlog.segmented.LogSegment: Successfully read 7440
> entries from segment file
> /data/1/ozone-0701/ratis/log/f7ddda32-45e0-4bec-a3e7-4d230
> ab58084/current/log_14847-22286
> 2019-07-01 23:22:47,664 INFO org.eclipse.jetty.server.handler.ContextHandler:
> Started
> o.e.j.w.WebAppContext@8a62297{/,file:///tmp/jetty-0.0.0.0-9882-hddsDatanode-_-any-7539213566265642568.di
> r/webapp/,AVAILABLE}{/hddsDatanode}
> 2019-07-01 23:22:47,681 INFO
> org.apache.ratis.server.raftlog.segmented.LogSegment: Successfully read 7353
> entries from segment file
> /data/1/ozone-0701/ratis/log/f7ddda32-45e0-4bec-a3e7-4d230
> ab58084/current/log_22287-29639
> 2019-07-01 23:22:47,695 INFO org.eclipse.jetty.server.AbstractConnector:
> Started ServerConnector@5116ac09{HTTP/1.1,[http/1.1]}{0.0.0.0:9882}
> 2019-07-01 23:22:47,695 INFO
> org.apache.ratis.server.raftlog.segmented.LogSegment: Successfully read 7291
> entries from segment file
> /data/1/ozone-0701/ratis/log/f7ddda32-45e0-4bec-a3e7-4d230
> ab58084/current/log_29640-36930
> 2019-07-01 23:22:47,695 INFO org.eclipse.jetty.server.Server: Started @56648ms
> 2019-07-01 23:22:47,695 INFO org.apache.hadoop.hdds.server.BaseHttpServer:
> HTTP server of HDDSDATANODE is listening at http://0.0.0.0:9882
> 2019-07-01 23:22:47,709 INFO
> org.apache.ratis.server.raftlog.segmented.LogSegment: Successfully read 7049
> entries from segment file
> /data/1/ozone-0701/ratis/log/f7ddda32-45e0-4bec-a3e7-4d230
> ab58084/current/log_36931-43979
> 2019-07-01 23:22:47,732 INFO
> org.apache.ratis.server.raftlog.segmented.LogSegment: Successfully read 7141
> entries from segment file
> /data/1/ozone-0701/ratis/log/f7ddda32-45e0-4bec-a3e7-4d230
> ab58084/current/log_43980-51120
> 2019-07-01 23:22:47,747 INFO
> org.apache.ratis.server.raftlog.segmented.LogSegment: Successfully read 7321
> entries from segment file
> /data/1/ozone-0701/ratis/log/f7ddda32-45e0-4bec-a3e7-4d230ab58084/current/log_51121-58441
> 2019-07-01 23:22:47,768 INFO
> org.apache.ratis.server.raftlog.segmented.LogSegment: Successfully read
[jira] [Commented] (RATIS-619) Ratis server on restart loads all the entries for the group
[
https://issues.apache.org/jira/browse/RATIS-619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916083#comment-16916083
]
Hadoop QA commented on RATIS-619:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 29m 40s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 42m 16s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | ratis.logservice.server.TestMetaServer |
| | ratis.logservice.TestLogServiceWithGrpc |
| | ratis.logservice.TestLogServiceWithNetty |
| | ratis.grpc.TestRaftStateMachineExceptionWithGrpc |
| | ratis.netty.TestRaftSnapshotWithNetty |
| | ratis.grpc.TestWatchRequestWithGrpc |
| | ratis.netty.TestRetryCacheWithNettyRpc |
| | ratis.netty.TestRaftStateMachineExceptionWithNetty |
| | ratis.server.simulation.TestRaftStateMachineExceptionWithSimulatedRpc |
| | ratis.examples.filestore.TestFileStoreWithGrpc |
| | ratis.examples.filestore.TestFileStoreWithNetty |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/ratis:date2019-08-26 |
| JIRA Issue | RATIS-619 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12978615/RATIS-619.02.patch |
| Optional Tests | dupname asflicense javac javadoc unit findbugs
checkstyle compile |
| uname | Linux eefd3d266b7e 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24
10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-RATIS-Build/yetus-personality.sh
|
| git revision | master / f401c7a |
| maven | version: Apache Maven 3.6.0
(97c98ec64a1fdfee7767ce5ffb20918da4f719f3; 2018-10-24T18:41:47Z) |
| Default Java | 1.8.0_222 |
| unit |
https://builds.apache.org/job/PreCommit-RATIS-Build/941/artifact/out/patch-unit-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-RATIS-Build/941/testReport/ |
| Max. process+thread count | 1566 (vs. ulimit of 5000) |
| modules | C: ratis-server U: ratis-server |
| Console output |
https://builds.apache.org/job/PreCommit-RATIS-Build/941/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Ratis server on restart loads all the entries for the group
>
[jira] [Commented] (RATIS-619) Ratis server on restart loads all the entries for the group
[
https://issues.apache.org/jira/browse/RATIS-619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916077#comment-16916077
]
Hadoop QA commented on RATIS-619:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 27m 12s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 40m 22s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | ratis.TestMultiRaftGroup |
| | ratis.examples.filestore.TestFileStoreWithNetty |
| | ratis.netty.TestRaftStateMachineExceptionWithNetty |
| | ratis.netty.TestRaftReconfigurationWithNetty |
| | ratis.grpc.TestRaftSnapshotWithGrpc |
| | ratis.grpc.TestRaftAsyncWithGrpc |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/ratis:date2019-08-26 |
| JIRA Issue | RATIS-619 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12978615/RATIS-619.02.patch |
| Optional Tests | dupname asflicense javac javadoc unit findbugs
checkstyle compile |
| uname | Linux aa9cfca3cb18 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-RATIS-Build/yetus-personality.sh
|
| git revision | master / f401c7a |
| maven | version: Apache Maven 3.6.0
(97c98ec64a1fdfee7767ce5ffb20918da4f719f3; 2018-10-24T18:41:47Z) |
| Default Java | 1.8.0_222 |
| unit |
https://builds.apache.org/job/PreCommit-RATIS-Build/940/artifact/out/patch-unit-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-RATIS-Build/940/testReport/ |
| Max. process+thread count | 1758 (vs. ulimit of 5000) |
| modules | C: ratis-server U: ratis-server |
| Console output |
https://builds.apache.org/job/PreCommit-RATIS-Build/940/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Ratis server on restart loads all the entries for the group
> ---
>
> Key: RATIS-619
> URL: https://issues.apache.org/jira/browse/RATIS-619
> Project: Ratis
> Issue Type: Bug
> Components: server
>Affects Versions: 0.3.0
>Reporter: Mukul Kumar Singh
>
[jira] [Commented] (RATIS-619) Ratis server on restart loads all the entries for the group
[
https://issues.apache.org/jira/browse/RATIS-619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916043#comment-16916043
]
Siddharth Wagle commented on RATIS-619:
---
Thanks, [~an...@apache.org] for the review. Updated patch.
> Ratis server on restart loads all the entries for the group
> ---
>
> Key: RATIS-619
> URL: https://issues.apache.org/jira/browse/RATIS-619
> Project: Ratis
> Issue Type: Bug
> Components: server
>Affects Versions: 0.3.0
>Reporter: Mukul Kumar Singh
>Assignee: Siddharth Wagle
>Priority: Blocker
> Labels: ozone
> Attachments: RATIS-619.01.patch, RATIS-619.02.patch
>
>
> Even after taking a snapshot, the raft log loads all the segment in the log
> {code}
> 2019-07-01 23:22:47,481 [pool-18-thread-1] INFO - Setting the last
> applied index to (t:2, i:15237039)
> {code}
> {code}
> 2019-07-01 23:22:47,516 INFO org.apache.ratis.server.RaftServerConfigKeys:
> raft.server.log.statemachine.data.caching.enabled = true (custom)
> 2019-07-01 23:22:47,531 INFO org.apache.ratis.server.impl.RaftServerImpl:
> 62941ca3-f244-4298-8497-f4c0bd57430a:group-4D230AB58084 set configuration 0:
> [1f3d7936-cb4e-4b68-86ed-578070472dea:1
> 0.17.213.36:9858, 62941ca3-f244-4298-8497-f4c0bd57430a:10.17.213.35:9858,
> f07c1f87-b377-40d9-8c56-4f1440c4fa77:10.17.213.37:9858], old=null at 0
> 2019-07-01 23:22:47,578 INFO org.apache.hadoop.http.HttpServer2: Jetty bound
> to port 9882
> 2019-07-01 23:22:47,579 INFO org.eclipse.jetty.server.Server:
> jetty-9.3.24.v20180605, build timestamp: 2018-06-05T10:11:56-07:00, git hash:
> 84205aa28f11a4f31f2a3b86d1bba2cc8ab69827
> 2019-07-01 23:22:47,601 INFO
> org.apache.ratis.server.raftlog.segmented.LogSegment: Successfully read 7461
> entries from segment file
> /data/1/ozone-0701/ratis/log/f7ddda32-45e0-4bec-a3e7-4d230
> ab58084/current/log_0-7460
> 2019-07-01 23:22:47,608 INFO org.eclipse.jetty.server.handler.ContextHandler:
> Started
> o.e.j.s.ServletContextHandler@6ce90bc5{/logs,file:///var/log/ozone/,AVAILABLE}
> 2019-07-01 23:22:47,608 INFO org.eclipse.jetty.server.handler.ContextHandler:
> Started
> o.e.j.s.ServletContextHandler@4b1c0397{/static,jar:file:/var/lib/hadoop-ozone/ozone-0.5.0-SNAPSHOT/share
> /ozone/lib/hadoop-hdds-container-service-0.5.0-SNAPSHOT.jar!/webapps/static,AVAILABLE}
> 2019-07-01 23:22:47,635 INFO
> org.apache.ratis.server.raftlog.segmented.LogSegment: Successfully read 7386
> entries from segment file
> /data/1/ozone-0701/ratis/log/f7ddda32-45e0-4bec-a3e7-4d230
> ab58084/current/log_7461-14846
> 2019-07-01 23:22:47,663 INFO
> org.apache.ratis.server.raftlog.segmented.LogSegment: Successfully read 7440
> entries from segment file
> /data/1/ozone-0701/ratis/log/f7ddda32-45e0-4bec-a3e7-4d230
> ab58084/current/log_14847-22286
> 2019-07-01 23:22:47,664 INFO org.eclipse.jetty.server.handler.ContextHandler:
> Started
> o.e.j.w.WebAppContext@8a62297{/,file:///tmp/jetty-0.0.0.0-9882-hddsDatanode-_-any-7539213566265642568.di
> r/webapp/,AVAILABLE}{/hddsDatanode}
> 2019-07-01 23:22:47,681 INFO
> org.apache.ratis.server.raftlog.segmented.LogSegment: Successfully read 7353
> entries from segment file
> /data/1/ozone-0701/ratis/log/f7ddda32-45e0-4bec-a3e7-4d230
> ab58084/current/log_22287-29639
> 2019-07-01 23:22:47,695 INFO org.eclipse.jetty.server.AbstractConnector:
> Started ServerConnector@5116ac09{HTTP/1.1,[http/1.1]}{0.0.0.0:9882}
> 2019-07-01 23:22:47,695 INFO
> org.apache.ratis.server.raftlog.segmented.LogSegment: Successfully read 7291
> entries from segment file
> /data/1/ozone-0701/ratis/log/f7ddda32-45e0-4bec-a3e7-4d230
> ab58084/current/log_29640-36930
> 2019-07-01 23:22:47,695 INFO org.eclipse.jetty.server.Server: Started @56648ms
> 2019-07-01 23:22:47,695 INFO org.apache.hadoop.hdds.server.BaseHttpServer:
> HTTP server of HDDSDATANODE is listening at http://0.0.0.0:9882
> 2019-07-01 23:22:47,709 INFO
> org.apache.ratis.server.raftlog.segmented.LogSegment: Successfully read 7049
> entries from segment file
> /data/1/ozone-0701/ratis/log/f7ddda32-45e0-4bec-a3e7-4d230
> ab58084/current/log_36931-43979
> 2019-07-01 23:22:47,732 INFO
> org.apache.ratis.server.raftlog.segmented.LogSegment: Successfully read 7141
> entries from segment file
> /data/1/ozone-0701/ratis/log/f7ddda32-45e0-4bec-a3e7-4d230
> ab58084/current/log_43980-51120
> 2019-07-01 23:22:47,747 INFO
> org.apache.ratis.server.raftlog.segmented.LogSegment: Successfully read 7321
> entries from segment file
> /data/1/ozone-0701/ratis/log/f7ddda32-45e0-4bec-a3e7-4d230ab58084/current/log_51121-58441
> 2019-07-01 23:22:47,768 INFO
> org.apache.ratis.server.raftlog.segmented.LogSegment: Successfully read 7081
> entries from segment file
>
[jira] [Commented] (RATIS-619) Ratis server on restart loads all the entries for the group
[
https://issues.apache.org/jira/browse/RATIS-619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16914635#comment-16914635
]
Ankit Singhal commented on RATIS-619:
-
Looks good to me. +1 (Failed tests looks flaky as they fail sometimes without
the patch as well when ran locally)
Nit:
can we change the log message here(just to say specifically that we are
skipping only entry cache as segments and index will anyways be loaded):-
from
{code}
} else {
238 if (LOG.isTraceEnabled()) {
239 LOG.trace("Skip loading of log segment {} older than last "
+
240 "applied index {}", pi.getPath(), lastIndexInSnapshot);
241 }
242 }
{code}
to
{code}
} else {
238 if (LOG.isTraceEnabled()) {
239 LOG.trace("Skip loading entries of the log segment {} in
the cache which are older than " +
240 "applied index {} of the latest snapshot",
pi.getPath(), lastIndexInSnapshot);
{code}
> Ratis server on restart loads all the entries for the group
> ---
>
> Key: RATIS-619
> URL: https://issues.apache.org/jira/browse/RATIS-619
> Project: Ratis
> Issue Type: Bug
> Components: server
>Affects Versions: 0.3.0
>Reporter: Mukul Kumar Singh
>Assignee: Siddharth Wagle
>Priority: Blocker
> Labels: ozone
> Attachments: RATIS-619.01.patch
>
>
> Even after taking a snapshot, the raft log loads all the segment in the log
> {code}
> 2019-07-01 23:22:47,481 [pool-18-thread-1] INFO - Setting the last
> applied index to (t:2, i:15237039)
> {code}
> {code}
> 2019-07-01 23:22:47,516 INFO org.apache.ratis.server.RaftServerConfigKeys:
> raft.server.log.statemachine.data.caching.enabled = true (custom)
> 2019-07-01 23:22:47,531 INFO org.apache.ratis.server.impl.RaftServerImpl:
> 62941ca3-f244-4298-8497-f4c0bd57430a:group-4D230AB58084 set configuration 0:
> [1f3d7936-cb4e-4b68-86ed-578070472dea:1
> 0.17.213.36:9858, 62941ca3-f244-4298-8497-f4c0bd57430a:10.17.213.35:9858,
> f07c1f87-b377-40d9-8c56-4f1440c4fa77:10.17.213.37:9858], old=null at 0
> 2019-07-01 23:22:47,578 INFO org.apache.hadoop.http.HttpServer2: Jetty bound
> to port 9882
> 2019-07-01 23:22:47,579 INFO org.eclipse.jetty.server.Server:
> jetty-9.3.24.v20180605, build timestamp: 2018-06-05T10:11:56-07:00, git hash:
> 84205aa28f11a4f31f2a3b86d1bba2cc8ab69827
> 2019-07-01 23:22:47,601 INFO
> org.apache.ratis.server.raftlog.segmented.LogSegment: Successfully read 7461
> entries from segment file
> /data/1/ozone-0701/ratis/log/f7ddda32-45e0-4bec-a3e7-4d230
> ab58084/current/log_0-7460
> 2019-07-01 23:22:47,608 INFO org.eclipse.jetty.server.handler.ContextHandler:
> Started
> o.e.j.s.ServletContextHandler@6ce90bc5{/logs,file:///var/log/ozone/,AVAILABLE}
> 2019-07-01 23:22:47,608 INFO org.eclipse.jetty.server.handler.ContextHandler:
> Started
> o.e.j.s.ServletContextHandler@4b1c0397{/static,jar:file:/var/lib/hadoop-ozone/ozone-0.5.0-SNAPSHOT/share
> /ozone/lib/hadoop-hdds-container-service-0.5.0-SNAPSHOT.jar!/webapps/static,AVAILABLE}
> 2019-07-01 23:22:47,635 INFO
> org.apache.ratis.server.raftlog.segmented.LogSegment: Successfully read 7386
> entries from segment file
> /data/1/ozone-0701/ratis/log/f7ddda32-45e0-4bec-a3e7-4d230
> ab58084/current/log_7461-14846
> 2019-07-01 23:22:47,663 INFO
> org.apache.ratis.server.raftlog.segmented.LogSegment: Successfully read 7440
> entries from segment file
> /data/1/ozone-0701/ratis/log/f7ddda32-45e0-4bec-a3e7-4d230
> ab58084/current/log_14847-22286
> 2019-07-01 23:22:47,664 INFO org.eclipse.jetty.server.handler.ContextHandler:
> Started
> o.e.j.w.WebAppContext@8a62297{/,file:///tmp/jetty-0.0.0.0-9882-hddsDatanode-_-any-7539213566265642568.di
> r/webapp/,AVAILABLE}{/hddsDatanode}
> 2019-07-01 23:22:47,681 INFO
> org.apache.ratis.server.raftlog.segmented.LogSegment: Successfully read 7353
> entries from segment file
> /data/1/ozone-0701/ratis/log/f7ddda32-45e0-4bec-a3e7-4d230
> ab58084/current/log_22287-29639
> 2019-07-01 23:22:47,695 INFO org.eclipse.jetty.server.AbstractConnector:
> Started ServerConnector@5116ac09{HTTP/1.1,[http/1.1]}{0.0.0.0:9882}
> 2019-07-01 23:22:47,695 INFO
> org.apache.ratis.server.raftlog.segmented.LogSegment: Successfully read 7291
> entries from segment file
> /data/1/ozone-0701/ratis/log/f7ddda32-45e0-4bec-a3e7-4d230
> ab58084/current/log_29640-36930
> 2019-07-01 23:22:47,695 INFO org.eclipse.jetty.server.Server: Started @56648ms
> 2019-07-01 23:22:47,695 INFO org.apache.hadoop.hdds.server.BaseHttpServer:
> HTTP server of HDDSDATANODE is listening at http://0.0.0.0:9882
> 2019-07-01 23:22:47,709 INFO
> org.apache.ratis.server.raftlog.segmented.LogSegment: Successfully read 7049
> entries from segment file
>
[jira] [Commented] (RATIS-619) Ratis server on restart loads all the entries for the group
[
https://issues.apache.org/jira/browse/RATIS-619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16906598#comment-16906598
]
Hadoop QA commented on RATIS-619:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 28m 46s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 39m 45s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | ratis.netty.TestRaftReconfigurationWithNetty |
| | ratis.server.raftlog.TestRaftLogMetrics |
| | ratis.netty.TestRaftWithNetty |
| | ratis.server.simulation.TestRaftWithSimulatedRpc |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/ratis:date2019-08-13 |
| JIRA Issue | RATIS-619 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12977535/RATIS-619.01.patch |
| Optional Tests | dupname asflicense javac javadoc unit findbugs
checkstyle compile |
| uname | Linux 69ef0f28df22 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24
10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-RATIS-Build/yetus-personality.sh
|
| git revision | master / 2739b4e |
| maven | version: Apache Maven 3.6.0
(97c98ec64a1fdfee7767ce5ffb20918da4f719f3; 2018-10-24T18:41:47Z) |
| Default Java | 1.8.0_222 |
| unit |
https://builds.apache.org/job/PreCommit-RATIS-Build/930/artifact/out/patch-unit-root.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-RATIS-Build/930/testReport/ |
| Max. process+thread count | 1988 (vs. ulimit of 5000) |
| modules | C: ratis-server U: ratis-server |
| Console output |
https://builds.apache.org/job/PreCommit-RATIS-Build/930/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Ratis server on restart loads all the entries for the group
> ---
>
> Key: RATIS-619
> URL: https://issues.apache.org/jira/browse/RATIS-619
> Project: Ratis
> Issue Type: Bug
> Components: server
>Affects Versions: 0.3.0
>Reporter: Mukul Kumar Singh
>Assignee: Siddharth Wagle
>Priority: Blocker
> Labels: ozone
>
