bharatviswa504 opened a new pull request #2299: URL: https://github.com/apache/ozone/pull/2299
## What changes were proposed in this pull request? Handle SIGTERM to perform cleanshut down in SCM. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-5290 ## How was this patch tested? Tested it on a cluster. ``` 2021-06-02 10:49:03,620 ERROR org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter: RECEIVED SIGNAL 15: SIGTERM 2021-06-02 10:49:03,624 INFO org.apache.hadoop.hdds.scm.server.StorageContainerManager: Stopping Replication Manager Service. 2021-06-02 10:49:03,624 INFO org.apache.hadoop.hdds.scm.container.ReplicationManager: Stopping Replication Monitor Thread. 2021-06-02 10:49:03,624 INFO org.apache.hadoop.hdds.scm.server.StorageContainerManager: Stopping the Datanode Admin Monitor. 2021-06-02 10:49:03,625 INFO org.apache.hadoop.hdds.scm.server.StorageContainerManager: Stopping Lease Manager of the command watchers 2021-06-02 10:49:03,625 INFO org.apache.hadoop.hdds.scm.server.StorageContainerManager: Stopping datanode service RPC server 2021-06-02 10:49:03,625 INFO org.apache.hadoop.hdds.scm.server.SCMDatanodeProtocolServer: Stopping the RPC server for DataNodes 2021-06-02 10:49:03,625 INFO org.apache.hadoop.ipc.Server: Stopping server on 9861 2021-06-02 10:49:03,630 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 9861 2021-06-02 10:49:03,632 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server Responder 2021-06-02 10:49:05,606 WARN org.apache.hadoop.hdds.scm.node.NodeStateManager: Current Thread is interrupted, shutting down HB processing thread for Node Manager. 2021-06-02 10:49:05,607 INFO org.apache.hadoop.hdds.scm.server.StorageContainerManager: Stopping block service RPC server 2021-06-02 10:49:05,607 INFO org.apache.hadoop.hdds.scm.server.SCMBlockProtocolServer: Stopping the RPC server for Block Protocol 2021-06-02 10:49:05,607 INFO org.apache.hadoop.ipc.Server: Stopping server on 9863 2021-06-02 10:49:05,610 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 9863 2021-06-02 10:49:05,611 INFO org.apache.hadoop.hdds.scm.server.StorageContainerManager: Stopping the StorageContainerLocationProtocol RPC server 2021-06-02 10:49:05,611 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server Responder 2021-06-02 10:49:05,611 INFO org.apache.hadoop.hdds.scm.server.SCMClientProtocolServer: Stopping the RPC server for Client Protocol 2021-06-02 10:49:05,611 INFO org.apache.hadoop.ipc.Server: Stopping server on 9860 2021-06-02 10:49:05,616 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 9860 2021-06-02 10:49:05,617 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server Responder 2021-06-02 10:49:05,617 INFO org.apache.hadoop.hdds.scm.server.StorageContainerManager: Stopping Storage Container Manager HTTP server. 2021-06-02 10:49:05,639 INFO org.eclipse.jetty.server.handler.ContextHandler: Stopped o.e.j.w.WebAppContext@73fb1d7f{scm,/,null,STOPPED}{jar:file:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.14269015/jars/hadoop-hdds-server-scm-1.1.0.7.1.7.0-414.jar!/webapps/scm} 2021-06-02 10:49:05,644 INFO org.eclipse.jetty.server.AbstractConnector: Stopped ServerConnector@332820f4{HTTP/1.1, (http/1.1)}{0.0.0.0:9876} 2021-06-02 10:49:05,644 INFO org.eclipse.jetty.server.session: node0 Stopped scavenging 2021-06-02 10:49:05,645 INFO org.eclipse.jetty.server.handler.ContextHandler: Stopped o.e.j.s.ServletContextHandler@12968227{static,/static,jar:file:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p0.14269015/jars/hadoop-hdds-server-scm-1.1.0.7.1.7.0-414.jar!/webapps/static,STOPPED} 2021-06-02 10:49:05,646 INFO org.eclipse.jetty.server.handler.ContextHandler: Stopped o.e.j.s.ServletContextHandler@58496c97{logs,/logs,file:///var/log/hadoop-ozone/,STOPPED} 2021-06-02 10:49:05,647 INFO org.apache.hadoop.hdds.scm.server.StorageContainerManager: Stopping Block Manager Service. 2021-06-02 10:49:05,647 INFO org.apache.hadoop.hdds.utils.BackgroundService: Shutting down service SCMBlockDeletingService 2021-06-02 10:49:05,647 INFO org.apache.hadoop.hdds.utils.BackgroundService: Shutting down service SCMBlockDeletingService 2021-06-02 10:49:05,648 INFO org.apache.hadoop.hdds.scm.server.StorageContainerManager: Stopping SCM Event Queue. 2021-06-02 10:49:05,652 INFO org.apache.hadoop.hdds.scm.server.StorageContainerManager: Stopping SCM HA services. 2021-06-02 10:49:05,652 INFO org.apache.hadoop.hdds.scm.ha.SCMRatisServerImpl: stopping ratis server 0.0.0.0:9894 2021-06-02 10:49:05,653 INFO org.apache.ratis.server.RaftServer: 3466735b-3f79-43d7-b9d0-de3452b5dacc: close 2021-06-02 10:49:05,654 INFO org.apache.ratis.server.RaftServer$Division: 3466735b-3f79-43d7-b9d0-de3452b5dacc@group-9FF618EF3790: shutdown 2021-06-02 10:49:05,654 INFO org.apache.ratis.util.JmxRegister: Successfully un-registered JMX Bean with object name Ratis:service=RaftServer,group=group-9FF618EF3790,id=3466735b-3f79-43d7-b9d0-de3452b5dacc 2021-06-02 10:49:05,655 INFO org.apache.ratis.server.impl.RoleInfo: 3466735b-3f79-43d7-b9d0-de3452b5dacc: shutdown 3466735b-3f79-43d7-b9d0-de3452b5dacc@group-9FF618EF3790-FollowerState 2021-06-02 10:49:05,655 INFO org.apache.ratis.server.impl.StateMachineUpdater: 3466735b-3f79-43d7-b9d0-de3452b5dacc@group-9FF618EF3790-StateMachineUpdater: set stopIndex = 1108 2021-06-02 10:49:05,656 INFO org.apache.ratis.server.impl.FollowerState: 3466735b-3f79-43d7-b9d0-de3452b5dacc@group-9FF618EF3790-FollowerState was interrupted: {} java.lang.InterruptedException: sleep interrupted at java.lang.Thread.sleep(Native Method) at java.lang.Thread.sleep(Thread.java:340) at java.util.concurrent.TimeUnit.sleep(TimeUnit.java:386) at org.apache.ratis.util.TimeDuration.sleep(TimeDuration.java:324) at org.apache.ratis.util.TimeDuration.sleep(TimeDuration.java:309) at org.apache.ratis.server.impl.FollowerState.run(FollowerState.java:118) 2021-06-02 10:49:05,656 INFO org.apache.hadoop.hdds.scm.ha.SCMStateMachine: Current Snapshot Index 1108, takeSnapshot took 1 ms 2021-06-02 10:49:05,659 INFO org.apache.ratis.server.impl.StateMachineUpdater: 3466735b-3f79-43d7-b9d0-de3452b5dacc@group-9FF618EF3790-StateMachineUpdater: Took a snapshot at index 1108 2021-06-02 10:49:05,659 INFO org.apache.ratis.server.impl.StateMachineUpdater: 3466735b-3f79-43d7-b9d0-de3452b5dacc@group-9FF618EF3790-StateMachineUpdater: snapshotIndex: updateIncreasingly 1105 -> 1108 2021-06-02 10:49:05,664 INFO org.apache.ratis.metrics.RatisMetrics: Unregistering Metrics Registry : ratis.state_machine.3466735b-3f79-43d7-b9d0-de3452b5dacc@group-9FF618EF3790 2021-06-02 10:49:05,665 INFO org.apache.ratis.server.RaftServer$Division: 3466735b-3f79-43d7-b9d0-de3452b5dacc@group-9FF618EF3790: closes. applyIndex: 1108 2021-06-02 10:49:05,666 INFO org.apache.ratis.server.raftlog.segmented.SegmentedRaftLogWorker: 3466735b-3f79-43d7-b9d0-de3452b5dacc@group-9FF618EF3790-SegmentedRaftLogWorker was interrupted, exiting. There are 0 tasks remaining in the queue. 2021-06-02 10:49:05,666 INFO org.apache.ratis.server.raftlog.segmented.SegmentedRaftLogWorker: 3466735b-3f79-43d7-b9d0-de3452b5dacc@group-9FF618EF3790-SegmentedRaftLogWorker close() 2021-06-02 10:49:05,667 INFO org.apache.ratis.metrics.RatisMetrics: Unregistering Metrics Registry : ratis.log_worker.3466735b-3f79-43d7-b9d0-de3452b5dacc@group-9FF618EF3790 2021-06-02 10:49:05,667 INFO org.apache.ratis.metrics.RatisMetrics: Unregistering Metrics Registry : ratis.leader_election.3466735b-3f79-43d7-b9d0-de3452b5dacc@group-9FF618EF3790 2021-06-02 10:49:05,667 INFO org.apache.ratis.metrics.RatisMetrics: Unregistering Metrics Registry : ratis.server.3466735b-3f79-43d7-b9d0-de3452b5dacc@group-9FF618EF3790 2021-06-02 10:49:05,667 INFO org.apache.ratis.grpc.server.GrpcService: 3466735b-3f79-43d7-b9d0-de3452b5dacc: shutdown server with port 9894 now 2021-06-02 10:49:05,682 INFO org.apache.ratis.grpc.server.GrpcService: 3466735b-3f79-43d7-b9d0-de3452b5dacc: shutdown server with port 9894 successfully 2021-06-02 10:49:05,682 INFO org.apache.ratis.util.JvmPauseMonitor: JvmPauseMonitor-3466735b-3f79-43d7-b9d0-de3452b5dacc: Stopped 2021-06-02 10:49:05,684 WARN org.apache.hadoop.hdds.scm.pipeline.BackgroundPipelineCreator: RatisPipelineUtilsThread is not running, just ignore. 2021-06-02 10:49:05,685 INFO org.apache.hadoop.hdds.scm.server.StorageContainerManager: Stopping SCM MetadataStore. 2021-06-02 10:49:05,686 WARN org.apache.ratis.grpc.server.GrpcServerProtocolService: 3466735b-3f79-43d7-b9d0-de3452b5dacc: installSnapshot onError, lastRequest: 23c8fce1-de0c-4630-b788-fa606b2d1ff1->3466735b-3f79-43d7-b9d0-de3452b5dacc#81826-t8,previous=(t:8, i:1107),leaderCommit=1107,initializing? true,entries: size=1, first=(t:8, i:1108), METADATAENTRY(c:1107): org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: CANCELLED: client cancelled 2021-06-02 10:49:05,689 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping StorageContainerManager metrics system... 2021-06-02 10:49:05,690 INFO org.apache.hadoop.metrics2.impl.MetricsSinkAdapter: prometheus thread interrupted. 2021-06-02 10:49:05,691 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: StorageContainerManager metrics system stopped. 2021-06-02 10:49:05,692 INFO org.apache.hadoop.hdds.scm.server.StorageContainerManagerStarter: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down StorageContainerManager at xxx ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
