ChenSammi commented on PR #6587:
URL: https://github.com/apache/ozone/pull/6587#issuecomment-2076343886
A DN shutdown due to Ratis server is shutdown. First ContainerStateMachine
is closed, "Container statemachine is closed by ratis, terminating
HddsDatanodeService", then XceiverServerRatis is stopped, "Stopping
XceiverServerRatis 01effdc6-dad1-4bf3-916a-749d9aa7e5e5".
```
2024-04-24 18:06:16,666 WARN util.JvmPauseMonitor
(JvmPauseMonitor.java:detectPause(168)) -
JvmPauseMonitor-01effdc6-dad1-4bf3-916a-749d9aa7e5e5: Detected pause in JVM or
host machine approximately 93.265s without any GCs.
2024-04-24 18:06:16,666 ERROR server.RaftServer
(RaftServerProxy.java:handleJvmPause(237)) -
01effdc6-dad1-4bf3-916a-749d9aa7e5e5: JVM pause detected 93.265s longer than
the close-threshold 60s, shutting down ...
2024-04-24 18:06:16,678 INFO server.RaftServer
(RaftServerProxy.java:lambda$close$9(416)) -
01effdc6-dad1-4bf3-916a-749d9aa7e5e5: close
2024-04-24 18:06:16,684 INFO server.RaftServer$Division
(RaftServerImpl.java:lambda$close$3(526)) -
01effdc6-dad1-4bf3-916a-749d9aa7e5e5@group-5EA60976374E: shutdown
2024-04-24 18:06:16,685 INFO server.GrpcService
(GrpcService.java:closeImpl(311)) - 01effdc6-dad1-4bf3-916a-749d9aa7e5e5:
shutdown server org.apache.ratis.grpc.server.GrpcClientProtocolService now
2024-04-24 18:06:16,690 INFO util.JmxRegister
(JmxRegister.java:unregister(73)) - Successfully un-registered JMX Bean with
object name
Ratis:service=RaftServer,group=group-5EA60976374E,id=01effdc6-dad1-4bf3-916a-749d9aa7e5e5
2024-04-24 18:06:16,691 INFO impl.RoleInfo
(RoleInfo.java:shutdownLeaderState(94)) - 01effdc6-dad1-4bf3-916a-749d9aa7e5e5:
shutdown 01effdc6-dad1-4bf3-916a-749d9aa7e5e5@group-5EA60976374E-LeaderStateImpl
2024-04-24 18:06:16,724 INFO impl.PendingRequests
(PendingRequests.java:sendNotLeaderResponses(289)) -
01effdc6-dad1-4bf3-916a-749d9aa7e5e5@group-5EA60976374E-PendingRequests:
sendNotLeaderResponses
2024-04-24 18:06:16,727 INFO server.GrpcService
(GrpcService.java:closeImpl(320)) - 01effdc6-dad1-4bf3-916a-749d9aa7e5e5:
shutdown server org.apache.ratis.grpc.server.GrpcClientProtocolService
successfully
2024-04-24 18:06:16,727 INFO server.GrpcService
(GrpcService.java:closeImpl(311)) - 01effdc6-dad1-4bf3-916a-749d9aa7e5e5:
shutdown server GrpcServerProtocolService now
2024-04-24 18:06:16,728 INFO impl.StateMachineUpdater
(StateMachineUpdater.java:stopAndJoin(157)) -
01effdc6-dad1-4bf3-916a-749d9aa7e5e5@group-5EA60976374E-StateMachineUpdater:
set stopIndex = 4
2024-04-24 18:06:16,729 INFO ratis.ContainerStateMachine
(ContainerStateMachine.java:takeSnapshot(359)) - group-5EA60976374E: Taking a
snapshot at:(t:3, i:4) file
/tmp/datanode1/ratis/e9e7ba3c-7686-4b3a-96fd-5ea60976374e/sm/snapshot.3_4
2024-04-24 18:06:16,729 INFO server.GrpcService
(GrpcService.java:closeImpl(320)) - 01effdc6-dad1-4bf3-916a-749d9aa7e5e5:
shutdown server GrpcServerProtocolService successfully
2024-04-24 18:06:16,729 INFO server.GrpcService
(GrpcService.java:closeImpl(311)) - 01effdc6-dad1-4bf3-916a-749d9aa7e5e5:
shutdown server org.apache.ratis.grpc.server.GrpcAdminProtocolService now
2024-04-24 18:06:16,732 INFO ratis.ContainerStateMachine
(ContainerStateMachine.java:takeSnapshot(370)) - group-5EA60976374E: Finished
taking a snapshot at:(t:3, i:4)
file:/tmp/datanode1/ratis/e9e7ba3c-7686-4b3a-96fd-5ea60976374e/sm/snapshot.3_4
took: 4 ms
2024-04-24 18:06:16,733 INFO server.GrpcService
(GrpcService.java:closeImpl(320)) - 01effdc6-dad1-4bf3-916a-749d9aa7e5e5:
shutdown server org.apache.ratis.grpc.server.GrpcAdminProtocolService
successfully
2024-04-24 18:06:16,734 INFO impl.StateMachineUpdater
(StateMachineUpdater.java:takeSnapshot(295)) -
01effdc6-dad1-4bf3-916a-749d9aa7e5e5@group-5EA60976374E-StateMachineUpdater:
Took a snapshot at index 4
2024-04-24 18:06:16,734 INFO impl.StateMachineUpdater
(StateMachineUpdater.java:lambda$new$0(98)) -
01effdc6-dad1-4bf3-916a-749d9aa7e5e5@group-5EA60976374E-StateMachineUpdater:
snapshotIndex: updateIncreasingly 2 -> 4
2024-04-24 18:06:16,740 ERROR ratis.ContainerStateMachine
(ContainerStateMachine.java:close(1142)) - Container statemachine is closed by
ratis, terminating HddsDatanodeService
2024-04-24 18:06:26,754 INFO ozoneimpl.OzoneContainer
(OzoneContainer.java:stop(482)) - Attempting to stop container services.
2024-04-24 18:06:26,754 WARN ozoneimpl.AbstractBackgroundContainerScanner
(AbstractBackgroundContainerScanner.java:handleRemainingSleep(134)) -
Background container scan was interrupted.
2024-04-24 18:06:26,754 INFO ozoneimpl.AbstractBackgroundContainerScanner
(AbstractBackgroundContainerScanner.java:run(61)) -
Thread[ContainerMetadataScanner,5,main] exiting.
2024-04-24 18:06:26,755 INFO ozoneimpl.BackgroundContainerDataScanner
(BackgroundContainerDataScanner.java:shutdown(141)) -
ContainerDataScanner(/tmp/datanode1/storage/hdds) is shutting down.
2024-04-24 18:06:26,755 WARN ozoneimpl.AbstractBackgroundContainerScanner
(AbstractBackgroundContainerScanner.java:handleRemainingSleep(134)) -
Background container scan was interrupted.
2024-04-24 18:06:26,755 INFO ozoneimpl.AbstractBackgroundContainerScanner
(AbstractBackgroundContainerScanner.java:run(61)) -
ContainerDataScanner(/tmp/datanode1/storage/hdds,
DS-af727dc0-66f9-4db9-8f1f-8ce487a40766) exiting.
2024-04-24 18:06:26,755 INFO ozoneimpl.OnDemandContainerDataScanner
(OnDemandContainerDataScanner.java:shutdownScanner(206)) - On-demand container
scanner is shutting down.
2024-04-24 18:06:26,756 INFO ratis.XceiverServerRatis
(XceiverServerRatis.java:stop(604)) - Stopping XceiverServerRatis
01effdc6-dad1-4bf3-916a-749d9aa7e5e5
2024-04-24 18:06:26,757 INFO util.JvmPauseMonitor
(JvmPauseMonitor.java:run(152)) -
JvmPauseMonitor-01effdc6-dad1-4bf3-916a-749d9aa7e5e5: Stopped
2024-04-24 18:06:28,892 INFO volume.HddsVolume
(HddsVolume.java:closeDbStore(470)) - SchemaV3 db is stopped at
/tmp/datanode1/storage/hdds/CID-9ba4109c-68b1-4311-9623-42f82149fb80/DS-af727dc0-66f9-4db9-8f1f-8ce487a40766/container.db
for volume DS-af727dc0-66f9-4db9-8f1f-8ce487a40766
2024-04-24 18:06:28,893 INFO utils.BackgroundService
(BackgroundService.java:shutdown(160)) - Shutting down service
BlockDeletingService
2024-04-24 18:06:28,893 INFO utils.BackgroundService
(BackgroundService.java:shutdown(160)) - Shutting down service
StaleRecoveringContainerScrubbingService
2024-04-24 18:06:28,894 INFO statemachine.DatanodeStateMachine
(DatanodeStateMachine.java:stopDaemon(640)) - Ozone container server stopped.
2024-04-24 18:06:28,899 INFO handler.ContextHandler
(ContextHandler.java:doStop(1159)) - Stopped
o.e.j.w.WebAppContext@5fbdc49b{hddsDatanode,/,null,STOPPED}{file:/Users/sammi/workspace/hadoop-ozone/hadoop-hdds/container-service/target/classes/webapps/hddsDatanode}
2024-04-24 18:06:28,903 INFO server.AbstractConnector
(AbstractConnector.java:doStop(383)) - Stopped
ServerConnector@7fc7c4a{HTTP/1.1, (http/1.1)}{SAMMICHEN-MB0:9882}
2024-04-24 18:06:28,903 INFO server.session
(HouseKeeper.java:stopScavenging(149)) - node0 Stopped scavenging
2024-04-24 18:06:28,903 INFO handler.ContextHandler
(ContextHandler.java:doStop(1159)) - Stopped
o.e.j.s.ServletContextHandler@76c387f9{static,/static,file:///Users/sammi/workspace/hadoop-ozone/hadoop-hdds/container-service/target/classes/webapps/static,STOPPED}
2024-04-24 18:06:28,904 INFO ozone.HddsDatanodeClientProtocolServer
(HddsDatanodeClientProtocolServer.java:stop(83)) - Stopping the RPC server for
Client Protocol
2024-04-24 18:06:28,905 INFO ipc.Server (Server.java:stop(3523)) - Stopping
server on 19864
2024-04-24 18:06:28,905 INFO ipc.Server (Server.java:run(1434)) - Stopping
IPC Server listener on 19864
2024-04-24 18:06:28,905 INFO ipc.Server (Server.java:run(1567)) - Stopping
IPC Server Responder
2024-04-24 18:06:28,908 INFO util.ExitUtil (ExitUtil.java:terminate(241)) -
Exiting with status 1: ExitException
2024-04-24 18:06:28,909 INFO ozone.HddsDatanodeService
(StringUtils.java:lambda$startupShutdownMessage$0(144)) - SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down HddsDatanodeService at SAMMICHEN-MB0/0.0.0.0
************************************************************/
Process finished with exit code 1
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
