[
https://issues.apache.org/jira/browse/HDDS-7374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17630087#comment-17630087
]
Hemant Kumar commented on HDDS-7374:
------------------------------------
Hi [~jyosin], I followed your instruction and couldn't reprodurce it.
Can you please fetch the latest code
(https://github.com/apache/ozone/pull/3824)and verify if it is reproducible?
{code:java}
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in
version 9.0 and will likely be removed in a future release.
2022-11-07 18:48:36,075 [main] INFO om.OzoneManagerStarter: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting OzoneManager
STARTUP_MSG: host = 3e97d46de699/172.18.0.6
STARTUP_MSG: args = [--init]
STARTUP_MSG: version = 1.3.0-SNAPSHOT
STARTUP_MSG: classpath =
/etc/hadoop:/opt/hadoop/share/ozone/lib/kerb-simplekdc-1.0.1.jar:/opt/hadoop/share/ozone/lib/slf4j-reload4j-1.7.36.jar:/opt/hadoop/share/ozone/lib/jna-platform-5.2.0.jar:/opt/hadoop/share/ozone/lib/protobuf-java-2.5.0.jar:/opt/hadoop/share/ozone/lib/netty-handler-4.1.79.Final.jar:/opt/hadoop/share/ozone/lib/commons-validator-1.6.jar:/opt/hadoop/share/ozone/lib/bcpkix-jdk15on-1.67.jar:/opt/hadoop/share/ozone/lib/hdds-interface-admin-1.3.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/kerb-core-1.0.1.jar:/opt/hadoop/share/ozone/lib/ozone-interface-client-1.3.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/orc-core-1.5.8.jar:/opt/hadoop/share/ozone/lib/hadoop-shaded-protobuf_3_7-1.1.1.jar:/opt/hadoop/share/ozone/lib/netty-common-4.1.79.Final.jar:/opt/hadoop/share/ozone/lib/ratis-thirdparty-misc-1.0.2.jar:/opt/hadoop/share/ozone/lib/jsr305-3.0.0.jar:/opt/hadoop/share/ozone/lib/j2objc-annotations-1.3.jar:/opt/hadoop/share/ozone/lib/netty-tcnative-boringssl-static-2.0.53.Final-linux-x86_64.jar:/opt/hadoop/share/ozone/lib/jetty-webapp-9.4.49.v20220914.jar:/opt/hadoop/share/ozone/lib/httpmime-4.5.6.jar:/opt/hadoop/share/ozone/lib/proto-google-common-protos-2.9.0.jar:/opt/hadoop/share/ozone/lib/picocli-4.6.1.jar:/opt/hadoop/share/ozone/lib/javassist-3.21.0-GA.jar:/opt/hadoop/share/ozone/lib/httpasyncclient-4.1.3.jar:/opt/hadoop/share/ozone/lib/kerb-util-1.0.1.jar:/opt/hadoop/share/ozone/lib/ratis-common-2.4.0.jar:/opt/hadoop/share/ozone/lib/jetty-client-9.4.31.v20200723.jar:/opt/hadoop/share/ozone/lib/hadoop-shaded-guava-1.1.1.jar:/opt/hadoop/share/ozone/lib/jackson-jaxrs-1.9.13.jar:/opt/hadoop/share/ozone/lib/kotlin-stdlib-1.6.21.jar:/opt/hadoop/share/ozone/lib/log4j-core-2.17.1.jar:/opt/hadoop/share/ozone/lib/commons-logging-1.2.jar:/opt/hadoop/share/ozone/lib/jetty-security-9.4.49.v20220914.jar:/opt/hadoop/share/ozone/lib/ranger-plugin-classloader-2.3.0.jar:/opt/hadoop/share/ozone/lib/accessors-smart-2.4.7.jar:/opt/hadoop/share/ozone/lib/kerby-xdr-1.0.1.jar:/opt/hadoop/share/ozone/lib/ratis-metrics-2.4.0.jar:/opt/hadoop/share/ozone/lib/jackson-core-asl-1.9.13.jar:/opt/hadoop/share/ozone/lib/ozone-interface-storage-1.3.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/grpc-protobuf-lite-1.48.1.jar:/opt/hadoop/share/ozone/lib/ratis-grpc-2.4.0.jar:/opt/hadoop/share/ozone/lib/httpcore-4.4.13.jar:/opt/hadoop/share/ozone/lib/grpc-context-1.48.1.jar:/opt/hadoop/share/ozone/lib/netty-tcnative-boringssl-static-2.0.53.Final-osx-aarch_64.jar:/opt/hadoop/share/ozone/lib/guava-31.1-jre.jar:/opt/hadoop/share/ozone/lib/opentracing-tracerresolver-0.1.8.jar:/opt/hadoop/share/ozone/lib/commons-net-3.6.jar:/opt/hadoop/share/ozone/lib/netty-transport-4.1.79.Final.jar:/opt/hadoop/share/ozone/lib/hadoop-auth-3.3.4.jar:/opt/hadoop/share/ozone/lib/libthrift-0.14.1.jar:/opt/hadoop/share/ozone/lib/grpc-protobuf-1.48.1.jar:/opt/hadoop/share/ozone/lib/error_prone_annotations-2.2.0.jar:/opt/hadoop/share/ozone/lib/kotlin-stdlib-common-1.4.10.jar:/opt/hadoop/share/ozone/lib/httpclient-4.5.13.jar:/opt/hadoop/share/ozone/lib/kerby-config-1.0.1.jar:/opt/hadoop/share/ozone/lib/jaeger-client-1.6.0.jar:/opt/hadoop/share/ozone/lib/jackson-annotations-2.13.4.jar:/opt/hadoop/share/ozone/lib/commons-lang-2.6.jar:/opt/hadoop/share/ozone/lib/commons-io-2.11.0.jar:/opt/hadoop/share/ozone/lib/metrics-core-3.2.4.jar:/opt/hadoop/share/ozone/lib/hadoop-annotations-3.3.4.jar:/opt/hadoop/share/ozone/lib/kerby-pkix-1.0.1.jar:/opt/hadoop/share/ozone/lib/ozone-client-1.3.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/jetty-util-9.4.49.v20220914.jar:/opt/hadoop/share/ozone/lib/jna-5.2.0.jar:/opt/hadoop/share/ozone/lib/ratis-proto-2.4.0.jar:/opt/hadoop/share/ozone/lib/aspectjweaver-1.9.7.jar:/opt/hadoop/share/ozone/lib/json-smart-2.4.7.jar:/opt/hadoop/share/ozone/lib/hdds-interface-client-1.3.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/jetty-io-9.4.49.v20220914.jar:/opt/hadoop/share/ozone/lib/jsr311-api-1.1.1.jar:/opt/hadoop/share/ozone/lib/jetty-server-9.4.49.v20220914.jar:/opt/hadoop/share/ozone/lib/hdds-annotation-processing-1.3.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/commons-beanutils-1.9.4.jar:/opt/hadoop/share/ozone/lib/grpc-netty-1.48.1.jar:/opt/hadoop/share/ozone/lib/grpc-api-1.48.1.jar:/opt/hadoop/share/ozone/lib/netty-codec-http2-4.1.79.Final.jar:/opt/hadoop/share/ozone/lib/hdds-server-framework-1.3.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/javax.servlet-api-3.1.0.jar:/opt/hadoop/share/ozone/lib/kerb-server-1.0.1.jar:/opt/hadoop/share/ozone/lib/annotations-4.1.1.4.jar:/opt/hadoop/share/ozone/lib/hdds-common-1.3.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/rocksdbjni-7.4.5.jar:/opt/hadoop/share/ozone/lib/kerb-client-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-xml-9.4.49.v20220914.jar:/opt/hadoop/share/ozone/lib/hadoop-common-3.3.4.jar:/opt/hadoop/share/ozone/lib/ranger-plugins-cred-2.3.0.jar:/opt/hadoop/share/ozone/lib/disruptor-3.4.2.jar:/opt/hadoop/share/ozone/lib/aspectjrt-1.9.7.jar:/opt/hadoop/share/ozone/lib/hppc-0.8.0.jar:/opt/hadoop/share/ozone/lib/gson-2.9.0.jar:/opt/hadoop/share/ozone/lib/reflections-0.9.11.jar:/opt/hadoop/share/ozone/lib/hdds-erasurecode-1.3.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/netty-tcnative-boringssl-static-2.0.53.Final-osx-x86_64.jar:/opt/hadoop/share/ozone/lib/snakeyaml-1.33.jar:/opt/hadoop/share/ozone/lib/hdds-interface-server-1.3.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/annotations-13.0.jar:/opt/hadoop/share/ozone/lib/simpleclient_common-0.7.0.jar:/opt/hadoop/share/ozone/lib/dnsjava-2.1.7.jar:/opt/hadoop/share/ozone/lib/jaeger-thrift-1.6.0.jar:/opt/hadoop/share/ozone/lib/netty-codec-http-4.1.79.Final.jar:/opt/hadoop/share/ozone/lib/bcprov-jdk15on-1.67.jar:/opt/hadoop/share/ozone/lib/commons-configuration2-2.1.1.jar:/opt/hadoop/share/ozone/lib/ranger-plugins-audit-2.3.0.jar:/opt/hadoop/share/ozone/lib/ozone-common-1.3.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/token-provider-1.0.1.jar:/opt/hadoop/share/ozone/lib/commons-daemon-1.0.13.jar:/opt/hadoop/share/ozone/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/opt/hadoop/share/ozone/lib/netty-buffer-4.1.79.Final.jar:/opt/hadoop/share/ozone/lib/jaeger-tracerresolver-1.6.0.jar:/opt/hadoop/share/ozone/lib/slf4j-api-1.7.36.jar:/opt/hadoop/share/ozone/lib/commons-lang3-3.7.jar:/opt/hadoop/share/ozone/lib/re2j-1.1.jar:/opt/hadoop/share/ozone/lib/jackson-datatype-jsr310-2.13.4.jar:/opt/hadoop/share/ozone/lib/jersey-client-1.19.jar:/opt/hadoop/share/ozone/lib/simpleclient_dropwizard-0.7.0.jar:/opt/hadoop/share/ozone/lib/simpleclient-0.7.0.jar:/opt/hadoop/share/ozone/lib/asm-5.0.4.jar:/opt/hadoop/share/ozone/lib/kerby-util-1.0.1.jar:/opt/hadoop/share/ozone/lib/netty-codec-4.1.79.Final.jar:/opt/hadoop/share/ozone/lib/netty-tcnative-boringssl-static-2.0.53.Final-windows-x86_64.jar:/opt/hadoop/share/ozone/lib/failureaccess-1.0.1.jar:/opt/hadoop/share/ozone/lib/ratis-server-2.4.0.jar:/opt/hadoop/share/ozone/lib/jackson-databind-2.13.4.2.jar:/opt/hadoop/share/ozone/lib/ratis-server-api-2.4.0.jar:/opt/hadoop/share/ozone/lib/okio-2.8.0.jar:/opt/hadoop/share/ozone/lib/jetty-servlet-9.4.49.v20220914.jar:/opt/hadoop/share/ozone/lib/jackson-mapper-asl-1.9.13.jar:/opt/hadoop/share/ozone/lib/hdds-config-1.3.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/jsp-api-2.1.jar:/opt/hadoop/share/ozone/lib/jsch-0.1.54.jar:/opt/hadoop/share/ozone/lib/jakarta.activation-api-1.2.1.jar:/opt/hadoop/share/ozone/lib/netty-tcnative-boringssl-static-2.0.53.Final-linux-aarch_64.jar:/opt/hadoop/share/ozone/lib/gethostname4j-0.0.2.jar:/opt/hadoop/share/ozone/lib/reload4j-1.2.22.jar:/opt/hadoop/share/ozone/lib/nimbus-jose-jwt-7.9.jar:/opt/hadoop/share/ozone/lib/woodstox-core-5.4.0.jar:/opt/hadoop/share/ozone/lib/netty-handler-proxy-4.1.79.Final.jar:/opt/hadoop/share/ozone/lib/commons-codec-1.11.jar:/opt/hadoop/share/ozone/lib/animal-sniffer-annotations-1.21.jar:/opt/hadoop/share/ozone/lib/netty-resolver-4.1.79.Final.jar:/opt/hadoop/share/ozone/lib/commons-pool2-2.6.0.jar:/opt/hadoop/share/ozone/lib/grpc-core-1.48.1.jar:/opt/hadoop/share/ozone/lib/ranger-intg-2.3.0.jar:/opt/hadoop/share/ozone/lib/netty-tcnative-classes-2.0.53.Final.jar:/opt/hadoop/share/ozone/lib/opentracing-api-0.33.0.jar:/opt/hadoop/share/ozone/lib/netty-codec-socks-4.1.79.Final.jar:/opt/hadoop/share/ozone/lib/jcip-annotations-1.0-1.jar:/opt/hadoop/share/ozone/lib/kerb-admin-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-util-ajax-9.4.49.v20220914.jar:/opt/hadoop/share/ozone/lib/ranger-plugins-common-2.3.0.jar:/opt/hadoop/share/ozone/lib/netty-transport-native-unix-common-4.1.79.Final.jar:/opt/hadoop/share/ozone/lib/hadoop-hdfs-client-3.3.4.jar:/opt/hadoop/share/ozone/lib/opentracing-noop-0.33.0.jar:/opt/hadoop/share/ozone/lib/jersey-core-1.19.jar:/opt/hadoop/share/ozone/lib/commons-compress-1.21.jar:/opt/hadoop/share/ozone/lib/hadoop-hdfs-3.3.4.jar:/opt/hadoop/share/ozone/lib/grpc-stub-1.48.1.jar:/opt/hadoop/share/ozone/lib/jackson-core-2.13.4.jar:/opt/hadoop/share/ozone/lib/opentracing-util-0.33.0.jar:/opt/hadoop/share/ozone/lib/ratis-client-2.4.0.jar:/opt/hadoop/share/ozone/lib/hdds-client-1.3.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/snappy-java-1.1.8.2.jar:/opt/hadoop/share/ozone/lib/hdds-hadoop-dependency-client-1.3.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/commons-cli-1.2.jar:/opt/hadoop/share/ozone/lib/log4j-api-2.17.1.jar:/opt/hadoop/share/ozone/lib/javax.annotation-api-1.2.jar:/opt/hadoop/share/ozone/lib/perfmark-api-0.25.0.jar:/opt/hadoop/share/ozone/lib/kerby-asn1-1.0.1.jar:/opt/hadoop/share/ozone/lib/okhttp-4.9.3.jar:/opt/hadoop/share/ozone/lib/netty-tcnative-boringssl-static-2.0.53.Final.jar:/opt/hadoop/share/ozone/lib/checker-qual-3.12.0.jar:/opt/hadoop/share/ozone/lib/stax2-api-3.1.4.jar:/opt/hadoop/share/ozone/lib/commons-collections-3.2.2.jar:/opt/hadoop/share/ozone/lib/commons-digester-1.8.1.jar:/opt/hadoop/share/ozone/lib/jaeger-core-1.6.0.jar:/opt/hadoop/share/ozone/lib/jersey-servlet-1.19.jar:/opt/hadoop/share/ozone/lib/jetty-http-9.4.49.v20220914.jar:/opt/hadoop/share/ozone/lib/kerb-identity-1.0.1.jar:/opt/hadoop/share/ozone/lib/ratis-netty-2.4.0.jar:/opt/hadoop/share/ozone/lib/kerb-common-1.0.1.jar:/opt/hadoop/share/ozone/lib/jersey-server-1.19.jar:/opt/hadoop/share/ozone/lib/httpcore-nio-4.4.6.jar:/opt/hadoop/share/ozone/lib/hdds-hadoop-dependency-server-1.3.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/kerb-crypto-1.0.1.jar:/opt/hadoop/share/ozone/web:/opt/hadoop/share/ozone/lib/ozone-manager-1.3.0-SNAPSHOT.jar
STARTUP_MSG: build =
https://github.com/hemantk-12/ozone.git/4ec46bc102b77ddf12aeb7241e32245e21f81229
; compiled by 'iamgroot' on 2022-11-07T18:37Z
STARTUP_MSG: java = 11.0.17
************************************************************/
2022-11-07 18:48:36,085 [main] INFO om.OzoneManagerStarter: registered UNIX
signal handlers for [TERM, HUP, INT]
2022-11-07 18:48:38,679 [main] INFO audit.AuditLogger: Refresh DebugCmdSet for
OMAudit to [].
2022-11-07 18:48:39,137 [main] INFO ha.OMHANodeDetails:
ozone.om.internal.service.id is not defined, falling back to
ozone.om.service.ids to find serviceID for OzoneManager if it is HA enabled
cluster
2022-11-07 18:48:39,180 [main] INFO ha.OMHANodeDetails: Configuration does not
have ozone.om.address set. Falling back to the default OM address
om/172.18.0.6:9862
2022-11-07 18:48:39,180 [main] INFO ha.OMHANodeDetails: OM Service ID is not
set. Setting it to the default ID: omServiceIdDefault
2022-11-07 18:48:39,180 [main] INFO ha.OMHANodeDetails: OM Node ID is not set.
Setting it to the default ID: om1
2022-11-07 18:48:39,192 [main] WARN server.ServerUtils: ozone.om.db.dirs is not
configured. We recommend adding this setting. Falling back to
ozone.metadata.dirs instead.
2022-11-07 18:48:39,594 [main] INFO
proxy.SCMBlockLocationFailoverProxyProvider: Created block location fail-over
proxy with 1 nodes: [nodeId=scmNodeId,nodeAddress=scm/172.18.0.4:9863]
2022-11-07 18:48:41,947 [main] INFO retry.RetryInvocationHandler:
com.google.protobuf.ServiceException: java.net.ConnectException: Call From
3e97d46de699/172.18.0.6 to scm:9863 failed on connection exception:
java.net.ConnectException: Connection refused; For more details see:
http://wiki.apache.org/hadoop/ConnectionRefused, while invoking $Proxy31.send
over nodeId=scmNodeId,nodeAddress=scm/172.18.0.4:9863 after 1 failover
attempts. Trying to failover after sleeping for 2000ms.
2022-11-07 18:48:43,949 [main] INFO retry.RetryInvocationHandler:
com.google.protobuf.ServiceException: java.net.ConnectException: Call From
3e97d46de699/172.18.0.6 to scm:9863 failed on connection exception:
java.net.ConnectException: Connection refused; For more details see:
http://wiki.apache.org/hadoop/ConnectionRefused, while invoking $Proxy31.send
over nodeId=scmNodeId,nodeAddress=scm/172.18.0.4:9863 after 2 failover
attempts. Trying to failover after sleeping for 2000ms.
2022-11-07 18:48:45,950 [main] INFO retry.RetryInvocationHandler:
com.google.protobuf.ServiceException: java.net.ConnectException: Call From
3e97d46de699/172.18.0.6 to scm:9863 failed on connection exception:
java.net.ConnectException: Connection refused; For more details see:
http://wiki.apache.org/hadoop/ConnectionRefused, while invoking $Proxy31.send
over nodeId=scmNodeId,nodeAddress=scm/172.18.0.4:9863 after 3 failover
attempts. Trying to failover after sleeping for 2000ms.
2022-11-07 18:48:47,951 [main] INFO retry.RetryInvocationHandler:
com.google.protobuf.ServiceException: java.net.ConnectException: Call From
3e97d46de699/172.18.0.6 to scm:9863 failed on connection exception:
java.net.ConnectException: Connection refused; For more details see:
http://wiki.apache.org/hadoop/ConnectionRefused, while invoking $Proxy31.send
over nodeId=scmNodeId,nodeAddress=scm/172.18.0.4:9863 after 4 failover
attempts. Trying to failover after sleeping for 2000ms.
2022-11-07 18:48:49,986 [main] INFO retry.RetryInvocationHandler:
com.google.protobuf.ServiceException:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdds.ratis.ServerNotLeaderException):
Server:05159085-eb94-4b89-984c-675a182384f3 is not the leader. Could not
determine the leader node.
at
org.apache.hadoop.hdds.ratis.ServerNotLeaderException.convertToNotLeaderException(ServerNotLeaderException.java:109)
at
org.apache.hadoop.hdds.scm.ha.RatisUtil.checkRatisException(RatisUtil.java:246)
at
org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocolServerSideTranslatorPB.send(ScmBlockLocationProtocolServerSideTranslatorPB.java:109)
at
org.apache.hadoop.hdds.protocol.proto.ScmBlockLocationProtocolProtos$ScmBlockLocationProtocolService$2.callBlockingMethod(ScmBlockLocationProtocolProtos.java:14202)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server.processCall(ProtobufRpcEngine.java:465)
at
org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:578)
at
org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:556)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1093)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1043)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:971)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2976)
, while invoking $Proxy31.send over
nodeId=scmNodeId,nodeAddress=scm/172.18.0.4:9863 after 5 failover attempts.
Trying to failover after sleeping for 2000ms.
2022-11-07 18:48:51,992 [main] INFO retry.RetryInvocationHandler:
com.google.protobuf.ServiceException:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdds.ratis.ServerNotLeaderException):
Server:05159085-eb94-4b89-984c-675a182384f3 is not the leader. Could not
determine the leader node.
at
org.apache.hadoop.hdds.ratis.ServerNotLeaderException.convertToNotLeaderException(ServerNotLeaderException.java:109)
at
org.apache.hadoop.hdds.scm.ha.RatisUtil.checkRatisException(RatisUtil.java:246)
at
org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocolServerSideTranslatorPB.send(ScmBlockLocationProtocolServerSideTranslatorPB.java:109)
at
org.apache.hadoop.hdds.protocol.proto.ScmBlockLocationProtocolProtos$ScmBlockLocationProtocolService$2.callBlockingMethod(ScmBlockLocationProtocolProtos.java:14202)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server.processCall(ProtobufRpcEngine.java:465)
at
org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:578)
at
org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:556)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1093)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1043)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:971)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2976)
, while invoking $Proxy31.send over
nodeId=scmNodeId,nodeAddress=scm/172.18.0.4:9863 after 6 failover attempts.
Trying to failover after sleeping for 2000ms.
2022-11-07 18:48:54,050 [shutdown-hook-0] INFO om.OzoneManagerStarter:
SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down OzoneManager at 3e97d46de699/172.18.0.6
************************************************************/
No '-XX:...' jvm parameters are set. Adding safer GC settings
'-XX:ParallelGCThreads=8 -XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled' to the
OZONE_OPTS
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in
version 9.0 and will likely be removed in a future release.
2022-11-07 18:48:55,155 [main] INFO om.OzoneManagerStarter: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting OzoneManager
STARTUP_MSG: host = 3e97d46de699/172.18.0.6
STARTUP_MSG: args = []
STARTUP_MSG: version = 1.3.0-SNAPSHOT
STARTUP_MSG: classpath =
/etc/hadoop:/opt/hadoop/share/ozone/lib/kerb-simplekdc-1.0.1.jar:/opt/hadoop/share/ozone/lib/slf4j-reload4j-1.7.36.jar:/opt/hadoop/share/ozone/lib/jna-platform-5.2.0.jar:/opt/hadoop/share/ozone/lib/protobuf-java-2.5.0.jar:/opt/hadoop/share/ozone/lib/netty-handler-4.1.79.Final.jar:/opt/hadoop/share/ozone/lib/commons-validator-1.6.jar:/opt/hadoop/share/ozone/lib/bcpkix-jdk15on-1.67.jar:/opt/hadoop/share/ozone/lib/hdds-interface-admin-1.3.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/kerb-core-1.0.1.jar:/opt/hadoop/share/ozone/lib/ozone-interface-client-1.3.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/orc-core-1.5.8.jar:/opt/hadoop/share/ozone/lib/hadoop-shaded-protobuf_3_7-1.1.1.jar:/opt/hadoop/share/ozone/lib/netty-common-4.1.79.Final.jar:/opt/hadoop/share/ozone/lib/ratis-thirdparty-misc-1.0.2.jar:/opt/hadoop/share/ozone/lib/jsr305-3.0.0.jar:/opt/hadoop/share/ozone/lib/j2objc-annotations-1.3.jar:/opt/hadoop/share/ozone/lib/netty-tcnative-boringssl-static-2.0.53.Final-linux-x86_64.jar:/opt/hadoop/share/ozone/lib/jetty-webapp-9.4.49.v20220914.jar:/opt/hadoop/share/ozone/lib/httpmime-4.5.6.jar:/opt/hadoop/share/ozone/lib/proto-google-common-protos-2.9.0.jar:/opt/hadoop/share/ozone/lib/picocli-4.6.1.jar:/opt/hadoop/share/ozone/lib/javassist-3.21.0-GA.jar:/opt/hadoop/share/ozone/lib/httpasyncclient-4.1.3.jar:/opt/hadoop/share/ozone/lib/kerb-util-1.0.1.jar:/opt/hadoop/share/ozone/lib/ratis-common-2.4.0.jar:/opt/hadoop/share/ozone/lib/jetty-client-9.4.31.v20200723.jar:/opt/hadoop/share/ozone/lib/hadoop-shaded-guava-1.1.1.jar:/opt/hadoop/share/ozone/lib/jackson-jaxrs-1.9.13.jar:/opt/hadoop/share/ozone/lib/kotlin-stdlib-1.6.21.jar:/opt/hadoop/share/ozone/lib/log4j-core-2.17.1.jar:/opt/hadoop/share/ozone/lib/commons-logging-1.2.jar:/opt/hadoop/share/ozone/lib/jetty-security-9.4.49.v20220914.jar:/opt/hadoop/share/ozone/lib/ranger-plugin-classloader-2.3.0.jar:/opt/hadoop/share/ozone/lib/accessors-smart-2.4.7.jar:/opt/hadoop/share/ozone/lib/kerby-xdr-1.0.1.jar:/opt/hadoop/share/ozone/lib/ratis-metrics-2.4.0.jar:/opt/hadoop/share/ozone/lib/jackson-core-asl-1.9.13.jar:/opt/hadoop/share/ozone/lib/ozone-interface-storage-1.3.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/grpc-protobuf-lite-1.48.1.jar:/opt/hadoop/share/ozone/lib/ratis-grpc-2.4.0.jar:/opt/hadoop/share/ozone/lib/httpcore-4.4.13.jar:/opt/hadoop/share/ozone/lib/grpc-context-1.48.1.jar:/opt/hadoop/share/ozone/lib/netty-tcnative-boringssl-static-2.0.53.Final-osx-aarch_64.jar:/opt/hadoop/share/ozone/lib/guava-31.1-jre.jar:/opt/hadoop/share/ozone/lib/opentracing-tracerresolver-0.1.8.jar:/opt/hadoop/share/ozone/lib/commons-net-3.6.jar:/opt/hadoop/share/ozone/lib/netty-transport-4.1.79.Final.jar:/opt/hadoop/share/ozone/lib/hadoop-auth-3.3.4.jar:/opt/hadoop/share/ozone/lib/libthrift-0.14.1.jar:/opt/hadoop/share/ozone/lib/grpc-protobuf-1.48.1.jar:/opt/hadoop/share/ozone/lib/error_prone_annotations-2.2.0.jar:/opt/hadoop/share/ozone/lib/kotlin-stdlib-common-1.4.10.jar:/opt/hadoop/share/ozone/lib/httpclient-4.5.13.jar:/opt/hadoop/share/ozone/lib/kerby-config-1.0.1.jar:/opt/hadoop/share/ozone/lib/jaeger-client-1.6.0.jar:/opt/hadoop/share/ozone/lib/jackson-annotations-2.13.4.jar:/opt/hadoop/share/ozone/lib/commons-lang-2.6.jar:/opt/hadoop/share/ozone/lib/commons-io-2.11.0.jar:/opt/hadoop/share/ozone/lib/metrics-core-3.2.4.jar:/opt/hadoop/share/ozone/lib/hadoop-annotations-3.3.4.jar:/opt/hadoop/share/ozone/lib/kerby-pkix-1.0.1.jar:/opt/hadoop/share/ozone/lib/ozone-client-1.3.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/jetty-util-9.4.49.v20220914.jar:/opt/hadoop/share/ozone/lib/jna-5.2.0.jar:/opt/hadoop/share/ozone/lib/ratis-proto-2.4.0.jar:/opt/hadoop/share/ozone/lib/aspectjweaver-1.9.7.jar:/opt/hadoop/share/ozone/lib/json-smart-2.4.7.jar:/opt/hadoop/share/ozone/lib/hdds-interface-client-1.3.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/jetty-io-9.4.49.v20220914.jar:/opt/hadoop/share/ozone/lib/jsr311-api-1.1.1.jar:/opt/hadoop/share/ozone/lib/jetty-server-9.4.49.v20220914.jar:/opt/hadoop/share/ozone/lib/hdds-annotation-processing-1.3.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/commons-beanutils-1.9.4.jar:/opt/hadoop/share/ozone/lib/grpc-netty-1.48.1.jar:/opt/hadoop/share/ozone/lib/grpc-api-1.48.1.jar:/opt/hadoop/share/ozone/lib/netty-codec-http2-4.1.79.Final.jar:/opt/hadoop/share/ozone/lib/hdds-server-framework-1.3.0-SNAPSHOT.jar:/opt/hado2022-11-07T18:48:54.048172208Z
OM initialization succeeded.Current cluster id for
sd=/data/metadata/om;cid=CID-21107a6f-ca40-4b6b-9059-6ea2da2e2897;layoutVersion=3
op/share/ozone/lib/javax.servlet-api-3.1.0.jar:/opt/hadoop/share/ozone/lib/kerb-server-1.0.1.jar:/opt/hadoop/share/ozone/lib/annotations-4.1.1.4.jar:/opt/hadoop/share/ozone/lib/hdds-common-1.3.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/rocksdbjni-7.4.5.jar:/opt/hadoop/share/ozone/lib/kerb-client-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-xml-9.4.49.v20220914.jar:/opt/hadoop/share/ozone/lib/hadoop-common-3.3.4.jar:/opt/hadoop/share/ozone/lib/ranger-plugins-cred-2.3.0.jar:/opt/hadoop/share/ozone/lib/disruptor-3.4.2.jar:/opt/hadoop/share/ozone/lib/aspectjrt-1.9.7.jar:/opt/hadoop/share/ozone/lib/hppc-0.8.0.jar:/opt/hadoop/share/ozone/lib/gson-2.9.0.jar:/opt/hadoop/share/ozone/lib/reflections-0.9.11.jar:/opt/hadoop/share/ozone/lib/hdds-erasurecode-1.3.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/netty-tcnative-boringssl-static-2.0.53.Final-osx-x86_64.jar:/opt/hadoop/share/ozone/lib/snakeyaml-1.33.jar:/opt/hadoop/share/ozone/lib/hdds-interface-server-1.3.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/annotations-13.0.jar:/opt/hadoop/share/ozone/lib/simpleclient_common-0.7.0.jar:/opt/hadoop/share/ozone/lib/dnsjava-2.1.7.jar:/opt/hadoop/share/ozone/lib/jaeger-thrift-1.6.0.jar:/opt/hadoop/share/ozone/lib/netty-codec-http-4.1.79.Final.jar:/opt/hadoop/share/ozone/lib/bcprov-jdk15on-1.67.jar:/opt/hadoop/share/ozone/lib/commons-configuration2-2.1.1.jar:/opt/hadoop/share/ozone/lib/ranger-plugins-audit-2.3.0.jar:/opt/hadoop/share/ozone/lib/ozone-common-1.3.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/token-provider-1.0.1.jar:/opt/hadoop/share/ozone/lib/commons-daemon-1.0.13.jar:/opt/hadoop/share/ozone/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/opt/hadoop/share/ozone/lib/netty-buffer-4.1.79.Final.jar:/opt/hadoop/share/ozone/lib/jaeger-tracerresolver-1.6.0.jar:/opt/hadoop/share/ozone/lib/slf4j-api-1.7.36.jar:/opt/hadoop/share/ozone/lib/commons-lang3-3.7.jar:/opt/hadoop/share/ozone/lib/re2j-1.1.jar:/opt/hadoop/share/ozone/lib/jackson-datatype-jsr310-2.13.4.jar:/opt/hadoop/share/ozone/lib/jersey-client-1.19.jar:/opt/hadoop/share/ozone/lib/simpleclient_dropwizard-0.7.0.jar:/opt/hadoop/share/ozone/lib/simpleclient-0.7.0.jar:/opt/hadoop/share/ozone/lib/asm-5.0.4.jar:/opt/hadoop/share/ozone/lib/kerby-util-1.0.1.jar:/opt/hadoop/share/ozone/lib/netty-codec-4.1.79.Final.jar:/opt/hadoop/share/ozone/lib/netty-tcnative-boringssl-static-2.0.53.Final-windows-x86_64.jar:/opt/hadoop/share/ozone/lib/failureaccess-1.0.1.jar:/opt/hadoop/share/ozone/lib/ratis-server-2.4.0.jar:/opt/hadoop/share/ozone/lib/jackson-databind-2.13.4.2.jar:/opt/hadoop/share/ozone/lib/ratis-server-api-2.4.0.jar:/opt/hadoop/share/ozone/lib/okio-2.8.0.jar:/opt/hadoop/share/ozone/lib/jetty-servlet-9.4.49.v20220914.jar:/opt/hadoop/share/ozone/lib/jackson-mapper-asl-1.9.13.jar:/opt/hadoop/share/ozone/lib/hdds-config-1.3.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/jsp-api-2.1.jar:/opt/hadoop/share/ozone/lib/jsch-0.1.54.jar:/opt/hadoop/share/ozone/lib/jakarta.activation-api-1.2.1.jar:/opt/hadoop/share/ozone/lib/netty-tcnative-boringssl-static-2.0.53.Final-linux-aarch_64.jar:/opt/hadoop/share/ozone/lib/gethostname4j-0.0.2.jar:/opt/hadoop/share/ozone/lib/reload4j-1.2.22.jar:/opt/hadoop/share/ozone/lib/nimbus-jose-jwt-7.9.jar:/opt/hadoop/share/ozone/lib/woodstox-core-5.4.0.jar:/opt/hadoop/share/ozone/lib/netty-handler-proxy-4.1.79.Final.jar:/opt/hadoop/share/ozone/lib/commons-codec-1.11.jar:/opt/hadoop/share/ozone/lib/animal-sniffer-annotations-1.21.jar:/opt/hadoop/share/ozone/lib/netty-resolver-4.1.79.Final.jar:/opt/hadoop/share/ozone/lib/commons-pool2-2.6.0.jar:/opt/hadoop/share/ozone/lib/grpc-core-1.48.1.jar:/opt/hadoop/share/ozone/lib/ranger-intg-2.3.0.jar:/opt/hadoop/share/ozone/lib/netty-tcnative-classes-2.0.53.Final.jar:/opt/hadoop/share/ozone/lib/opentracing-api-0.33.0.jar:/opt/hadoop/share/ozone/lib/netty-codec-socks-4.1.79.Final.jar:/opt/hadoop/share/ozone/lib/jcip-annotations-1.0-1.jar:/opt/hadoop/share/ozone/lib/kerb-admin-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-util-ajax-9.4.49.v20220914.jar:/opt/hadoop/share/ozone/lib/ranger-plugins-common-2.3.0.jar:/opt/hadoop/share/ozone/lib/netty-transport-native-unix-common-4.1.79.Final.jar:/opt/hadoop/share/ozone/lib/hadoop-hdfs-client-3.3.4.jar:/opt/hadoop/share/ozone/lib/opentracing-noop-0.33.0.jar:/opt/hadoop/share/ozone/lib/jersey-core-1.19.jar:/opt/hadoop/share/ozone/lib/commons-compress-1.21.jar:/opt/hadoop/share/ozone/lib/hadoop-hdfs-3.3.4.jar:/opt/hadoop/share/ozone/lib/grpc-stub-1.48.1.jar:/opt/hadoop/share/ozone/lib/jackson-core-2.13.4.jar:/opt/hadoop/share/ozone/lib/opentracing-util-0.33.0.jar:/opt/hadoop/share/ozone/lib/ratis-client-2.4.0.jar:/opt/hadoop/share/ozone/lib/hdds-client-1.3.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/snappy-java-1.1.8.2.jar:/opt/hadoop/share/ozone/lib/hdds-hadoop-dependency-client-1.3.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/commons-cli-1.2.jar:/opt/hadoop/share/ozone/lib/log4j-api-2.17.1.jar:/opt/hadoop/share/ozone/lib/javax.annotation-api-1.2.jar:/opt/hadoop/share/ozone/lib/perfmark-api-0.25.0.jar:/opt/hadoop/share/ozone/lib/kerby-asn1-1.0.1.jar:/opt/hadoop/share/ozone/lib/okhttp-4.9.3.jar:/opt/hadoop/share/ozone/lib/netty-tcnative-boringssl-static-2.0.53.Final.jar:/opt/hadoop/share/ozone/lib/checker-qual-3.12.0.jar:/opt/hadoop/share/ozone/lib/stax2-api-3.1.4.jar:/opt/hadoop/share/ozone/lib/commons-collections-3.2.2.jar:/opt/hadoop/share/ozone/lib/commons-digester-1.8.1.jar:/opt/hadoop/share/ozone/lib/jaeger-core-1.6.0.jar:/opt/hadoop/share/ozone/lib/jersey-servlet-1.19.jar:/opt/hadoop/share/ozone/lib/jetty-http-9.4.49.v20220914.jar:/opt/hadoop/share/ozone/lib/kerb-identity-1.0.1.jar:/opt/hadoop/share/ozone/lib/ratis-netty-2.4.0.jar:/opt/hadoop/share/ozone/lib/kerb-common-1.0.1.jar:/opt/hadoop/share/ozone/lib/jersey-server-1.19.jar:/opt/hadoop/share/ozone/lib/httpcore-nio-4.4.6.jar:/opt/hadoop/share/ozone/lib/hdds-hadoop-dependency-server-1.3.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/kerb-crypto-1.0.1.jar:/opt/hadoop/share/ozone/web:/opt/hadoop/share/ozone/lib/ozone-manager-1.3.0-SNAPSHOT.jar
STARTUP_MSG: build =
https://github.com/hemantk-12/ozone.git/4ec46bc102b77ddf12aeb7241e32245e21f81229
; compiled by 'iamgroot' on 2022-11-07T18:37Z
STARTUP_MSG: java = 11.0.17
************************************************************/
2022-11-07 18:48:55,159 [main] INFO om.OzoneManagerStarter: registered UNIX
signal handlers for [TERM, HUP, INT]
2022-11-07 18:48:55,723 [main] INFO audit.AuditLogger: Refresh DebugCmdSet for
OMAudit to [].
2022-11-07 18:48:55,848 [main] INFO ha.OMHANodeDetails:
ozone.om.internal.service.id is not defined, falling back to
ozone.om.service.ids to find serviceID for OzoneManager if it is HA enabled
cluster
2022-11-07 18:48:55,874 [main] INFO ha.OMHANodeDetails: Configuration does not
have ozone.om.address set. Falling back to the default OM address
om/172.18.0.6:9862
2022-11-07 18:48:55,874 [main] INFO ha.OMHANodeDetails: OM Service ID is not
set. Setting it to the default ID: omServiceIdDefault
2022-11-07 18:48:55,874 [main] INFO ha.OMHANodeDetails: OM Node ID is not set.
Setting it to the default ID: om1
2022-11-07 18:48:55,880 [main] WARN server.ServerUtils: ozone.om.db.dirs is not
configured. We recommend adding this setting. Falling back to
ozone.metadata.dirs instead.
2022-11-07 18:48:55,911 [main] INFO upgrade.AbstractLayoutVersionManager:
Initializing Layout version manager with metadata layout = MULTITENANCY_SCHEMA
(version = 3), software layout = MULTITENANCY_SCHEMA (version = 3)
2022-11-07 18:48:56,234 [main] INFO reflections.Reflections: Reflections took
293 ms to scan 1 urls, producing 114 keys and 335 values [using 5 cores]
2022-11-07 18:48:56,247 [main] WARN server.ServerUtils: ozone.om.db.dirs is not
configured. We recommend adding this setting. Falling back to
ozone.metadata.dirs instead.
2022-11-07 18:48:56,358 [main] INFO
proxy.SCMBlockLocationFailoverProxyProvider: Created block location fail-over
proxy with 1 nodes: [nodeId=scmNodeId,nodeAddress=scm/172.18.0.4:9863]
2022-11-07 18:48:56,385 [main] INFO
proxy.SCMBlockLocationFailoverProxyProvider: Created block location fail-over
proxy with 1 nodes: [nodeId=scmNodeId,nodeAddress=scm/172.18.0.4:9863]
2022-11-07 18:48:56,693 [main] WARN server.ServerUtils: ozone.om.db.dirs is not
configured. We recommend adding this setting. Falling back to
ozone.metadata.dirs instead.
2022-11-07 18:48:56,798 [main] INFO codec.OmKeyInfoCodec: OmKeyInfoCodec
ignorePipeline = true
2022-11-07 18:48:56,798 [main] INFO codec.RepeatedOmKeyInfoCodec:
RepeatedOmKeyInfoCodec ignorePipeline = true
2022-11-07 18:48:57,030 [main] INFO om.OzoneManager: S3 Multi-Tenancy is
disabled
2022-11-07 18:48:57,092 [main] INFO om.OzoneManager: Created Volume s3v With
Owner hadoop required for S3Gateway operations.
2022-11-07 18:48:57,176 [main] WARN server.ServerUtils: Storage directory for
Ratis is not configured. It is a good idea to map this to an SSD disk. Falling
back to ozone.metadata.dirs
2022-11-07 18:48:57,176 [main] WARN utils.OzoneManagerRatisUtils:
ozone.om.ratis.snapshot.dir is not configured. Falling back to
ozone.metadata.dirs config
2022-11-07 18:48:57,190 [main] INFO metrics.MetricRegistries: Loaded
MetricRegistries class org.apache.ratis.metrics.impl.MetricRegistriesImpl
2022-11-07 18:48:57,194 [main] WARN server.ServerUtils: Storage directory for
Ratis is not configured. It is a good idea to map this to an SSD disk. Falling
back to ozone.metadata.dirs
2022-11-07 18:48:57,214 [main] INFO ratis.OzoneManagerRatisServer:
Instantiating OM Ratis server with groupID: omServiceIdDefault and peers:
om:9872
2022-11-07 18:48:57,224 [main] INFO ratis.OzoneManagerStateMachine:
LastAppliedIndex is set from TransactionInfo from OM DB as (t:0, i:~)
2022-11-07 18:48:57,282 [main] INFO server.RaftServer: raft.rpc.type = GRPC
(default)
2022-11-07 18:48:57,314 [main] INFO grpc.GrpcConfigKeys: raft.grpc.admin.host =
null (fallback to raft.grpc.server.host)
2022-11-07 18:48:57,314 [main] INFO grpc.GrpcConfigKeys: raft.grpc.admin.port =
9872 (fallback to raft.grpc.server.port)
2022-11-07 18:48:57,314 [main] INFO grpc.GrpcConfigKeys: raft.grpc.client.host
= null (fallback to raft.grpc.server.host)
2022-11-07 18:48:57,315 [main] INFO grpc.GrpcConfigKeys: raft.grpc.client.port
= 9872 (fallback to raft.grpc.server.port)
2022-11-07 18:48:57,315 [main] INFO grpc.GrpcConfigKeys: raft.grpc.server.host
= null (default)
2022-11-07 18:48:57,315 [main] INFO grpc.GrpcConfigKeys: raft.grpc.server.port
= 9872 (custom)
2022-11-07 18:48:57,315 [main] INFO server.GrpcService:
raft.grpc.message.size.max = 33554432 (custom)
2022-11-07 18:48:57,316 [main] INFO server.RaftServerConfigKeys:
raft.server.log.appender.buffer.byte-limit = 33554432 (custom)
2022-11-07 18:48:57,316 [main] INFO server.GrpcService:
raft.grpc.flow.control.window = 1MB (=1048576) (default)
2022-11-07 18:48:57,316 [main] INFO server.RaftServerConfigKeys:
raft.server.rpc.request.timeout = 3000ms (default)
2022-11-07 18:48:57,322 [main] INFO grpc.GrpcConfigKeys:
raft.grpc.server.heartbeat.channel = true (default)
2022-11-07 18:48:57,324 [main] INFO grpc.GrpcConfigKeys:
raft.grpc.server.async.request.thread.pool.cached = true (default)
2022-11-07 18:48:57,325 [main] INFO grpc.GrpcConfigKeys:
raft.grpc.server.async.request.thread.pool.size = 32 (default)
2022-11-07 18:48:57,445 [main] INFO impl.DataStreamServerImpl:
raft.datastream.type = DISABLED (default)
2022-11-07 18:48:57,446 [main] INFO server.RaftServerConfigKeys:
raft.server.threadpool.proxy.cached = true (default)
2022-11-07 18:48:57,446 [main] INFO server.RaftServerConfigKeys:
raft.server.threadpool.proxy.size = 0 (default)
2022-11-07 18:48:57,446 [main] INFO server.RaftServerConfigKeys:
raft.server.rpc.slowness.timeout = 120s (custom)
2022-11-07 18:48:57,446 [main] INFO server.RaftServerConfigKeys:
raft.server.leaderelection.leader.step-down.wait-time = 10s (default)
2022-11-07 18:48:57,448 [main] INFO server.RaftServerConfigKeys:
raft.server.storage.dir = [/data/metadata/ratis] (custom)
2022-11-07 18:48:57,455 [main] INFO server.RaftServer: om1: addNew
group-C5BA1605619E:[om1|rpc:om:9872|priority:0|startupRole:FOLLOWER] returns
group-C5BA1605619E:java.util.concurrent.CompletableFuture@1e3f0aea[Not
completed]
2022-11-07 18:48:57,455 [main] INFO om.OzoneManager: OzoneManager Ratis server
initialized at port 9872
2022-11-07 18:48:57,466 [main] INFO om.OzoneManager: Creating RPC Server
2022-11-07 18:48:57,473 [pool-26-thread-1] INFO server.RaftServer$Division:
om1: new RaftServerImpl for
group-C5BA1605619E:[om1|rpc:om:9872|priority:0|startupRole:FOLLOWER] with
OzoneManagerStateMachine:uninitialized
2022-11-07 18:48:57,480 [pool-26-thread-1] INFO server.RaftServerConfigKeys:
raft.server.rpc.timeout.min = 5s (custom)
2022-11-07 18:48:57,480 [pool-26-thread-1] INFO server.RaftServerConfigKeys:
raft.server.rpc.timeout.max = 5200ms (custom)
2022-11-07 18:48:57,480 [pool-26-thread-1] INFO server.RaftServerConfigKeys:
raft.server.rpc.sleep.time = 25ms (default)
2022-11-07 18:48:57,480 [pool-26-thread-1] INFO server.RaftServerConfigKeys:
raft.server.rpc.slowness.timeout = 120s (custom)
2022-11-07 18:48:57,481 [pool-26-thread-1] INFO server.RaftServerConfigKeys:
raft.server.leaderelection.leader.step-down.wait-time = 10s (default)
2022-11-07 18:48:57,481 [pool-26-thread-1] INFO server.RaftServerConfigKeys:
raft.server.sleep.deviation.threshold = 300ms (default)
2022-11-07 18:48:57,485 [pool-26-thread-1] INFO server.RaftServer$Division:
om1@group-C5BA1605619E: ConfigurationManager, init=-1:
peers:[om1|rpc:om:9872|priority:0|startupRole:FOLLOWER]|listeners:[], old=null,
confs=<EMPTY_MAP>
2022-11-07 18:48:57,486 [pool-26-thread-1] INFO server.RaftServerConfigKeys:
raft.server.storage.dir = [/data/metadata/ratis] (custom)
2022-11-07 18:48:57,501 [pool-26-thread-1] INFO server.RaftServerConfigKeys:
raft.server.log.corruption.policy = EXCEPTION (default)
2022-11-07 18:48:57,502 [pool-26-thread-1] INFO server.RaftServerConfigKeys:
raft.server.storage.free-space.min = 0MB (=0) (default)
2022-11-07 18:48:57,517 [pool-26-thread-1] INFO server.RaftServerConfigKeys:
raft.server.notification.no-leader.timeout = 120s (custom)
2022-11-07 18:48:57,521 [pool-26-thread-1] INFO server.RaftServerConfigKeys:
raft.server.retrycache.expirytime = 300s (custom)
2022-11-07 18:48:57,522 [pool-26-thread-1] INFO server.RaftServerConfigKeys:
raft.server.retrycache.statistics.expirytime = 100?s (default)
2022-11-07 18:48:57,874 [pool-26-thread-1] INFO server.RaftServerConfigKeys:
raft.server.log.appender.install.snapshot.enabled = false (custom)
2022-11-07 18:48:57,874 [pool-26-thread-1] INFO server.RaftServerConfigKeys:
raft.server.threadpool.server.cached = true (default)
2022-11-07 18:48:57,874 [pool-26-thread-1] INFO server.RaftServerConfigKeys:
raft.server.threadpool.server.size = 0 (default)
2022-11-07 18:48:57,875 [pool-26-thread-1] INFO server.RaftServerConfigKeys:
raft.server.threadpool.client.cached = true (default)
2022-11-07 18:48:57,875 [pool-26-thread-1] INFO server.RaftServerConfigKeys:
raft.server.threadpool.client.size = 0 (default)
2022-11-07 18:48:58,218 [main] INFO reflections.Reflections: Reflections took
696 ms to scan 8 urls, producing 23 keys and 525 values [using 5 cores]
2022-11-07 18:48:58,383 [main] INFO ipc.CallQueueManager: Using callQueue:
class java.util.concurrent.LinkedBlockingQueue, queueCapacity: 10000,
scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler, ipcBackoff: false.
2022-11-07 18:48:58,390 [Socket Reader #1 for port 9862] INFO ipc.Server:
Starting Socket Reader #1 for port 9862
2022-11-07 18:48:58,978 [Listener at om/9862] INFO impl.MetricsConfig: Loaded
properties from hadoop-metrics2.properties
2022-11-07 18:48:58,985 [Listener at om/9862] INFO impl.MetricsSystemImpl:
Scheduled Metric snapshot period at 10 second(s).
2022-11-07 18:48:58,985 [Listener at om/9862] INFO impl.MetricsSystemImpl:
OzoneManager metrics system started
2022-11-07 18:48:59,014 [Listener at om/9862] INFO om.OzoneManager:
OzoneManager RPC server is listening at om/172.18.0.6:9862
2022-11-07 18:48:59,014 [Listener at om/9862] INFO
ratis.OzoneManagerRatisServer: Starting OzoneManagerRatisServer om1 at port 9872
2022-11-07 18:48:59,016 [om1-impl-thread1] INFO storage.RaftStorageDirectory:
The storage directory /data/metadata/ratis/bf265839-605b-3f16-9796-c5ba1605619e
does not exist. Creating ...
2022-11-07 18:48:59,020 [om1-impl-thread1] INFO storage.RaftStorageDirectory:
Lock on /data/metadata/ratis/bf265839-605b-3f16-9796-c5ba1605619e/in_use.lock
acquired by nodename 7@3e97d46de699
2022-11-07 18:48:59,025 [om1-impl-thread1] INFO storage.RaftStorage: Storage
directory /data/metadata/ratis/bf265839-605b-3f16-9796-c5ba1605619e has been
successfully formatted.
2022-11-07 18:48:59,027 [om1-impl-thread1] INFO server.RaftServerConfigKeys:
raft.server.log.use.memory = false (default)
2022-11-07 18:48:59,031 [om1-impl-thread1] INFO server.RaftServerConfigKeys:
raft.server.log.purge.gap = 1000000 (custom)
2022-11-07 18:48:59,031 [om1-impl-thread1] INFO server.RaftServerConfigKeys:
raft.server.log.appender.buffer.byte-limit = 33554432 (custom)
2022-11-07 18:48:59,032 [om1-impl-thread1] INFO server.RaftServerConfigKeys:
raft.server.log.statemachine.data.read.timeout = 1000ms (default)
2022-11-07 18:48:59,032 [om1-impl-thread1] INFO server.RaftServerConfigKeys:
raft.server.log.purge.preservation.log.num = 0 (default)
2022-11-07 18:48:59,035 [om1-impl-thread1] INFO server.RaftServerConfigKeys:
raft.server.log.segment.size.max = 4194304 (custom)
2022-11-07 18:48:59,038 [om1-impl-thread1] INFO server.RaftServerConfigKeys:
raft.server.log.segment.cache.num.max = 2 (custom)
2022-11-07 18:48:59,038 [om1-impl-thread1] INFO server.RaftServerConfigKeys:
raft.server.log.segment.cache.size.max = 200MB (=209715200) (default)
2022-11-07 18:48:59,041 [om1-impl-thread1] INFO
segmented.SegmentedRaftLogWorker: new
om1@group-C5BA1605619E-SegmentedRaftLogWorker for RaftStorageImpl:Storage
Directory /data/metadata/ratis/bf265839-605b-3f16-9796-c5ba1605619e
2022-11-07 18:48:59,041 [om1-impl-thread1] INFO server.RaftServerConfigKeys:
raft.server.log.queue.byte-limit = 64MB (=67108864) (default)
2022-11-07 18:48:59,041 [om1-impl-thread1] INFO server.RaftServerConfigKeys:
raft.server.log.queue.element-limit = 4096 (default)
2022-11-07 18:48:59,042 [om1-impl-thread1] INFO server.RaftServerConfigKeys:
raft.server.log.segment.size.max = 4194304 (custom)
2022-11-07 18:48:59,042 [om1-impl-thread1] INFO server.RaftServerConfigKeys:
raft.server.log.preallocated.size = 4194304 (custom)
2022-11-07 18:48:59,042 [om1-impl-thread1] INFO server.RaftServerConfigKeys:
raft.server.log.force.sync.num = 128 (default)
2022-11-07 18:48:59,043 [om1-impl-thread1] INFO server.RaftServerConfigKeys:
raft.server.log.statemachine.data.sync = true (default)
2022-11-07 18:48:59,043 [om1-impl-thread1] INFO server.RaftServerConfigKeys:
raft.server.log.statemachine.data.sync.timeout = 10s (default)
2022-11-07 18:48:59,043 [om1-impl-thread1] INFO server.RaftServerConfigKeys:
raft.server.log.statemachine.data.sync.timeout.retry = -1 (default)
2022-11-07 18:48:59,048 [om1-impl-thread1] INFO server.RaftServerConfigKeys:
raft.server.log.write.buffer.size = 64KB (=65536) (default)
2022-11-07 18:48:59,048 [om1-impl-thread1] INFO server.RaftServerConfigKeys:
raft.server.log.unsafe-flush.enabled = false (default)
2022-11-07 18:48:59,048 [om1-impl-thread1] INFO server.RaftServerConfigKeys:
raft.server.log.async-flush.enabled = false (default)
2022-11-07 18:48:59,048 [om1-impl-thread1] INFO server.RaftServerConfigKeys:
raft.server.log.statemachine.data.caching.enabled = false (default)
2022-11-07 18:48:59,052 [om1-impl-thread1] INFO
segmented.SegmentedRaftLogWorker:
om1@group-C5BA1605619E-SegmentedRaftLogWorker: flushIndex: setUnconditionally 0
-> -1
2022-11-07 18:48:59,052 [om1-impl-thread1] INFO
segmented.SegmentedRaftLogWorker:
om1@group-C5BA1605619E-SegmentedRaftLogWorker: safeCacheEvictIndex:
setUnconditionally 0 -> -1
2022-11-07 18:48:59,060 [om1-impl-thread1] INFO server.RaftServer$Division:
om1@group-C5BA1605619E: start as a follower, conf=-1:
peers:[om1|rpc:om:9872|priority:0|startupRole:FOLLOWER]|listeners:[], old=null
2022-11-07 18:48:59,061 [om1-impl-thread1] INFO server.RaftServer$Division:
om1@group-C5BA1605619E: changes role from null to FOLLOWER at term 0 for
startAsFollower
2022-11-07 18:48:59,062 [om1-impl-thread1] INFO impl.RoleInfo: om1: start
om1@group-C5BA1605619E-FollowerState
2022-11-07 18:48:59,062 [om1@group-C5BA1605619E-FollowerState] INFO
server.RaftServerConfigKeys: raft.server.rpc.first-election.timeout.min = 5s
(fallback to raft.server.rpc.timeout.min)
2022-11-07 18:48:59,062 [om1@group-C5BA1605619E-FollowerState] INFO
server.RaftServerConfigKeys: raft.server.rpc.first-election.timeout.max =
5200ms (fallback to raft.server.rpc.timeout.max)
2022-11-07 18:48:59,066 [om1-impl-thread1] INFO util.JmxRegister: Successfully
registered JMX Bean with object name
Ratis:service=RaftServer,group=group-C5BA1605619E,id=om1
2022-11-07 18:48:59,067 [om1-impl-thread1] INFO server.RaftServerConfigKeys:
raft.server.snapshot.auto.trigger.enabled = true (custom)
2022-11-07 18:48:59,067 [om1-impl-thread1] INFO server.RaftServerConfigKeys:
raft.server.snapshot.auto.trigger.threshold = 400000 (default)
2022-11-07 18:48:59,068 [om1-impl-thread1] INFO server.RaftServerConfigKeys:
raft.server.snapshot.retention.file.num = -1 (default)
2022-11-07 18:48:59,068 [om1-impl-thread1] INFO server.RaftServerConfigKeys:
raft.server.log.purge.upto.snapshot.index = true (custom)
2022-11-07 18:48:59,069 [Listener at om/9862] INFO server.RaftServer: om1:
start RPC server
2022-11-07 18:48:59,097 [Listener at om/9862] INFO server.GrpcService: om1:
GrpcService started, listening on 9872
2022-11-07 18:48:59,098 [JvmPauseMonitor0] INFO util.JvmPauseMonitor:
JvmPauseMonitor-om1: Started
2022-11-07 18:48:59,098 [Listener at om/9862] INFO om.OzoneManager: Version
File has different layout version (3) than OM DB (null). That is expected if
this OM has never been finalized to a newer layout version.
2022-11-07 18:48:59,128 [Listener at om/9862] INFO http.BaseHttpServer:
Starting Web-server for ozoneManager at: http://0.0.0.0:9874
2022-11-07 18:48:59,128 [Listener at om/9862] INFO http.BaseHttpServer: Hadoop
Security Enabled: false Ozone Security Enabled: false Ozone HTTP Security
Enabled: false
2022-11-07 18:48:59,149 [Listener at om/9862] INFO util.log: Logging
initialized @4851ms to org.eclipse.jetty.util.log.Slf4jLog
2022-11-07 18:48:59,310 [Listener at om/9862] WARN server.AuthenticationFilter:
Unable to initialize FileSignerSecretProvider, falling back to use random
secrets. Reason: Could not read signature secret file:
/opt/hadoop/hadoop-http-auth-signature-secret
2022-11-07 18:48:59,315 [Listener at om/9862] INFO http.HttpRequestLog: Http
request log for http.requests.ozoneManager is not defined
2022-11-07 18:48:59,324 [Listener at om/9862] INFO http.HttpServer2: Added
global filter 'safety'
(class=org.apache.hadoop.hdds.server.http.HttpServer2$QuotingInputFilter)
2022-11-07 18:48:59,325 [Listener at om/9862] INFO http.HttpServer2: Added
filter static_user_filter
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
context ozoneManager
2022-11-07 18:48:59,325 [Listener at om/9862] INFO http.HttpServer2: Added
filter static_user_filter
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
context logs
2022-11-07 18:48:59,326 [Listener at om/9862] INFO http.HttpServer2: Added
filter static_user_filter
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
context static
2022-11-07 18:48:59,376 [Listener at om/9862] INFO http.HttpServer2: Jetty
bound to port 9874
2022-11-07 18:48:59,377 [Listener at om/9862] INFO server.Server:
jetty-9.4.49.v20220914; built: 2022-09-14T01:07:36.601Z; git:
4231a3b2e4cb8548a412a789936d640a97b1aa0a; jvm 11.0.17+8-LTS
2022-11-07 18:48:59,416 [Listener at om/9862] INFO server.session:
DefaultSessionIdManager workerName=node0
2022-11-07 18:48:59,416 [Listener at om/9862] INFO server.session: No
SessionScavenger set, using defaults
2022-11-07 18:48:59,417 [Listener at om/9862] INFO server.session: node0
Scavenging every 600000ms
2022-11-07 18:48:59,430 [Listener at om/9862] INFO handler.ContextHandler:
Started
o.e.j.s.ServletContextHandler@25fa6889{logs,/logs,file:///var/log/hadoop/,AVAILABLE}
2022-11-07 18:48:59,430 [Listener at om/9862] INFO handler.ContextHandler:
Started
o.e.j.s.ServletContextHandler@43755e2f{static,/static,jar:file:/opt/hadoop/share/ozone/lib/ozone-manager-1.3.0-SNAPSHOT.jar!/webapps/static,AVAILABLE}
2022-11-07 18:48:59,829 [Listener at om/9862] INFO handler.ContextHandler:
Started
o.e.j.w.WebAppContext@280d5a82{ozoneManager,/,file:///tmp/jetty-0_0_0_0-9874-ozone-manager-1_3_0-SNAPSHOT_jar-_-any-16605797659923762101/webapp/,AVAILABLE}{jar:file:/opt/hadoop/share/ozone/lib/ozone-manager-1.3.0-SNAPSHOT.jar!/webapps/ozoneManager}
2022-11-07 18:48:59,834 [Listener at om/9862] INFO server.AbstractConnector:
Started ServerConnector@222d9d4f{HTTP/1.1, (http/1.1)}{0.0.0.0:9874}
2022-11-07 18:48:59,834 [Listener at om/9862] INFO server.Server: Started
@5537ms
2022-11-07 18:48:59,836 [Listener at om/9862] INFO impl.MetricsSinkAdapter:
Sink prometheus started
2022-11-07 18:48:59,836 [Listener at om/9862] INFO impl.MetricsSystemImpl:
Registered sink prometheus
2022-11-07 18:48:59,842 [Listener at om/9862] INFO http.BaseHttpServer: HTTP
server of ozoneManager listening at http://0.0.0.0:9874
2022-11-07 18:48:59,843 [IPC Server Responder] INFO ipc.Server: IPC Server
Responder: starting
2022-11-07 18:48:59,844 [IPC Server listener on 9862] INFO ipc.Server: IPC
Server listener on 9862: starting
2022-11-07 18:48:59,929 [Listener at om/9862] INFO om.TrashPolicyOzone: The
configured checkpoint interval is 0 minutes. Using an interval of 1 minutes
that is used for deletion instead
2022-11-07 18:48:59,930 [Listener at om/9862] INFO om.TrashPolicyOzone: Ozone
Manager trash configuration: Deletion interval = 1 minutes, Emptier interval =
1 minutes.
2022-11-07 18:49:00,009 [Listener at om/9862] INFO om.GrpcOzoneManagerServer:
GrpcOzoneManagerServer is started using port 8981
2022-11-07 18:49:00,012
[org.apache.hadoop.util.JvmPauseMonitor$Monitor@7d18338b] INFO
util.JvmPauseMonitor: Starting JVM pause monitor
2022-11-07 18:49:04,070 [om1@group-C5BA1605619E-FollowerState] INFO
impl.FollowerState: om1@group-C5BA1605619E-FollowerState: change to CANDIDATE,
lastRpcElapsedTime:5008407961ns, electionTimeout:5007ms
2022-11-07 18:49:04,071 [om1@group-C5BA1605619E-FollowerState] INFO
impl.RoleInfo: om1: shutdown om1@group-C5BA1605619E-FollowerState
2022-11-07 18:49:04,072 [om1@group-C5BA1605619E-FollowerState] INFO
server.RaftServer$Division: om1@group-C5BA1605619E: changes role from FOLLOWER
to CANDIDATE at term 0 for changeToCandidate
2022-11-07 18:49:04,074 [om1@group-C5BA1605619E-FollowerState] INFO
server.RaftServerConfigKeys: raft.server.leaderelection.pre-vote = false
(custom)
2022-11-07 18:49:04,074 [om1@group-C5BA1605619E-FollowerState] INFO
impl.RoleInfo: om1: start om1@group-C5BA1605619E-LeaderElection1
2022-11-07 18:49:04,080 [om1@group-C5BA1605619E-LeaderElection1] INFO
impl.LeaderElection: om1@group-C5BA1605619E-LeaderElection1 ELECTION round 0:
submit vote requests at term 1 for -1:
peers:[om1|rpc:om:9872|priority:0|startupRole:FOLLOWER]|listeners:[], old=null
2022-11-07 18:49:04,081 [om1@group-C5BA1605619E-LeaderElection1] INFO
impl.LeaderElection: om1@group-C5BA1605619E-LeaderElection1 ELECTION round 0:
result PASSED (term=1)
2022-11-07 18:49:04,081 [om1@group-C5BA1605619E-LeaderElection1] INFO
impl.RoleInfo: om1: shutdown om1@group-C5BA1605619E-LeaderElection1
2022-11-07 18:49:04,081 [om1@group-C5BA1605619E-LeaderElection1] INFO
server.RaftServer$Division: om1@group-C5BA1605619E: changes role from CANDIDATE
to LEADER at term 1 for changeToLeader
2022-11-07 18:49:04,081 [om1@group-C5BA1605619E-LeaderElection1] INFO
server.RaftServer$Division: om1@group-C5BA1605619E: change Leader from null to
om1 at term 1 for becomeLeader, leader elected after 6564ms
2022-11-07 18:49:04,085 [om1@group-C5BA1605619E-LeaderElection1] INFO
server.RaftServerConfigKeys: raft.server.staging.catchup.gap = 1000 (default)
2022-11-07 18:49:04,088 [om1@group-C5BA1605619E-LeaderElection1] INFO
server.RaftServerConfigKeys: raft.server.write.element-limit = 4096 (default)
2022-11-07 18:49:04,088 [om1@group-C5BA1605619E-LeaderElection1] INFO
server.RaftServerConfigKeys: raft.server.write.byte-limit = 64MB (=67108864)
(default)
2022-11-07 18:49:04,091 [om1@group-C5BA1605619E-LeaderElection1] INFO
server.RaftServerConfigKeys: raft.server.watch.timeout = 10s (default)
2022-11-07 18:49:04,091 [om1@group-C5BA1605619E-LeaderElection1] INFO
server.RaftServerConfigKeys: raft.server.watch.timeout.denomination = 1s
(default)
2022-11-07 18:49:04,091 [om1@group-C5BA1605619E-LeaderElection1] INFO
server.RaftServerConfigKeys: raft.server.watch.element-limit = 65536 (default)
2022-11-07 18:49:04,096 [om1@group-C5BA1605619E-LeaderElection1] INFO
server.RaftServerConfigKeys: raft.server.write.element-limit = 4096 (default)
2022-11-07 18:49:04,097 [om1@group-C5BA1605619E-LeaderElection1] INFO
server.RaftServerConfigKeys: raft.server.write.follower.gap.ratio.max = -1.0
(default)
2022-11-07 18:49:04,098 [om1@group-C5BA1605619E-LeaderElection1] INFO
impl.RoleInfo: om1: start om1@group-C5BA1605619E-LeaderStateImpl
2022-11-07 18:49:04,111 [om1@group-C5BA1605619E-LeaderElection1] INFO
segmented.SegmentedRaftLogWorker:
om1@group-C5BA1605619E-SegmentedRaftLogWorker: Starting segment from index:0
2022-11-07 18:49:04,128 [om1@group-C5BA1605619E-LeaderElection1] INFO
server.RaftServer$Division: om1@group-C5BA1605619E: set configuration 0:
peers:[om1|rpc:om:9872|admin:|client:|dataStream:|priority:0|startupRole:FOLLOWER]|listeners:[],
old=null
2022-11-07 18:49:04,167 [om1@group-C5BA1605619E-SegmentedRaftLogWorker] INFO
segmented.SegmentedRaftLogWorker:
om1@group-C5BA1605619E-SegmentedRaftLogWorker: created new log segment
/data/metadata/ratis/bf265839-605b-3f16-9796-c5ba1605619e/current/log_inprogress_0
2022-11-07 18:49:04,270 [om1@group-C5BA1605619E-StateMachineUpdater] INFO
ratis.OzoneManagerStateMachine: Received Configuration change notification from
Ratis. New Peer list:
[id: "om1"
address: "om:9872"
startupRole: FOLLOWER
]
2022-11-07 18:49:46,482 [qtp2063309971-52] INFO utils.DBCheckpointServlet:
Received request to obtain DB checkpoint snapshot
2022-11-07 18:49:46,497 [qtp2063309971-52] INFO db.RDBCheckpointManager:
Created checkpoint at
/data/metadata/db.checkpoints/om.db_checkpoint_1667846986483 in 13 milliseconds
2022-11-07 18:49:46,543 [qtp2063309971-52] INFO utils.DBCheckpointServlet: Time
taken to write the checkpoint to response output stream: 46 milliseconds
2022-11-07 18:49:46,544 [qtp2063309971-52] INFO db.RocksDBCheckpoint: Cleaning
up RocksDB checkpoint at
/data/metadata/db.checkpoints/om.db_checkpoint_1667846986483
2022-11-07 18:50:50,895 [OM StateMachine ApplyTransaction Thread - 0] INFO
volume.OMVolumeCreateRequest: created volume:vol1 for user:hadoop
2022-11-07 18:50:58,742 [OM StateMachine ApplyTransaction Thread - 0] INFO
bucket.OMBucketCreateRequest: created bucket: buck1 of layout LEGACY in volume:
vol1
No '-XX:...' jvm parameters are set. Adding safer GC settings
'-XX:ParallelGCThreads=8 -XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=70 -XX:+CMSParallelRemarkEnabled' to the
OZONE_OPTS
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in
version 9.0 and will likely be removed in a future release.
2022-11-07 19:48:59,132 [main] INFO om.OzoneManagerStarter: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting OzoneManager
STARTUP_MSG: host = 3e97d46de699/172.18.0.4
STARTUP_MSG: args = []
STARTUP_MSG: version = 1.3.0-SNAPSHOT
STARTUP_MSG: classpath =
/etc/hadoop:/opt/hadoop/share/ozone/lib/kerb-simplekdc-1.0.1.jar:/opt/hadoop/share/ozone/lib/protobuf-java-2.5.0.jar:/opt/hadoop/share/ozone/lib/netty-handler-4.1.79.Final.jar:/opt/hadoop/share/ozone/lib/bcpkix-jdk15on-1.67.jar:/opt/hadoop/share/ozone/lib/jackson-annotations-2.13.2.jar:/opt/hadoop/share/ozone/lib/ozone-interface-client-1.3.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/orc-core-1.5.8.jar:/opt/hadoop/share/ozone/lib/netty-common-4.1.79.Final.jar:/opt/hadoop/share/ozone/lib/jsr305-3.0.0.jar:/opt/hadoop/share/ozone/lib/j2objc-annotations-1.3.jar:/opt/hadoop/share/ozone/lib/netty-tcnative-boringssl-static-2.0.53.Final-linux-x86_64.jar:/opt/hadoop/share/ozone/lib/httpmime-4.5.6.jar:/opt/hadoop/share/ozone/lib/jackson-datatype-jsr310-2.13.2.jar:/opt/hadoop/share/ozone/lib/htrace-core4-4.1.0-incubating.jar:/opt/hadoop/share/ozone/lib/picocli-4.6.1.jar:/opt/hadoop/share/ozone/lib/httpasyncclient-4.1.3.jar:/opt/hadoop/share/ozone/lib/jackson-jaxrs-1.9.13.jar:/opt/hadoop/share/ozone/lib/ratis-metrics-2.3.0.jar:/opt/hadoop/share/ozone/lib/kotlin-stdlib-1.6.21.jar:/opt/hadoop/share/ozone/lib/ranger-plugin-classloader-2.3.0.jar:/opt/hadoop/share/ozone/lib/accessors-smart-2.4.7.jar:/opt/hadoop/share/ozone/lib/kerby-xdr-1.0.1.jar:/opt/hadoop/share/ozone/lib/ozone-interface-storage-1.3.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/netty-tcnative-boringssl-static-2.0.53.Final-osx-aarch_64.jar:/opt/hadoop/share/ozone/lib/libthrift-0.14.1.jar:/opt/hadoop/share/ozone/lib/grpc-protobuf-1.48.1.jar:/opt/hadoop/share/ozone/lib/error_prone_annotations-2.2.0.jar:/opt/hadoop/share/ozone/lib/httpclient-4.5.13.jar:/opt/hadoop/share/ozone/lib/jgrapht-ext-1.4.0.jar:/opt/hadoop/share/ozone/lib/commons-lang-2.6.jar:/opt/hadoop/share/ozone/lib/metrics-core-3.2.4.jar:/opt/hadoop/share/ozone/lib/kerby-pkix-1.0.1.jar:/opt/hadoop/share/ozone/lib/ozone-client-1.3.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/jna-5.2.0.jar:/opt/hadoop/share/ozone/lib/aspectjweaver-1.9.7.jar:/opt/hadoop/share/ozone/lib/json-smart-2.4.7.jar:/opt/hadoop/share/ozone/lib/jsr311-api-1.1.1.jar:/opt/hadoop/share/ozone/lib/hdds-annotation-processing-1.3.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/commons-beanutils-1.9.4.jar:/opt/hadoop/share/ozone/lib/grpc-netty-1.48.1.jar:/opt/hadoop/share/ozone/lib/jackson-core-2.13.2.jar:/opt/hadoop/share/ozone/lib/hdds-server-framework-1.3.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/kerb-server-1.0.1.jar:/opt/hadoop/share/ozone/lib/hdds-common-1.3.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/rocksdbjni-7.4.5.jar:/opt/hadoop/share/ozone/lib/jgraphx-4.2.2.jar:/opt/hadoop/share/ozone/lib/ranger-plugins-cred-2.3.0.jar:/opt/hadoop/share/ozone/lib/disruptor-3.4.2.jar:/opt/hadoop/share/ozone/lib/aspectjrt-1.9.7.jar:/opt/hadoop/share/ozone/lib/hppc-0.8.0.jar:/opt/hadoop/share/ozone/lib/hdds-interface-server-1.3.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/simpleclient_common-0.7.0.jar:/opt/hadoop/share/ozone/lib/dnsjava-2.1.7.jar:/opt/hadoop/share/ozone/lib/commons-configuration2-2.1.1.jar:/opt/hadoop/share/ozone/lib/ozone-common-1.3.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/opt/hadoop/share/ozone/lib/netty-buffer-4.1.79.Final.jar:/opt/hadoop/share/ozone/lib/commons-lang3-3.7.jar:/opt/hadoop/share/ozone/lib/re2j-1.1.jar:/opt/hadoop/share/ozone/lib/commons-math3-3.1.1.jar:/opt/hadoop/share/ozone/lib/simpleclient_dropwizard-0.7.0.jar:/opt/hadoop/share/ozone/lib/jackson-databind-2.13.2.2.jar:/opt/hadoop/share/ozone/lib/simpleclient-0.7.0.jar:/opt/hadoop/share/ozone/lib/asm-5.0.4.jar:/opt/hadoop/share/ozone/lib/kerby-util-1.0.1.jar:/opt/hadoop/share/ozone/lib/netty-codec-4.1.79.Final.jar:/opt/hadoop/share/ozone/lib/netty-tcnative-boringssl-static-2.0.53.Final-windows-x86_64.jar:/opt/hadoop/share/ozone/lib/failureaccess-1.0.1.jar:/opt/hadoop/share/ozone/lib/jackson-mapper-asl-1.9.13.jar:/opt/hadoop/share/ozone/lib/hdds-config-1.3.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/jsp-api-2.1.jar:/opt/hadoop/share/ozone/lib/jakarta.activation-api-1.2.1.jar:/opt/hadoop/share/ozone/lib/gethostname4j-0.0.2.jar:/opt/hadoop/share/ozone/lib/hadoop-hdfs-client-3.3.1.jar:/opt/hadoop/share/ozone/lib/commons-codec-1.11.jar:/opt/hadoop/share/ozone/lib/netty-resolver-4.1.79.Final.jar:/opt/hadoop/share/ozone/lib/jetty-util-9.4.43.v20210629.jar:/opt/hadoop/share/ozone/lib/grpc-core-1.48.1.jar:/opt/hadoop/share/ozone/lib/jetty-util-ajax-9.4.43.v20210629.jar:/opt/hadoop/share/ozone/lib/opentracing-api-0.33.0.jar:/opt/hadoop/share/ozone/lib/jcip-annotations-1.0-1.jar:/opt/hadoop/share/ozone/lib/ratis-thirdparty-misc-1.0.0.jar:/opt/hadoop/share/ozone/lib/kotlin-stdlib-common-1.6.21.jar:/opt/hadoop/share/ozone/lib/opentracing-noop-0.33.0.jar:/opt/hadoop/share/ozone/lib/commons-compress-1.21.jar:/opt/hadoop/share/ozone/lib/ratis-server-api-2.3.0.jar:/opt/hadoop/share/ozone/lib/opentracing-util-0.33.0.jar:/opt/hadoop/share/ozone/lib/hdds-client-1.3.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/snappy-java-1.1.8.2.jar:/opt/hadoop/share/ozone/lib/hdds-hadoop-dependency-client-1.3.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/commons-cli-1.2.jar:/opt/hadoop/share/ozone/lib/perfmark-api-0.25.0.jar:/opt/hadoop/share/ozone/lib/hadoop-annotations-3.3.1.jar:/opt/hadoop/share/ozone/lib/netty-tcnative-boringssl-static-2.0.53.Final.jar:/opt/hadoop/share/ozone/lib/commons-collections-3.2.2.jar:/opt/hadoop/share/ozone/lib/commons-digester-1.8.1.jar:/opt/hadoop/share/ozone/lib/jaeger-core-1.6.0.jar:/opt/hadoop/share/ozone/lib/jersey-servlet-1.19.jar:/opt/hadoop/share/ozone/lib/hadoop-auth-3.3.1.jar:/opt/hadoop/share/ozone/lib/kerb-identity-1.0.1.jar:/opt/hadoop/share/ozone/lib/hdds-hadoop-dependency-server-1.3.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/slf4j-reload4j-1.7.36.jar:/opt/hadoop/share/ozone/lib/jna-platform-5.2.0.jar:/opt/hadoop/share/ozone/lib/commons-validator-1.6.jar:/opt/hadoop/share/ozone/lib/hdds-interface-admin-1.3.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/jgrapht-core-1.5.0.jar:/opt/hadoop/share/ozone/lib/jetty-security-9.4.43.v20210629.jar:/opt/hadoop/share/ozone/lib/kerb-core-1.0.1.jar:/opt/hadoop/share/ozone/lib/hadoop-shaded-protobuf_3_7-1.1.1.jar:/opt/hadoop/share/ozone/lib/okhttp-4.9.0.jar:/opt/hadoop/share/ozone/lib/proto-google-common-protos-2.9.0.jar:/opt/hadoop/share/ozone/lib/jgrapht-guava-1.5.0.jar:/opt/hadoop/share/ozone/lib/javassist-3.21.0-GA.jar:/opt/hadoop/share/ozone/lib/kerb-util-1.0.1.jar:/opt/hadoop/share/ozone/lib/jetty-client-9.4.31.v20200723.jar:/opt/hadoop/share/ozone/lib/hadoop-shaded-guava-1.1.1.jar:/opt/hadoop/share/ozone/lib/log4j-core-2.17.1.jar:/opt/hadoop/share/ozone/lib/commons-logging-1.2.jar:/opt/hadoop/share/ozone/lib/jackson-core-asl-1.9.13.jar:/opt/hadoop/share/ozone/lib/grpc-protobuf-lite-1.48.1.jar:/opt/hadoop/share/ozone/lib/httpcore-4.4.13.jar:/opt/hadoop/share/ozone/lib/grpc-context-1.48.1.jar:/opt/hadoop/share/ozone/lib/jheaps-0.13.jar:/opt/hadoop/share/ozone/lib/guava-31.1-jre.jar:/opt/hadoop/share/ozone/lib/opentracing-tracerresolver-0.1.8.jar:/opt/hadoop/share/ozone/lib/commons-net-3.6.jar:/opt/hadoop/share/ozone/lib/netty-transport-4.1.79.Final.jar:/opt/hadoop/share/ozone/lib/jetty-servlet-9.4.43.v20210629.jar:/opt/hadoop/share/ozone/lib/ratis-common-2.3.0.jar:/opt/hadoop/share/ozone/lib/kerby-config-1.0.1.jar:/opt/hadoop/share/ozone/lib/jaeger-client-1.6.0.jar:/opt/hadoop/share/ozone/lib/commons-io-2.11.0.jar:/opt/hadoop/share/ozone/lib/snakeyaml-1.32.jar:/opt/hadoop/share/ozone/lib/hdds-interface-client-1.3.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/ratis-grpc-2.3.0.jar:/opt/hadoop/share/ozone/lib/jetty-webapp-9.4.43.v20210629.jar:/opt/hadoop/share/ozone/lib/grpc-api-1.48.1.jar:/opt/hadoop/share/ozone/lib/netty-codec-http2-4.1.79.Final.jar:/opt/hadoop/share/ozone/lib/javax.servlet-api-3.1.0.jar:/opt/hadoop/share/ozone/lib/annotations-4.1.1.4.jar:/opt/hadoop/share/ozone/lib/kerb-client-1.0.1.jar:/opt/hadoop/share/ozone/lib/gson-2.9.0.jar:/opt/hadoop/share/ozone/lib/reflections-0.9.11.jar:/opt/hadoop/share/ozone/lib/hdds-erasurecode-1.3.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/netty-tcnative-boringssl-static-2.0.53.Final-osx-x86_64.jar:/opt/hadoop/share/ozone/lib/jetty-io-9.4.43.v20210629.jar:/opt/hadoop/share/ozone/lib/hadoop-hdfs-3.3.1.jar:/opt/hadoop/share/ozone/lib/ratis-proto-2.3.0.jar:/opt/hadoop/share/ozone/lib/annotations-13.0.jar:/opt/hadoop/share/ozone/lib/jetty-server-9.4.43.v20210629.jar:/opt/hadoop/share/ozone/lib/hadoop-common-3.3.1.jar:/opt/hadoop/share/ozone/lib/jaeger-thrift-1.6.0.jar:/opt/hadoop/share/ozone/lib/netty-codec-http-4.1.79.Final.jar:/opt/hadoop/share/ozone/lib/bcprov-jdk15on-1.67.jar:/opt/hadoop/share/ozone/lib/ranger-plugins-audit-2.3.0.jar:/opt/hadoop/share/ozone/lib/token-provider-1.0.1.jar:/opt/hadoop/share/ozone/lib/commons-daemon-1.0.13.jar:/opt/hadoop/share/ozone/lib/jetty-xml-9.4.43.v20210629.jar:/opt/hadoop/share/ozone/lib/jaeger-tracerresolver-1.6.0.jar:/opt/hadoop/share/ozone/lib/slf4j-api-1.7.36.jar:/opt/hadoop/share/ozone/lib/ratis-netty-2.3.0.jar:/opt/hadoop/share/ozone/lib/jersey-client-1.19.jar:/opt/hadoop/share/ozone/lib/okio-2.8.0.jar:/opt/hadoop/share/ozone/lib/jetty-http-9.4.43.v20210629.jar:/opt/hadoop/share/ozone/lib/jsch-0.1.54.jar:/opt/hadoop/share/ozone/lib/netty-tcnative-boringssl-static-2.0.53.Final-linux-aarch_64.jar:/opt/hadoop/share/ozone/lib/reload4j-1.2.22.jar:/opt/hadoop/share/ozone/lib/nimbus-jose-jwt-7.9.jar:/opt/hadoop/share/ozone/lib/netty-handler-proxy-4.1.79.Final.jar:/opt/hadoop/share/ozone/lib/animal-sniffer-annotations-1.21.jar:/opt/hadoop/share/ozone/lib/commons-pool2-2.6.0.jar:/opt/hadoop/share/ozone/lib/ranger-intg-2.3.0.jar:/opt/hadoop/share/ozone/lib/netty-tcnative-classes-2.0.53.Final.jar:/opt/hadoop/share/ozone/lib/netty-codec-socks-4.1.79.Final.jar:/opt/hadoop/share/ozone/lib/kerb-admin-1.0.1.jar:/opt/hadoop/share/ozone/lib/ranger-plugins-common-2.3.0.jar:/opt/hadoop/share/ozone/lib/netty-transport-native-unix-common-4.1.79.Final.jar:/opt/hadoop/share/ozone/lib/jersey-core-1.19.jar:/opt/hadoop/share/ozone/lib/grpc-stub-1.48.1.jar:/opt/hadoop/share/ozone/lib/ratis-server-2.3.0.jar:/opt/hadoop/share/ozone/lib/woodstox-core-5.0.3.jar:/opt/hadoop/share/ozone/lib/log4j-api-2.17.1.jar:/opt/hadoop/share/ozone/lib/javax.annotation-api-1.2.jar:/opt/hadoop/share/ozone/lib/kerby-asn1-1.0.1.jar:/opt/hadoop/share/ozone/lib/rocksdb-checkpoint-differ-1.3.0-SNAPSHOT.jar:/opt/hadoop/share/ozone/lib/checker-qual-3.12.0.jar:/opt/hadoop/share/ozone/lib/stax2-api-3.1.4.jar:/opt/hadoop/share/ozone/lib/commons-text-1.4.jar:/opt/hadoop/share/ozone/lib/ratis-client-2.3.0.jar:/opt/hadoop/share/ozone/lib/kerb-common-1.0.1.jar:/opt/hadoop/share/ozone/lib/jersey-server-1.19.jar:/opt/hadoop/share/ozone/lib/httpcore-nio-4.4.6.jar:/opt/hadoop/share/ozone/lib/kerb-crypto-1.0.1.jar:/opt/hadoop/share/ozone/web:/opt/hadoop/share/ozone/lib/ozone-manager-1.3.0-SNAPSHOT.jar
STARTUP_MSG: build =
https://github.com/hemantk-12/ozone.git/e642ddeb55edd2bd2580a7ce6a1d30c0e081b1a3
; compiled by 'iamgroot' on 2022-11-07T19:30Z
STARTUP_MSG: java = 11.0.17
************************************************************/
2022-11-07 19:48:59,137 [main] INFO om.OzoneManagerStarter: registered UNIX
signal handlers for [TERM, HUP, INT]
2022-11-07 19:49:01,554 [main] INFO audit.AuditLogger: Refresh DebugCmdSet for
OMAudit to [].
2022-11-07 19:49:02,286 [main] INFO ha.OMHANodeDetails:
ozone.om.internal.service.id is not defined, falling back to
ozone.om.service.ids to find serviceID for OzoneManager if it is HA enabled
cluster
2022-11-07 19:49:02,368 [main] INFO ha.OMHANodeDetails: Configuration does not
have ozone.om.address set. Falling back to the default OM address
om/172.18.0.4:9862
2022-11-07 19:49:02,368 [main] INFO ha.OMHANodeDetails: OM Service ID is not
set. Setting it to the default ID: omServiceIdDefault
2022-11-07 19:49:02,369 [main] INFO ha.OMHANodeDetails: OM Node ID is not set.
Setting it to the default ID: om1
2022-11-07 19:49:02,402 [main] WARN server.ServerUtils: ozone.om.db.dirs is not
configured. We recommend adding this setting. Falling back to
ozone.metadata.dirs instead.
2022-11-07 19:49:02,448 [main] INFO upgrade.AbstractLayoutVersionManager:
Initializing Layout version manager with metadata layout = MULTITENANCY_SCHEMA
(version = 3), software layout = MULTITENANCY_SCHEMA (version = 3)
2022-11-07 19:49:03,557 [main] INFO reflections.Reflections: Reflections took
990 ms to scan 1 urls, producing 116 keys and 347 values [using 5 cores]
2022-11-07 19:49:03,636 [main] WARN server.ServerUtils: ozone.om.db.dirs is not
configured. We recommend adding this setting. Falling back to
ozone.metadata.dirs instead.
2022-11-07 19:49:04,601 [main] INFO
proxy.SCMBlockLocationFailoverProxyProvider: Created block location fail-over
proxy with 1 nodes: [nodeId=scmNodeId,nodeAddress=scm/172.18.0.3:9863]
2022-11-07 19:49:04,682 [main] INFO
proxy.SCMBlockLocationFailoverProxyProvider: Created block location fail-over
proxy with 1 nodes: [nodeId=scmNodeId,nodeAddress=scm/172.18.0.3:9863]
2022-11-07 19:49:06,997 [main] INFO retry.RetryInvocationHandler:
com.google.protobuf.ServiceException: java.net.ConnectException: Call From
3e97d46de699/172.18.0.4 to scm:9863 failed on connection exception:
java.net.ConnectException: Connection refused; For more details see:
http://wiki.apache.org/hadoop/ConnectionRefused, while invoking $Proxy33.send
over nodeId=scmNodeId,nodeAddress=scm/172.18.0.3:9863 after 1 failover
attempts. Trying to failover after sleeping for 2000ms.
2022-11-07 19:49:09,243 [main] INFO retry.RetryInvocationHandler:
com.google.protobuf.ServiceException:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdds.ratis.ServerNotLeaderException):
Server:05159085-eb94-4b89-984c-675a182384f3 is not the leader. Could not
determine the leader node.
at
org.apache.hadoop.hdds.ratis.ServerNotLeaderException.convertToNotLeaderException(ServerNotLeaderException.java:109)
at
org.apache.hadoop.hdds.scm.ha.RatisUtil.checkRatisException(RatisUtil.java:246)
at
org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocolServerSideTranslatorPB.send(ScmBlockLocationProtocolServerSideTranslatorPB.java:109)
at
org.apache.hadoop.hdds.protocol.proto.ScmBlockLocationProtocolProtos$ScmBlockLocationProtocolService$2.callBlockingMethod(ScmBlockLocationProtocolProtos.java:14202)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server.processCall(ProtobufRpcEngine.java:466)
at
org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:574)
at
org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:552)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1093)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1035)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:963)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2966)
, while invoking $Proxy33.send over
nodeId=scmNodeId,nodeAddress=scm/172.18.0.3:9863 after 2 failover attempts.
Trying to failover after sleeping for 2000ms.
2022-11-07 19:49:11,249 [main] INFO retry.RetryInvocationHandler:
com.google.protobuf.ServiceException:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdds.ratis.ServerNotLeaderException):
Server:05159085-eb94-4b89-984c-675a182384f3 is not the leader. Could not
determine the leader node.
at
org.apache.hadoop.hdds.ratis.ServerNotLeaderException.convertToNotLeaderException(ServerNotLeaderException.java:109)
at
org.apache.hadoop.hdds.scm.ha.RatisUtil.checkRatisException(RatisUtil.java:246)
at
org.apache.hadoop.hdds.scm.protocol.ScmBlockLocationProtocolServerSideTranslatorPB.send(ScmBlockLocationProtocolServerSideTranslatorPB.java:109)
at
org.apache.hadoop.hdds.protocol.proto.ScmBlockLocationProtocolProtos$ScmBlockLocationProtocolService$2.callBlockingMethod(ScmBlockLocationProtocolProtos.java:14202)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server.processCall(ProtobufRpcEngine.java:466)
at
org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:574)
at
org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:552)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1093)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1035)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:963)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2966)
, while invoking $Proxy33.send over
nodeId=scmNodeId,nodeAddress=scm/172.18.0.3:9863 after 3 failover attempts.
Trying to failover after sleeping for 2000ms.
2022-11-07 19:49:13,609 [main] WARN server.ServerUtils: ozone.om.db.dirs is not
configured. We recommend adding this setting. Falling back to
ozone.metadata.dirs instead.
2022-11-07 19:49:13,703 [main] INFO codec.OmKeyInfoCodec: OmKeyInfoCodec
ignorePipeline = true
2022-11-07 19:49:13,704 [main] INFO codec.RepeatedOmKeyInfoCodec:
RepeatedOmKeyInfoCodec ignorePipeline = true
2022-11-07 19:49:13,864 [main] INFO om.OzoneManager: S3 Multi-Tenancy is
disabled
2022-11-07 19:49:14,034 [main] WARN server.ServerUtils: Storage directory for
Ratis is not configured. It is a good idea to map this to an SSD disk. Falling
back to ozone.metadata.dirs
2022-11-07 19:49:14,034 [main] WARN utils.OzoneManagerRatisUtils:
ozone.om.ratis.snapshot.dir is not configured. Falling back to
ozone.metadata.dirs config
2022-11-07 19:49:14,048 [main] INFO metrics.MetricRegistries: Loaded
MetricRegistries class org.apache.ratis.metrics.impl.MetricRegistriesImpl
2022-11-07 19:49:14,053 [main] WARN server.ServerUtils: Storage directory for
Ratis is not configured. It is a good idea to map this to an SSD disk. Falling
back to ozone.metadata.dirs
2022-11-07 19:49:14,070 [main] INFO ratis.OzoneManagerRatisServer:
Instantiating OM Ratis server with groupID: omServiceIdDefault and peers:
om:9872
2022-11-07 19:49:14,078 [main] INFO ratis.OzoneManagerStateMachine:
LastAppliedIndex is set from TransactionInfo from OM DB as (t:1, i:103)
2022-11-07 19:49:14,132 [main] INFO server.RaftServer: raft.rpc.type = GRPC
(default)
2022-11-07 19:49:14,177 [main] INFO grpc.GrpcConfigKeys: raft.grpc.admin.port =
-1 (default)
2022-11-07 19:49:14,177 [main] INFO grpc.GrpcConfigKeys: raft.grpc.server.port
= 9872 (custom)
2022-11-07 19:49:14,178 [main] INFO grpc.GrpcConfigKeys: raft.grpc.client.port
= -1 (default)
2022-11-07 19:49:14,178 [main] INFO grpc.GrpcConfigKeys: raft.grpc.server.port
= 9872 (custom)
2022-11-07 19:49:14,178 [main] INFO grpc.GrpcConfigKeys: raft.grpc.server.port
= 9872 (custom)
2022-11-07 19:49:14,178 [main] INFO server.GrpcService:
raft.grpc.message.size.max = 33554432 (custom)
2022-11-07 19:49:14,179 [main] INFO server.RaftServerConfigKeys:
raft.server.log.appender.buffer.byte-limit = 33554432 (custom)
2022-11-07 19:49:14,180 [main] INFO server.GrpcService:
raft.grpc.flow.control.window = 1MB (=1048576) (default)
2022-11-07 19:49:14,180 [main] INFO server.RaftServerConfigKeys:
raft.server.rpc.request.timeout = 3000ms (default)
2022-11-07 19:49:14,189 [main] INFO grpc.GrpcConfigKeys:
raft.grpc.server.async.request.thread.pool.cached = true (default)
2022-11-07 19:49:14,189 [main] INFO grpc.GrpcConfigKeys:
raft.grpc.server.async.request.thread.pool.size = 32 (default)
2022-11-07 19:49:14,337 [main] INFO impl.DataStreamServerImpl:
raft.datastream.type = DISABLED (default)
2022-11-07 19:49:14,338 [main] INFO server.RaftServerConfigKeys:
raft.server.threadpool.proxy.cached = true (default)
2022-11-07 19:49:14,338 [main] INFO server.RaftServerConfigKeys:
raft.server.threadpool.proxy.size = 0 (default)
2022-11-07 19:49:14,338 [main] INFO server.RaftServerConfigKeys:
raft.server.rpc.slowness.timeout = 120s (custom)
2022-11-07 19:49:14,338 [main] INFO server.RaftServerConfigKeys:
raft.server.leaderelection.leader.step-down.wait-time = 10s (default)
2022-11-07 19:49:14,340 [main] INFO server.RaftServerConfigKeys:
raft.server.storage.dir = [/data/metadata/ratis] (custom)
2022-11-07 19:49:14,342 [om1-impl-thread1] INFO server.RaftServer: om1: found a
subdirectory /data/metadata/ratis/bf265839-605b-3f16-9796-c5ba1605619e
2022-11-07 19:49:14,345 [main] INFO server.RaftServer: om1: addNew
group-C5BA1605619E:[om1|rpc:om:9872|priority:0] returns
group-C5BA1605619E:java.util.concurrent.CompletableFuture@4c731956[Not
completed]
2022-11-07 19:49:14,346 [main] INFO om.OzoneManager: OzoneManager Ratis server
initialized at port 9872
2022-11-07 19:49:14,349 [main] INFO om.OzoneManager: Creating RPC Server
2022-11-07 19:49:14,355 [pool-27-thread-1] INFO server.RaftServer$Division:
om1: new RaftServerImpl for group-C5BA1605619E:[om1|rpc:om:9872|priority:0]
with OzoneManagerStateMachine:uninitialized
2022-11-07 19:49:14,359 [pool-27-thread-1] INFO server.RaftServerConfigKeys:
raft.server.rpc.timeout.min = 5s (custom)
2022-11-07 19:49:14,361 [pool-27-thread-1] INFO server.RaftServerConfigKeys:
raft.server.rpc.timeout.max = 5200ms (custom)
2022-11-07 19:49:14,361 [pool-27-thread-1] INFO server.RaftServerConfigKeys:
raft.server.rpc.sleep.time = 25ms (default)
2022-11-07 19:49:14,361 [pool-27-thread-1] INFO server.RaftServerConfigKeys:
raft.server.rpc.slowness.timeout = 120s (custom)
2022-11-07 19:49:14,361 [pool-27-thread-1] INFO server.RaftServerConfigKeys:
raft.server.leaderelection.leader.step-down.wait-time = 10s (default)
2022-11-07 19:49:14,361 [pool-27-thread-1] INFO server.RaftServerConfigKeys:
raft.server.sleep.deviation.threshold = 300ms (default)
2022-11-07 19:49:14,367 [pool-27-thread-1] INFO server.RaftServer$Division:
om1@group-C5BA1605619E: ConfigurationManager, init=-1:
[om1|rpc:om:9872|priority:0], old=null, confs=<EMPTY_MAP>
2022-11-07 19:49:14,367 [pool-27-thread-1] INFO server.RaftServerConfigKeys:
raft.server.storage.dir = [/data/metadata/ratis] (custom)
2022-11-07 19:49:14,369 [pool-27-thread-1] INFO server.RaftServerConfigKeys:
raft.server.log.corruption.policy = EXCEPTION (default)
2022-11-07 19:49:14,369 [pool-27-thread-1] INFO server.RaftServerConfigKeys:
raft.server.storage.free-space.min = 0MB (=0) (default)
2022-11-07 19:49:14,379 [pool-27-thread-1] INFO storage.RaftStorageDirectory:
Lock on /data/metadata/ratis/bf265839-605b-3f16-9796-c5ba1605619e/in_use.lock
acquired by nodename 6@3e97d46de699
2022-11-07 19:49:14,396 [pool-27-thread-1] INFO storage.RaftStorage: Read
RaftStorageMetadata{term=1, votedFor=om1} from
/data/metadata/ratis/bf265839-605b-3f16-9796-c5ba1605619e/current/raft-meta
2022-11-07 19:49:14,472 [pool-27-thread-1] INFO server.RaftServer$Division:
om1@group-C5BA1605619E: set configuration 0:
[om1|rpc:om:9872|admin:|client:|dataStream:|priority:0], old=null
2022-11-07 19:49:14,472 [pool-27-thread-1] INFO server.RaftServerConfigKeys:
raft.server.notification.no-leader.timeout = 120s (custom)
2022-11-07 19:49:14,473 [pool-27-thread-1] INFO server.RaftServerConfigKeys:
raft.server.log.use.memory = false (default)
2022-11-07 19:49:14,477 [pool-27-thread-1] INFO server.RaftServerConfigKeys:
raft.server.log.purge.gap = 1000000 (custom)
2022-11-07 19:49:14,478 [pool-27-thread-1] INFO server.RaftServerConfigKeys:
raft.server.log.appender.buffer.byte-limit = 33554432 (custom)
2022-11-07 19:49:14,478 [pool-27-thread-1] INFO server.RaftServerConfigKeys:
raft.server.log.statemachine.data.read.timeout = 1000ms (default)
2022-11-07 19:49:14,501 [pool-27-thread-1] INFO server.RaftServerConfigKeys:
raft.server.log.segment.size.max = 4194304 (custom)
2022-11-07 19:49:14,514 [pool-27-thread-1] INFO server.RaftServerConfigKeys:
raft.server.log.segment.cache.num.max = 2 (custom)
2022-11-07 19:49:14,516 [pool-27-thread-1] INFO server.RaftServerConfigKeys:
raft.server.log.segment.cache.size.max = 200MB (=209715200) (default)
2022-11-07 19:49:14,520 [pool-27-thread-1] INFO
segmented.SegmentedRaftLogWorker: new
om1@group-C5BA1605619E-SegmentedRaftLogWorker for RaftStorageImpl:Storage
Directory /data/metadata/ratis/bf265839-605b-3f16-9796-c5ba1605619e
2022-11-07 19:49:14,520 [pool-27-thread-1] INFO server.RaftServerConfigKeys:
raft.server.log.queue.byte-limit = 64MB (=67108864) (default)
2022-11-07 19:49:14,520 [pool-27-thread-1] INFO server.RaftServerConfigKeys:
raft.server.log.queue.element-limit = 4096 (default)
2022-11-07 19:49:14,521 [pool-27-thread-1] INFO server.RaftServerConfigKeys:
raft.server.log.segment.size.max = 4194304 (custom)
2022-11-07 19:49:14,522 [pool-27-thread-1] INFO server.RaftServerConfigKeys:
raft.server.log.preallocated.size = 4194304 (custom)
2022-11-07 19:49:14,522 [pool-27-thread-1] INFO server.RaftServerConfigKeys:
raft.server.log.force.sync.num = 128 (default)
2022-11-07 19:49:14,523 [pool-27-thread-1] INFO server.RaftServerConfigKeys:
raft.server.log.statemachine.data.sync = true (default)
2022-11-07 19:49:14,523 [pool-27-thread-1] INFO server.RaftServerConfigKeys:
raft.server.log.statemachine.data.sync.timeout = 10s (default)
2022-11-07 19:49:14,523 [pool-27-thread-1] INFO server.RaftServerConfigKeys:
raft.server.log.statemachine.data.sync.timeout.retry = -1 (default)
2022-11-07 19:49:14,528 [pool-27-thread-1] INFO server.RaftServerConfigKeys:
raft.server.log.write.buffer.size = 64KB (=65536) (default)
2022-11-07 19:49:14,528 [pool-27-thread-1] INFO server.RaftServerConfigKeys:
raft.server.log.unsafe-flush.enabled = false (default)
2022-11-07 19:49:14,528 [pool-27-thread-1] INFO server.RaftServerConfigKeys:
raft.server.log.statemachine.data.caching.enabled = false (default)
2022-11-07 19:49:14,544 [pool-27-thread-1] INFO server.RaftServer$Division:
om1@group-C5BA1605619E: set configuration 0:
[om1|rpc:om:9872|admin:|client:|dataStream:|priority:0], old=null
2022-11-07 19:49:14,589 [pool-27-thread-1] INFO segmented.LogSegment:
Successfully read 105 entries from segment file
/data/metadata/ratis/bf265839-605b-3f16-9796-c5ba1605619e/current/log_inprogress_0
2022-11-07 19:49:14,593 [pool-27-thread-1] INFO
segmented.SegmentedRaftLogWorker:
om1@group-C5BA1605619E-SegmentedRaftLogWorker: flushIndex: setUnconditionally 0
-> 104
2022-11-07 19:49:14,595 [pool-27-thread-1] INFO
segmented.SegmentedRaftLogWorker:
om1@group-C5BA1605619E-SegmentedRaftLogWorker: safeCacheEvictIndex:
setUnconditionally 0 -> -1
2022-11-07 19:49:14,660 [main] INFO reflections.Reflections: Reflections took
277 ms to scan 8 urls, producing 24 keys and 534 values [using 5 cores]
2022-11-07 19:49:14,672 [pool-27-thread-1] INFO raftlog.RaftLog:
om1@group-C5BA1605619E-SegmentedRaftLog: commitIndex: updateToMax old=103,
new=103, updated? false
2022-11-07 19:49:14,674 [pool-27-thread-1] INFO server.RaftServerConfigKeys:
raft.server.snapshot.auto.trigger.enabled = true (custom)
2022-11-07 19:49:14,677 [pool-27-thread-1] INFO server.RaftServerConfigKeys:
raft.server.snapshot.auto.trigger.threshold = 400000 (default)
2022-11-07 19:49:14,678 [pool-27-thread-1] INFO server.RaftServerConfigKeys:
raft.server.snapshot.retention.file.num = -1 (default)
2022-11-07 19:49:14,679 [pool-27-thread-1] INFO server.RaftServerConfigKeys:
raft.server.log.purge.upto.snapshot.index = true (custom)
2022-11-07 19:49:14,680 [pool-27-thread-1] INFO server.RaftServerConfigKeys:
raft.server.retrycache.expirytime = 300s (custom)
2022-11-07 19:49:14,681 [pool-27-thread-1] INFO server.RaftServerConfigKeys:
raft.server.retrycache.statistics.expirytime = 100?s (default)
2022-11-07 19:49:14,703 [pool-27-thread-1] INFO server.RaftServerConfigKeys:
raft.server.log.appender.install.snapshot.enabled = false (custom)
2022-11-07 19:49:14,704 [pool-27-thread-1] INFO server.RaftServerConfigKeys:
raft.server.threadpool.server.cached = true (default)
2022-11-07 19:49:14,704 [pool-27-thread-1] INFO server.RaftServerConfigKeys:
raft.server.threadpool.server.size = 0 (default)
2022-11-07 19:49:14,705 [pool-27-thread-1] INFO server.RaftServerConfigKeys:
raft.server.threadpool.client.cached = true (default)
2022-11-07 19:49:14,705 [pool-27-thread-1] INFO server.RaftServerConfigKeys:
raft.server.threadpool.client.size = 0 (default)
2022-11-07 19:49:14,775 [main] INFO ipc.CallQueueManager: Using callQueue:
class java.util.concurrent.LinkedBlockingQueue, queueCapacity: 10000,
scheduler: class org.apache.hadoop.ipc.DefaultRpcScheduler, ipcBackoff: false.
2022-11-07 19:49:14,780 [Socket Reader #1 for port 9862] INFO ipc.Server:
Starting Socket Reader #1 for port 9862
2022-11-07 19:49:15,266 [Listener at om/9862] INFO impl.MetricsConfig: Loaded
properties from hadoop-metrics2.properties
2022-11-07 19:49:15,275 [Listener at om/9862] INFO impl.MetricsSystemImpl:
Scheduled Metric snapshot period at 10 second(s).
2022-11-07 19:49:15,275 [Listener at om/9862] INFO impl.MetricsSystemImpl:
OzoneManager metrics system started
2022-11-07 19:49:15,298 [Listener at om/9862] INFO om.OzoneManager:
OzoneManager RPC server is listening at om/172.18.0.4:9862
2022-11-07 19:49:15,298 [Listener at om/9862] INFO
ratis.OzoneManagerRatisServer: Starting OzoneManagerRatisServer om1 at port 9872
2022-11-07 19:49:15,298 [om1-impl-thread1] INFO server.RaftServer$Division:
om1@group-C5BA1605619E: start as a follower, conf=0:
[om1|rpc:om:9872|admin:|client:|dataStream:|priority:0], old=null
2022-11-07 19:49:15,299 [om1-impl-thread1] INFO server.RaftServer$Division:
om1@group-C5BA1605619E: changes role from null to FOLLOWER at term 1 for
startAsFollower
2022-11-07 19:49:15,300 [om1-impl-thread1] INFO impl.RoleInfo: om1: start
om1@group-C5BA1605619E-FollowerState
2022-11-07 19:49:15,303 [om1-impl-thread1] INFO util.JmxRegister: Successfully
registered JMX Bean with object name
Ratis:service=RaftServer,group=group-C5BA1605619E,id=om1
2022-11-07 19:49:15,306 [Listener at om/9862] INFO server.RaftServer: om1:
start RPC server
2022-11-07 19:49:15,331 [Listener at om/9862] INFO server.GrpcService: om1:
GrpcService started, listening on 9872
2022-11-07 19:49:15,333
[org.apache.ratis.util.JvmPauseMonitor$$Lambda$416/0x0000000840502840@5f45bc8e]
INFO util.JvmPauseMonitor: JvmPauseMonitor-om1: Started
2022-11-07 19:49:15,333 [Listener at om/9862] INFO om.OzoneManager: Version
File has different layout version (3) than OM DB (null). That is expected if
this OM has never been finalized to a newer layout version.
2022-11-07 19:49:15,371 [Listener at om/9862] INFO http.BaseHttpServer:
Starting Web-server for ozoneManager at: http://0.0.0.0:9874
2022-11-07 19:49:15,371 [Listener at om/9862] INFO http.BaseHttpServer: Hadoop
Security Enabled: false Ozone Security Enabled: false Ozone HTTP Security
Enabled: false
2022-11-07 19:49:15,388 [Listener at om/9862] INFO util.log: Logging
initialized @19881ms to org.eclipse.jetty.util.log.Slf4jLog
2022-11-07 19:49:15,456 [Listener at om/9862] INFO server.AuthenticationFilter:
Unable to initialize FileSignerSecretProvider, falling back to use random
secrets.
2022-11-07 19:49:15,459 [Listener at om/9862] INFO http.HttpRequestLog: Http
request log for http.requests.ozoneManager is not defined
2022-11-07 19:49:15,466 [Listener at om/9862] INFO http.HttpServer2: Added
global filter 'safety'
(class=org.apache.hadoop.hdds.server.http.HttpServer2$QuotingInputFilter)
2022-11-07 19:49:15,467 [Listener at om/9862] INFO http.HttpServer2: Added
filter static_user_filter
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
context ozoneManager
2022-11-07 19:49:15,467 [Listener at om/9862] INFO http.HttpServer2: Added
filter static_user_filter
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
context static
2022-11-07 19:49:15,467 [Listener at om/9862] INFO http.HttpServer2: Added
filter static_user_filter
(class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter) to
context logs
2022-11-07 19:49:15,489 [Listener at om/9862] INFO http.HttpServer2: Jetty
bound to port 9874
2022-11-07 19:49:15,490 [Listener at om/9862] INFO server.Server:
jetty-9.4.43.v20210629; built: 2021-06-30T11:07:22.254Z; git:
526006ecfa3af7f1a27ef3a288e2bef7ea9dd7e8; jvm 11.0.17+8-LTS
2022-11-07 19:49:15,506 [Listener at om/9862] INFO server.session:
DefaultSessionIdManager workerName=node0
2022-11-07 19:49:15,506 [Listener at om/9862] INFO server.session: No
SessionScavenger set, using defaults
2022-11-07 19:49:15,507 [Listener at om/9862] INFO server.session: node0
Scavenging every 660000ms
2022-11-07 19:49:15,515 [Listener at om/9862] INFO handler.ContextHandler:
Started
o.e.j.s.ServletContextHandler@29892a77{logs,/logs,file:///var/log/hadoop/,AVAILABLE}
2022-11-07 19:49:15,515 [Listener at om/9862] INFO handler.ContextHandler:
Started
o.e.j.s.ServletContextHandler@1a7e799e{static,/static,jar:file:/opt/hadoop/share/ozone/lib/ozone-manager-1.3.0-SNAPSHOT.jar!/webapps/static,AVAILABLE}
2022-11-07 19:49:15,879 [Listener at om/9862] INFO handler.ContextHandler:
Started
o.e.j.w.WebAppContext@7b79ff1c{ozoneManager,/,file:///tmp/jetty-0_0_0_0-9874-ozone-manager-1_3_0-SNAPSHOT_jar-_-any-5410988415102375404/webapp/,AVAILABLE}{jar:file:/opt/hadoop/share/ozone/lib/ozone-manager-1.3.0-SNAPSHOT.jar!/webapps/ozoneManager}
2022-11-07 19:49:15,909 [Listener at om/9862] INFO server.AbstractConnector:
Started ServerConnector@70cf7d1e{HTTP/1.1, (http/1.1)}{0.0.0.0:9874}
2022-11-07 19:49:15,909 [Listener at om/9862] INFO server.Server: Started
@20402ms
2022-11-07 19:49:15,912 [Listener at om/9862] INFO impl.MetricsSinkAdapter:
Sink prometheus started
2022-11-07 19:49:15,912 [Listener at om/9862] INFO impl.MetricsSystemImpl:
Registered sink prometheus
2022-11-07 19:49:15,924 [Listener at om/9862] INFO http.BaseHttpServer: HTTP
server of ozoneManager listening at http://0.0.0.0:9874
2022-11-07 19:49:15,924 [IPC Server Responder] INFO ipc.Server: IPC Server
Responder: starting
2022-11-07 19:49:15,925 [IPC Server listener on 9862] INFO ipc.Server: IPC
Server listener on 9862: starting
2022-11-07 19:49:16,008 [Listener at om/9862] INFO om.TrashPolicyOzone: The
configured checkpoint interval is 0 minutes. Using an interval of 1 minutes
that is used for deletion instead
2022-11-07 19:49:16,008 [Listener at om/9862] INFO om.TrashPolicyOzone: Ozone
Manager trash configuration: Deletion interval = 1 minutes, Emptier interval =
1 minutes.
2022-11-07 19:49:16,099 [Listener at om/9862] INFO om.GrpcOzoneManagerServer:
GrpcOzoneManagerServer is started using port 8981
2022-11-07 19:49:16,103
[org.apache.hadoop.util.JvmPauseMonitor$Monitor@22899683] INFO
util.JvmPauseMonitor: Starting JVM pause monitor
2022-11-07 19:49:20,426 [om1@group-C5BA1605619E-FollowerState] INFO
impl.FollowerState: om1@group-C5BA1605619E-FollowerState: change to CANDIDATE,
lastRpcElapsedTime:5126860294ns, electionTimeout:5124ms
2022-11-07 19:49:20,428 [om1@group-C5BA1605619E-FollowerState] INFO
impl.RoleInfo: om1: shutdown om1@group-C5BA1605619E-FollowerState
2022-11-07 19:49:20,429 [om1@group-C5BA1605619E-FollowerState] INFO
server.RaftServer$Division: om1@group-C5BA1605619E: changes role from FOLLOWER
to CANDIDATE at term 1 for changeToCandidate
2022-11-07 19:49:20,432 [om1@group-C5BA1605619E-FollowerState] INFO
server.RaftServerConfigKeys: raft.server.leaderelection.pre-vote = false
(custom)
2022-11-07 19:49:20,432 [om1@group-C5BA1605619E-FollowerState] INFO
impl.RoleInfo: om1: start om1@group-C5BA1605619E-LeaderElection1
2022-11-07 19:49:20,441 [om1@group-C5BA1605619E-LeaderElection1] INFO
impl.LeaderElection: om1@group-C5BA1605619E-LeaderElection1 ELECTION round 0:
submit vote requests at term 2 for 0:
[om1|rpc:om:9872|admin:|client:|dataStream:|priority:0], old=null
2022-11-07 19:49:20,443 [om1@group-C5BA1605619E-LeaderElection1] INFO
impl.LeaderElection: om1@group-C5BA1605619E-LeaderElection1 ELECTION round 0:
result PASSED (term=2)
2022-11-07 19:49:20,443 [om1@group-C5BA1605619E-LeaderElection1] INFO
impl.RoleInfo: om1: shutdown om1@group-C5BA1605619E-LeaderElection1
2022-11-07 19:49:20,444 [om1@group-C5BA1605619E-LeaderElection1] INFO
server.RaftServer$Division: om1@group-C5BA1605619E: changes role from CANDIDATE
to LEADER at term 2 for changeToLeader
2022-11-07 19:49:20,444 [om1@group-C5BA1605619E-LeaderElection1] INFO
server.RaftServer$Division: om1@group-C5BA1605619E: change Leader from null to
om1 at term 2 for becomeLeader, leader elected after 5971ms
2022-11-07 19:49:20,450 [om1@group-C5BA1605619E-LeaderElection1] INFO
server.RaftServerConfigKeys: raft.server.staging.catchup.gap = 1000 (default)
2022-11-07 19:49:20,453 [om1@group-C5BA1605619E-LeaderElection1] INFO
server.RaftServerConfigKeys: raft.server.write.element-limit = 4096 (default)
2022-11-07 19:49:20,454 [om1@group-C5BA1605619E-LeaderElection1] INFO
server.RaftServerConfigKeys: raft.server.write.byte-limit = 64MB (=67108864)
(default)
2022-11-07 19:49:20,458 [om1@group-C5BA1605619E-LeaderElection1] INFO
server.RaftServerConfigKeys: raft.server.watch.timeout = 10s (default)
2022-11-07 19:49:20,458 [om1@group-C5BA1605619E-LeaderElection1] INFO
server.RaftServerConfigKeys: raft.server.watch.timeout.denomination = 1s
(default)
2022-11-07 19:49:20,458 [om1@group-C5BA1605619E-LeaderElection1] INFO
server.RaftServerConfigKeys: raft.server.watch.element-limit = 65536 (default)
2022-11-07 19:49:20,463 [om1@group-C5BA1605619E-LeaderElection1] INFO
server.RaftServerConfigKeys: raft.server.write.element-limit = 4096 (default)
2022-11-07 19:49:20,464 [om1@group-C5BA1605619E-LeaderElection1] INFO
server.RaftServerConfigKeys: raft.server.write.follower.gap.ratio.max = -1.0
(default)
2022-11-07 19:49:20,466 [om1@group-C5BA1605619E-LeaderElection1] INFO
impl.RoleInfo: om1: start om1@group-C5BA1605619E-LeaderStateImpl
2022-11-07 19:49:20,472 [om1@group-C5BA1605619E-LeaderElection1] INFO
segmented.SegmentedRaftLogWorker:
om1@group-C5BA1605619E-SegmentedRaftLogWorker: Rolling segment log-0_104 to
index:104
2022-11-07 19:49:20,475 [om1@group-C5BA1605619E-SegmentedRaftLogWorker] INFO
segmented.SegmentedRaftLogWorker:
om1@group-C5BA1605619E-SegmentedRaftLogWorker: Rolled log segment from
/data/metadata/ratis/bf265839-605b-3f16-9796-c5ba1605619e/current/log_inprogress_0
to /data/metadata/ratis/bf265839-605b-3f16-9796-c5ba1605619e/current/log_0-104
2022-11-07 19:49:20,477 [om1@group-C5BA1605619E-LeaderElection1] INFO
server.RaftServer$Division: om1@group-C5BA1605619E: set configuration 105:
[om1|rpc:om:9872|admin:|client:|dataStream:|priority:0], old=null
2022-11-07 19:49:20,486 [om1@group-C5BA1605619E-SegmentedRaftLogWorker] INFO
segmented.SegmentedRaftLogWorker:
om1@group-C5BA1605619E-SegmentedRaftLogWorker: created new log segment
/data/metadata/ratis/bf265839-605b-3f16-9796-c5ba1605619e/current/log_inprogress_105
2022-11-07 19:49:20,572 [om1@group-C5BA1605619E-StateMachineUpdater] INFO
ratis.OzoneManagerStateMachine: Received Configuration change notification from
Ratis. New Peer list:
[id: "om1"
address: "om:9872"
]
2022-11-07 19:50:11,033 [IPC Server handler 80 on default port 9862] WARN
db.RDBStore: Unable to get delta updates since sequenceNumber 61. This
exception will be thrown to the client
org.apache.hadoop.hdds.utils.db.SequenceNumberNotFoundException: Invalid
transaction log iterator when getting updates since sequence number 61
at
org.apache.hadoop.hdds.utils.db.RDBStore.getUpdatesSince(RDBStore.java:364)
at
org.apache.hadoop.ozone.om.OzoneManager.getDBUpdates(OzoneManager.java:3810)
at
org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.getOMDBUpdates(OzoneManagerRequestHandler.java:321)
at
org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleReadRequest(OzoneManagerRequestHandler.java:223)
at
org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitReadRequestToOM(OzoneManagerProtocolServerSideTranslatorPB.java:226)
at
org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.processRequest(OzoneManagerProtocolServerSideTranslatorPB.java:175)
at
org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:87)
at
org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:147)
at
org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server.processCall(ProtobufRpcEngine.java:466)
at
org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:574)
at
org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:552)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1093)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1035)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:963)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2966)
2022-11-07 19:50:11,180 [qtp8302165-57] INFO utils.DBCheckpointServlet:
Received request to obtain DB checkpoint snapshot
2022-11-07 19:50:11,184 [qtp8302165-57] INFO db.RDBCheckpointManager: Created
checkpoint at /data/metadata/db.checkpoints/om.db_checkpoint_1667850611180 in 4
milliseconds
2022-11-07 19:50:11,231 [qtp8302165-57] INFO utils.DBCheckpointServlet: Time
taken to write the checkpoint to response output stream: 46 milliseconds
2022-11-07 19:50:11,232 [qtp8302165-57] INFO db.RocksDBCheckpoint: Cleaning up
RocksDB checkpoint at
/data/metadata/db.checkpoints/om.db_checkpoint_1667850611180
2022-11-07 20:31:44,988 [OM StateMachine ApplyTransaction Thread - 0] ERROR
volume.OMVolumeCreateRequest: Volume creation failed for user:hadoop volume:vol1
VOLUME_ALREADY_EXISTS org.apache.hadoop.ozone.om.exceptions.OMException: Volume
already exists
at
org.apache.hadoop.ozone.om.request.volume.OMVolumeCreateRequest.validateAndUpdateCache(OMVolumeCreateRequest.java:153)
at
org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleWriteRequest(OzoneManagerRequestHandler.java:304)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.runCommand(OzoneManagerStateMachine.java:529)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.lambda$1(OzoneManagerStateMachine.java:324)
at
java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2022-11-07 20:31:52,566 [OM StateMachine ApplyTransaction Thread - 0] ERROR
bucket.OMBucketCreateRequest: Bucket creation failed for bucket:buck1 in
volume:vol1
BUCKET_ALREADY_EXISTS org.apache.hadoop.ozone.om.exceptions.OMException: Bucket
already exist
at
org.apache.hadoop.ozone.om.request.bucket.OMBucketCreateRequest.validateAndUpdateCache(OMBucketCreateRequest.java:206)
at
org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleWriteRequest(OzoneManagerRequestHandler.java:304)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.runCommand(OzoneManagerStateMachine.java:529)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.lambda$1(OzoneManagerStateMachine.java:324)
at
java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2022-11-07 20:32:04,307 [OM StateMachine ApplyTransaction Thread - 0] INFO
snapshot.OMSnapshotCreateRequest: created snapshot: name snap1 in snapshotPath:
vol1/buck1
2022-11-07 20:32:04,324 [OMDoubleBufferFlushThread] INFO
db.RDBCheckpointManager: Created checkpoint at
/data/metadata/db.snapshots/om.db-2fedb192-6194-4367-8353-fd607b520f4e in 15
milliseconds
2022-11-07 20:32:11,872 [IPC Server handler 9 on default port 9862] WARN
db.RDBStore: Unable to get delta updates since sequenceNumber 62. This
exception will be thrown to the client
org.apache.hadoop.hdds.utils.db.SequenceNumberNotFoundException: Unable to read
data from RocksDB wal to get delta updates. It may have already been flushed to
SSTs.
at
org.apache.hadoop.hdds.utils.db.RDBStore.getUpdatesSince(RDBStore.java:383)
at
org.apache.hadoop.ozone.om.OzoneManager.getDBUpdates(OzoneManager.java:3810)
at
org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.getOMDBUpdates(OzoneManagerRequestHandler.java:321)
at
org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleReadRequest(OzoneManagerRequestHandler.java:223)
at
org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitReadRequestToOM(OzoneManagerProtocolServerSideTranslatorPB.java:226)
at
org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.processRequest(OzoneManagerProtocolServerSideTranslatorPB.java:175)
at
org.apache.hadoop.hdds.server.OzoneProtocolMessageDispatcher.processRequest(OzoneProtocolMessageDispatcher.java:87)
at
org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB.submitRequest(OzoneManagerProtocolServerSideTranslatorPB.java:147)
at
org.apache.hadoop.ozone.protocol.proto.OzoneManagerProtocolProtos$OzoneManagerService$2.callBlockingMethod(OzoneManagerProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server.processCall(ProtobufRpcEngine.java:466)
at
org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:574)
at
org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:552)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1093)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1035)
at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:963)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2966)
2022-11-07 20:32:11,886 [qtp8302165-55] INFO utils.DBCheckpointServlet:
Received request to obtain DB checkpoint snapshot
2022-11-07 20:32:11,910 [qtp8302165-55] INFO db.RDBCheckpointManager: Created
checkpoint at /data/metadata/db.checkpoints/om.db_checkpoint_1667853131886 in
23 milliseconds
2022-11-07 20:32:11,917 [qtp8302165-55] INFO utils.DBCheckpointServlet: Time
taken to write the checkpoint to response output stream: 7 milliseconds
2022-11-07 20:32:11,918 [qtp8302165-55] INFO db.RocksDBCheckpoint: Cleaning up
RocksDB checkpoint at
/data/metadata/db.checkpoints/om.db_checkpoint_1667853131886
2022-11-07 20:32:12,394 [IPC Server handler 86 on default port 9862] WARN
server.ServerUtils: ozone.om.db.dirs is not configured. We recommend adding
this setting. Falling back to ozone.metadata.dirs instead.
2022-11-07 20:32:12,395 [IPC Server handler 86 on default port 9862] INFO
codec.OmKeyInfoCodec: OmKeyInfoCodec ignorePipeline = true
2022-11-07 20:32:12,395 [IPC Server handler 86 on default port 9862] INFO
codec.RepeatedOmKeyInfoCodec: RepeatedOmKeyInfoCodec ignorePipeline = true
2022-11-07 20:32:12,450 [IPC Server handler 86 on default port 9862] INFO
audit.AuditLogger: Refresh DebugCmdSet for OMAudit to [].
2022-11-07 20:34:02,694 [OM StateMachine ApplyTransaction Thread - 0] INFO
snapshot.OMSnapshotCreateRequest: created snapshot: name s20221107-203402.694
in snapshotPath: vol1/buck1
2022-11-07 20:34:02,703 [OMDoubleBufferFlushThread] INFO
db.RDBCheckpointManager: Created checkpoint at
/data/metadata/db.snapshots/om.db-62de62b7-60ab-4345-826a-ae0089ffdc88 in 6
milliseconds
2022-11-07 20:34:02,703 [OMDoubleBufferFlushThread] WARN
rocksdiff.RocksDBCheckpointDiffer: Compaction log exists:
/data/metadata/compaction-log/0000000000000000000.log. Will append
2022-11-07 21:14:59,162 [IPC Server handler 24 on default port 9862] WARN
server.ServerUtils: ozone.om.db.dirs is not configured. We recommend adding
this setting. Falling back to ozone.metadata.dirs instead.
2022-11-07 21:14:59,163 [IPC Server handler 24 on default port 9862] INFO
codec.OmKeyInfoCodec: OmKeyInfoCodec ignorePipeline = true
2022-11-07 21:14:59,163 [IPC Server handler 24 on default port 9862] INFO
codec.RepeatedOmKeyInfoCodec: RepeatedOmKeyInfoCodec ignorePipeline = true
2022-11-07 21:43:41,651 [OM StateMachine ApplyTransaction Thread - 0] ERROR
volume.OMVolumeCreateRequest: Volume creation failed for user:hadoop volume:vol1
VOLUME_ALREADY_EXISTS org.apache.hadoop.ozone.om.exceptions.OMException: Volume
already exists
at
org.apache.hadoop.ozone.om.request.volume.OMVolumeCreateRequest.validateAndUpdateCache(OMVolumeCreateRequest.java:153)
at
org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleWriteRequest(OzoneManagerRequestHandler.java:304)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.runCommand(OzoneManagerStateMachine.java:529)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.lambda$1(OzoneManagerStateMachine.java:324)
at
java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2022-11-07 22:03:55,719 [OM StateMachine ApplyTransaction Thread - 0] ERROR
volume.OMVolumeCreateRequest: Volume creation failed for user:hadoop volume:vol1
VOLUME_ALREADY_EXISTS org.apache.hadoop.ozone.om.exceptions.OMException: Volume
already exists
at
org.apache.hadoop.ozone.om.request.volume.OMVolumeCreateRequest.validateAndUpdateCache(OMVolumeCreateRequest.java:153)
at
org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleWriteRequest(OzoneManagerRequestHandler.java:304)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.runCommand(OzoneManagerStateMachine.java:529)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.lambda$1(OzoneManagerStateMachine.java:324)
at
java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2022-11-07 22:18:27,182
[org.apache.ratis.util.JvmPauseMonitor$$Lambda$416/0x0000000840502840@5f45bc8e]
WARN util.JvmPauseMonitor: JvmPauseMonitor-om1: Detected pause in JVM or host
machine (eg GC): pause of approximately 128755167ns. No GCs detected.
2022-11-07 23:17:52,722
[org.apache.ratis.util.JvmPauseMonitor$$Lambda$416/0x0000000840502840@5f45bc8e]
WARN util.JvmPauseMonitor: JvmPauseMonitor-om1: Detected pause in JVM or host
machine (eg GC): pause of approximately 601302626ns. No GCs detected.
2022-11-07 23:36:58,352 [OM StateMachine ApplyTransaction Thread - 0] ERROR
volume.OMVolumeCreateRequest: Volume creation failed for user:hadoop volume:vol1
VOLUME_ALREADY_EXISTS org.apache.hadoop.ozone.om.exceptions.OMException: Volume
already exists
at
org.apache.hadoop.ozone.om.request.volume.OMVolumeCreateRequest.validateAndUpdateCache(OMVolumeCreateRequest.java:153)
at
org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleWriteRequest(OzoneManagerRequestHandler.java:304)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.runCommand(OzoneManagerStateMachine.java:529)
at
org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.lambda$1(OzoneManagerStateMachine.java:324)
at
java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829){code}
> [snapshot] OM Process exiting on idle cluster
> ---------------------------------------------
>
> Key: HDDS-7374
> URL: https://issues.apache.org/jira/browse/HDDS-7374
> Project: Apache Ozone
> Issue Type: Bug
> Components: Ozone Manager
> Affects Versions: 1.3.0
> Environment: docker-compose on HDDS-6517-Snapshot
> Reporter: Jyotirmoy Sinha
> Assignee: Hemant Kumar
> Priority: Critical
> Labels: ozone-snapshot
>
> Snapshots were created on the cluster, after which the system was kept idle.
> After certain interval the OM Process exited and shut down.
> Docker logs -
> {code:java}
> 2022-10-20 12:53:31,082
> [org.apache.ratis.util.JvmPauseMonitor$$Lambda$398/0x00000008404cac40@bfe47a8]
> WARN util.JvmPauseMonitor: JvmPauseMonitor-om1: Detected pause in JVM or
> host machine (eg GC): pause of approximately 7196529648812ns. No GCs detected.
> 2022-10-20 12:53:27,062
> [org.apache.ratis.util.JvmPauseMonitor$$Lambda$398/0x00000008404cac40@bfe47a8]
> INFO server.RaftServer: om1: close
> 2022-10-20 12:53:27,128
> [org.apache.hadoop.util.JvmPauseMonitor$Monitor@60844992] WARN
> util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of
> approximately 7197056ms
> No GCs detected
> 2022-10-20 12:53:27,166
> [org.apache.ratis.util.JvmPauseMonitor$$Lambda$398/0x00000008404cac40@bfe47a8]
> INFO server.RaftServer$Division: om1@group-C5BA1605619E: shutdown
> 2022-10-20 12:53:27,177
> [org.apache.ratis.util.JvmPauseMonitor$$Lambda$398/0x00000008404cac40@bfe47a8]
> INFO util.JmxRegister: Successfully un-registered JMX Bean with object name
> Ratis:service=RaftServer,group=group-C5BA1605619E,id=om1
> 2022-10-20 12:53:27,196
> [org.apache.ratis.util.JvmPauseMonitor$$Lambda$398/0x00000008404cac40@bfe47a8]
> INFO impl.RoleInfo: om1: shutdown om1@group-C5BA1605619E-LeaderStateImpl
> 2022-10-20 12:53:27,212
> [org.apache.ratis.util.JvmPauseMonitor$$Lambda$398/0x00000008404cac40@bfe47a8]
> INFO impl.PendingRequests: om1@group-C5BA1605619E-PendingRequests:
> sendNotLeaderResponses
> 2022-10-20 12:53:27,366
> [org.apache.ratis.util.JvmPauseMonitor$$Lambda$398/0x00000008404cac40@bfe47a8]
> INFO impl.StateMachineUpdater: om1@group-C5BA1605619E-StateMachineUpdater:
> set stopIndex = 10
> 2022-10-20 12:53:27,368 [om1@group-C5BA1605619E-StateMachineUpdater] INFO
> ratis.OzoneManagerStateMachine: Current Snapshot Index (t:1, i:10)
> 2022-10-20 12:53:27,390 [om1@group-C5BA1605619E-StateMachineUpdater] INFO
> impl.StateMachineUpdater: om1@group-C5BA1605619E-StateMachineUpdater: Took a
> snapshot at index 10
> 2022-10-20 12:53:27,398 [om1@group-C5BA1605619E-StateMachineUpdater] INFO
> impl.StateMachineUpdater: om1@group-C5BA1605619E-StateMachineUpdater:
> snapshotIndex: updateIncreasingly -1 -> 10
> 2022-10-20 12:53:27,398 [om1@group-C5BA1605619E-StateMachineUpdater] INFO
> ratis.OzoneManagerStateMachine: StateMachine has shutdown. Shutdown
> OzoneManager if not already shutdown.
> 2022-10-20 12:53:27,399 [om1@group-C5BA1605619E-StateMachineUpdater] INFO
> om.OzoneManager: om1[om:9862]: Stopping Ozone Manager
> 2022-10-20 12:53:27,474 [om1@group-C5BA1605619E-StateMachineUpdater] INFO
> ipc.Server: Stopping server on 9862
> 2022-10-20 12:53:27,609 [IPC Server listener on 9862] INFO ipc.Server:
> Stopping IPC Server listener on 9862
> 2022-10-20 12:53:27,628 [IPC Server Responder] INFO ipc.Server: Stopping IPC
> Server Responder {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
