Hello Sreejesh, My read is that servers are not being able to connect to each other for leader election.
-Flavio > On 3 May 2022, at 15:50, Sreejesh Radhakrishnan > <sreejesh.radhakrish...@db.com.INVALID> wrote: > > Classification: Public > > Hi > > Sorry not sure if this is the right email. But still trying my luck, > > I am getting an NPE as pointed on the subject line, I am using Strimzi KAFKA > on GKE (1.21.9-gke.1002) using n1-standard-4 infra. > > Zookeeper is throwing below error, is that something which you guys have seen > or have idea of what causing it? > > get ZK log > > Detected Zookeeper ID 2 > Preparing truststore > Adding /opt/kafka/cluster-ca-certs/ca.crt to truststore > /tmp/zookeeper/cluster.truststore.p12 with alias ca > Certificate was added to keystore > Preparing truststore is complete > Looking for the right CA > Found the right CA: /opt/kafka/cluster-ca-certs/ca.crt > Preparing keystore for client and quorum listeners > Preparing keystore for client and quorum listeners is complete > Starting Zookeeper with configuration: > # The directory where the snapshot is stored. > dataDir=/var/lib/zookeeper/data > > # Other options > 4lw.commands.whitelist=* > standaloneEnabled=false > reconfigEnabled=true > clientPort=12181 > clientPortAddress=127.0.0.1 > > # TLS options > serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory > ssl.clientAuth=need > ssl.quorum.clientAuth=need > secureClientPort=2181 > sslQuorum=true > > ssl.trustStore.location=/tmp/zookeeper/cluster.truststore.p12 > ssl.trustStore.*** > ssl.trustStore.type=PKCS12 > ssl.quorum.trustStore.location=/tmp/zookeeper/cluster.truststore.p12 > ssl.quorum.trustStore.*** > ssl.quorum.trustStore.type=PKCS12 > > ssl.keyStore.location=/tmp/zookeeper/cluster.keystore.p12 > ssl.keyStore.*** > ssl.keyStore.type=PKCS12 > ssl.quorum.keyStore.location=/tmp/zookeeper/cluster.keystore.p12 > ssl.quorum.keyStore.*** > ssl.quorum.keyStore.type=PKCS12 > > # Provided configuration > tickTime=2000 > initLimit=5 > syncLimit=2 > autopurge.purgeInterval=1 > > > # Zookeeper nodes configuration > server.1=uwq-cluster-zookeeper-0.uwq-cluster-zookeeper-nodes.uwq-kafka.svc:2888:3888:participant;127.0.0.1:12181 > server.2=uwq-cluster-zookeeper-1.uwq-cluster-zookeeper-nodes.uwq-kafka.svc:2888:3888:participant;127.0.0.1:12181 > server.3=uwq-cluster-zookeeper-2.uwq-cluster-zookeeper-nodes.uwq-kafka.svc:2888:3888:participant;127.0.0.1:12181 > > + exec /usr/bin/tini -w -e 143 -- /opt/kafka/bin/zookeeper-server-start.sh > /tmp/zookeeper.properties > 2022-05-03 09:15:52,181 INFO Reading configuration from: > /tmp/zookeeper.properties > (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [main] > 2022-05-03 09:15:52,193 INFO clientPortAddress is 127.0.0.1:12181 > (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [main] > 2022-05-03 09:15:52,193 INFO secureClientPortAddress is 0.0.0.0:2181 > (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [main] > 2022-05-03 09:15:52,197 INFO Setting -D > jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated > TLS renegotiation (org.apache.zookeeper.common.X509Util) [main] > 2022-05-03 09:15:52,198 INFO observerMasterPort is not set > (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [main] > 2022-05-03 09:15:52,198 INFO metricsProvider.className is > org.apache.zookeeper.metrics.impl.DefaultMetricsProvider > (org.apache.zookeeper.server.quorum.QuorumPeerConfig) [main] > 2022-05-03 09:15:52,233 INFO autopurge.snapRetainCount set to 3 > (org.apache.zookeeper.server.DatadirCleanupManager) [main] > 2022-05-03 09:15:52,233 INFO autopurge.purgeInterval set to 1 > (org.apache.zookeeper.server.DatadirCleanupManager) [main] > 2022-05-03 09:15:52,241 INFO Log4j 1.2 jmx support found and enabled. > (org.apache.zookeeper.jmx.ManagedUtil) [main] > 2022-05-03 09:15:52,235 INFO Purge task started. > (org.apache.zookeeper.server.DatadirCleanupManager) [PurgeTask] > 2022-05-03 09:15:52,266 INFO zookeeper.snapshot.trust.empty : false > (org.apache.zookeeper.server.persistence.FileTxnSnapLog) [PurgeTask] > 2022-05-03 09:15:52,268 INFO Starting quorum peer, myid=2 > (org.apache.zookeeper.server.quorum.QuorumPeerMain) [main] > 2022-05-03 09:15:52,291 INFO zookeeper.snapshot.compression.method = CHECKED > (org.apache.zookeeper.server.persistence.SnapStream) [PurgeTask] > 2022-05-03 09:15:52,298 INFO Purge task completed. > (org.apache.zookeeper.server.DatadirCleanupManager) [PurgeTask] > 2022-05-03 09:15:52,306 INFO ServerMetrics initialized with provider > org.apache.zookeeper.metrics.impl.DefaultMetricsProvider@5276e6b0 > <mailto:org.apache.zookeeper.metrics.impl.DefaultMetricsProvider@5276e6b0> > (org.apache.zookeeper.server.ServerMetrics) [main] > 2022-05-03 09:15:52,404 INFO zookeeper.client.portUnification=false > (org.apache.zookeeper.server.NettyServerCnxnFactory) [main] > 2022-05-03 09:15:52,404 INFO zookeeper.netty.advancedFlowControl.enabled = > false (org.apache.zookeeper.server.NettyServerCnxnFactory) [main] > 2022-05-03 09:15:52,404 INFO handshakeThrottlingEnabled = false, > zookeeper.netty.server.outstandingHandshake.limit = -1 > (org.apache.zookeeper.server.NettyServerCnxnFactory) [main] > 2022-05-03 09:15:52,464 INFO Using > org.apache.zookeeper.server.NettyServerCnxnFactory as server connection > factory (org.apache.zookeeper.server.ServerCnxnFactory) [main] > 2022-05-03 09:15:52,466 WARN maxCnxns is not configured, using default value > 0. (org.apache.zookeeper.server.ServerCnxnFactory) [main] > 2022-05-03 09:15:52,466 INFO zookeeper.client.portUnification=false > (org.apache.zookeeper.server.NettyServerCnxnFactory) [main] > 2022-05-03 09:15:52,466 INFO zookeeper.netty.advancedFlowControl.enabled = > false (org.apache.zookeeper.server.NettyServerCnxnFactory) [main] > 2022-05-03 09:15:52,466 INFO handshakeThrottlingEnabled = false, > zookeeper.netty.server.outstandingHandshake.limit = -1 > (org.apache.zookeeper.server.NettyServerCnxnFactory) [main] > 2022-05-03 09:15:52,467 INFO Using > org.apache.zookeeper.server.NettyServerCnxnFactory as server connection > factory (org.apache.zookeeper.server.ServerCnxnFactory) [main] > 2022-05-03 09:15:52,467 WARN maxCnxns is not configured, using default value > 0. (org.apache.zookeeper.server.ServerCnxnFactory) [main] > 2022-05-03 09:15:52,476 INFO zookeeper.quorumCnxnTimeoutMs=-1 > (org.apache.zookeeper.server.quorum.QuorumPeer) [main] > 2022-05-03 09:15:52,507 INFO Logging initialized @932ms to > org.eclipse.jetty.util.log.Slf4jLog (org.eclipse.jetty.util.log) [main] > 2022-05-03 09:15:52,633 WARN > o.e.j.s.ServletContextHandler@1a75e76a{/,null,STOPPED} > <mailto:o.e.j.s.ServletContextHandler@1a75e76a%7b/,null,STOPPED%7d> > contextPath ends with /* (org.eclipse.jetty.server.handler.ContextHandler) > [main] > 2022-05-03 09:15:52,633 WARN Empty contextPath > (org.eclipse.jetty.server.handler.ContextHandler) [main] > 2022-05-03 09:15:52,666 INFO zookeeper.snapshot.trust.empty : false > (org.apache.zookeeper.server.persistence.FileTxnSnapLog) [main] > 2022-05-03 09:15:52,666 INFO Local sessions disabled > (org.apache.zookeeper.server.quorum.QuorumPeer) [main] > 2022-05-03 09:15:52,667 INFO Local session upgrading disabled > (org.apache.zookeeper.server.quorum.QuorumPeer) [main] > 2022-05-03 09:15:52,667 INFO tickTime set to 2000 > (org.apache.zookeeper.server.quorum.QuorumPeer) [main] > 2022-05-03 09:15:52,667 INFO minSessionTimeout set to 4000 > (org.apache.zookeeper.server.quorum.QuorumPeer) [main] > 2022-05-03 09:15:52,667 INFO maxSessionTimeout set to 40000 > (org.apache.zookeeper.server.quorum.QuorumPeer) [main] > 2022-05-03 09:15:52,667 INFO initLimit set to 5 > (org.apache.zookeeper.server.quorum.QuorumPeer) [main] > 2022-05-03 09:15:52,667 INFO syncLimit set to 2 > (org.apache.zookeeper.server.quorum.QuorumPeer) [main] > 2022-05-03 09:15:52,667 INFO connectToLearnerMasterLimit set to 0 > (org.apache.zookeeper.server.quorum.QuorumPeer) [main] > 2022-05-03 09:15:52,686 INFO (org.apache.zookeeper.server.ZooKeeperServer) > [main] > 2022-05-03 09:15:52,688 INFO ______ _ > (org.apache.zookeeper.server.ZooKeeperServer) [main] > 2022-05-03 09:15:52,689 INFO |___ / | | > (org.apache.zookeeper.server.ZooKeeperServer) [main] > 2022-05-03 09:15:52,689 INFO / / ___ ___ | | __ ___ ___ _ > __ ___ _ __ (org.apache.zookeeper.server.ZooKeeperServer) [main] > 2022-05-03 09:15:52,689 INFO / / / _ \ / _ \ | |/ / / _ \ / _ \ | > '_ \ / _ \ | '__| (org.apache.zookeeper.server.ZooKeeperServer) [main] > 2022-05-03 09:15:52,689 INFO / /__ | (_) | | (_) | | < | __/ | __/ | > |_) | | __/ | | (org.apache.zookeeper.server.ZooKeeperServer) [main] > 2022-05-03 09:15:52,689 INFO /_____| \___/ \___/ |_|\_\ \___| \___| | > .__/ \___| |_| (org.apache.zookeeper.server.ZooKeeperServer) [main] > 2022-05-03 09:15:52,689 INFO | > | (org.apache.zookeeper.server.ZooKeeperServer) [main] > 2022-05-03 09:15:52,689 INFO > |_| (org.apache.zookeeper.server.ZooKeeperServer) [main] > 2022-05-03 09:15:52,689 INFO (org.apache.zookeeper.server.ZooKeeperServer) > [main] > 2022-05-03 09:15:52,692 INFO Server > environment:zookeeper.version=3.6.3--6401e4ad2087061bc6b9f80dec2d69f2e3c8660a, > built on 04/08/2021 16:35 GMT (org.apache.zookeeper.server.ZooKeeperServer) > [main] > 2022-05-03 09:15:52,692 INFO Server > environment:host.name=uwq-cluster-zookeeper-1.uwq-cluster-zookeeper-nodes.uwq-kafka.svc.cluster.local > (org.apache.zookeeper.server.ZooKeeperServer) [main] > 2022-05-03 09:15:52,692 INFO Server environment:java.version=11.0.14 > (org.apache.zookeeper.server.ZooKeeperServer) [main] > 2022-05-03 09:15:52,692 INFO Server environment:java.vendor=Red Hat, Inc. > (org.apache.zookeeper.server.ZooKeeperServer) [main] > 2022-05-03 09:15:52,692 INFO Server > environment:java.home=/usr/lib/jvm/java-11-openjdk-11.0.14.0.9-2.el8_5.x86_64 > (org.apache.zookeeper.server.ZooKeeperServer) [main] > 2022-05-03 09:15:52,692 INFO Server > environment:java.class.path=/opt/kafka/bin/../libs/accessors-smart-2.4.7.jar:/opt/kafka/bin/../libs/activation-1.1.1.jar:/opt/kafka/bin/../libs/annotations-13.0.jar:/opt/kafka/bin/../libs/aopalliance-repackaged-2.6.1.jar:/opt/kafka/bin/../libs/argparse4j-0.7.0.jar:/opt/kafka/bin/../libs/audience-annotations-0.5.0.jar:/opt/kafka/bin/../libs/automaton-1.11-8.jar:/opt/kafka/bin/../libs/checker-qual-3.5.0.jar:/opt/kafka/bin/../libs/commons-cli-1.4.jar:/opt/kafka/bin/../libs/commons-lang-2.6.jar:/opt/kafka/bin/../libs/commons-lang3-3.8.1.jar:/opt/kafka/bin/../libs/connect-api-3.1.0.jar:/opt/kafka/bin/../libs/connect-basic-auth-extension-3.1.0.jar:/opt/kafka/bin/../libs/connect-file-3.1.0.jar:/opt/kafka/bin/../libs/connect-json-3.1.0.jar:/opt/kafka/bin/../libs/connect-mirror-3.1.0.jar:/opt/kafka/bin/../libs/connect-mirror-client-3.1.0.jar:/opt/kafka/bin/../libs/connect-runtime-3.1.0.jar:/opt/kafka/bin/../libs/connect-transforms-3.1.0.jar:/opt/kafka/bin/../libs/cruise-control-metrics-reporter-2.5.82.jar:/opt/kafka/bin/../libs/error_prone_annotations-2.3.4.jar:/opt/kafka/bin/../libs/failureaccess-1.0.1.jar:/opt/kafka/bin/../libs/generex-1.0.2.jar:/opt/kafka/bin/../libs/gson-2.8.6.jar:/opt/kafka/bin/../libs/guava-30.1-jre.jar:/opt/kafka/bin/../libs/hk2-api-2.6.1.jar:/opt/kafka/bin/../libs/hk2-locator-2.6.1.jar:/opt/kafka/bin/../libs/hk2-utils-2.6.1.jar:/opt/kafka/bin/../libs/j2objc-annotations-1.3.jar:/opt/kafka/bin/../libs/jackson-annotations-2.12.3.jar:/opt/kafka/bin/../libs/jackson-core-2.12.3.jar:/opt/kafka/bin/../libs/jackson-databind-2.12.3.jar:/opt/kafka/bin/../libs/jackson-dataformat-csv-2.12.3.jar:/opt/kafka/bin/../libs/jackson-dataformat-yaml-2.12.3.jar:/opt/kafka/bin/../libs/jackson-datatype-jdk8-2.12.3.jar:/opt/kafka/bin/../libs/jackson-datatype-jsr310-2.12.3.jar:/opt/kafka/bin/../libs/jackson-jaxrs-base-2.12.3.jar:/opt/kafka/bin/../libs/jackson-jaxrs-json-provider-2.12.3.jar:/opt/kafka/bin/../libs/jackson-module-jaxb-annotations-2.12.3.jar:/opt/kafka/bin/../libs/jackson-module-scala_2.13-2.12.3.jar:/opt/kafka/bin/../libs/jaeger-client-1.6.0.jar:/opt/kafka/bin/../libs/jaeger-core-1.6.0.jar:/opt/kafka/bin/../libs/jaeger-thrift-1.6.0.jar:/opt/kafka/bin/../libs/jaeger-tracerresolver-1.6.0.jar:/opt/kafka/bin/../libs/jakarta.activation-api-1.2.1.jar:/opt/kafka/bin/../libs/jakarta.annotation-api-1.3.5.jar:/opt/kafka/bin/../libs/jakarta.inject-2.6.1.jar:/opt/kafka/bin/../libs/jakarta.validation-api-2.0.2.jar:/opt/kafka/bin/../libs/jakarta.ws.rs > > <http://jakarta.ws.rs/>-api-2.1.6.jar:/opt/kafka/bin/../libs/jakarta.xml.bind-api-2.3.2.jar:/opt/kafka/bin/../libs/javassist-3.27.0-GA.jar:/opt/kafka/bin/../libs/javax.servlet-api-3.1.0.jar:/opt/kafka/bin/../libs/javax.ws.rs > > <http://javax.ws.rs/>-api-2.1.1.jar:/opt/kafka/bin/../libs/jaxb-api-2.3.0.jar:/opt/kafka/bin/../libs/jersey-client-2.34.jar:/opt/kafka/bin/../libs/jersey-common-2.34.jar:/opt/kafka/bin/../libs/jersey-container-servlet-2.34.jar:/opt/kafka/bin/../libs/jersey-container-servlet-core-2.34.jar:/opt/kafka/bin/../libs/jersey-hk2-2.34.jar:/opt/kafka/bin/../libs/jersey-server-2.34.jar:/opt/kafka/bin/../libs/jetty-client-9.4.43.v20210629.jar:/opt/kafka/bin/../libs/jetty-continuation-9.4.43.v20210629.jar:/opt/kafka/bin/../libs/jetty-http-9.4.43.v20210629.jar:/opt/kafka/bin/../libs/jetty-io-9.4.43.v20210629.jar:/opt/kafka/bin/../libs/jetty-security-9.4.43.v20210629.jar:/opt/kafka/bin/../libs/jetty-server-9.4.43.v20210629.jar:/opt/kafka/bin/../libs/jetty-servlet-9.4.43.v20210629.jar:/opt/kafka/bin/../libs/jetty-servlets-9.4.43.v20210629.jar:/opt/kafka/bin/../libs/jetty-util-9.4.43.v20210629.jar:/opt/kafka/bin/../libs/jetty-util-ajax-9.4.43.v20210629.jar:/opt/kafka/bin/../libs/jline-3.12.1.jar:/opt/kafka/bin/../libs/jmx_prometheus_javaagent-0.16.1.jar:/opt/kafka/bin/../libs/jopt-simple-5.0.4.jar:/opt/kafka/bin/../libs/jose4j-0.7.8.jar:/opt/kafka/bin/../libs/json-path-2.6.0.jar:/opt/kafka/bin/../libs/json-smart-2.4.7.jar:/opt/kafka/bin/../libs/jsonevent-layout-1.7.jar:/opt/kafka/bin/../libs/jsr305-3.0.2.jar:/opt/kafka/bin/../libs/kafka-agent-0.28.0.jar:/opt/kafka/bin/../libs/kafka-clients-3.1.0.jar:/opt/kafka/bin/../libs/kafka-env-var-config-provider-1.0.0.jar:/opt/kafka/bin/../libs/kafka-kubernetes-config-provider-1.0.0.jar:/opt/kafka/bin/../libs/kafka-log4j-appender-3.1.0.jar:/opt/kafka/bin/../libs/kafka-metadata-3.1.0.jar:/opt/kafka/bin/../libs/kafka-oauth-client-0.10.0.jar:/opt/kafka/bin/../libs/kafka-oauth-common-0.10.0.jar:/opt/kafka/bin/../libs/kafka-oauth-keycloak-authorizer-0.10.0.jar:/opt/kafka/bin/../libs/kafka-oauth-server-0.10.0.jar:/opt/kafka/bin/../libs/kafka-oauth-server-plain-0.10.0.jar:/opt/kafka/bin/../libs/kafka-quotas-plugin-0.2.0.jar:/opt/kafka/bin/../libs/kafka-raft-3.1.0.jar:/opt/kafka/bin/../libs/kafka-server-common-3.1.0.jar:/opt/kafka/bin/../libs/kafka-shell-3.1.0.jar:/opt/kafka/bin/../libs/kafka-storage-3.1.0.jar:/opt/kafka/bin/../libs/kafka-storage-api-3.1.0.jar:/opt/kafka/bin/../libs/kafka-streams-3.1.0.jar:/opt/kafka/bin/../libs/kafka-streams-examples-3.1.0.jar:/opt/kafka/bin/../libs/kafka-streams-scala_2.13-3.1.0.jar:/opt/kafka/bin/../libs/kafka-streams-test-utils-3.1.0.jar:/opt/kafka/bin/../libs/kafka-tools-3.1.0.jar:/opt/kafka/bin/../libs/kafka_2.13-3.1.0.jar:/opt/kafka/bin/../libs/kotlin-stdlib-1.4.10.jar:/opt/kafka/bin/../libs/kotlin-stdlib-common-1.4.0.jar:/opt/kafka/bin/../libs/kubernetes-client-5.12.0.jar:/opt/kafka/bin/../libs/kubernetes-model-apps-5.12.0.jar:/opt/kafka/bin/../libs/kubernetes-model-batch-5.12.0.jar:/opt/kafka/bin/../libs/kubernetes-model-certificates-5.12.0.jar:/opt/kafka/bin/../libs/kubernetes-model-common-5.12.0.jar:/opt/kafka/bin/../libs/kubernetes-model-core-5.12.0.jar:/opt/kafka/bin/../libs/kubernetes-model-extensions-5.12.0.jar:/opt/kafka/bin/../libs/libthrift-0.14.1.jar:/opt/kafka/bin/../libs/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/opt/kafka/bin/../libs/log4j-1.2.17.jar:/opt/kafka/bin/../libs/logging-interceptor-3.12.12.jar:/opt/kafka/bin/../libs/lz4-java-1.8.0.jar:/opt/kafka/bin/../libs/maven-artifact-3.8.1.jar:/opt/kafka/bin/../libs/metrics-core-2.2.0.jar:/opt/kafka/bin/../libs/metrics-core-4.1.12.1.jar:/opt/kafka/bin/../libs/mirror-maker-2-extensions-1.2.0.jar:/opt/kafka/bin/../libs/mirror-maker-agent-0.28.0.jar:/opt/kafka/bin/../libs/netty-buffer-4.1.68.Final.jar:/opt/kafka/bin/../libs/netty-codec-4.1.68.Final.jar:/opt/kafka/bin/../libs/netty-common-4.1.68.Final.jar:/opt/kafka/bin/../libs/netty-handler-4.1.68.Final.jar:/opt/kafka/bin/../libs/netty-resolver-4.1.68.Final.jar:/opt/kafka/bin/../libs/netty-transport-4.1.68.Final.jar:/opt/kafka/bin/../libs/netty-transport-native-epoll-4.1.68.Final.jar:/opt/kafka/bin/../libs/netty-transport-native-unix-common-4.1.68.Final.jar:/opt/kafka/bin/../libs/nimbus-jose-jwt-9.10.jar:/opt/kafka/bin/../libs/okhttp-4.9.0.jar:/opt/kafka/bin/../libs/okio-2.8.0.jar:/opt/kafka/bin/../libs/opa-authorizer-1.4.0.jar:/opt/kafka/bin/../libs/opentracing-api-0.33.0.jar:/opt/kafka/bin/../libs/opentracing-kafka-client-0.1.15.jar:/opt/kafka/bin/../libs/opentracing-noop-0.33.0.jar:/opt/kafka/bin/../libs/opentracing-tracerresolver-0.1.8.jar:/opt/kafka/bin/../libs/opentracing-util-0.33.0.jar:/opt/kafka/bin/../libs/osgi-resource-locator-1.0.3.jar:/opt/kafka/bin/../libs/paranamer-2.8.jar:/opt/kafka/bin/../libs/plexus-utils-3.2.1.jar:/opt/kafka/bin/../libs/reflections-0.9.12.jar:/opt/kafka/bin/../libs/rocksdbjni-6.22.1.1.jar:/opt/kafka/bin/../libs/scala-collection-compat_2.13-2.4.4.jar:/opt/kafka/bin/../libs/scala-java8-compat_2.13-1.0.0.jar:/opt/kafka/bin/../libs/scala-library-2.13.6.jar:/opt/kafka/bin/../libs/scala-logging_2.13-3.9.3.jar:/opt/kafka/bin/../libs/scala-reflect-2.13.6.jar:/opt/kafka/bin/../libs/slf4j-api-1.7.30.jar:/opt/kafka/bin/../libs/slf4j-log4j12-1.7.30.jar:/opt/kafka/bin/../libs/snakeyaml-1.27.jar:/opt/kafka/bin/../libs/snappy-java-1.1.8.4.jar:/opt/kafka/bin/../libs/tracing-agent-0.28.0.jar:/opt/kafka/bin/../libs/trogdor-3.1.0.jar:/opt/kafka/bin/../libs/zjsonpatch-0.3.0.jar:/opt/kafka/bin/../libs/zookeeper-3.6.3.jar:/opt/kafka/bin/../libs/zookeeper-jute-3.6.3.jar:/opt/kafka/bin/../libs/zstd-jni-1.5.0-4.jar > (org.apache.zookeeper.server.ZooKeeperServer) [main] > 2022-05-03 09:15:52,692 INFO Server > environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib > (org.apache.zookeeper.server.ZooKeeperServer) [main] > 2022-05-03 09:15:52,692 INFO Server environment:java.io.tmpdir=/tmp > (org.apache.zookeeper.server.ZooKeeperServer) [main] > 2022-05-03 09:15:52,692 INFO Server environment:java.compiler=<NA> > (org.apache.zookeeper.server.ZooKeeperServer) [main] > 2022-05-03 09:15:52,692 INFO Server environment:os.name=Linux > (org.apache.zookeeper.server.ZooKeeperServer) [main] > 2022-05-03 09:15:52,692 INFO Server environment:os.arch=amd64 > (org.apache.zookeeper.server.ZooKeeperServer) [main] > 2022-05-03 09:15:52,692 INFO Server environment:os.version=5.4.170+ > (org.apache.zookeeper.server.ZooKeeperServer) [main] > 2022-05-03 09:15:52,692 INFO Server environment:user.name=kafka > (org.apache.zookeeper.server.ZooKeeperServer) [main] > 2022-05-03 09:15:52,693 INFO Server environment:user.home=/home/kafka > (org.apache.zookeeper.server.ZooKeeperServer) [main] > 2022-05-03 09:15:52,693 INFO Server environment:user.dir=/opt/kafka > (org.apache.zookeeper.server.ZooKeeperServer) [main] > 2022-05-03 09:15:52,693 INFO Server environment:os.memory.free=242MB > (org.apache.zookeeper.server.ZooKeeperServer) [main] > 2022-05-03 09:15:52,693 INFO Server environment:os.memory.max=3118MB > (org.apache.zookeeper.server.ZooKeeperServer) [main] > 2022-05-03 09:15:52,693 INFO Server environment:os.memory.total=260MB > (org.apache.zookeeper.server.ZooKeeperServer) [main] > 2022-05-03 09:15:52,693 INFO zookeeper.enableEagerACLCheck = false > (org.apache.zookeeper.server.ZooKeeperServer) [main] > 2022-05-03 09:15:52,693 INFO zookeeper.skipACL=="yes", ACL checks will be > skipped (org.apache.zookeeper.server.ZooKeeperServer) [main] > 2022-05-03 09:15:52,693 INFO zookeeper.digest.enabled = true > (org.apache.zookeeper.server.ZooKeeperServer) [main] > 2022-05-03 09:15:52,693 INFO zookeeper.closeSessionTxn.enabled = true > (org.apache.zookeeper.server.ZooKeeperServer) [main] > 2022-05-03 09:15:52,693 INFO zookeeper.flushDelay=0 > (org.apache.zookeeper.server.ZooKeeperServer) [main] > 2022-05-03 09:15:52,693 INFO zookeeper.maxWriteQueuePollTime=0 > (org.apache.zookeeper.server.ZooKeeperServer) [main] > 2022-05-03 09:15:52,693 INFO zookeeper.maxBatchSize=1000 > (org.apache.zookeeper.server.ZooKeeperServer) [main] > 2022-05-03 09:15:52,693 INFO zookeeper.intBufferStartingSizeBytes = 1024 > (org.apache.zookeeper.server.ZooKeeperServer) [main] > 2022-05-03 09:15:52,699 INFO Using > org.apache.zookeeper.server.watch.WatchManager as watch manager > (org.apache.zookeeper.server.watch.WatchManagerFactory) [main] > 2022-05-03 09:15:52,700 INFO Using > org.apache.zookeeper.server.watch.WatchManager as watch manager > (org.apache.zookeeper.server.watch.WatchManagerFactory) [main] > 2022-05-03 09:15:52,701 INFO zookeeper.snapshotSizeFactor = 0.33 > (org.apache.zookeeper.server.ZKDatabase) [main] > 2022-05-03 09:15:52,701 INFO zookeeper.commitLogCount=500 > (org.apache.zookeeper.server.ZKDatabase) [main] > 2022-05-03 09:15:52,712 INFO Using TLS encrypted quorum communication > (org.apache.zookeeper.server.quorum.QuorumPeer) [main] > 2022-05-03 09:15:52,712 INFO Port unification disabled > (org.apache.zookeeper.server.quorum.QuorumPeer) [main] > 2022-05-03 09:15:52,713 INFO multiAddress.enabled set to false > (org.apache.zookeeper.server.quorum.QuorumPeer) [main] > 2022-05-03 09:15:52,713 INFO multiAddress.reachabilityCheckEnabled set to > true (org.apache.zookeeper.server.quorum.QuorumPeer) [main] > 2022-05-03 09:15:52,713 INFO multiAddress.reachabilityCheckTimeoutMs set to > 1000 (org.apache.zookeeper.server.quorum.QuorumPeer) [main] > 2022-05-03 09:15:52,713 INFO QuorumPeer communication is not secured! (SASL > auth disabled) (org.apache.zookeeper.server.quorum.QuorumPeer) [main] > 2022-05-03 09:15:52,713 INFO quorum.cnxn.threads.size set to 20 > (org.apache.zookeeper.server.quorum.QuorumPeer) [main] > 2022-05-03 09:15:52,713 INFO Reading snapshot > /var/lib/zookeeper/data/version-2/snapshot.0 > (org.apache.zookeeper.server.persistence.FileSnap) [main] > 2022-05-03 09:15:52,718 INFO The digest value is empty in snapshot > (org.apache.zookeeper.server.DataTree) [main] > 2022-05-03 09:15:52,721 INFO Snapshot loaded in 8 ms, highest zxid is 0x0, > digest is 1371985504 (org.apache.zookeeper.server.ZKDatabase) [main] > 2022-05-03 09:15:52,722 INFO binding to port /127.0.0.1:12181 > (org.apache.zookeeper.server.NettyServerCnxnFactory) [main] > 2022-05-03 09:15:52,812 INFO bound to port 12181 > (org.apache.zookeeper.server.NettyServerCnxnFactory) [main] > 2022-05-03 09:15:52,812 INFO binding to port 0.0.0.0/0.0.0.0:2181 > (org.apache.zookeeper.server.NettyServerCnxnFactory) [main] > 2022-05-03 09:15:52,814 INFO bound to port 2181 > (org.apache.zookeeper.server.NettyServerCnxnFactory) [main] > 2022-05-03 09:15:52,818 INFO jetty-9.4.43.v20210629; built: > 2021-06-30T11:07:22.254Z; git: 526006ecfa3af7f1a27ef3a288e2bef7ea9dd7e8; jvm > 11.0.14+9-LTS (org.eclipse.jetty.server.Server) [main] > 2022-05-03 09:15:52,852 INFO DefaultSessionIdManager workerName=node0 > (org.eclipse.jetty.server.session) [main] > 2022-05-03 09:15:52,852 INFO No SessionScavenger set, using defaults > (org.eclipse.jetty.server.session) [main] > 2022-05-03 09:15:52,854 INFO node0 Scavenging every 600000ms > (org.eclipse.jetty.server.session) [main] > 2022-05-03 09:15:52,857 WARN > ServletContext@o.e.j.s.ServletContextHandler@1a75e76a{/,null,STARTING} > <mailto:ServletContext@o.e.j.s.ServletContextHandler@1a75e76a%7b/,null,STARTING%7d> > has uncovered http methods for path: /* > (org.eclipse.jetty.security.SecurityHandler) [main] > 2022-05-03 09:15:52,866 INFO Started > o.e.j.s.ServletContextHandler@1a75e76a{/,null,AVAILABLE} > <mailto:o.e.j.s.ServletContextHandler@1a75e76a%7b/,null,AVAILABLE%7d> > (org.eclipse.jetty.server.handler.ContextHandler) [main] > 2022-05-03 09:15:52,873 INFO Started ServerConnector@60bd273d{HTTP/1.1 > <mailto:ServerConnector@60bd273d%7bHTTP/1.1>, (http/1.1)}{0.0.0.0:8080} > (org.eclipse.jetty.server.AbstractConnector) [main] > 2022-05-03 09:15:52,873 INFO Started @1299ms > (org.eclipse.jetty.server.Server) [main] > 2022-05-03 09:15:52,874 INFO Started AdminServer on address 0.0.0.0, port > 8080 and command URL /commands > (org.apache.zookeeper.server.admin.JettyAdminServer) [main] > 2022-05-03 09:15:52,875 INFO Using 4000ms as the quorum cnxn socket timeout > (org.apache.zookeeper.server.quorum.QuorumPeer) [main] > 2022-05-03 09:15:52,884 INFO Election port bind maximum retries is 3 > (org.apache.zookeeper.server.quorum.QuorumCnxManager) [main] > 2022-05-03 09:15:52,893 INFO zookeeper.fastleader.minNotificationInterval=200 > (org.apache.zookeeper.server.quorum.FastLeaderElection) [main] > 2022-05-03 09:15:52,894 INFO > zookeeper.fastleader.maxNotificationInterval=60000 > (org.apache.zookeeper.server.quorum.FastLeaderElection) [main] > 2022-05-03 09:15:52,894 INFO Creating TLS-only quorum server socket > (org.apache.zookeeper.server.quorum.QuorumCnxManager) > [ListenerHandler-uwq-cluster-zookeeper-1.uwq-cluster-zookeeper-nodes.uwq-kafka.svc/240.1.185.38:3888] > 2022-05-03 09:15:52,898 INFO 2 is accepting connections now, my election bind > port: > uwq-cluster-zookeeper-1.uwq-cluster-zookeeper-nodes.uwq-kafka.svc/240.1.185.38:3888 > (org.apache.zookeeper.server.quorum.QuorumCnxManager) > [ListenerHandler-uwq-cluster-zookeeper-1.uwq-cluster-zookeeper-nodes.uwq-kafka.svc/240.1.185.38:3888] > 2022-05-03 09:15:52,901 INFO ZooKeeper audit is disabled. > (org.apache.zookeeper.audit.ZKAuditProvider) [main] > 2022-05-03 09:15:52,922 INFO LOOKING > (org.apache.zookeeper.server.quorum.QuorumPeer) > [QuorumPeer[myid=2](plain=127.0.0.1:12181)(secure=0.0.0.0:2181)] > 2022-05-03 09:15:52,938 INFO New election. My id = 2, proposed zxid=0x0 > (org.apache.zookeeper.server.quorum.FastLeaderElection) > [QuorumPeer[myid=2](plain=127.0.0.1:12181)(secure=0.0.0.0:2181)] > 2022-05-03 09:15:52,951 INFO Notification: my state:LOOKING; n.sid:2, > n.state:LOOKING, n.leader:2, n.round:0x1, n.peerEpoch:0x0, n.zxid:0x0, > message format version:0x2, n.config version:0x0 > (org.apache.zookeeper.server.quorum.FastLeaderElection) > [WorkerReceiver[myid=2]] > 2022-05-03 09:15:53,164 INFO Notification time out: 400 > (org.apache.zookeeper.server.quorum.FastLeaderElection) > [QuorumPeer[myid=2](plain=127.0.0.1:12181)(secure=0.0.0.0:2181)] > 2022-05-03 09:15:53,565 INFO Notification time out: 800 > (org.apache.zookeeper.server.quorum.FastLeaderElection) > [QuorumPeer[myid=2](plain=127.0.0.1:12181)(secure=0.0.0.0:2181)] > 2022-05-03 09:15:54,366 INFO Notification time out: 1600 > (org.apache.zookeeper.server.quorum.FastLeaderElection) > [QuorumPeer[myid=2](plain=127.0.0.1:12181)(secure=0.0.0.0:2181)] > 2022-05-03 09:15:55,966 INFO Notification time out: 3200 > (org.apache.zookeeper.server.quorum.FastLeaderElection) > [QuorumPeer[myid=2](plain=127.0.0.1:12181)(secure=0.0.0.0:2181)] > 2022-05-03 09:15:58,440 WARN Cannot open channel to 1 at election address > uwq-cluster-zookeeper-0.uwq-cluster-zookeeper-nodes.uwq-kafka.svc/240.1.185.68:3888 > (org.apache.zookeeper.server.quorum.QuorumCnxManager) > [QuorumConnectionThread-[myid=2]-1] > java.net.SocketTimeoutException: connect timed out > at java.base/java.net.PlainSocketImpl.socketConnect(Native > Method) > at > java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412) > at > java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255) > at > java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237) > at > java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > at java.base/java.net.Socket.connect(Socket.java:609) > at > java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:300) > at > org.apache.zookeeper.server.quorum.QuorumCnxManager.initiateConnection(QuorumCnxManager.java:383) > at > org.apache.zookeeper.server.quorum.QuorumCnxManager$QuorumConnectionReqThread.run(QuorumCnxManager.java:457) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at java.base/java.lang.Thread.run(Thread.java:829) > 2022-05-03 09:15:58,440 WARN Cannot open channel to 3 at election address > uwq-cluster-zookeeper-2.uwq-cluster-zookeeper-nodes.uwq-kafka.svc/240.1.185.170:3888 > (org.apache.zookeeper.server.quorum.QuorumCnxManager) > [QuorumConnectionThread-[myid=2]-2] > java.net.SocketTimeoutException: connect timed out > at java.base/java.net.PlainSocketImpl.socketConnect(Native > Method) > at > java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412) > at > java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255) > at > java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237) > at > java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > at java.base/java.net.Socket.connect(Socket.java:609) > at > java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:300) > at > org.apache.zookeeper.server.quorum.QuorumCnxManager.initiateConnection(QuorumCnxManager.java:383) > at > org.apache.zookeeper.server.quorum.QuorumCnxManager$QuorumConnectionReqThread.run(QuorumCnxManager.java:457) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at java.base/java.lang.Thread.run(Thread.java:829) > 2022-05-03 09:15:59,168 INFO Notification time out: 6400 > (org.apache.zookeeper.server.quorum.FastLeaderElection) > [QuorumPeer[myid=2](plain=127.0.0.1:12181)(secure=0.0.0.0:2181)] > 2022-05-03 09:16:04,173 WARN Cannot open channel to 3 at election address > uwq-cluster-zookeeper-2.uwq-cluster-zookeeper-nodes.uwq-kafka.svc/240.1.185.170:3888 > (org.apache.zookeeper.server.quorum.QuorumCnxManager) > [QuorumConnectionThread-[myid=2]-1] > java.net.SocketTimeoutException: connect timed out > at java.base/java.net.PlainSocketImpl.socketConnect(Native > Method) > at > java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412) > at > java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255) > at > java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237) > at > java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > at java.base/java.net.Socket.connect(Socket.java:609) > at > java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:300) > at > org.apache.zookeeper.server.quorum.QuorumCnxManager.initiateConnection(QuorumCnxManager.java:383) > at > org.apache.zookeeper.server.quorum.QuorumCnxManager$QuorumConnectionReqThread.run(QuorumCnxManager.java:457) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at java.base/java.lang.Thread.run(Thread.java:829) > 2022-05-03 09:16:04,174 WARN Cannot open channel to 1 at election address > uwq-cluster-zookeeper-0.uwq-cluster-zookeeper-nodes.uwq-kafka.svc/240.1.185.68:3888 > (org.apache.zookeeper.server.quorum.QuorumCnxManager) > [QuorumConnectionThread-[myid=2]-3] > java.net.SocketTimeoutException: connect timed out > at java.base/java.net.PlainSocketImpl.socketConnect(Native > Method) > at > java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412) > at > java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255) > at > java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237) > at > java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > at java.base/java.net.Socket.connect(Socket.java:609) > at > java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:300) > at > org.apache.zookeeper.server.quorum.QuorumCnxManager.initiateConnection(QuorumCnxManager.java:383) > at > org.apache.zookeeper.server.quorum.QuorumCnxManager$QuorumConnectionReqThread.run(QuorumCnxManager.java:457) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at java.base/java.lang.Thread.run(Thread.java:829) > 2022-05-03 09:16:05,570 INFO Notification time out: 12800 > (org.apache.zookeeper.server.quorum.FastLeaderElection) > [QuorumPeer[myid=2](plain=127.0.0.1:12181)(secure=0.0.0.0:2181)] > 2022-05-03 09:16:09,563 WARN Exception caught > (org.apache.zookeeper.server.NettyServerCnxnFactory) [nioEventLoopGroup-4-1] > java.lang.NullPointerException > at > org.apache.zookeeper.server.NettyServerCnxnFactory$CnxnChannelHandler.channelActive(NettyServerCnxnFactory.java:260) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:230) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:216) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:209) > at > io.netty.channel.DefaultChannelPipeline$HeadContext.channelActive(DefaultChannelPipeline.java:1398) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:230) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:216) > at > io.netty.channel.DefaultChannelPipeline.fireChannelActive(DefaultChannelPipeline.java:895) > at > io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:522) > at > io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:429) > at > io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:486) > at > io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) > at > io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) > at > io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) > at > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) > at > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.base/java.lang.Thread.run(Thread.java:829) > 2022-05-03 09:16:09,616 WARN Exception caught > (org.apache.zookeeper.server.NettyServerCnxnFactory) [nioEventLoopGroup-4-2] > java.lang.NullPointerException > at > org.apache.zookeeper.server.NettyServerCnxnFactory$CnxnChannelHandler.channelActive(NettyServerCnxnFactory.java:260) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:230) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:216) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:209) > at > io.netty.channel.DefaultChannelPipeline$HeadContext.channelActive(DefaultChannelPipeline.java:1398) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:230) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:216) > at > io.netty.channel.DefaultChannelPipeline.fireChannelActive(DefaultChannelPipeline.java:895) > at > io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:522) > at > io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:429) > at > io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:486) > at > io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) > at > io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) > at > io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) > at > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) > at > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.base/java.lang.Thread.run(Thread.java:829) > 2022-05-03 09:16:10,576 WARN Cannot open channel to 3 at election address > uwq-cluster-zookeeper-2.uwq-cluster-zookeeper-nodes.uwq-kafka.svc/240.1.185.170:3888 > (org.apache.zookeeper.server.quorum.QuorumCnxManager) > [QuorumConnectionThread-[myid=2]-1] > java.net.SocketTimeoutException: connect timed out > at java.base/java.net.PlainSocketImpl.socketConnect(Native > Method) > at > java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412) > at > java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255) > at > java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237) > at > java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > at java.base/java.net.Socket.connect(Socket.java:609) > at > java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:300) > at > org.apache.zookeeper.server.quorum.QuorumCnxManager.initiateConnection(QuorumCnxManager.java:383) > at > org.apache.zookeeper.server.quorum.QuorumCnxManager$QuorumConnectionReqThread.run(QuorumCnxManager.java:457) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at java.base/java.lang.Thread.run(Thread.java:829) > 2022-05-03 09:16:10,576 WARN Cannot open channel to 1 at election address > uwq-cluster-zookeeper-0.uwq-cluster-zookeeper-nodes.uwq-kafka.svc/240.1.185.68:3888 > (org.apache.zookeeper.server.quorum.QuorumCnxManager) > [QuorumConnectionThread-[myid=2]-3] > java.net.SocketTimeoutException: connect timed out > at java.base/java.net.PlainSocketImpl.socketConnect(Native > Method) > at > java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412) > at > java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255) > at > java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237) > at > java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > at java.base/java.net.Socket.connect(Socket.java:609) > at > java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:300) > at > org.apache.zookeeper.server.quorum.QuorumCnxManager.initiateConnection(QuorumCnxManager.java:383) > at > org.apache.zookeeper.server.quorum.QuorumCnxManager$QuorumConnectionReqThread.run(QuorumCnxManager.java:457) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at java.base/java.lang.Thread.run(Thread.java:829) > 2022-05-03 09:16:18,371 INFO Notification time out: 25600 > (org.apache.zookeeper.server.quorum.FastLeaderElection) > [QuorumPeer[myid=2](plain=127.0.0.1:12181)(secure=0.0.0.0:2181)] > 2022-05-03 09:16:19,531 WARN Exception caught > (org.apache.zookeeper.server.NettyServerCnxnFactory) [nioEventLoopGroup-4-1] > java.lang.NullPointerException > at > org.apache.zookeeper.server.NettyServerCnxnFactory$CnxnChannelHandler.channelActive(NettyServerCnxnFactory.java:260) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:230) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:216) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:209) > at > io.netty.channel.DefaultChannelPipeline$HeadContext.channelActive(DefaultChannelPipeline.java:1398) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:230) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:216) > at > io.netty.channel.DefaultChannelPipeline.fireChannelActive(DefaultChannelPipeline.java:895) > at > io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:522) > at > io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:429) > at > io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:486) > at > io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) > at > io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) > at > io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) > at > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) > at > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.base/java.lang.Thread.run(Thread.java:829) > 2022-05-03 09:16:19,583 WARN Exception caught > (org.apache.zookeeper.server.NettyServerCnxnFactory) [nioEventLoopGroup-4-2] > java.lang.NullPointerException > at > org.apache.zookeeper.server.NettyServerCnxnFactory$CnxnChannelHandler.channelActive(NettyServerCnxnFactory.java:260) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:230) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:216) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:209) > at > io.netty.channel.DefaultChannelPipeline$HeadContext.channelActive(DefaultChannelPipeline.java:1398) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:230) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:216) > at > io.netty.channel.DefaultChannelPipeline.fireChannelActive(DefaultChannelPipeline.java:895) > at > io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:522) > at > io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:429) > at > io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:486) > at > io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) > at > io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) > at > io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) > at > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) > at > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.base/java.lang.Thread.run(Thread.java:829) > 2022-05-03 09:16:23,377 WARN Cannot open channel to 1 at election address > uwq-cluster-zookeeper-0.uwq-cluster-zookeeper-nodes.uwq-kafka.svc/240.1.185.68:3888 > (org.apache.zookeeper.server.quorum.QuorumCnxManager) > [QuorumConnectionThread-[myid=2]-3] > java.net.SocketTimeoutException: connect timed out > at java.base/java.net.PlainSocketImpl.socketConnect(Native > Method) > at > java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412) > at > java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255) > at > java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237) > at > java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > at java.base/java.net.Socket.connect(Socket.java:609) > at > java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:300) > at > org.apache.zookeeper.server.quorum.QuorumCnxManager.initiateConnection(QuorumCnxManager.java:383) > at > org.apache.zookeeper.server.quorum.QuorumCnxManager$QuorumConnectionReqThread.run(QuorumCnxManager.java:457) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at java.base/java.lang.Thread.run(Thread.java:829) > 2022-05-03 09:16:23,377 WARN Cannot open channel to 3 at election address > uwq-cluster-zookeeper-2.uwq-cluster-zookeeper-nodes.uwq-kafka.svc/240.1.185.170:3888 > (org.apache.zookeeper.server.quorum.QuorumCnxManager) > [QuorumConnectionThread-[myid=2]-1] > java.net.SocketTimeoutException: connect timed out > at java.base/java.net.PlainSocketImpl.socketConnect(Native > Method) > at > java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412) > at > java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255) > at > java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237) > at > java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > at java.base/java.net.Socket.connect(Socket.java:609) > at > java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:300) > at > org.apache.zookeeper.server.quorum.QuorumCnxManager.initiateConnection(QuorumCnxManager.java:383) > at > org.apache.zookeeper.server.quorum.QuorumCnxManager$QuorumConnectionReqThread.run(QuorumCnxManager.java:457) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at java.base/java.lang.Thread.run(Thread.java:829) > 2022-05-03 09:16:29,538 WARN Exception caught > (org.apache.zookeeper.server.NettyServerCnxnFactory) [nioEventLoopGroup-4-1] > java.lang.NullPointerException > at > org.apache.zookeeper.server.NettyServerCnxnFactory$CnxnChannelHandler.channelActive(NettyServerCnxnFactory.java:260) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:230) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:216) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:209) > at > io.netty.channel.DefaultChannelPipeline$HeadContext.channelActive(DefaultChannelPipeline.java:1398) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:230) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:216) > at > io.netty.channel.DefaultChannelPipeline.fireChannelActive(DefaultChannelPipeline.java:895) > at > io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:522) > at > io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:429) > at > io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:486) > at > io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) > at > io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) > at > io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) > at > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) > at > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.base/java.lang.Thread.run(Thread.java:829) > 2022-05-03 09:16:29,596 WARN Exception caught > (org.apache.zookeeper.server.NettyServerCnxnFactory) [nioEventLoopGroup-4-2] > java.lang.NullPointerException > at > org.apache.zookeeper.server.NettyServerCnxnFactory$CnxnChannelHandler.channelActive(NettyServerCnxnFactory.java:260) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:230) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:216) > at > io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:209) > at > io.netty.channel.DefaultChannelPipeline$HeadContext.channelActive(DefaultChannelPipeline.java:1398) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:230) > at > io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:216) > at > io.netty.channel.DefaultChannelPipeline.fireChannelActive(DefaultChannelPipeline.java:895) > at > io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:522) > at > io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:429) > at > io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:486) > at > io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) > at > io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) > at > io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500) > at > io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) > at > io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.base/java.lang.Thread.run(Thread.java:829) > get Operator log > > > Kind Regards, > > Sree > > ____________________________________________________ > > > > Sreejesh Radhakrishnan > > Deutsche Bank AG, Filiale London > CB & IB Tech > 10 Upper Bank Street, Canary Wharf, E14 5JJ London, United Kingdom > Mobile +44 7918-824-905 > Email sreejesh.radhakrish...@db.com <mailto:sreejesh.radhakrish...@db.com> > > <https://www.db.com/company/en/positiveimpact.htm> > > > > --- > This e-mail may contain confidential and/or privileged information. If you > are not the intended recipient (or have received this e-mail in error) please > notify the sender immediately and delete this e-mail. Any unauthorized > copying, disclosure or distribution of the material in this e-mail is > strictly forbidden. > > Please refer to https://www.db.com/disclosures > <https://www.db.com/disclosures> for additional EU corporate and regulatory > disclosures and to http://www.db.com/unitedkingdom/content/privacy.htm > <http://www.db.com/unitedkingdom/content/privacy.htm> for information about > privacy.
