rolish commented on issue #2080: Geo-replication setup on K8s cluster URL: https://github.com/apache/bookkeeper/issues/2080#issuecomment-489025425 Bookie startup log: ``` Environment Vars for bookie: [metadata service] BK_zkServers is zookeeper-1.abcdefg.com;zookeeper-2.abcdefg.com;zookeeper-3.abcdefg.com;zookeeper-4.abcdefg.com;zookeeper-5.abcdefg.com;zookeeper-6.abcdefg.com BK_CLUSTER_ROOT_PATH is BK_metadataServiceUri is zk://zookeeper-1.abcdefg.com;zookeeper-2.abcdefg.com;zookeeper-3.abcdefg.com;zookeeper-4.abcdefg.com;zookeeper-5.abcdefg.com;zookeeper-6.abcdefg.com/ledgers [bookie] BK_bookiePort bookie service port is 3181 BK_DATA_DIR is /data/bookkeeper BK_journalDirectory is /bookkeeper/data/journal BK_ledgerDirectories are /bookkeeper/data/ledgers BK_indexDirectories are /bookkeeper/data/ledgers [bookie http] BK_httpServerEnabled is true BK_httpServerPort is [dlog] BK_dlogRootPath is /distributedlog [stream storage] BK_STREAM_STORAGE_ROOT_PATH is /stream BK_NUM_STORAGE_CONTAINERS is 32 BOOKIE_GRPC_PORT is conf files: ['/opt/bookkeeper/conf/zookeeper.conf', '/opt/bookkeeper/conf/bkenv.sh', '/opt/bookkeeper/conf/zookeeper.conf.dynamic', '/opt/bookkeeper/conf/bk_server.conf', '/opt/bookkeeper/conf/nettyenv.sh', '/opt/bookkeeper/conf/log4j.properties', '/opt/bookkeeper/conf/log4j.shell.properties', '/opt/bookkeeper/conf/standalone.conf', '/opt/bookkeeper/conf/log4j.cli.properties', '/opt/bookkeeper/conf/bk_cli_env.sh', '/opt/bookkeeper/conf/jaas_example.conf'] [/opt/bookkeeper/conf/bkenv.sh] Applying config BOOKIE_EXTRA_OPTS = "-Xms1g -Xmx1g -XX:MaxDirectMemorySize=1g -XX:+UseG1GC -XX:MaxGCPauseMillis=10 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+AggressiveOpts -XX:+DoEscapeAnalysis -XX:ParallelGCThreads=32 -XX:ConcGCThreads=32 -XX:G1NewSizePercent=50 -XX:+DisableExplicitGC -XX:-ResizePLAB" [/opt/bookkeeper/conf/bk_server.conf] Applying config bookiePort = 3181 [/opt/bookkeeper/conf/bk_server.conf] Applying config httpServerEnabled = true [/opt/bookkeeper/conf/bk_server.conf] Applying config httpServerPort = [/opt/bookkeeper/conf/bk_server.conf] Applying config indexDirectories = /bookkeeper/data/ledgers [/opt/bookkeeper/conf/bk_server.conf] Applying config journalDirectory = /bookkeeper/data/journal [/opt/bookkeeper/conf/bk_server.conf] Applying config ledgerDirectories = /bookkeeper/data/ledgers [/opt/bookkeeper/conf/bk_server.conf] Applying config metadataServiceUri = zk://zookeeper-1.abcdefg.com;zookeeper-2.abcdefg.com;zookeeper-3.abcdefg.com;zookeeper-4.abcdefg.com;zookeeper-5.abcdefg.com;zookeeper-6.abcdefg.com/ledgers [/opt/bookkeeper/conf/bk_server.conf] Applying config useHostNameAsBookieID = true [/opt/bookkeeper/conf/bk_server.conf] Applying config zkLedgersRootPath = /ledgers [/opt/bookkeeper/conf/bk_server.conf] Applying config zkServers = zookeeper-1.abcdefg.com;zookeeper-2.abcdefg.com;zookeeper-3.abcdefg.com;zookeeper-4.abcdefg.com;zookeeper-5.abcdefg.com;zookeeper-6.abcdefg.com Created bookie dirs : journal = /bookkeeper/data/journal ledger = /bookkeeper/data/ledgers index = /bookkeeper/data/ledgers wait for zookeeper Connecting to zookeeper-1.abcdefg.com;zookeeper-2.abcdefg.com;zookeeper-3.abcdefg.com;zookeeper-4.abcdefg.com;zookeeper-5.abcdefg.com;zookeeper-6.abcdefg.com 2019-05-03 07:47:33,088 - INFO - [main:Environment@100] - Client environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 00:39 GMT 2019-05-03 07:47:33,091 - INFO - [main:Environment@100] - Client environment:host.name=k8s-agentpool1-24779565-0 2019-05-03 07:47:33,091 - INFO - [main:Environment@100] - Client environment:java.version=1.8.0_201 2019-05-03 07:47:33,093 - INFO - [main:Environment@100] - Client environment:java.vendor=Oracle Corporation 2019-05-03 07:47:33,093 - INFO - [main:Environment@100] - Client environment:java.home=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.201.b09-2.el7_6.x86_64/jre 2019-05-03 07:47:33,094 - INFO - [main:Environment@100] - Client environment:java.class.path=/opt/bookkeeper/conf:/opt/bookkeeper/lib/org.apache.bookkeeper-bookkeeper-server-4.9.0.jar::/opt/bookkeeper/lib/com.beust-jcommander-1.48.jar:/opt/bookkeeper/lib/com.fasterxml.jackson.core-jackson-annotations-2.9.7.jar:/opt/bookkeeper/lib/com.fasterxml.jackson.core-jackson-core-2.9.7.jar:/opt/bookkeeper/lib/com.fasterxml.jackson.core-jackson-databind-2.9.7.jar:/opt/bookkeeper/lib/com.google.api.grpc-proto-google-common-protos-1.0.0.jar:/opt/bookkeeper/lib/com.google.auth-google-auth-library-credentials-0.9.0.jar:/opt/bookkeeper/lib/com.google.code.gson-gson-2.7.jar:/opt/bookkeeper/lib/com.google.errorprone-error_prone_annotations-2.1.2.jar:/opt/bookkeeper/lib/com.google.guava-guava-21.0.jar:/opt/bookkeeper/lib/com.google.protobuf-protobuf-java-3.5.1.jar:/opt/bookkeeper/lib/com.google.protobuf-protobuf-java-util-3.5.1.jar:/opt/bookkeeper/lib/com.google.protobuf.nano-protobuf-javanano-3.0.0-alpha-5.jar:/opt/bookkeeper/lib/com.squareup.okhttp-okhttp-2.5.0.jar:/opt/bookkeeper/lib/com.squareup.okio-okio-1.13.0.jar:/opt/bookkeeper/lib/com.yahoo.datasketches-memory-0.8.3.jar:/opt/bookkeeper/lib/com.yahoo.datasketches-sketches-core-0.8.3.jar:/opt/bookkeeper/lib/commons-cli-commons-cli-1.2.jar:/opt/bookkeeper/lib/commons-codec-commons-codec-1.6.jar:/opt/bookkeeper/lib/commons-configuration-commons-configuration-1.10.jar:/opt/bookkeeper/lib/commons-io-commons-io-2.4.jar:/opt/bookkeeper/lib/commons-lang-commons-lang-2.6.jar:/opt/bookkeeper/lib/commons-logging-commons-logging-1.1.1.jar:/opt/bookkeeper/lib/io.grpc-grpc-all-1.12.0.jar:/opt/bookkeeper/lib/io.grpc-grpc-auth-1.12.0.jar:/opt/bookkeeper/lib/io.grpc-grpc-context-1.12.0.jar:/opt/bookkeeper/lib/io.grpc-grpc-core-1.12.0.jar:/opt/bookkeeper/lib/io.grpc-grpc-netty-1.12.0.jar:/opt/bookkeeper/lib/io.grpc-grpc-okhttp-1.12.0.jar:/opt/bookkeeper/lib/io.grpc-grpc-protobuf-1.12.0.jar:/opt/bookkeeper/lib/io.grpc-grpc-protobuf-lite-1.12.0.jar:/opt/bookkeeper/lib/io.grpc-grpc-protobuf-nano-1.12.0.jar:/opt/bookkeeper/lib/io.grpc-grpc-stub-1.12.0.jar:/opt/bookkeeper/lib/io.grpc-grpc-testing-1.12.0.jar:/opt/bookkeeper/lib/io.netty-netty-buffer-4.1.31.Final.jar:/opt/bookkeeper/lib/io.netty-netty-codec-4.1.31.Final.jar:/opt/bookkeeper/lib/io.netty-netty-codec-dns-4.1.31.Final.jar:/opt/bookkeeper/lib/io.netty-netty-codec-http-4.1.31.Final.jar:/opt/bookkeeper/lib/io.netty-netty-codec-http2-4.1.31.Final.jar:/opt/bookkeeper/lib/io.netty-netty-codec-socks-4.1.31.Final.jar:/opt/bookkeeper/lib/io.netty-netty-common-4.1.31.Final.jar:/opt/bookkeeper/lib/io.netty-netty-handler-4.1.31.Final.jar:/opt/bookkeeper/lib/io.netty-netty-handler-proxy-4.1.31.Final.jar:/opt/bookkeeper/lib/io.netty-netty-resolver-4.1.31.Final.jar:/opt/bookkeeper/lib/io.netty-netty-resolver-dns-4.1.31.Final.jar:/opt/bookkeeper/lib/io.netty-netty-tcnative-boringssl-static-2.0.19.Final.jar:/opt/bookkeeper/lib/io.netty-netty-transport-4.1.31.Final.jar:/opt/bookkeeper/lib/io.netty-netty-transport-native-epoll-4.1.31.Final.jar:/opt/bookkeeper/lib/io.netty-netty-transport-native-unix-common-4.1.31.Final.jar:/opt/bookkeeper/lib/io.opencensus-opencensus-api-0.11.0.jar:/opt/bookkeeper/lib/io.opencensus-opencensus-contrib-grpc-metrics-0.11.0.jar:/opt/bookkeeper/lib/io.prometheus-simpleclient-0.0.21.jar:/opt/bookkeeper/lib/io.prometheus-simpleclient_common-0.0.21.jar:/opt/bookkeeper/lib/io.prometheus-simpleclient_hotspot-0.0.21.jar:/opt/bookkeeper/lib/io.prometheus-simpleclient_servlet-0.0.21.jar:/opt/bookkeeper/lib/io.vertx-vertx-auth-common-3.4.1.jar:/opt/bookkeeper/lib/io.vertx-vertx-core-3.4.1.jar:/opt/bookkeeper/lib/io.vertx-vertx-web-3.4.1.jar:/opt/bookkeeper/lib/javax.servlet-javax.servlet-api-3.1.0.jar:/opt/bookkeeper/lib/jline-jline-2.11.jar:/opt/bookkeeper/lib/log4j-log4j-1.2.17.jar:/opt/bookkeeper/lib/net.java.dev.jna-jna-3.2.7.jar:/opt/bookkeeper/lib/net.jpountz.lz4-lz4-1.3.0.jar:/opt/bookkeeper/lib/org.apache.bookkeeper-bookkeeper-common-4.9.0.jar:/opt/bookkeeper/lib/org.apache.bookkeeper-bookkeeper-common-allocator-4.9.0.jar:/opt/bookkeeper/lib/org.apache.bookkeeper-bookkeeper-proto-4.9.0.jar:/opt/bookkeeper/lib/org.apache.bookkeeper-bookkeeper-server-4.9.0.jar:/opt/bookkeeper/lib/org.apache.bookkeeper-bookkeeper-tools-4.9.0.jar:/opt/bookkeeper/lib/org.apache.bookkeeper-bookkeeper-tools-framework-4.9.0.jar:/opt/bookkeeper/lib/org.apache.bookkeeper-bookkeeper-tools-ledger-4.9.0.jar:/opt/bookkeeper/lib/org.apache.bookkeeper-circe-checksum-4.9.0.jar:/opt/bookkeeper/lib/org.apache.bookkeeper-cpu-affinity-4.9.0.jar:/opt/bookkeeper/lib/org.apache.bookkeeper-statelib-4.9.0.jar:/opt/bookkeeper/lib/org.apache.bookkeeper-stream-storage-cli-4.9.0.jar:/opt/bookkeeper/lib/org.apache.bookkeeper-stream-storage-java-client-4.9.0.jar:/opt/bookkeeper/lib/org.apache.bookkeeper-stream-storage-server-4.9.0.jar:/opt/bookkeeper/lib/org.apache.bookkeeper-stream-storage-service-api-4.9.0.jar:/opt/bookkeeper/lib/org.apache.bookkeeper-stream-storage-service-impl-4.9.0.jar:/opt/bookkeeper/lib/org.apache.bookkeeper.http-http-server-4.9.0.jar:/opt/bookkeeper/lib/org.apache.bookkeeper.http-vertx-http-server-4.9.0.jar:/opt/bookkeeper/lib/org.apache.bookkeeper.stats-bookkeeper-stats-api-4.9.0.jar:/opt/bookkeeper/lib/org.apache.bookkeeper.stats-prometheus-metrics-provider-4.9.0.jar:/opt/bookkeeper/lib/org.apache.bookkeeper.tests-stream-storage-tests-common-4.9.0.jar:/opt/bookkeeper/lib/org.apache.commons-commons-collections4-4.1.jar:/opt/bookkeeper/lib/org.apache.commons-commons-lang3-3.6.jar:/opt/bookkeeper/lib/org.apache.curator-curator-client-4.0.1.jar:/opt/bookkeeper/lib/org.apache.curator-curator-framework-4.0.1.jar:/opt/bookkeeper/lib/org.apache.curator-curator-recipes-4.0.1.jar:/opt/bookkeeper/lib/org.apache.distributedlog-distributedlog-common-4.9.0.jar:/opt/bookkeeper/lib/org.apache.distributedlog-distributedlog-core-4.9.0-tests.jar:/opt/bookkeeper/lib/org.apache.distributedlog-distributedlog-core-4.9.0.jar:/opt/bookkeeper/lib/org.apache.distributedlog-distributedlog-protocol-4.9.0.jar:/opt/bookkeeper/lib/org.apache.yetus-audience-annotations-0.5.0.jar:/opt/bookkeeper/lib/org.apache.zookeeper-zookeeper-3.4.13.jar:/opt/bookkeeper/lib/org.codehaus.jackson-jackson-core-asl-1.9.11.jar:/opt/bookkeeper/lib/org.codehaus.jackson-jackson-mapper-asl-1.9.11.jar:/opt/bookkeeper/lib/org.eclipse.jetty-jetty-http-9.4.5.v20170502.jar:/opt/bookkeeper/lib/org.eclipse.jetty-jetty-io-9.4.5.v20170502.jar:/opt/bookkeeper/lib/org.eclipse.jetty-jetty-security-9.4.5.v20170502.jar:/opt/bookkeeper/lib/org.eclipse.jetty-jetty-server-9.4.5.v20170502.jar:/opt/bookkeeper/lib/org.eclipse.jetty-jetty-servlet-9.4.5.v20170502.jar:/opt/bookkeeper/lib/org.eclipse.jetty-jetty-util-9.4.5.v20170502.jar:/opt/bookkeeper/lib/org.inferred-freebuilder-1.14.9.jar:/opt/bookkeeper/lib/org.jctools-jctools-core-2.1.2.jar:/opt/bookkeeper/lib/org.rocksdb-rocksdbjni-5.13.1.jar:/opt/bookkeeper/lib/org.slf4j-slf4j-api-1.7.25.jar:/opt/bookkeeper/lib/org.slf4j-slf4j-log4j12-1.7.25.jar: 2019-05-03 07:47:33,094 - INFO - [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib 2019-05-03 07:47:33,094 - INFO - [main:Environment@100] - Client environment:java.io.tmpdir=/tmp 2019-05-03 07:47:33,094 - INFO - [main:Environment@100] - Client environment:java.compiler=<NA> 2019-05-03 07:47:33,094 - INFO - [main:Environment@100] - Client environment:os.name=Linux 2019-05-03 07:47:33,094 - INFO - [main:Environment@100] - Client environment:os.arch=amd64 2019-05-03 07:47:33,094 - INFO - [main:Environment@100] - Client environment:os.version=4.15.0-1041-azure 2019-05-03 07:47:33,095 - INFO - [main:Environment@100] - Client environment:user.name=root 2019-05-03 07:47:33,095 - INFO - [main:Environment@100] - Client environment:user.home=/root 2019-05-03 07:47:33,095 - INFO - [main:Environment@100] - Client environment:user.dir=/opt/bookkeeper 2019-05-03 07:47:33,096 - INFO - [main:ZooKeeper@442] - Initiating client connection, connectString=zookeeper-1.abcdefg.com;zookeeper-2.abcdefg.com;zookeeper-3.abcdefg.com;zookeeper-4.abcdefg.com;zookeeper-5.abcdefg.com;zookeeper-6.abcdefg.com sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@782830e 2019-05-03 07:47:33,171 - INFO - [main-SendThread(zookeeper-1.abcdefg.com;zookeeper-2.abcdefg.com;zookeeper-3.abcdefg.com;zookeeper-4.abcdefg.com;zookeeper-5.abcdefg.com;zookeeper-6.abcdefg.com:2181):ClientCnxn$SendThread@1029] - Opening socket connection to server zookeeper-1.abcdefg.com;zookeeper-2.abcdefg.com;zookeeper-3.abcdefg.com;zookeeper-4.abcdefg.com;zookeeper-5.abcdefg.com;zookeeper-6.abcdefg.com:2181. Will not attempt to authenticate using SASL (unknown error) 2019-05-03 07:47:33,234 - WARN - [main-SendThread(zookeeper-1.abcdefg.com;zookeeper-2.abcdefg.com;zookeeper-3.abcdefg.com;zookeeper-4.abcdefg.com;zookeeper-5.abcdefg.com;zookeeper-6.abcdefg.com:2181):ClientCnxn$SendThread@1168] - Session 0x0 for server zookeeper-1.abcdefg.com;zookeeper-2.abcdefg.com;zookeeper-3.abcdefg.com;zookeeper-4.abcdefg.com;zookeeper-5.abcdefg.com;zookeeper-6.abcdefg.com:2181, unexpected error, closing socket connection and attempting reconnect java.nio.channels.UnresolvedAddressException at sun.nio.ch.Net.checkAddress(Net.java:101) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622) at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:277) at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:287) at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1021) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1064) Exception in thread "main" org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for / at org.apache.zookeeper.KeeperException.create(KeeperException.java:102) at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1541) at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1569) at org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:732) at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:600) at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:363) at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:291) Connecting to zookeeper-1.abcdefg.com;zookeeper-2.abcdefg.com;zookeeper-3.abcdefg.com;zookeeper-4.abcdefg.com;zookeeper-5.abcdefg.com;zookeeper-6.abcdefg.com ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
