vishalananth07 opened a new issue #5630: Error when trying to run Producer-Consumer commands in a 2 node cluster URL: https://github.com/apache/pulsar/issues/5630 **Describe the bug** Producer command ``` $ bin/pulsar-client consume persistent://public/default/test -n 100 -s "consumer-test" -t "Exclusive" ``` and Consumer command ``` bin/pulsar-client consume persistent://public/default/test -n 100 -s "consumer-test" -t "Exclusive" ``` are giving the following error messages: Producer error message: ``` bin/pulsar-client produce persistent://public/default/test -n 1 -m "hello pulsar" 14:28:32.171 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConnectionPool - [[id: 0x2c91e691, L:/192.168.1.13:35714 - R:192.168.1.5/192.168.1.5:6650]] Connected to server 14:28:32.707 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ProducerStatsRecorderImpl - Starting Pulsar producer perf with config: { "topicName" : "persistent://public/default/test", "producerName" : null, "sendTimeoutMs" : 30000, "blockIfQueueFull" : false, "maxPendingMessages" : 1000, "maxPendingMessagesAcrossPartitions" : 50000, "messageRoutingMode" : "RoundRobinPartition", "hashingScheme" : "JavaStringHash", "cryptoFailureAction" : "FAIL", "batchingMaxPublishDelayMicros" : 1000, "batchingMaxMessages" : 1000, "batchingEnabled" : true, "compressionType" : "NONE", "initialSequenceId" : null, "autoUpdatePartitions" : true, "properties" : { } } 14:28:32.710 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ProducerStatsRecorderImpl - Pulsar client config: { "serviceUrl" : "pulsar://192.168.1.13:6650,192.168.1.5:6650", "authPluginClassName" : null, "authParams" : null, "operationTimeoutMs" : 30000, "statsIntervalSeconds" : 60, "numIoThreads" : 1, "numListenerThreads" : 1, "connectionsPerBroker" : 1, "useTcpNoDelay" : true, "useTls" : false, "tlsTrustCertsFilePath" : "", "tlsAllowInsecureConnection" : false, "tlsHostnameVerificationEnable" : false, "concurrentLookupRequest" : 5000, "maxLookupRequest" : 50000, "maxNumberOfRejectedRequestPerConnection" : 50, "keepAliveIntervalSeconds" : 30, "connectionTimeoutMs" : 10000, "requestTimeoutMs" : 60000, "defaultBackoffIntervalNanos" : 100000000, "maxBackoffIntervalNanos" : 30000000000 } 14:28:32.714 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConnectionPool - [[id: 0x6e57db24, L:/192.168.1.13:43896 - R:192.168.1.13/192.168.1.13:6650]] Connected to server 14:28:32.734 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConnectionPool - [[id: 0x009288aa, L:/127.0.0.1:40736 - R:sundar-UX303UB/127.0.1.1:6650]] Connected to server 14:28:32.741 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ProducerImpl - [persistent://public/default/test] [null] Creating producer on cnx [id: 0x009288aa, L:/127.0.0.1:40736 - R:sundar-UX303UB/127.0.1.1:6650] 14:28:32.864 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ProducerImpl - [persistent://public/default/test] [test-cluster-1-0-3] Created producer on cnx [id: 0x009288aa, L:/127.0.0.1:40736 - R:sundar-UX303UB/127.0.1.1:6650] 14:28:32.951 [main] WARN com.scurrilous.circe.checksum.Crc32cIntChecksum - Failed to load Circe JNI library. Falling back to Java based CRC32c provider 14:29:02.954 [pulsar-timer-4-1] INFO org.apache.pulsar.client.impl.ProducerImpl - [persistent://public/default/test] [test-cluster-1-0-3] Message send timed out. Failing 1 messages 14:29:02.964 [main] ERROR org.apache.pulsar.client.cli.PulsarClientTool - Error while producing messages 14:29:02.965 [main] ERROR org.apache.pulsar.client.cli.PulsarClientTool - Could not send message to broker within given timeout org.apache.pulsar.client.api.PulsarClientException$TimeoutException: Could not send message to broker within given timeout at org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:247) ~[org.apache.pulsar-pulsar-client-api-2.4.1.jar:2.4.1] at org.apache.pulsar.client.impl.ProducerBase.send(ProducerBase.java:86) ~[org.apache.pulsar-pulsar-client-original-2.4.1.jar:2.4.1] at org.apache.pulsar.client.impl.TypedMessageBuilderImpl.send(TypedMessageBuilderImpl.java:61) ~[org.apache.pulsar-pulsar-client-original-2.4.1.jar:2.4.1] at org.apache.pulsar.client.impl.ProducerBase.send(ProducerBase.java:51) ~[org.apache.pulsar-pulsar-client-original-2.4.1.jar:2.4.1] at org.apache.pulsar.client.cli.CmdProduce.publish(CmdProduce.java:189) [org.apache.pulsar-pulsar-client-tools-2.4.1.jar:2.4.1] at org.apache.pulsar.client.cli.CmdProduce.run(CmdProduce.java:169) [org.apache.pulsar-pulsar-client-tools-2.4.1.jar:2.4.1] at org.apache.pulsar.client.cli.PulsarClientTool.run(PulsarClientTool.java:131) [org.apache.pulsar-pulsar-client-tools-2.4.1.jar:2.4.1] at org.apache.pulsar.client.cli.PulsarClientTool.main(PulsarClientTool.java:165) [org.apache.pulsar-pulsar-client-tools-2.4.1.jar:2.4.1] 14:29:02.978 [main] INFO org.apache.pulsar.client.cli.PulsarClientTool - 0 messages successfully produced ``` Consumer error message: ``` bin/pulsar-client consume persistent://public/default/test -n 100 -s "consumer-test" -t "Exclusive" 14:29:02.442 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConnectionPool - [[id: 0xc40711ef, L:/192.168.1.5:46150 - R:192.168.1.13/192.168.1.13:6650]] Connected to server 14:29:02.653 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - Starting Pulsar consumer perf with config: { "topicNames" : [ "persistent://public/default/test" ], "topicsPattern" : null, "subscriptionName" : "consumer-test", "subscriptionType" : "Exclusive", "receiverQueueSize" : 1000, "acknowledgementsGroupTimeMicros" : 100000, "negativeAckRedeliveryDelayMicros" : 60000000, "maxTotalReceiverQueueSizeAcrossPartitions" : 50000, "consumerName" : null, "ackTimeoutMillis" : 0, "tickDurationMillis" : 1000, "priorityLevel" : 0, "cryptoFailureAction" : "FAIL", "properties" : { }, "readCompacted" : false, "subscriptionInitialPosition" : "Latest", "patternAutoDiscoveryPeriod" : 1, "regexSubscriptionMode" : "PersistentOnly", "deadLetterPolicy" : null, "autoUpdatePartitions" : true, "replicateSubscriptionState" : false, "resetIncludeHead" : false } 14:29:02.660 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConsumerStatsRecorderImpl - Pulsar client config: { "serviceUrl" : "pulsar://192.168.1.13:6650,192.168.1.5:6650", "authPluginClassName" : null, "authParams" : null, "operationTimeoutMs" : 30000, "statsIntervalSeconds" : 60, "numIoThreads" : 1, "numListenerThreads" : 1, "connectionsPerBroker" : 1, "useTcpNoDelay" : true, "useTls" : false, "tlsTrustCertsFilePath" : "", "tlsAllowInsecureConnection" : false, "tlsHostnameVerificationEnable" : false, "concurrentLookupRequest" : 5000, "maxLookupRequest" : 50000, "maxNumberOfRejectedRequestPerConnection" : 50, "keepAliveIntervalSeconds" : 30, "connectionTimeoutMs" : 10000, "requestTimeoutMs" : 60000, "defaultBackoffIntervalNanos" : 100000000, "maxBackoffIntervalNanos" : 30000000000 } 14:29:02.679 [pulsar-client-io-1-1] INFO org.apache.pulsar.client.impl.ConnectionPool - [[id: 0xa7cf0295, L:/192.168.1.5:42154 - R:192.168.1.5/192.168.1.5:6650]] Connected to server 14:29:02.911 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ConnectionPool - Failed to open connection to sundar-UX303UB:6650 : java.net.UnknownHostException: failed to resolve 'sundar-UX303UB' after 2 queries 14:29:02.913 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ConnectionHandler - [persistent://public/default/test] [consumer-test] Error connecting to broker: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: java.net.UnknownHostException: failed to resolve 'sundar-UX303UB' after 2 queries 14:29:02.913 [pulsar-client-io-1-1] WARN org.apache.pulsar.client.impl.ConnectionHandler - [persistent://public/default/test] [consumer-test] Could not get connection to broker: org.apache.pulsar.client.api.PulsarClientException: java.util.concurrent.CompletionException: java.net.UnknownHostException: failed to resolve 'sundar-UX303UB' after 2 queries -- Will try again in 0.1 s ``` **To Reproduce** Steps to reproduce the behavior: 1. Start zookeeper, set cluster meta data, start bookie, start broker after modifying necessary configurations. 2. Bookiesanity is succeeding but simpletest fails with the following error, we are using 2 systems which run zookeeper,bookkeeper and broker ``` bin/bookkeeper shell simpletest --ensemble 2 --writeQuorum 2 --ackQuorum 2 --numEntries 2 JMX enabled by default 14:43:53.364 [main] INFO org.apache.bookkeeper.meta.MetadataDrivers - BookKeeper metadata driver manager initialized 14:43:53.366 [main] INFO org.apache.bookkeeper.meta.MetadataDrivers - BookKeeper metadata driver manager initialized 14:43:53.366 [main] INFO org.apache.bookkeeper.meta.MetadataDrivers - BookKeeper metadata driver manager initialized 14:43:53.373 [main] INFO org.apache.bookkeeper.meta.zk.ZKMetadataDriverBase - Initialize zookeeper metadata driver at metadata service uri zk+null://192.168.1.13:2181;192.168.1.5:2181/ledgers : zkServers = 192.168.1.13:2181,192.168.1.5:2181, ledgersRootPath = /ledgers. 14:43:53.381 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 00:39 GMT 14:43:53.381 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:host.name=sundar-UX303UB 14:43:53.382 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.version=1.8.0_222 14:43:53.382 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.vendor=Private Build 14:43:53.382 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.home=/usr/lib/jvm/java-8-openjdk-amd64/jre 14:43:53.382 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.class.path=/home/sundar/apache-pulsar-2.4.1/conf:::/home/sundar/apache-pulsar-2.4.1/lib/com.beust-jcommander-1.48.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.carrotsearch-hppc-0.7.3.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.fasterxml.jackson.core-jackson-annotations-2.9.9.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.fasterxml.jackson.core-jackson-core-2.9.9.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.fasterxml.jackson.core-jackson-databind-2.9.9.3.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.fasterxml.jackson.dataformat-jackson-dataformat-yaml-2.9.9.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.fasterxml.jackson.jaxrs-jackson-jaxrs-base-2.9.9.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.fasterxml.jackson.jaxrs-jackson-jaxrs-json-provider-2.9.9.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.fasterxml.jackson.module-jackson-module-jaxb-annotations-2.9.9.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.fasterxml.jackson.module-jackson-module-jsonSchema-2.9.9.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.github.ben-manes.caffeine-caffeine-2.6.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.github.luben-zstd-jni-1.3.7-3.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.github.zafarkhaja-java-semver-0.9.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.google.api.grpc-proto-google-common-protos-1.12.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.google.auth-google-auth-library-credentials-0.9.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.google.code.findbugs-jsr305-3.0.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.google.code.gson-gson-2.8.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.google.errorprone-error_prone_annotations-2.2.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.google.guava-guava-21.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.google.protobuf.nano-protobuf-javanano-3.0.0-alpha-5.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.google.protobuf-protobuf-java-3.5.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.google.protobuf-protobuf-java-util-3.5.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/commons-cli-commons-cli-1.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/commons-codec-commons-codec-1.10.jar:/home/sundar/apache-pulsar-2.4.1/lib/commons-collections-commons-collections-3.2.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/commons-configuration-commons-configuration-1.10.jar:/home/sundar/apache-pulsar-2.4.1/lib/commons-io-commons-io-2.5.jar:/home/sundar/apache-pulsar-2.4.1/lib/commons-lang-commons-lang-2.6.jar:/home/sundar/apache-pulsar-2.4.1/lib/commons-logging-commons-logging-1.1.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.squareup.okhttp-logging-interceptor-2.7.5.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.squareup.okhttp-okhttp-2.5.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.squareup.okhttp-okhttp-ws-2.7.5.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.squareup.okio-okio-1.13.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.sun.activation-javax.activation-1.2.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.thoughtworks.paranamer-paranamer-2.7.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.typesafe.netty-netty-reactive-streams-2.0.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.yahoo.datasketches-memory-0.8.3.jar:/home/sundar/apache-pulsar-2.4.1/lib/com.yahoo.datasketches-sketches-core-0.8.3.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.dropwizard.metrics-metrics-core-3.1.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.dropwizard.metrics-metrics-graphite-3.1.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.dropwizard.metrics-metrics-jvm-3.1.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.grpc-grpc-all-1.18.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.grpc-grpc-auth-1.18.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.grpc-grpc-context-1.18.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.grpc-grpc-core-1.18.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.grpc-grpc-netty-1.18.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.grpc-grpc-okhttp-1.18.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.grpc-grpc-protobuf-1.18.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.grpc-grpc-protobuf-lite-1.18.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.grpc-grpc-protobuf-nano-1.18.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.grpc-grpc-stub-1.18.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.grpc-grpc-testing-1.12.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.jsonwebtoken-jjwt-api-0.10.5.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.jsonwebtoken-jjwt-impl-0.10.5.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.jsonwebtoken-jjwt-jackson-0.10.5.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.kubernetes-client-java-2.0.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.kubernetes-client-java-api-2.0.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.kubernetes-client-java-proto-2.0.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.netty-netty-3.10.1.Final.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.netty-netty-all-4.1.32.Final.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.netty-netty-tcnative-boringssl-static-2.0.20.Final.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.opencensus-opencensus-api-0.18.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.opencensus-opencensus-contrib-grpc-metrics-0.18.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.prometheus-simpleclient-0.5.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.prometheus-simpleclient_common-0.5.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.prometheus-simpleclient_hotspot-0.5.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.prometheus-simpleclient_httpserver-0.5.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.prometheus-simpleclient_jetty-0.5.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.prometheus-simpleclient_log4j2-0.5.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.prometheus-simpleclient_servlet-0.5.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.swagger-swagger-annotations-1.5.21.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.swagger-swagger-core-1.5.21.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.swagger-swagger-models-1.5.21.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.vertx-vertx-auth-common-3.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.vertx-vertx-core-3.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/io.vertx-vertx-web-3.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/javax.annotation-javax.annotation-api-1.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/javax.inject-javax.inject-1.jar:/home/sundar/apache-pulsar-2.4.1/lib/javax.servlet-javax.servlet-api-3.1.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/javax.validation-validation-api-1.1.0.Final.jar:/home/sundar/apache-pulsar-2.4.1/lib/javax.websocket-javax.websocket-client-api-1.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/javax.ws.rs-javax.ws.rs-api-2.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/javax.xml.bind-jaxb-api-2.3.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/jline-jline-0.9.94.jar:/home/sundar/apache-pulsar-2.4.1/lib/joda-time-joda-time-2.10.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/log4j-log4j-1.2.17.jar:/home/sundar/apache-pulsar-2.4.1/lib/net.java.dev.jna-jna-4.2.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/net.jodah-typetools-0.5.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.avro-avro-1.8.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.avro-avro-protobuf-1.8.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-bookkeeper-common-allocator-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-bookkeeper-proto-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-bookkeeper-tools-framework-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-circe-checksum-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-cpu-affinity-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper.http-http-server-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper.http-vertx-http-server-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-statelib-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper.stats-bookkeeper-stats-api-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper.stats-codahale-metrics-provider-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper.stats-prometheus-metrics-provider-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-stream-storage-api-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-stream-storage-common-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-stream-storage-java-client-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-stream-storage-java-client-base-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-stream-storage-proto-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-stream-storage-server-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-stream-storage-service-api-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper-stream-storage-service-impl-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.bookkeeper.tests-stream-storage-tests-common-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.commons-commons-collections4-4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.commons-commons-compress-1.15.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.commons-commons-lang3-3.4.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.curator-curator-client-4.0.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.curator-curator-framework-4.0.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.curator-curator-recipes-4.0.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.distributedlog-distributedlog-common-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.distributedlog-distributedlog-core-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.distributedlog-distributedlog-core-4.9.2-tests.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.distributedlog-distributedlog-protocol-4.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.httpcomponents-httpclient-4.5.5.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.httpcomponents-httpcore-4.4.9.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.logging.log4j-log4j-api-2.10.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.logging.log4j-log4j-core-2.10.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.logging.log4j-log4j-slf4j-impl-2.10.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.logging.log4j-log4j-web-2.10.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-managed-ledger-original-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-protobuf-shaded-2.1.0-incubating.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-broker-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-broker-auth-sasl-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-broker-common-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-client-admin-original-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-client-api-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-client-auth-sasl-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-client-original-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-client-tools-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-common-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-discovery-service-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-functions-api-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-functions-api-examples-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-functions-instance-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-functions-local-runner-original-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-functions-proto-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-functions-runtime-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-functions-secrets-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-functions-utils-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-functions-worker-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-io-core-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-proxy-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-testclient-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-transaction-common-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-websocket-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-zookeeper-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.pulsar-pulsar-zookeeper-utils-2.4.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.apache.yetus-audience-annotations-0.5.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.aspectj-aspectjrt-1.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.aspectj-aspectjweaver-1.9.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.asynchttpclient-async-http-client-2.7.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.asynchttpclient-async-http-client-netty-utils-2.7.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.bouncycastle-bcpkix-jdk15on-1.60.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.bouncycastle-bcprov-ext-jdk15on-1.60.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.bouncycastle-bcprov-jdk15on-1.60.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.checkerframework-checker-compat-qual-2.5.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.codehaus.jackson-jackson-core-asl-1.9.13.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.codehaus.jackson-jackson-mapper-asl-1.9.13.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.codehaus.mojo-animal-sniffer-annotations-1.17.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.eclipse.jetty-jetty-client-9.4.12.v20180830.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.eclipse.jetty-jetty-continuation-9.4.12.v20180830.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.eclipse.jetty-jetty-http-9.4.12.v20180830.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.eclipse.jetty-jetty-io-9.4.12.v20180830.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.eclipse.jetty-jetty-proxy-9.4.12.v20180830.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.eclipse.jetty-jetty-security-9.4.12.v20180830.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.eclipse.jetty-jetty-server-9.4.12.v20180830.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.eclipse.jetty-jetty-servlet-9.4.12.v20180830.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.eclipse.jetty-jetty-servlets-9.4.12.v20180830.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.eclipse.jetty-jetty-util-9.4.12.v20180830.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.eclipse.jetty-jetty-xml-9.4.12.v20180830.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.eclipse.jetty.websocket-javax-websocket-client-impl-9.4.12.v20180830.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.eclipse.jetty.websocket-websocket-api-9.4.12.v20180830.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.eclipse.jetty.websocket-websocket-client-9.4.12.v20180830.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.eclipse.jetty.websocket-websocket-common-9.4.12.v20180830.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.eclipse.jetty.websocket-websocket-server-9.4.12.v20180830.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.eclipse.jetty.websocket-websocket-servlet-9.4.12.v20180830.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.glassfish.hk2.external-aopalliance-repackaged-2.5.0-b42.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.glassfish.hk2.external-javax.inject-2.5.0-b42.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.glassfish.hk2-hk2-api-2.5.0-b42.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.glassfish.hk2-hk2-locator-2.5.0-b42.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.glassfish.hk2-hk2-utils-2.5.0-b42.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.glassfish.hk2-osgi-resource-locator-1.0.1.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.glassfish.jersey.containers-jersey-container-servlet-2.27.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.glassfish.jersey.containers-jersey-container-servlet-core-2.27.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.glassfish.jersey.core-jersey-client-2.27.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.glassfish.jersey.core-jersey-common-2.27.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.glassfish.jersey.core-jersey-server-2.27.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.glassfish.jersey.ext-jersey-entity-filtering-2.27.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.glassfish.jersey.inject-jersey-hk2-2.27.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.glassfish.jersey.media-jersey-media-jaxb-2.27.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.glassfish.jersey.media-jersey-media-json-jackson-2.27.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.glassfish.jersey.media-jersey-media-multipart-2.27.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.hdrhistogram-HdrHistogram-2.1.9.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.inferred-freebuilder-1.14.9.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.javassist-javassist-3.25.0-GA.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.jctools-jctools-core-2.1.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.jvnet.mimepull-mimepull-1.9.6.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.lz4-lz4-java-1.5.0.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.objenesis-objenesis-2.6.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.reactivestreams-reactive-streams-1.0.2.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.rocksdb-rocksdbjni-5.13.3.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.slf4j-jcl-over-slf4j-1.7.25.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.slf4j-jul-to-slf4j-1.7.25.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.slf4j-slf4j-api-1.7.25.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.tukaani-xz-1.5.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.xerial.snappy-snappy-java-1.1.1.3.jar:/home/sundar/apache-pulsar-2.4.1/lib/org.yaml-snakeyaml-1.23.jar: 14:43:53.384 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=:/home/sundar/ns-allinone-2.35/otcl-1.14/:/home/sundar/ns-allinone-2.35/lib/:/usr/local/lib/:/usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib 14:43:53.384 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp 14:43:53.384 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA> 14:43:53.384 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux 14:43:53.384 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64 14:43:53.384 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.version=4.15.0-66-generic 14:43:53.384 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.name=sundar 14:43:53.384 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.home=/home/sundar 14:43:53.384 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/home/sundar/apache-pulsar-2.4.1 14:43:53.389 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=192.168.1.13:2181,192.168.1.5:2181 sessionTimeout=30000 watcher=org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase@50468873 14:43:53.510 [main-SendThread(192.168.1.5:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server 192.168.1.5/192.168.1.5:2181. Will not attempt to authenticate using SASL (unknown error) 14:43:53.517 [main-SendThread(192.168.1.5:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established to 192.168.1.5/192.168.1.5:2181, initiating session 14:43:53.536 [main-SendThread(192.168.1.5:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server 192.168.1.5/192.168.1.5:2181, sessionid = 0x2000535ffc2000e, negotiated timeout = 30000 14:43:53.541 [main-EventThread] INFO org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is connected now. 14:43:54.506 [main] INFO org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to initialize DNS Resolver org.apache.bookkeeper.net.ScriptBasedMapping, used default subnet resolver : java.lang.RuntimeException: No network topology script is found when using script based DNS resolver. 14:43:54.539 [main] INFO org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Initialize rackaware ensemble placement policy @ <Bookie:127.0.1.1:0> @ /default-rack : org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl$DefaultResolver. 14:43:54.540 [main] INFO org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Not weighted 14:43:54.548 [main] INFO org.apache.bookkeeper.client.BookKeeper - Weighted ledger placement is not enabled 14:43:54.592 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /default-rack/sundar-UX303UB:3181 14:43:54.593 [BookKeeperClientScheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.net.NetworkTopologyImpl - Adding a new node: /default-rack/vishal-Satellite-C55-B:3181 14:43:54.641 [main] WARN org.apache.bookkeeper.client.BookieWatcherImpl - New ensemble: [vishal-Satellite-C55-B:3181, sundar-UX303UB:3181] is not adhering to Placement Policy. quarantinedBookies: [] 14:43:54.808 [main-EventThread] ERROR org.apache.bookkeeper.proto.checksum.CRC32CDigestManager - Sse42Crc32C is not supported, will use a slower CRC32C implementation. 14:43:54.818 [main-EventThread] INFO org.apache.bookkeeper.client.LedgerCreateOp - Ensemble: [vishal-Satellite-C55-B:3181, sundar-UX303UB:3181] for ledger: 1371 Ledger ID: 1371 14:43:54.888 [BookKeeperClientWorker-OrderedExecutor-3-0] WARN com.scurrilous.circe.checksum.Crc32cIntChecksum - Failed to load Circe JNI library. Falling back to Java based CRC32c provider 14:43:55.006 [bookkeeper-io-3-1] WARN org.apache.bookkeeper.proto.PerChannelBookieClient - Could not connect to bookie: [id: 0x6515e025]/vishal-Satellite-C55-B:3181, current state CONNECTING : vishal-Satellite-C55-B 14:43:55.008 [bookkeeper-io-3-2] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: [id: 0xdb053f5e, L:/127.0.0.1:41224 - R:sundar-UX303UB/127.0.1.1:3181] 14:43:55.019 [bookkeeper-io-3-2] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Successfully connected to bookie: sundar-UX303UB:3181 14:43:55.020 [BookKeeperClientWorker-OrderedExecutor-3-0] WARN org.apache.bookkeeper.client.PendingAddOp - Failed to write entry (1371, 0): Bookie handle is not available 14:43:55.037 [BookKeeperClientWorker-OrderedExecutor-3-0] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [<Bookie:sundar-UX303UB:3181>, <Bookie:vishal-Satellite-C55-B:3181>], allBookies [<Bookie:sundar-UX303UB:3181>, <Bookie:vishal-Satellite-C55-B:3181>]. 14:43:55.039 [BookKeeperClientWorker-OrderedExecutor-3-0] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to choose a bookie: excluded [<Bookie:sundar-UX303UB:3181>, <Bookie:vishal-Satellite-C55-B:3181>], fallback to choose bookie randomly from the cluster. 14:43:55.039 [BookKeeperClientWorker-OrderedExecutor-3-0] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [<Bookie:sundar-UX303UB:3181>, <Bookie:vishal-Satellite-C55-B:3181>], allBookies [<Bookie:vishal-Satellite-C55-B:3181>, <Bookie:sundar-UX303UB:3181>]. 14:43:55.040 [BookKeeperClientWorker-OrderedExecutor-3-0] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [<Bookie:sundar-UX303UB:3181>, <Bookie:vishal-Satellite-C55-B:3181>], allBookies [<Bookie:sundar-UX303UB:3181>, <Bookie:vishal-Satellite-C55-B:3181>]. 14:43:55.043 [BookKeeperClientWorker-OrderedExecutor-3-0] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to choose a bookie: excluded [<Bookie:sundar-UX303UB:3181>, <Bookie:vishal-Satellite-C55-B:3181>], fallback to choose bookie randomly from the cluster. 14:43:55.043 [BookKeeperClientWorker-OrderedExecutor-3-0] WARN org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl - Failed to find 1 bookies : excludeBookies [<Bookie:sundar-UX303UB:3181>, <Bookie:vishal-Satellite-C55-B:3181>], allBookies [<Bookie:vishal-Satellite-C55-B:3181>, <Bookie:sundar-UX303UB:3181>]. 14:43:55.043 [BookKeeperClientWorker-OrderedExecutor-3-0] ERROR org.apache.bookkeeper.client.MetadataUpdateLoop - UpdateLoop(ledgerId=1371,loopId=2d2b0a86) Exception updating org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandomInternal(RackawareEnsemblePlacementPolicyImpl.java:989) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandom(RackawareEnsemblePlacementPolicyImpl.java:907) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:797) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:200) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:757) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:221) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.replaceBookie(RackawareEnsemblePlacementPolicyImpl.java:659) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.replaceBookie(RackawareEnsemblePlacementPolicy.java:114) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.client.BookieWatcherImpl.replaceBookie(BookieWatcherImpl.java:295) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.client.EnsembleUtils.replaceBookiesInEnsemble(EnsembleUtils.java:71) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.client.LedgerHandle.lambda$ensembleChangeLoop$2(LedgerHandle.java:1908) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.client.MetadataUpdateLoop.writeLoop(MetadataUpdateLoop.java:122) [org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.client.MetadataUpdateLoop.run(MetadataUpdateLoop.java:111) [org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.client.LedgerHandle.ensembleChangeLoop(LedgerHandle.java:1927) [org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.client.LedgerHandle.handleBookieFailure(LedgerHandle.java:1876) [org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:360) [org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.proto.BookieClientImpl$1.safeRun(BookieClientImpl.java:280) [org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_222] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222] 14:43:55.052 [bookkeeper-io-3-2] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - connection [id: 0xdb053f5e, L:/127.0.0.1:41224 - R:sundar-UX303UB/127.0.1.1:3181] authenticated as BookKeeperPrincipal{ANONYMOUS} 14:43:55.056 [BookKeeperClientWorker-OrderedExecutor-3-0] WARN org.apache.bookkeeper.client.LedgerHandle - [EnsembleChange(ledger:1371, change-id:0000000001)][attempt:1] Exception changing ensemble org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandomInternal(RackawareEnsemblePlacementPolicyImpl.java:989) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectRandom(RackawareEnsemblePlacementPolicyImpl.java:907) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:797) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:200) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.selectFromNetworkLocation(RackawareEnsemblePlacementPolicyImpl.java:757) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.selectFromNetworkLocation(RackawareEnsemblePlacementPolicy.java:221) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicyImpl.replaceBookie(RackawareEnsemblePlacementPolicyImpl.java:659) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.client.RackawareEnsemblePlacementPolicy.replaceBookie(RackawareEnsemblePlacementPolicy.java:114) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.client.BookieWatcherImpl.replaceBookie(BookieWatcherImpl.java:295) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.client.EnsembleUtils.replaceBookiesInEnsemble(EnsembleUtils.java:71) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.client.LedgerHandle.lambda$ensembleChangeLoop$2(LedgerHandle.java:1908) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.client.MetadataUpdateLoop.writeLoop(MetadataUpdateLoop.java:122) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.client.MetadataUpdateLoop.run(MetadataUpdateLoop.java:111) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.client.LedgerHandle.ensembleChangeLoop(LedgerHandle.java:1927) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.client.LedgerHandle.handleBookieFailure(LedgerHandle.java:1876) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.client.PendingAddOp.writeComplete(PendingAddOp.java:360) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.proto.BookieClientImpl$1.safeRun(BookieClientImpl.java:280) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) ~[org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_222] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222] 14:43:55.058 [BookKeeperClientWorker-OrderedExecutor-3-0] ERROR org.apache.bookkeeper.client.LedgerHandle - Closing ledger 1371 due to NotEnoughBookiesException: Not enough non-faulty bookies available 14:43:55.061 [BookKeeperClientWorker-OrderedExecutor-3-0] ERROR org.apache.bookkeeper.client.PendingAddOp - Write of ledger entry to quorum failed: L1371 E0 14:43:55.186 [main] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel bookie client for sundar-UX303UB:3181 14:43:55.190 [main] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel bookie client for vishal-Satellite-C55-B:3181 14:43:55.190 [bookkeeper-io-3-2] INFO org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie channel [id: 0xdb053f5e, L:/127.0.0.1:41224 ! R:sundar-UX303UB/127.0.1.1:3181] 14:43:55.210 [main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x2000535ffc2000e closed 14:43:55.210 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x2000535ffc2000e 14:43:55.211 [main] ERROR org.apache.bookkeeper.tools.cli.helpers.ClientCommand - Failed to process command 'simpletest' org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not enough non-faulty bookies available at org.apache.bookkeeper.client.SyncCallbackUtils.finish(SyncCallbackUtils.java:83) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.client.SyncCallbackUtils$SyncAddCallback.addComplete(SyncCallbackUtils.java:251) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.client.AsyncCallback$AddCallback.addCompleteWithLatency(AsyncCallback.java:91) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.client.PendingAddOp.submitCallback(PendingAddOp.java:390) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.client.LedgerHandle.errorOutPendingAdds(LedgerHandle.java:1772) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.client.LedgerHandle$5.safeRun(LedgerHandle.java:572) ~[org.apache.bookkeeper-bookkeeper-server-4.9.2.jar:4.9.2] at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) ~[org.apache.bookkeeper-bookkeeper-common-4.9.2.jar:4.9.2] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_222] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_222] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-all-4.1.32.Final.jar:4.1.32.Final] at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_222] ``` 3. Try running the producer and consumer commands on different broker systems after modifying the service urls in client.conf(we are using the following url webServiceUrl=http://192.168.1.13:8080,192.168.1.5:8080, brokerServiceUrl=pulsar://192.168.1.13:6650,192.168.1.5:6650) **Expected behavior** Producer should produce message and Consumer should consume it. **Desktop (please complete the following information):** - OS: Ubuntu 16.04, Ubuntu 18.04 **Additional context** All of Zookeeper, Bookkeeper and Broker are starting, we are able to see all the processes running when we give jps command. But when we try to run producer or consumer command consumer broker crashes and error message is displayed.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
