jiangmincong opened a new issue #8077:
URL: https://github.com/apache/pulsar/issues/8077
**Describe the bug**
A clear and concise description of what the bug is.
21:28:02.513 [BookieJournal-3181] INFO
org.apache.bookkeeper.bookie.JournalChannel - Opening journal
/journal/bk-journal/current/17260dc8a5e.txn
21:28:35.954 [GarbageCollectorThread-35-1] INFO
org.apache.bookkeeper.bookie.EntryLogCompactor - Removing entry log 6058 after
compaction
21:28:36.128 [GarbageCollectorThread-35-1] INFO
org.apache.bookkeeper.bookie.GarbageCollectorThread - Removing entry log
metadata for 6058
21:28:36.128 [GarbageCollectorThread-35-1] INFO
org.apache.bookkeeper.bookie.GarbageCollectorThread - Compaction: entry log
usage buckets[10% 20% 30% 40% 50% 60% 70% 80% 90% 100%] = [0, 7, 2, 1, 1, 0, 0,
2, 2, 327]
21:29:20.531 [main-SendThread(jxnc-bd-olmc-009:2181)] INFO
org.apache.zookeeper.ClientCnxn - Unable to read additional data from server
sessionid 0x902a3f7df0500d2, likely server has closed socket, closing socket
connection and attempting reconnect
21:29:20.636 [main-EventThread] INFO
org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is
disconnected from zookeeper now, but it is OK unless we received EXPIRED event.
21:29:21.547 [main-SendThread(jxnc-bd-olmc-006:2181)] INFO
org.apache.zookeeper.ClientCnxn - Opening socket connection to server
jxnc-bd-olmc-006/10.242.63.176:2181. Will not attempt to authenticate using
SASL (unknown error)
21:29:21.548 [main-SendThread(jxnc-bd-olmc-006:2181)] INFO
org.apache.zookeeper.ClientCnxn - Socket connection established, initiating
session, client: /10.242.63.177:36774, server:
jxnc-bd-olmc-006/10.242.63.176:2181
21:29:21.608 [main-SendThread(jxnc-bd-olmc-006:2181)] WARN
org.apache.zookeeper.ClientCnxn - Unable to reconnect to ZooKeeper service,
session 0x902a3f7df0500d2 has expired
21:29:21.608 [main-EventThread] ERROR
org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client
connection to the ZooKeeper server has expired!
21:29:21.608 [main-SendThread(jxnc-bd-olmc-006:2181)] INFO
org.apache.zookeeper.ClientCnxn - Unable to reconnect to ZooKeeper service,
session 0x902a3f7df0500d2 has expired, closing socket connection
21:29:21.608 [main-EventThread] INFO
org.apache.bookkeeper.zookeeper.ZooKeeperClient - ZooKeeper session
902a3f7df0500d2 is expired from
jxnc-bd-olmc-006:2181,jxnc-bd-olmc-007:2181,jxnc-bd-olmc-008:2181,jxnc-bd-olmc-009:2181,jxnc-bd-olmc-010:2181.
21:29:21.615 [main-EventThread] ERROR
org.apache.bookkeeper.replication.AutoRecoveryMain - Client connection to the
Metadata server has expired, so shutting down AutoRecoveryMain!
21:29:21.615 [main-EventThread] INFO
org.apache.bookkeeper.replication.AutoRecoveryMain - Shutting down auto
recovery: 3
21:29:21.615 [main-EventThread] INFO
org.apache.bookkeeper.replication.AutoRecoveryMain - Shutting down AutoRecovery
21:29:21.669 [ZKC-connect-executor-0] INFO
org.apache.bookkeeper.zookeeper.ZooKeeperClient - Reconnecting zookeeper
jxnc-bd-olmc-006:2181,jxnc-bd-olmc-007:2181,jxnc-bd-olmc-008:2181,jxnc-bd-olmc-009:2181,jxnc-bd-olmc-010:2181.
21:29:21.683 [ZKC-connect-executor-0] INFO org.apache.zookeeper.ZooKeeper -
Initiating client connection,
connectString=jxnc-bd-olmc-006:2181,jxnc-bd-olmc-007:2181,jxnc-bd-olmc-008:2181,jxnc-bd-olmc-009:2181,jxnc-bd-olmc-010:2181
sessionTimeout=30000
watcher=org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase@420745d7
21:29:21.683 [main-EventThread] INFO
org.apache.bookkeeper.replication.ReplicationWorker - Shutting down replication
worker
21:29:21.683 [main-EventThread] INFO
org.apache.bookkeeper.replication.ReplicationWorker - Shutting down
ReplicationWorker
21:29:21.683 [AuditorElector-10.242.63.177:3181] INFO
org.apache.bookkeeper.replication.AuditorElector - Shutting down AuditorElector
21:29:21.714 [ZKC-connect-executor-0] INFO
org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 10485760 Bytes
21:29:21.714 [ZKC-connect-executor-0] INFO org.apache.zookeeper.ClientCnxn
- zookeeper.request.timeout value is 0. feature enabled=
21:29:21.717 [ZKC-connect-executor-0-SendThread(jxnc-bd-olmc-007:2181)] INFO
org.apache.zookeeper.ClientCnxn - Opening socket connection to server
jxnc-bd-olmc-007/10.242.63.177:2181. Will not attempt to authenticate using
SASL (unknown error)
21:29:21.717 [ZKC-connect-executor-0-SendThread(jxnc-bd-olmc-007:2181)] INFO
org.apache.zookeeper.ClientCnxn - Socket connection established, initiating
session, client: /10.242.63.177:33116, server:
jxnc-bd-olmc-007/10.242.63.177:2181
21:29:21.718 [ZKC-connect-executor-0-SendThread(jxnc-bd-olmc-007:2181)] INFO
org.apache.zookeeper.ClientCnxn - Session establishment complete on server
jxnc-bd-olmc-007/10.242.63.177:2181, sessionid = 0x702a3f516f40636, negotiated
timeout = 30000
21:29:21.718 [ZKC-connect-executor-0-EventThread] INFO
org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is
connected now.
21:29:21.718 [ZKC-connect-executor-0] INFO
org.apache.bookkeeper.zookeeper.ZooKeeperClient - ZooKeeper session
702a3f516f40636 is created to
jxnc-bd-olmc-006:2181,jxnc-bd-olmc-007:2181,jxnc-bd-olmc-008:2181,jxnc-bd-olmc-009:2181,jxnc-bd-olmc-010:2181.
21:29:21.717 [ReplicationWorker] ERROR
org.apache.bookkeeper.replication.ReplicationWorker - UnavailableException
while replicating fragments
org.apache.bookkeeper.replication.ReplicationException$UnavailableException:
Interrupted while contacting zookeeper
at
org.apache.bookkeeper.meta.ZkLedgerUnderreplicationManager.isLedgerReplicationEnabled(ZkLedgerUnderreplicationManager.java:732)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.meta.ZkLedgerUnderreplicationManager.waitIfLedgerReplicationDisabled(ZkLedgerUnderreplicationManager.java:619)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.meta.ZkLedgerUnderreplicationManager.getLedgerToRereplicate(ZkLedgerUnderreplicationManager.java:600)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.replication.ReplicationWorker.rereplicate(ReplicationWorker.java:272)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.replication.ReplicationWorker.run(ReplicationWorker.java:238)
[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
[io.netty-netty-common-4.1.43.Final.jar:4.1.43.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_192]
Caused by: java.lang.InterruptedException
at java.lang.Object.wait(Native Method) ~[?:1.8.0_192]
at java.lang.Object.wait(Object.java:502) ~[?:1.8.0_192]
at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1529)
~[org.apache.pulsar-pulsar-zookeeper-2.5.0.jar:2.5.0]
at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1512)
~[org.apache.pulsar-pulsar-zookeeper-2.5.0.jar:2.5.0]
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:2016)
~[org.apache.pulsar-pulsar-zookeeper-2.5.0.jar:2.5.0]
at
org.apache.bookkeeper.zookeeper.ZooKeeperClient.access$2301(ZooKeeperClient.java:70)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.zookeeper.ZooKeeperClient$13.call(ZooKeeperClient.java:830)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.zookeeper.ZooKeeperClient$13.call(ZooKeeperClient.java:824)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.zookeeper.ZooWorker.syncCallWithRetries(ZooWorker.java:140)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.zookeeper.ZooKeeperClient.exists(ZooKeeperClient.java:824)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:2049)
~[org.apache.pulsar-pulsar-zookeeper-2.5.0.jar:2.5.0]
at
org.apache.bookkeeper.zookeeper.ZooKeeperClient.access$2401(ZooKeeperClient.java:70)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.zookeeper.ZooKeeperClient$14.call(ZooKeeperClient.java:851)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.zookeeper.ZooKeeperClient$14.call(ZooKeeperClient.java:845)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.zookeeper.ZooWorker.syncCallWithRetries(ZooWorker.java:140)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.zookeeper.ZooKeeperClient.exists(ZooKeeperClient.java:845)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.meta.ZkLedgerUnderreplicationManager.isLedgerReplicationEnabled(ZkLedgerUnderreplicationManager.java:723)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
... 6 more
21:29:21.745 [ReplicationWorker] INFO
org.apache.bookkeeper.replication.ReplicationWorker - ReplicationWorker exited
loop!
21:29:21.756 [main-EventThread] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel
bookie client for 10.242.63.179:3181
21:29:21.777 [main-EventThread] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel
bookie client for 10.242.63.199:3181
21:29:21.779 [bookkeeper-io-50-52] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie
channel [id: 0xcf0cda8c, L:/10.242.63.177:52874 ! R:/10.242.63.179:3181]
21:29:21.779 [main-EventThread] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel
bookie client for 10.242.63.177:3181
21:29:21.779 [bookkeeper-io-50-50] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie
channel [id: 0x4d645ee5, L:/10.242.63.177:53768 ! R:/10.242.63.199:3181]
21:29:21.781 [main-EventThread] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel
bookie client for 10.242.63.178:3181
21:29:21.781 [bookkeeper-io-50-45] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie
channel [id: 0xbcd90045, L:/10.242.63.177:51154 ! R:/10.242.63.177:3181]
21:29:21.783 [bookie-io-1-33] INFO
org.apache.bookkeeper.proto.BookieRequestHandler - Channels disconnected: [id:
0x1a719422, L:/10.242.63.177:3181 ! R:/10.242.63.177:51154]
21:29:21.783 [main-EventThread] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel
bookie client for 10.242.63.175:3181
21:29:21.783 [bookkeeper-io-50-44] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie
channel [id: 0x79a921b7, L:/10.242.63.177:55774 ! R:/10.242.63.178:3181]
21:29:21.785 [main-EventThread] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel
bookie client for 10.242.63.197:3181
21:29:21.785 [bookkeeper-io-50-76] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie
channel [id: 0xedd7ec8a, L:/10.242.63.177:50894 ! R:/10.242.63.175:3181]
21:29:21.786 [main-EventThread] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel
bookie client for 10.242.63.198:3181
21:29:21.786 [bookkeeper-io-50-40] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie
channel [id: 0x9eef5b31, L:/10.242.63.177:44234 ! R:/10.242.63.197:3181]
21:29:21.788 [main-EventThread] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel
bookie client for 10.242.63.176:3181
21:29:21.789 [bookkeeper-io-50-46] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie
channel [id: 0x8b324bea, L:/10.242.63.177:36426 !
R:jxnc-bd-olmc-028/10.242.63.198:3181]
21:29:21.789 [main-EventThread] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel
bookie client for 10.242.63.195:3181
21:29:21.789 [bookkeeper-io-50-60] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie
channel [id: 0x6505713a, L:/10.242.63.177:54268 ! R:/10.242.63.176:3181]
21:29:21.791 [main-EventThread] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel
bookie client for 10.242.63.173:3181
21:29:21.791 [bookkeeper-io-50-37] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie
channel [id: 0x259ea28e, L:/10.242.63.177:50694 ! R:/10.242.63.195:3181]
21:29:21.793 [main-EventThread] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel
bookie client for 10.242.63.196:3181
21:29:21.793 [bookkeeper-io-50-83] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie
channel [id: 0x564ddd2e, L:/10.242.63.177:38982 !
R:jxnc-bd-olmc-003/10.242.63.173:3181]
21:29:21.794 [main-EventThread] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel
bookie client for 10.242.63.174:3181
21:29:21.794 [bookkeeper-io-50-39] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie
channel [id: 0x2967b15b, L:/10.242.63.177:46508 ! R:/10.242.63.196:3181]
21:29:21.796 [main-EventThread] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel
bookie client for 10.242.63.171:3181
21:29:21.796 [bookkeeper-io-50-58] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie
channel [id: 0x0255fa8b, L:/10.242.63.177:37096 ! R:/10.242.63.174:3181]
21:29:21.797 [main-EventThread] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel
bookie client for 10.242.63.172:3181
21:29:21.797 [bookkeeper-io-50-59] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie
channel [id: 0xb0f71766, L:/10.242.63.177:56610 ! R:/10.242.63.171:3181]
21:29:21.799 [main-EventThread] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel
bookie client for 10.242.63.194:3181
21:29:21.800 [bookkeeper-io-50-43] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie
channel [id: 0x744b9bc0, L:/10.242.63.177:34736 ! R:/10.242.63.172:3181]
21:29:21.800 [main-EventThread] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel
bookie client for 10.242.63.191:3181
21:29:21.800 [bookkeeper-io-50-57] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie
channel [id: 0xdf0ae60a, L:/10.242.63.177:57296 ! R:/10.242.63.194:3181]
21:29:21.802 [main-EventThread] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel
bookie client for 10.242.63.192:3181
21:29:21.802 [bookkeeper-io-50-54] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie
channel [id: 0x2d8f2f79, L:/10.242.63.177:33988 ! R:/10.242.63.191:3181]
21:29:21.804 [main-EventThread] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel
bookie client for 10.242.63.190:3181
21:29:21.807 [bookkeeper-io-50-41] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie
channel [id: 0x3d27521e, L:/10.242.63.177:42342 ! R:/10.242.63.192:3181]
21:29:21.807 [main-EventThread] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel
bookie client for 10.242.63.188:3181
21:29:21.807 [bookkeeper-io-50-55] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie
channel [id: 0xacc73cb1, L:/10.242.63.177:35546 ! R:/10.242.63.190:3181]
21:29:21.809 [main-EventThread] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel
bookie client for 10.242.63.200:3181
21:29:21.809 [bookkeeper-io-50-56] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie
channel [id: 0x3ab7591d, L:/10.242.63.177:55812 ! R:/10.242.63.188:3181]
21:29:21.810 [main-EventThread] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel
bookie client for 10.242.63.189:3181
21:29:21.810 [bookkeeper-io-50-47] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie
channel [id: 0x6e9f7b8d, L:/10.242.63.177:52520 ! R:/10.242.63.200:3181]
21:29:21.812 [main-EventThread] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel
bookie client for 10.242.63.186:3181
21:29:21.812 [bookkeeper-io-50-48] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie
channel [id: 0x70bf160e, L:/10.242.63.177:52438 ! R:/10.242.63.189:3181]
21:29:21.813 [main-EventThread] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel
bookie client for 10.242.63.187:3181
21:29:21.813 [bookkeeper-io-50-35] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie
channel [id: 0x778a8f55, L:/10.242.63.177:40600 !
R:jxnc-bd-olmc-016/10.242.63.186:3181]
21:29:21.813 [main-EventThread] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel
bookie client for 10.242.63.184:3181
21:29:21.813 [bookkeeper-io-50-36] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie
channel [id: 0x6271e956, L:/10.242.63.177:59924 !
R:jxnc-bd-olmc-017/10.242.63.187:3181]
21:29:21.815 [main-EventThread] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel
bookie client for 10.242.63.185:3181
21:29:21.815 [bookkeeper-io-50-49] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie
channel [id: 0xd38880f1, L:/10.242.63.177:37420 ! R:/10.242.63.184:3181]
21:29:21.817 [main-EventThread] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel
bookie client for 10.242.63.182:3181
21:29:21.817 [bookkeeper-io-50-51] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie
channel [id: 0x59002b72, L:/10.242.63.177:46156 !
R:jxnc-bd-olmc-015/10.242.63.185:3181]
21:29:21.818 [main-EventThread] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel
bookie client for 10.242.63.183:3181
21:29:21.818 [bookkeeper-io-50-61] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie
channel [id: 0x066deb9c, L:/10.242.63.177:53474 ! R:/10.242.63.182:3181]
21:29:21.819 [main-EventThread] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel
bookie client for 10.242.63.180:3181
21:29:21.819 [bookkeeper-io-50-53] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie
channel [id: 0xed6e3cc6, L:/10.242.63.177:39112 ! R:/10.242.63.183:3181]
21:29:21.821 [main-EventThread] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Closing the per channel
bookie client for 10.242.63.181:3181
21:29:21.821 [bookkeeper-io-50-38] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie
channel [id: 0x102b4a79, L:/10.242.63.177:60566 !
R:jxnc-bd-olmc-010/10.242.63.180:3181]
21:29:21.821 [bookkeeper-io-50-42] INFO
org.apache.bookkeeper.proto.PerChannelBookieClient - Disconnected from bookie
channel [id: 0xbb2b5056, L:/10.242.63.177:46174 ! R:/10.242.63.181:3181]
21:29:21.887 [AuditorElector-10.242.63.177:3181] ERROR
org.apache.bookkeeper.replication.AuditorElector - Exception while deleting
myVote:/ledgers/underreplication/auditorelection/V_0000000169
org.apache.zookeeper.KeeperException$SessionExpiredException:
KeeperErrorCode = Session expired for
/ledgers/underreplication/auditorelection/V_0000000169
at
org.apache.zookeeper.KeeperException.create(KeeperException.java:134)
~[org.apache.pulsar-pulsar-zookeeper-2.5.0.jar:2.5.0]
at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
~[org.apache.pulsar-pulsar-zookeeper-2.5.0.jar:2.5.0]
at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:1793)
~[org.apache.pulsar-pulsar-zookeeper-2.5.0.jar:2.5.0]
at
org.apache.bookkeeper.zookeeper.ZooKeeperClient.access$2101(ZooKeeperClient.java:70)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.zookeeper.ZooKeeperClient$11.call(ZooKeeperClient.java:770)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.zookeeper.ZooKeeperClient$11.call(ZooKeeperClient.java:764)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.zookeeper.ZooWorker.syncCallWithRetries(ZooWorker.java:140)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.zookeeper.ZooKeeperClient.delete(ZooKeeperClient.java:764)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.replication.AuditorElector$2.run(AuditorElector.java:262)
[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[?:1.8.0_192]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[?:1.8.0_192]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_192]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_192]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_192]
21:29:21.950 [main-EventThread] INFO org.apache.zookeeper.ZooKeeper -
Session: 0x702a3f516f40636 closed
21:29:21.950 [main-EventThread] INFO
org.apache.bookkeeper.meta.AbstractZkLedgerManager - ZooKeeper client expired
on ledger manager.
21:29:21.950 [ZKC-connect-executor-0-EventThread] INFO
org.apache.zookeeper.ClientCnxn - EventThread shut down for session:
0x702a3f516f40636
21:29:21.952 [main-EventThread] WARN
org.apache.bookkeeper.discover.ZKRegistrationClient - Failed to schedule watch
bookies task
java.util.concurrent.RejectedExecutionException: Task
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@522f30a
rejected from
java.util.concurrent.ScheduledThreadPoolExecutor@6ac3341d[Terminated, pool size
= 0, active threads = 0, queued tasks = 0, completed tasks = 2749179]
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
~[?:1.8.0_192]
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
~[?:1.8.0_192]
at
java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:326)
~[?:1.8.0_192]
at
java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533)
~[?:1.8.0_192]
at
com.google.common.util.concurrent.MoreExecutors$ScheduledListeningDecorator.schedule(MoreExecutors.java:555)
~[com.google.guava-guava-25.1-jre.jar:?]
at
org.apache.bookkeeper.common.util.BoundedScheduledExecutorService.schedule(BoundedScheduledExecutorService.java:63)
~[org.apache.bookkeeper-bookkeeper-common-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.common.util.OrderedScheduler$OrderedSchedulerDecoratedThread.schedule(OrderedScheduler.java:317)
~[org.apache.bookkeeper-bookkeeper-common-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.common.util.OrderedScheduler.schedule(OrderedScheduler.java:273)
~[org.apache.bookkeeper-bookkeeper-common-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.discover.ZKRegistrationClient$WatchTask.scheduleWatchTask(ZKRegistrationClient.java:107)
[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.discover.ZKRegistrationClient$WatchTask.process(ZKRegistrationClient.java:149)
[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535)
[org.apache.pulsar-pulsar-zookeeper-2.5.0.jar:2.5.0]
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
[org.apache.pulsar-pulsar-zookeeper-2.5.0.jar:2.5.0]
21:29:21.953 [main-EventThread] WARN
org.apache.bookkeeper.discover.ZKRegistrationClient - Failed to schedule watch
bookies task
java.util.concurrent.RejectedExecutionException: Task
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@31652151
rejected from
java.util.concurrent.ScheduledThreadPoolExecutor@6ac3341d[Terminated, pool size
= 0, active threads = 0, queued tasks = 0, completed tasks = 2749179]
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
~[?:1.8.0_192]
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
~[?:1.8.0_192]
at
java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:326)
~[?:1.8.0_192]
at
java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533)
~[?:1.8.0_192]
at
com.google.common.util.concurrent.MoreExecutors$ScheduledListeningDecorator.schedule(MoreExecutors.java:555)
~[com.google.guava-guava-25.1-jre.jar:?]
at
org.apache.bookkeeper.common.util.BoundedScheduledExecutorService.schedule(BoundedScheduledExecutorService.java:63)
~[org.apache.bookkeeper-bookkeeper-common-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.common.util.OrderedScheduler$OrderedSchedulerDecoratedThread.schedule(OrderedScheduler.java:317)
~[org.apache.bookkeeper-bookkeeper-common-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.common.util.OrderedScheduler.schedule(OrderedScheduler.java:273)
~[org.apache.bookkeeper-bookkeeper-common-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.discover.ZKRegistrationClient$WatchTask.scheduleWatchTask(ZKRegistrationClient.java:107)
[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.discover.ZKRegistrationClient$WatchTask.process(ZKRegistrationClient.java:149)
[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535)
[org.apache.pulsar-pulsar-zookeeper-2.5.0.jar:2.5.0]
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
[org.apache.pulsar-pulsar-zookeeper-2.5.0.jar:2.5.0]
21:29:21.953 [main-EventThread] ERROR
org.apache.bookkeeper.replication.AuditorElector - Lost ZK connection, shutting
down
21:29:21.953 [main-EventThread] ERROR org.apache.zookeeper.ClientCnxn -
Error while calling watcher
java.util.concurrent.RejectedExecutionException: Task
java.util.concurrent.FutureTask@6856522d rejected from
java.util.concurrent.ThreadPoolExecutor@7afe568e[Terminated, pool size = 0,
active threads = 0, queued tasks = 0, completed tasks = 2]
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
~[?:1.8.0_192]
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
~[?:1.8.0_192]
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
~[?:1.8.0_192]
at
java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112)
~[?:1.8.0_192]
at
java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678)
~[?:1.8.0_192]
at
org.apache.bookkeeper.replication.AuditorElector.submitShutdownTask(AuditorElector.java:254)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.replication.AuditorElector.access$100(AuditorElector.java:80)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.bookkeeper.replication.AuditorElector$ElectionWatcher.process(AuditorElector.java:238)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
at
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535)
[org.apache.pulsar-pulsar-zookeeper-2.5.0.jar:2.5.0]
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510)
[org.apache.pulsar-pulsar-zookeeper-2.5.0.jar:2.5.0]
21:29:21.953 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn -
EventThread shut down for session: 0x902a3f7df0500d2
21:29:22.383 [AutoRecoveryDeathWatcher-3181] INFO
org.apache.bookkeeper.replication.AutoRecoveryMain - AutoRecoveryDeathWatcher
noticed the AutoRecovery is not running any more,exiting the watch loop!
21:29:22.383 [AutoRecoveryDeathWatcher-3181] ERROR
org.apache.bookkeeper.common.component.ComponentStarter - Triggered
exceptionHandler of Component: bookie-server because of Exception in Thread:
Thread[AutoRecoveryDeathWatcher-3181,5,main]
java.lang.RuntimeException: AutoRecovery is not running any more
at
org.apache.bookkeeper.replication.AutoRecoveryMain$AutoRecoveryDeathWatcher.run(AutoRecoveryMain.java:237)
~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0]
21:29:22.385 [component-shutdown-thread] INFO
org.apache.bookkeeper.common.component.ComponentStarter - Closing component
bookie-server in shutdown hook.
21:29:22.387 [component-shutdown-thread] INFO
org.apache.bookkeeper.replication.AutoRecoveryMain - Shutting down auto
recovery: 0
21:29:22.387 [component-shutdown-thread] INFO
org.apache.bookkeeper.proto.BookieServer - Shutting down BookieServer
21:29:22.387 [component-shutdown-thread] INFO
org.apache.bookkeeper.proto.BookieNettyServer - Shutting down BookieNettyServer
**To Reproduce**
Steps to reproduce the behavior:
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error
**Expected behavior**
A clear and concise description of what you expected to happen.
**Screenshots**
If applicable, add screenshots to help explain your problem.
**Desktop (please complete the following information):**
- OS: [e.g. iOS]
**Additional context**
Add any other context about the problem here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
