2019-10-29 15:43:23 UTC - Natalie: @Natalie has joined the channel
----
2019-10-29 18:13:36 UTC - tihomir: Hi guys, we run out of space on our server
and after increasing the storage we cannot get our broker to work. It looks
like there is some disrupt with the zk state and we are constantly getting
org.apache.pulsar.client.api.PulsarClientException$LookupException: No active
broker is available
----
2019-10-29 18:31:47 UTC - Addison Higham: ran out of space on bookies?
----
2019-10-29 18:31:57 UTC - Addison Higham: or zk?
----
2019-10-29 18:32:32 UTC - tihomir: on bookies
----
2019-10-29 19:02:57 UTC - Addison Higham: do your logs say something to the
effect of them being unable to form a healthy ensemble? Bookkeeper has a config
value, `diskUsageThreshold`, that once that threshold is reached, the bookie
marks itself as unavailable for new segments. If for some reason, it hasn't
picked up on the increased disk size (maybe it needs a restart?) that could be
why it is still not healthy. I was able to get out of situation like this by
stopping producers, slightly increasing the diskUsageThreshold, deleting some
topics to free up space, then resuming
----
2019-10-29 19:07:40 UTC - Matteo Merli: > that once that threshold is
reached, the bookie marks itself as unavailable for new segments
----
2019-10-29 19:08:06 UTC - Matteo Merli: not only for new segments, after the
threshold is reached, the bookie turn into read-only mode, no more writes are
accepted
----
2019-10-29 19:11:48 UTC - Addison Higham: that makes more sense
----
2019-10-29 19:18:25 UTC - tihomir: it looks like our zookeeper has got in
corrupt state
----
2019-10-29 19:18:55 UTC - Matteo Merli: was disk full on ZK then ?
----
2019-10-29 19:19:03 UTC - tihomir: we have the following errors
org.apache.bookkeeper.meta.AbstractZkLedgerManager - Could not read metadata
for ledger: 1473
org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode =
ConnectionLoss for /ledgers/00/0000/L1473
at org.apache.zookeeper.KeeperException.create(KeeperException.java:10
----
2019-10-29 19:19:14 UTC - tihomir: no only on bookies
----
2019-10-29 19:19:42 UTC - tihomir: but it was full only on 2 of 7 nodes
----
2019-10-29 19:49:39 UTC - tihomir: even listing of active brokers in that
region result in error
----
2019-10-29 19:49:55 UTC - tihomir:
----
2019-10-30 01:15:52 UTC - Sijie Guo: the error message indicates that you can
not resolve the domain name “streams-broker”
----
2019-10-30 04:43:56 UTC - PRASAD VUNNAM: @PRASAD VUNNAM has joined the channel
----
2019-10-30 05:16:08 UTC - Saurabh J: @Saurabh J has joined the channel
----
