2018-04-20 12:50:25 UTC - Vasily Yanov: Hi! Today we had our first issue with pulsar. Even not with pulsar but bookkeeper. Do you have any ideas why it could happened: ---- 2018-04-20 12:50:30 UTC - Vasily Yanov: ```Apr 20 12:42:46 localhost bookkeeper[32582]: 2018-04-20 12:42:46,377 - WARN - [bookie-io-14-8:ReadEntryProcessor@63] - Ledger 305000 fenced by /1.1.1.1:47758 Apr 20 12:42:47 localhost bookkeeper[32582]: 2018-04-20 12:42:47,725 - WARN - [bookie-io-14-14:ReadEntryProcessor@63] - Ledger 117386 fenced by /1.1.1.1:47770 Apr 20 12:42:48 localhost bookkeeper[32582]: 2018-04-20 12:42:48,134 - WARN - [bookie-io-14-8:ReadEntryProcessor@63] - Ledger 221224 fenced by /1.1.1.1:47758 Apr 20 12:42:49 localhost bookkeeper[32582]: 2018-04-20 12:42:49,493 - WARN - [bookie-io-14-14:ReadEntryProcessor@63] - Ledger 304986 fenced by /1.1.1.1:47770 Apr 20 12:42:49 localhost bookkeeper[32582]: 2018-04-20 12:42:49,890 - WARN - [bookie-io-14-8:ReadEntryProcessor@63] - Ledger 162056 fenced by /1.1.1.1:47758 Apr 20 12:42:51 localhost bookkeeper[32582]: 2018-04-20 12:42:51,260 - WARN - [bookie-io-14-14:ReadEntryProcessor@63] - Ledger 205578 fenced by /1.1.1.1:47770 Apr 20 12:42:51 localhost bookkeeper[32582]: 2018-04-20 12:42:51,647 - WARN - [bookie-io-14-8:ReadEntryProcessor@63] - Ledger 67144 fenced by /1.1.1.1:47758``` ---- 2018-04-20 12:50:59 UTC - Vasily Yanov: and a lot of pulsar logs: ---- 2018-04-20 12:51:03 UTC - Vasily Yanov: ``` Apr 20 12:58:07 localhost broker[25751]: org.apache.bookkeeper.mledger.ManagedLedgerException: Error while recovering ledger Apr 20 12:58:07 localhost broker[25751]: 2018-04-20 12:58:07,301 - ERROR - [bookkeeper-ml-workers-40-1:ManagedLedgerImpl$1@268] - [server-eagle/prod-pulsar-cluster-1/session_init/persistent/376d8f49-967b-41c6-899f-bdcb772fb5d5] Failed to open ledger 104654: Error while recovering ledger Apr 20 12:58:07 localhost broker[25751]: 2018-04-20 12:58:07,301 - WARN - [bookkeeper-ml-workers-40-1:BrokerService$3@613] - Failed to create topic <persistent://server-eagle/prod-pulsar-cluster-1/session_init/376d8f49-967b-41c6-899f-bdcb772fb5d5> ``` ---- 2018-04-20 12:51:37 UTC - Vasily Yanov: just spoke with developers. They tried to create a lot of topics and as a result on 1 of 3 servers I saw a lot of "Failed to create topic" and on another 2 server in cluser "Ledger XXXXX fenced by <first server with Failed to create topic error>" ---- 2018-04-20 15:09:36 UTC - Brent Hughes: @Brent Hughes has joined the channel ---- 2018-04-20 16:30:04 UTC - Sijie Guo: @Vasily Yanov
- “Ledger xyz fenced” happens when a topic is moving from one broker to the other broker. when the topic ownership is moved, for data consistency consideration, pulsar will seal the segment (aka ledger) that previous broker open. that’s why you will see “Ledger xyz fenced”. The loggings there are general safe logging (although sometimes might be just annoying :)) - so the question would become why topics are moved or in the other way, why do you see “Failed to create topic”. in order to help with your questions, do you mind sharing a bit more information? - what is your current setup? how many bookies/brokers? - how does your configuration settings look like? most likely are just default. ---- 2018-04-20 17:48:10 UTC - Vasily Yanov: @Sijie Guo, thank you for your reply and detailed explanation. My current setup is: 3 servers with zookeeper+bookkeeper+pulsan at each one. Configuration settings mostly default excent zookeeper servers configuration. ---- 2018-04-20 20:02:49 UTC - Asish Skaria: @Asish Skaria has joined the channel ---- 2018-04-20 20:21:55 UTC - Asish Skaria: Is there any metrics page : Like maximum message size ( is it 5MB), max subscriber limit, how does the latency and throughput depend upon number of subscribers. ---- 2018-04-20 22:58:58 UTC - Sijie Guo: okay. this sounds like one broker can’t not talk to the other bookies? was there any network problems? are these 3 servers physical machines or vms? ---- 2018-04-20 23:05:21 UTC - Sijie Guo: @Asish Skaria currently we don’t have this page. there is FAQ page, that accumulating bunch of questions, which can be helpful - <https://github.com/apache/incubator-pulsar/blob/master/faq.md> regarding your questions: - maximum message size: currently it is 5MB. you can set `nettyMaxFrameSizeBytes` to increase. - max subscriber limit: you mean maximum number of throughput or number of subscribers? either is kind of related to your hardware as well. the number of subscribers is kind of bound by metadata, which is limited by zookeeper. the throughput is bound with the network bandwidth. - how does latency and throughput depend upon number of subscribers. latency is kind of related to disk settings, whether you enable fsync or not, and also the bandwidth you used (both disk/network) throughput is typically an easy way to figure out, pulsar can easily exhaust all the network bandwidth, so it most likely only bound by the network bandwidth. the number of subscribers: the number of subscribers depends on the throughput of topics and the network bandwidth of a broker. hope this can help you understand. ---- 2018-04-20 23:31:40 UTC - Asish Skaria: @Sijie Guo Than you. Also your post about Kafka vs pulsar was very helpful ---- 2018-04-21 05:32:02 UTC - Vasily Yanov: it is 3 separated physical machines in same DC but in different VLANs. Just tried to telnet between those machines to tcp/3181,6650,8080 with possitive result. BTW: zookeeper can form cluster and elect leader. ---- 2018-04-21 09:03:51 UTC - Vasily Yanov: Does pulsar has a nice grafana dashboard? :slightly_smiling_face: ---- 2018-04-21 09:08:47 UTC - Ali Ahmed: yes there is Prometheus integration and default grafana dashboard in the repo ----
