[GitHub] [pulsar] evfurman opened a new issue #8069: Consumption Stops When Losing a Bookie.

[GitBox]

evfurman opened a new issue #8069:
URL: https://github.com/apache/pulsar/issues/8069


   **Describe the bug**
   Consumers stop consuming when bookie service is stopped.
   
   Slack thread: 
https://apache-pulsar.slack.com/archives/C5Z4T36F7/p1600113955395000
   
   We have a 4 node bookie cluster utilizing rack awareness, running version 
`2.6.1.` While testing failure scenarios we stopped the `pulsar-bookie` service 
on one of the nodes. In the Pulsar Manager consumption throughput dropped to 
zero across all partitioned topics. Production continued fine. This caused an 
immediate backlog (obviously). We restarted the `pulsar-bookie` service but 
consumption did not resume. @addisonj recommended trying to unload the 
partition which caused consumption to resume.
   
   We are using the default namespace with the following config:
   ```
   # Number of bookies to use when creating a ledger
   managedLedgerDefaultEnsembleSize=3
   # Number of copies to store for each message
   managedLedgerDefaultWriteQuorum=2
   # Number of guaranteed copies (acks to wait before write is complete)
   managedLedgerDefaultAckQuorum=2
   ```
   
   **Expected behavior**
   We expected higher latencies briefly with the loss of a bookie but would 
have thought production/consumption would not stop working entirely.
   
   **Screenshots**
   Consumption stops.
   <img width="1608" alt="Screen Shot 2020-09-14 at 4 31 57 PM" 
src="https://user-images.githubusercontent.com/7016955/93243653-1c264000-f74e-11ea-9ab9-bcf763b5da68.png";>
   
   After unloading partition, consumption resumes.
   <img width="1550" alt="Screen Shot 2020-09-15 at 11 50 32 AM" 
src="https://user-images.githubusercontent.com/7016955/93243590-06b11600-f74e-11ea-834a-20f826ba62ad.png";>
   
   **Additional context**
   Attaching logs from the brokers.
   
[pulsar-broker-1.log-09-14-2020-1.log.gz](https://github.com/apache/pulsar/files/5226610/pulsar-broker-1.log-09-14-2020-1.log.gz)
   
[pulsar-broker-2.log-09-14-2020-1.log.gz](https://github.com/apache/pulsar/files/5226611/pulsar-broker-2.log-09-14-2020-1.log.gz)
   
[pulsar-broker-3.log-09-14-2020-1.log.gz](https://github.com/apache/pulsar/files/5226612/pulsar-broker-3.log-09-14-2020-1.log.gz)
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org