Lanayx opened a new issue #6154: Brokers constantly crash in the k8s cluster
URL: https://github.com/apache/pulsar/issues/6154
 
 
   **Describe the bug**
   Brokers constantly failing in a cluster with a _CrashLoopBackOff_ . They 
work for several hours and then fail. Tried installing clean cluster 3 days in 
a raw and the behavior persisted.
   
   **To Reproduce**
   Steps to reproduce the behavior:
   1. Use default helm chart values-mini.yaml on GKE 3 nodes
   2. Use following values:
   ```
     # Pulsar proxy
     proxy: yes
     # Bookkeeper auto-recovery
     autoRecovery: no
     # Pulsar dashboard
     # Deprecated
     # Replace pulsar-dashboard with pulsar-manager
     dashboard: yes
     # pulsar manager
     pulsar_manager: no
     # Bastion pod for administrative commands
     bastion: yes
     # Monitoring stack (prometheus and grafana)
     monitoring: no
   ```
   3. Wait for some time (4+ hours)
   4. See error
   
   **Expected behavior**
   Brokers should fail in the CrashLoopBackOff
   
   **Screenshots**
   
![image](https://user-images.githubusercontent.com/3329606/73246774-80e15a00-41c0-11ea-9913-a24725282f8e.png)
   
   **Additional context**
   Adding logs from 2 failing brokers and 3 bookies. Also adding last logs from 
one of the evicted pods.
   
[bookeeper-1.log](https://github.com/apache/pulsar/files/4121328/bookeeper-1.log)
   
[bookeeper-2.log](https://github.com/apache/pulsar/files/4121329/bookeeper-2.log)
   
[bookeeper-3.log](https://github.com/apache/pulsar/files/4121331/bookeeper-3.log)
   [broker-1.log](https://github.com/apache/pulsar/files/4121332/broker-1.log)
   [broker-2.log](https://github.com/apache/pulsar/files/4121333/broker-2.log)
   
[evicted_broker.txt](https://github.com/apache/pulsar/files/4121424/evicted_broker.txt)
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[email protected]


With regards,
Apache Git Services

Reply via email to