[GitHub] [pulsar] codelipenghui opened a new issue, #16998: Broker direct memory leak due to BKNotEnoughBookiesException

[GitBox]

codelipenghui opened a new issue, #16998:
URL: https://github.com/apache/pulsar/issues/16998

   **Describe the bug**
   
   ```
   Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: 
2022-08-08T17:44:23,714+0000 [BookKeeperClientWorker-OrderedExecutor-20-0] 
ERROR org.apache.bookkeeper.client.MetadataUpdateLoop - 
UpdateLoop(ledgerId=223240,loopId=5cca03e4) Exception updating
   Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: 
org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not 
enough non-faulty bookies available
   Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: 
2022-08-08T17:44:23,714+0000 [BookKeeperClientWorker-OrderedExecutor-20-0] WARN 
 org.apache.bookkeeper.client.LedgerHandle - [EnsembleChange(ledger:223240, 
change-id:0000000001)][attempt:1] Exception changing ensemble
   Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: 
org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not 
enough non-faulty bookies available
   Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: 
2022-08-08T17:44:23,714+0000 [BookKeeperClientWorker-OrderedExecutor-20-0] 
ERROR org.apache.bookkeeper.client.LedgerHandle - Closing ledger 223240 due to 
NotEnoughBookiesException: Not enough non-faulty bookies available
   Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: 
2022-08-08T17:44:23,714+0000 [BookKeeperClientWorker-OrderedExecutor-12-0] 
ERROR org.apache.bookkeeper.client.MetadataUpdateLoop - 
UpdateLoop(ledgerId=233784,loopId=0147ac3a) Exception updating
   Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: 
org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not 
enough non-faulty bookies available
   Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: 
2022-08-08T17:44:23,714+0000 [BookKeeperClientWorker-OrderedExecutor-12-0] WARN 
 org.apache.bookkeeper.client.LedgerHandle - [EnsembleChange(ledger:233784, 
change-id:0000000001)][attempt:1] Exception changing ensemble
   Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: 
org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not 
enough non-faulty bookies available
   Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: 
2022-08-08T17:44:23,714+0000 [BookKeeperClientWorker-OrderedExecutor-12-0] 
ERROR org.apache.bookkeeper.client.LedgerHandle - Closing ledger 233784 due to 
NotEnoughBookiesException: Not enough non-faulty bookies available
   Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: 
2022-08-08T17:44:23,752+0000 [BookKeeperClientWorker-OrderedExecutor-10-0] 
ERROR org.apache.bookkeeper.client.MetadataUpdateLoop - 
UpdateLoop(ledgerId=238581,loopId=3062cb0d) Exception updating
   Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: 
org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not 
enough non-faulty bookies available
   Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: 
2022-08-08T17:44:23,752+0000 [BookKeeperClientWorker-OrderedExecutor-10-0] WARN 
 org.apache.bookkeeper.client.LedgerHandle - [EnsembleChange(ledger:238581, 
change-id:0000000001)][attempt:1] Exception changing ensemble
   Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: 
org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not 
enough non-faulty bookies available
   Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: 
2022-08-08T17:44:23,752+0000 [BookKeeperClientWorker-OrderedExecutor-10-0] 
ERROR org.apache.bookkeeper.client.LedgerHandle - Closing ledger 238581 due to 
NotEnoughBookiesException: Not enough non-faulty bookies available
   Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: 
2022-08-08T17:44:23,777+0000 [BookKeeperClientWorker-OrderedExecutor-7-0] ERROR 
org.apache.bookkeeper.client.MetadataUpdateLoop - 
UpdateLoop(ledgerId=259791,loopId=15db4235) Exception updating
   Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: 
org.apache.bookkeeper.client.BKException$BKNotEnoughBookiesException: Not 
enough non-faulty bookies available
   Aug 08 17:44:23 ip-10-0-0-158.us-west-2.compute.internal pulsar[25809]: 
2022-08-08T17:44:23,777+0000 [BookKeeperClientWorker-OrderedExecutor-7-0] WARN  
org.apache.bookkeeper.client.LedgerHandle - [EnsembleChange(ledger:259791, 
change-id:0000000001)][attempt:1] Exception changing ensemble
   ```
   
   <img width="1807" alt="image" 
src="https://user-images.githubusercontent.com/12592133/183480581-fa3575b1-2ea6-4ae0-a62a-0fe83fedfd1d.png";>
   
   The second broker does not have the above error logs.
   
   **Additional context**
   Build from branch-2.11 with bookkeeper 4.15.1
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org