merlimat opened a new pull request #3487: When cursor recovery encounters empty cursor ledger, fallback to latest snapshot URL: https://github.com/apache/pulsar/pull/3487 ### Motivation In some conditions, the ledger used for cursor recovery might appear to be empty after re-opening. In these cases we're currently getting runtime exception from BK client since we're trying to read entry -1. In these case, use the same fallback mechanism to fallback to the previous cursor snapshot and re-open the cursor. Example of logs of this error: ``` 21:16:35.837 [bookkeeper-ml-workers-OrderedExecutor-6-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/functions/persistent/coordinate] Created ledger 1988 21:16:35.841 [bookkeeper-ml-workers-OrderedExecutor-6-0] INFO org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [public/functions/persistent/coordinate] Recovering from bookkeeper ledger cursor: participants 21:16:35.842 [bookkeeper-ml-workers-OrderedExecutor-6-0] INFO org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [public/functions/persistent/coordinate] Consumer participants meta-data recover from ledger 3 21:16:35.844 [pulsar-ordered-OrderedExecutor-3-0-EventThread] ERROR org.apache.bookkeeper.client.LedgerHandle - IncorrectParameterException on ledgerId:3 firstEntry:-1 lastEntry:-1 21:16:35.844 [pulsar-ordered-OrderedExecutor-3-0-EventThread] WARN org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [public/functions/persistent/coordinate] Error reading from metadata ledger 3 for consumer participants: Incorrect parameter input 21:16:35.844 [pulsar-ordered-OrderedExecutor-3-0-EventThread] WARN org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [public/functions/persistent/coordinate] Recovery for cursor participants failed org.apache.bookkeeper.mledger.ManagedLedgerException: Incorrect parameter input 21:16:35.845 [pulsar-ordered-OrderedExecutor-3-0-EventThread] WARN org.apache.pulsar.broker.service.BrokerService - Failed to create topic persistent://public/functions/coordinate org.apache.bookkeeper.mledger.ManagedLedgerException: Incorrect parameter input 21:16:35.847 [pulsar-ordered-OrderedExecutor-3-0-EventThread] WARN org.apache.pulsar.broker.service.ServerCnx - [/10.244.1.24:49452][persistent://public/functions/coordinate][participants] Failed to create consumer: org.apache.bookkeeper.mledger.ManagedLedgerException: Incorrect parameter input java.util.concurrent.CompletionException: org.apache.pulsar.broker.service.BrokerServiceException$PersistenceException: org.apache.bookkeeper.mledger.ManagedLedgerException: Incorrect parameter input at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) ~[?:1.8.0_181] at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) ~[?:1.8.0_181] at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593) ~[?:1.8.0_181] at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) ~[?:1.8.0_181] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_181] at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_181] at org.apache.pulsar.broker.service.BrokerService$3.openLedgerFailed(BrokerService.java:680) ~[org.apache.pulsar-pulsar-broker-2.2.1.jar:2.2.1] at org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl.lambda$asyncOpen$5(ManagedLedgerFactoryImpl.java:254) ~[org.apache.pulsar-managed-ledger-original-2.2.1.jar:2.2.1] at java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:870) ~[?:1.8.0_181] at java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:852) ~[?:1.8.0_181] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[?:1.8.0_181] at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[?:1.8.0_181] at org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl$2.initializeFailed(ManagedLedgerFactoryImpl.java:247) ~[org.apache.pulsar-managed-ledger-original-2.2.1.jar:2.2.1] at org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl$3$1.operationFailed(ManagedLedgerImpl.java:452) ~[org.apache.pulsar-managed-ledger-original-2.2.1.jar:2.2.1] at org.apache.bookkeeper.mledger.impl.ManagedCursorImpl.lambda$null$0(ManagedCursorImpl.java:299) ~[org.apache.pulsar-managed-ledger-original-2.2.1.jar:2.2.1] at org.apache.bookkeeper.client.LedgerHandle.asyncReadEntries(LedgerHandle.java:663) ~[org.apache.bookkeeper-bookkeeper-server-4.7.3.jar:4.7.3] at org.apache.bookkeeper.mledger.impl.ManagedCursorImpl.lambda$recoverFromLedger$1(ManagedCursorImpl.java:285) ~[org.apache.pulsar-managed-ledger-original-2.2.1.jar:2.2.1] at org.apache.bookkeeper.client.LedgerOpenOp.openComplete(LedgerOpenOp.java:225) [org.apache.bookkeeper-bookkeeper-server-4.7.3.jar:4.7.3] at org.apache.bookkeeper.client.LedgerOpenOp.operationComplete(LedgerOpenOp.java:181) [org.apache.bookkeeper-bookkeeper-server-4.7.3.jar:4.7.3] at org.apache.bookkeeper.client.LedgerOpenOp.operationComplete(LedgerOpenOp.java:49) [org.apache.bookkeeper-bookkeeper-server-4.7.3.jar:4.7.3] at org.apache.bookkeeper.meta.CleanupLedgerManager$CleanupGenericCallback.operationComplete(CleanupLedgerManager.java:56) [org.apache.bookkeeper-bookkeeper-server-4.7.3.jar:4.7.3] at org.apache.bookkeeper.meta.AbstractZkLedgerManager$3.processResult(AbstractZkLedgerManager.java:421) [org.apache.bookkeeper-bookkeeper-server-4.7.3.jar:4.7.3] at org.apache.bookkeeper.zookeeper.ZooKeeperClient$19$1.processResult(ZooKeeperClient.java:994) [org.apache.bookkeeper-bookkeeper-server-4.7.3.jar:4.7.3] at org.apache.zookeeper.ClientCnxn$EventThread.processEvent_aroundBody0(ClientCnxn.java:572) [org.apache.pulsar-pulsar-broker-2.2.1.jar:2.2.1] at org.apache.zookeeper.ClientCnxn$EventThread$AjcClosure1.run(ClientCnxn.java:1) [org.apache.pulsar-pulsar-broker-2.2.1.jar:2.2.1] at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149) [org.aspectj-aspectjrt-1.9.1.jar:?] at org.apache.pulsar.broker.zookeeper.aspectj.ClientCnxnAspect.timedProcessEvent(ClientCnxnAspect.java:73) [org.apache.pulsar-pulsar-broker-2.2.1.jar:2.2.1] at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:528) [org.apache.pulsar-pulsar-broker-2.2.1.jar:2.2.1] at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:508) [org.apache.pulsar-pulsar-broker-2.2.1.jar:2.2.1] Caused by: org.apache.pulsar.broker.service.BrokerServiceException$PersistenceException: org.apache.bookkeeper.mledger.ManagedLedgerException: Incorrect parameter input ... 23 more Caused by: org.apache.bookkeeper.mledger.ManagedLedgerException: Incorrect parameter input 21:16:35.853 [pulsar-client-io-44-1] WARN org.apache.pulsar.client.impl.ClientCnx - [id: 0xa29958a3, L:/10.244.1.24:49452 - R:10.244.1.24/10.244.1.24:6650] Received error from server: org.apache.bookkeeper.mledger.ManagedLedgerException: Incorrect parameter input 21:16:35.853 [pulsar-client-io-44-1] WARN org.apache.pulsar.client.impl.ConsumerImpl - [persistent://public/functions/coordinate][participants] Failed to subscribe to topic on 10.244.1.24/10.244.1.24:6650 21:16:35.854 [main] ERROR org.apache.pulsar.functions.worker.WorkerService - Error Starting up in worker ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
