bardock commented on issue #2013: Unable to consume messages from a partition URL: https://github.com/apache/incubator-pulsar/issues/2013#issuecomment-406054594 Sorry for the delay guys. We faced the same issue past week. This time we detected the issue 5 hours later, recreated the subscriptions, but when we moved back 5h the cursor the problem appeared again (clients did not receive messages from partition 4). We finally decided to restart the entire cluster (bookies and brokers) and recreated the subscriptions. However, we didn't moved back the cursor this time because we realized we prefer to lose messages and start consuming the most recent ones (only for this particular topic). We found some logs starting at the same time the issue started and they were resolved when we restarted the cluster: ``` 2018-07-13 19:05:49,194 - WARN - [BookKeeperClientWorker-20-1:ManagedCursorImpl$11@958] - [fury/global/0f9b2d5d1c3d495d9081e530b18111a3-notifications-news/persistent/0f9b2d5d1c3d495d9081e530b18111a3-notifications-news-partition-4][f6f49168781f402c99ddfa871bc0e90c-fury-news-sender-android-meli.notifications-providers] Error while replaying entries ``` ``` 2018-07-13 19:05:49,226 - ERROR - [BookKeeperClientWorker-20-1:PersistentDispatcherMultipleConsumers@399] - [persistent://fury/global/0f9b2d5d1c3d495d9081e530b18111a3-notifications-news/0f9b2d5d1c3d495d9081e530b18111a3-notifications-news-partition-4 / f6f49168781f402c99ddfa871bc0e90c-fury-news-sender-android-meli.notifications-providers] Error reading entries at 1199746:2412 : org.apache.bookkeeper.client.BKException$BKTooManyRequestsException, Read Type Replay - Retrying to read in 15.0 seconds org.apache.bookkeeper.mledger.ManagedLedgerException: org.apache.bookkeeper.client.BKException$BKTooManyRequestsException Caused by: org.apache.bookkeeper.client.BKException$BKTooManyRequestsException at org.apache.bookkeeper.client.BKException.create(BKException.java:92) at org.apache.bookkeeper.mledger.impl.EntryCacheImpl.lambda$asyncReadEntry$1(EntryCacheImpl.java:187) at org.apache.bookkeeper.client.PendingReadOp.submitCallback(PendingReadOp.java:430) at org.apache.bookkeeper.client.PendingReadOp.access$000(PendingReadOp.java:59) at org.apache.bookkeeper.client.PendingReadOp$LedgerEntryRequest.sendNextRead(PendingReadOp.java:171) at org.apache.bookkeeper.client.PendingReadOp$LedgerEntryRequest.logErrorAndReattemptRead(PendingReadOp.java:227) at org.apache.bookkeeper.client.PendingReadOp.readEntryComplete(PendingReadOp.java:380) at org.apache.bookkeeper.proto.PerChannelBookieClient$ReadCompletion.readEntryComplete(PerChannelBookieClient.java:1116) at org.apache.bookkeeper.proto.PerChannelBookieClient.handleReadResponse(PerChannelBookieClient.java:1048) at org.apache.bookkeeper.proto.PerChannelBookieClient$ReadV2ResponseCallback.safeRun(PerChannelBookieClient.java:908) at org.apache.bookkeeper.util.SafeRunnable.run(SafeRunnable.java:31) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) ``` An this one starting 10 minutes later: ``` 2018-07-13 19:15:28,557 - ERROR - [BookKeeperClientWorker-20-1:PersistentDispatcherMultipleConsumers@399] - [persistent://fury/global/0f9b2d5d1c3d495d9081e530b18111a3-notifications-news/0f9b2d5d1c3d495d9081e530b18111a3-notifications-news-partition-4 / f6f49168781f402c99ddfa871bc0e90c-fury-news-sender-android-meli.notifications-providers] Error reading entries at 1199746:2412 : org.apache.bookkeeper.client.BKException$BKTimeoutException, Read Type Replay - Retrying to read in 56.043 seconds org.apache.bookkeeper.mledger.ManagedLedgerException: org.apache.bookkeeper.client.BKException$BKTimeoutException Caused by: org.apache.bookkeeper.client.BKException$BKTimeoutException at org.apache.bookkeeper.client.BKException.create(BKException.java:100) at org.apache.bookkeeper.mledger.impl.EntryCacheImpl.lambda$asyncReadEntry$1(EntryCacheImpl.java:187) at org.apache.bookkeeper.client.PendingReadOp.submitCallback(PendingReadOp.java:430) at org.apache.bookkeeper.client.PendingReadOp.access$000(PendingReadOp.java:59) at org.apache.bookkeeper.client.PendingReadOp$LedgerEntryRequest.sendNextRead(PendingReadOp.java:171) at org.apache.bookkeeper.client.PendingReadOp$LedgerEntryRequest.logErrorAndReattemptRead(PendingReadOp.java:227) at org.apache.bookkeeper.client.PendingReadOp.readEntryComplete(PendingReadOp.java:380) at org.apache.bookkeeper.proto.PerChannelBookieClient$ReadCompletion.readEntryComplete(PerChannelBookieClient.java:1116) at org.apache.bookkeeper.proto.PerChannelBookieClient.handleReadResponse(PerChannelBookieClient.java:1048) at org.apache.bookkeeper.proto.PerChannelBookieClient$ReadV2ResponseCallback.safeRun(PerChannelBookieClient.java:908) at org.apache.bookkeeper.util.SafeRunnable.run(SafeRunnable.java:31) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) at java.lang.Thread.run(Thread.java:748) ``` Thank you so much for the help!
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
