TianFeng111 commented on issue #17076: URL: https://github.com/apache/pulsar/issues/17076#issuecomment-1214555603
> @TianFeng111 Do you know why so many reset cursor requests? This morning ,we observed that the problem recurs, some supplementary descriptions below 1、It's a delay queue, consumers will delay consumption by one minute. 2、This topics is a partitioned-topic, we only find this problem on partition-2. 3、Firlstly, we observed backlog sudden increaseand storage size reduction，at the same time we got so many reset cursor request in this broker's log.( Log is attached at the bottom) 4、Last week, as this issue described, we foud consumers on this partition got some acked history msg and at the same time, the backlog of the sudden increase returned to normal., which would affect downstream services. 5、But today ,after we foud the phenomenon described in point 3 (problems recurs probably)，the consumers on this partition seemed to be working fine, but the backlog still high .(monitor of this partition attached at the bottom) 6、We just observed the above phenomenon, but do not know it's cause and causality. What caused the initial backlog sudden increase? Here are some log of this partition's broker and the monitor of this partition(today) below:  2022-08-15T08:57:44,581+0800 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [ACE/0/persistent/0_949687666_20482_2014005_0-partition-0] Initiate reset position to 60734:1 on cursor ACE_SUBSCRIPTION_1001_10804_2515 2022-08-15T08:57:44,586+0800 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [ACE/0/persistent/0_949687666_20482_2014005_0-partition-0] reset position to 60734:1 before current read position 60734:1 on cursor ACE_SUBSCRIPTION_1001_10804_2515 2022-08-15T08:57:44,586+0800 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [ACE/0/persistent/0_949687666_20482_2014005_0-partition-0] Initiate reset position to 60734:1 on cursor ACE_SUBSCRIPTION_1001_10804_2515 2022-08-15T08:57:44,588+0800 [pulsar-io-4-4] INFO org.apache.pulsar.broker.service.ServerCnx - [/30.43.156.124:41976][persistent://ACE/0/0_949687666_20482_2014005_0-partition-0] Creating producer. producerId=274662 2022-08-15T08:57:44,591+0800 [Thread-69266] INFO org.apache.pulsar.broker.service.ServerCnx - [/30.43.156.124:41976] persistent://ACE/0/0_949687666_20482_2014005_0-partition-0 configured with schema false 2022-08-15T08:57:44,591+0800 [Thread-69266] INFO org.apache.pulsar.broker.service.ServerCnx - [/30.43.156.124:41976] Created new producer: Producer{topic=PersistentTopic{topic=persistent://ACE/0/0_949687666_20482_2014005_0-partition-0}, client=/30.43.156.124:41976, producerName=ACE_PRODUCER_1001_10821_5714_3813511, producerId=274662} 2022-08-15T08:57:44,592+0800 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [ACE/0/persistent/0_949687666_20482_2014005_0-partition-0] reset position to 60734:1 before current read position 60734:1 on cursor ACE_SUBSCRIPTION_1001_10804_2515 2022-08-15T08:57:44,592+0800 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [ACE/0/persistent/0_949687666_20482_2014005_0-partition-0] Initiate reset position to 60734:1 on cursor ACE_SUBSCRIPTION_1001_10804_2515 2022-08-15T08:57:44,597+0800 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [ACE/0/persistent/0_949687666_20482_2014005_0-partition-0] reset position to 60734:1 before current read position 60734:1 on cursor ACE_SUBSCRIPTION_1001_10804_2515 2022-08-15T08:57:44,597+0800 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [ACE/0/persistent/0_949687666_20482_2014005_0-partition-0] Initiate reset position to 60734:1 on cursor ACE_SUBSCRIPTION_1001_10804_2515 2022-08-15T08:57:44,600+0800 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [ACE/0/persistent/0_949687666_20482_2014005_0-partition-0] Creating a new ledger 2022-08-15T08:57:44,600+0800 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [ACE/0/persistent/0_949687666_20482_2014005_0-partition-0] Creating ledger, metadata: {component=[109, 97, 110, 97, 103, 101, 100, 45, 108, 101, 100, 103, 101, 114], pulsar/managed-ledger=[65, 67, 69, 47, 48, 47, 112, 101, 114, 115, 105, 115, 116, 101, 110, 116, 47, 48, 95, 57, 52, 57, 54, 56, 55, 54, 54, 54, 95, 50, 48, 52, 56, 50, 95, 50, 48, 49, 52, 48, 48, 53, 95, 48, 45, 112, 97, 114, 116, 105, 116, 105, 111, 110, 45, 48], application=[112, 117, 108, 115, 97, 114]} - metadata ops timeout : 60 seconds 2022-08-15T08:57:44,600+0800 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.bookkeeper.client.BookieWatcherImpl - New ensemble: [11.177.85.74:3181, 11.186.71.172:3181, 11.177.86.228:3181, 11.177.87.59:3181] is not adhering to Placement Policy. quarantinedBookies: [] 2022-08-15T08:57:44,604+0800 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [ACE/0/persistent/0_949687666_20482_2014005_0-partition-0] reset position to 60734:1 before current read position 60734:1 on cursor ACE_SUBSCRIPTION_1001_10804_2515 2022-08-15T08:57:44,604+0800 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [ACE/0/persistent/0_949687666_20482_2014005_0-partition-0] Initiate reset position to 60734:1 on cursor ACE_SUBSCRIPTION_1001_10804_2515 2022-08-15T08:57:44,608+0800 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [ACE/0/persistent/0_949687666_20482_2014005_0-partition-0] reset position to 60734:1 before current read position 60734:1 on cursor ACE_SUBSCRIPTION_1001_10804_2515 2022-08-15T08:57:44,608+0800 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [ACE/0/persistent/0_949687666_20482_2014005_0-partition-0] Initiate reset position to 60734:1 on cursor ACE_SUBSCRIPTION_1001_10804_2515 2022-08-15T08:57:44,612+0800 [main-EventThread] INFO org.apache.bookkeeper.client.LedgerCreateOp - Ensemble: [11.177.85.74:3181, 11.186.71.172:3181, 11.177.86.228:3181, 11.177.87.59:3181] for ledger: 107006 2022-08-15T08:57:44,612+0800 [main-EventThread] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [ACE/0/persistent/0_949687666_20482_2014005_0-partition-0] Created new ledger 107006 2022-08-15T08:57:44,613+0800 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [ACE/0/persistent/0_949687666_20482_2014005_0-partition-0] reset position to 60734:1 before current read position 60734:1 on cursor ACE_SUBSCRIPTION_1001_10804_2515 2022-08-15T08:57:44,613+0800 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [ACE/0/persistent/0_949687666_20482_2014005_0-partition-0] Initiate reset position to 60734:1 on cursor ACE_SUBSCRIPTION_1001_10804_2515 2022-08-15T08:57:44,618+0800 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [ACE/0/persistent/0_949687666_20482_2014005_0-partition-0] reset position to 60734:1 before current read position 60734:1 on cursor ACE_SUBSCRIPTION_1001_10804_2515 2022-08-15T08:57:44,618+0800 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [ACE/0/persistent/0_949687666_20482_2014005_0-partition-0] Initiate reset position to 60734:1 on cursor ACE_SUBSCRIPTION_1001_10804_2515 2022-08-15T08:57:44,618+0800 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - Reset cursor:ManagedCursorImpl{ledger=ACE/0/persistent/0_949687666_20482_2014005_0-partition-0, name=ACE_SUBSCRIPTION_1001_10804_2515, ackPos=60734:0, readPos=60734:1} to 107006:-1 since ledger consumed completely 2022-08-15T08:57:44,618+0800 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [ACE/0/persistent/0_949687666_20482_2014005_0-partition-0] Ledger 60734 contains the current last confirmed entry 60734:0, and it is going to be deleted 2022-08-15T08:57:44,622+0800 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [ACE/0/persistent/0_949687666_20482_2014005_0-partition-0] End TrimConsumedLedgers. ledgers=1 totalSize=0 2022-08-15T08:57:44,622+0800 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [ACE/0/persistent/0_949687666_20482_2014005_0-partition-0] Removing ledger 60734 - size: 552 2022-08-15T08:57:44,622+0800 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [ACE/0/persistent/0_949687666_20482_2014005_0-partition-0] reset position to 60734:1 before current read position 60734:1 on cursor ACE_SUBSCRIPTION_1001_10804_2515 2022-08-15T08:57:44,623+0800 [pulsar-backlog-quota-checker-26-1] ERROR org.apache.pulsar.broker.service.BacklogQuotaManager - [persistent://ACE/0/0_949687666_20482_2014005_0-partition-0] Error resetting cursor for slowest consumer [ACE_SUBSCRIPTION_1001_10804_2515] java.lang.NullPointerException: null 2022-08-15T08:57:44,623+0800 [pulsar-backlog-quota-checker-26-1] INFO org.apache.pulsar.broker.service.BacklogQuotaManager - Backlog quota type message_age exceeded for topic [persistent://ACE/0/0_0_20483_isasync_0-partition-2]. Applying [consumer_backlog_eviction] policy 2022-08-15T08:57:44,623+0800 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [ACE/0/persistent/0_0_20483_isasync_0-partition-2] Initiate reset position to 88545:0 on cursor ACE_SUBSCRIPTION_1001_10597_2511 2022-08-15T08:57:44,628+0800 [BookKeeperClientWorker-OrderedExecutor-1-0] INFO org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [ACE/0/persistent/0_0_20483_isasync_0-partition-2] reset position to 88545:0 before current read position 105731:17659 on cursor ACE_SUBSCRIPTION_1001_10597_2511 2022-08-15T08:57:44,628+0800 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [ACE/0/persistent/0_0_20483_isasync_0-partition-2] Failed to mark delete while trimming data ledgers: Reset cursor in progress - unable to mark delete position 88545:-1 2022-08-15T08:57:44,628+0800 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [ACE/0/persistent/0_0_20483_isasync_0-partition-2] Initiate reset position to 88545:0 on cursor ACE_SUBSCRIPTION_1001_10597_2511 2022-08-15T08:57:44,632+0800 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [ACE/0/persistent/0_0_20483_isasync_0-partition-2] End TrimConsumedLedgers. ledgers=11 totalSize=915274097 2022-08-15T08:57:44,632+0800 [bookkeeper-ml-scheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [ACE/0/persistent/0_0_20483_isasync_0-partition-2] Removing ledger 87121 - size: 102888483 2022-08-15T08:57:44,632+0800 [BookKeeperClientWorker-OrderedExecutor-1-0] INFO org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [ACE/0/persistent/0_0_20483_isasync_0-partition-2] reset position to 88545:0 before current read position 105731:17659 on cursor ACE_SUBSCRIPTION_1001_10597_2511 2022-08-15T08:57:44,632+0800 [BookKeeperClientWorker-OrderedExecutor-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [ACE/0/persistent/0_0_20483_isasync_0-partition-2] Initiate reset position to 90184:0 on cursor ACE_SUBSCRIPTION_1001_10597_2511 2022-08-15T08:57:44,637+0800 [BookKeeperClientWorker-OrderedExecutor-0-0] WARN org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [ACE/0/persistent/0_0_20483_isasync_0-partition-2] Failed to mark delete while trimming data ledgers: Reset cursor in progress - unable to mark delete position 90184:-1 2022-08-15T08:57:44,637+0800 [BookKeeperClientWorker-OrderedExecutor-1-0] INFO org.apache.bookkeeper.mledger.impl.ManagedCursorImpl - [ACE/0/persistent/0_0_20483_isasync_0-partition-2] reset position to 90184:0 skipping from current read position 88545:0 on cursor ACE_SUBSCRIPTION_1001_10597_2511 2022-08-15T08:57:44,638+0800 [pulsar-backlog-quota-checker-26-1] ERROR org.apache.pulsar.broker.service.BacklogQuotaManager - [persistent://ACE/0/0_0_20483_isasync_0-partition-2] Error resetting cursor for slowest consumer [ACE_SUBSCRIPTION_1001_10597_2511] java.lang.NullPointerException: null at org.apache.pulsar.broker.service.BacklogQuotaManager.dropBacklogForTimeLimit(BacklogQuotaManager.java:218) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at org.apache.pulsar.broker.service.BacklogQuotaManager.handleExceededBacklogQuota(BacklogQuotaManager.java:98) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at org.apache.pulsar.broker.service.BrokerService.lambda$null$79(BrokerService.java:1768) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:670) ~[?:1.8.0_292] at java.util.concurrent.CompletableFuture.uniAcceptStage(CompletableFuture.java:683) ~[?:1.8.0_292] at java.util.concurrent.CompletableFuture.thenAccept(CompletableFuture.java:2010) ~[?:1.8.0_292] at org.apache.pulsar.broker.service.BrokerService.lambda$monitorBacklogQuota$81(BrokerService.java:1766) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at java.util.Optional.ifPresent(Optional.java:159) ~[?:1.8.0_292] at org.apache.pulsar.broker.service.BrokerService.lambda$forEachTopic$78(BrokerService.java:1750) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:413) ~[org.apache.pulsar-pulsar-common-2.10.0.jar:2.10.0] at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:185) ~[org.apache.pulsar-pulsar-common-2.10.0.jar:2.10.0] at org.apache.pulsar.broker.service.BrokerService.forEachTopic(BrokerService.java:1748) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at org.apache.pulsar.broker.service.BrokerService.monitorBacklogQuota(BrokerService.java:1759) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at org.apache.bookkeeper.mledger.util.SafeRun$1.safeRun(SafeRun.java:32) [org.apache.pulsar-managed-ledger-2.10.0.jar:2.10.0] at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [org.apache.bookkeeper-bookkeeper-common-4.14.4.jar:4.14.4] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_292] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_292] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_292] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_292] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_292] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_292] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292] 2022-08-15T08:57:44,638+0800 [pulsar-backlog-quota-checker-26-1] INFO org.apache.pulsar.broker.service.BacklogQuotaManager - Backlog quota type message_age exceeded for topic [persistent://ACE/0/0_0_20483_isasync_0-partition-2]. Applying [consumer_backlog_eviction] policy 2022-08-15T08:57:44,639+0800 [pulsar-backlog-quota-checker-26-1] ERROR org.apache.pulsar.broker.service.BacklogQuotaManager - [persistent://ACE/0/0_0_20483_isasync_0-partition-2] Error resetting cursor for slowest consumer [ACE_SUBSCRIPTION_1001_10597_2511] java.lang.NullPointerException: null at org.apache.pulsar.broker.service.BacklogQuotaManager.dropBacklogForTimeLimit(BacklogQuotaManager.java:218) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at org.apache.pulsar.broker.service.BacklogQuotaManager.handleExceededBacklogQuota(BacklogQuotaManager.java:98) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at org.apache.pulsar.broker.service.BrokerService.lambda$null$79(BrokerService.java:1768) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:670) ~[?:1.8.0_292] at java.util.concurrent.CompletableFuture.uniAcceptStage(CompletableFuture.java:683) ~[?:1.8.0_292] at java.util.concurrent.CompletableFuture.thenAccept(CompletableFuture.java:2010) ~[?:1.8.0_292] at org.apache.pulsar.broker.service.BrokerService.lambda$monitorBacklogQuota$81(BrokerService.java:1766) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at java.util.Optional.ifPresent(Optional.java:159) ~[?:1.8.0_292] at org.apache.pulsar.broker.service.BrokerService.lambda$forEachTopic$78(BrokerService.java:1750) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:413) ~[org.apache.pulsar-pulsar-common-2.10.0.jar:2.10.0] at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:185) ~[org.apache.pulsar-pulsar-common-2.10.0.jar:2.10.0] at org.apache.pulsar.broker.service.BrokerService.forEachTopic(BrokerService.java:1748) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at org.apache.pulsar.broker.service.BrokerService.monitorBacklogQuota(BrokerService.java:1759) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at org.apache.bookkeeper.mledger.util.SafeRun$1.safeRun(SafeRun.java:32) [org.apache.pulsar-managed-ledger-2.10.0.jar:2.10.0] at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [org.apache.bookkeeper-bookkeeper-common-4.14.4.jar:4.14.4] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_292] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_292] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_292] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_292] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_292] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_292] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292] 2022-08-15T08:57:44,639+0800 [pulsar-backlog-quota-checker-26-1] INFO org.apache.pulsar.broker.service.BacklogQuotaManager - Backlog quota type message_age exceeded for topic [persistent://ACE/0/0_0_20483_isasync_0-partition-2]. Applying [consumer_backlog_eviction] policy 2022-08-15T08:57:44,639+0800 [pulsar-backlog-quota-checker-26-1] ERROR org.apache.pulsar.broker.service.BacklogQuotaManager - [persistent://ACE/0/0_0_20483_isasync_0-partition-2] Error resetting cursor for slowest consumer [ACE_SUBSCRIPTION_1001_10597_2511] java.lang.NullPointerException: null at org.apache.pulsar.broker.service.BacklogQuotaManager.dropBacklogForTimeLimit(BacklogQuotaManager.java:218) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at org.apache.pulsar.broker.service.BacklogQuotaManager.handleExceededBacklogQuota(BacklogQuotaManager.java:98) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at org.apache.pulsar.broker.service.BrokerService.lambda$null$79(BrokerService.java:1768) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:670) ~[?:1.8.0_292] at java.util.concurrent.CompletableFuture.uniAcceptStage(CompletableFuture.java:683) ~[?:1.8.0_292] at java.util.concurrent.CompletableFuture.thenAccept(CompletableFuture.java:2010) ~[?:1.8.0_292] at org.apache.pulsar.broker.service.BrokerService.lambda$monitorBacklogQuota$81(BrokerService.java:1766) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at java.util.Optional.ifPresent(Optional.java:159) ~[?:1.8.0_292] at org.apache.pulsar.broker.service.BrokerService.lambda$forEachTopic$78(BrokerService.java:1750) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:413) ~[org.apache.pulsar-pulsar-common-2.10.0.jar:2.10.0] at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:185) ~[org.apache.pulsar-pulsar-common-2.10.0.jar:2.10.0] at org.apache.pulsar.broker.service.BrokerService.forEachTopic(BrokerService.java:1748) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at org.apache.pulsar.broker.service.BrokerService.monitorBacklogQuota(BrokerService.java:1759) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at org.apache.bookkeeper.mledger.util.SafeRun$1.safeRun(SafeRun.java:32) [org.apache.pulsar-managed-ledger-2.10.0.jar:2.10.0] at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [org.apache.bookkeeper-bookkeeper-common-4.14.4.jar:4.14.4] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_292] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_292] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_292] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_292] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_292] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_292] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292] 2022-08-15T08:57:44,640+0800 [pulsar-backlog-quota-checker-26-1] INFO org.apache.pulsar.broker.service.BacklogQuotaManager - Backlog quota type message_age exceeded for topic [persistent://ACE/0/0_0_20483_isasync_0-partition-2]. Applying [consumer_backlog_eviction] policy 2022-08-15T08:57:44,640+0800 [pulsar-backlog-quota-checker-26-1] ERROR org.apache.pulsar.broker.service.BacklogQuotaManager - [persistent://ACE/0/0_0_20483_isasync_0-partition-2] Error resetting cursor for slowest consumer [ACE_SUBSCRIPTION_1001_10597_2511] java.lang.NullPointerException: null at org.apache.pulsar.broker.service.BacklogQuotaManager.dropBacklogForTimeLimit(BacklogQuotaManager.java:218) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at org.apache.pulsar.broker.service.BacklogQuotaManager.handleExceededBacklogQuota(BacklogQuotaManager.java:98) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at org.apache.pulsar.broker.service.BrokerService.lambda$null$79(BrokerService.java:1768) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:670) ~[?:1.8.0_292] at java.util.concurrent.CompletableFuture.uniAcceptStage(CompletableFuture.java:683) ~[?:1.8.0_292] at java.util.concurrent.CompletableFuture.thenAccept(CompletableFuture.java:2010) ~[?:1.8.0_292] at org.apache.pulsar.broker.service.BrokerService.lambda$monitorBacklogQuota$81(BrokerService.java:1766) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at java.util.Optional.ifPresent(Optional.java:159) ~[?:1.8.0_292] at org.apache.pulsar.broker.service.BrokerService.lambda$forEachTopic$78(BrokerService.java:1750) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:413) ~[org.apache.pulsar-pulsar-common-2.10.0.jar:2.10.0] at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:185) ~[org.apache.pulsar-pulsar-common-2.10.0.jar:2.10.0] at org.apache.pulsar.broker.service.BrokerService.forEachTopic(BrokerService.java:1748) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at org.apache.pulsar.broker.service.BrokerService.monitorBacklogQuota(BrokerService.java:1759) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at org.apache.bookkeeper.mledger.util.SafeRun$1.safeRun(SafeRun.java:32) [org.apache.pulsar-managed-ledger-2.10.0.jar:2.10.0] at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [org.apache.bookkeeper-bookkeeper-common-4.14.4.jar:4.14.4] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_292] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_292] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_292] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_292] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_292] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_292] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292] 2022-08-15T08:57:44,640+0800 [pulsar-backlog-quota-checker-26-1] INFO org.apache.pulsar.broker.service.BacklogQuotaManager - Backlog quota type message_age exceeded for topic [persistent://ACE/0/0_0_20483_isasync_0-partition-2]. Applying [consumer_backlog_eviction] policy 2022-08-15T08:57:44,640+0800 [pulsar-backlog-quota-checker-26-1] ERROR org.apache.pulsar.broker.service.BacklogQuotaManager - [persistent://ACE/0/0_0_20483_isasync_0-partition-2] Error resetting cursor for slowest consumer [ACE_SUBSCRIPTION_1001_10597_2511] java.lang.NullPointerException: null at org.apache.pulsar.broker.service.BacklogQuotaManager.dropBacklogForTimeLimit(BacklogQuotaManager.java:218) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at org.apache.pulsar.broker.service.BacklogQuotaManager.handleExceededBacklogQuota(BacklogQuotaManager.java:98) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at org.apache.pulsar.broker.service.BrokerService.lambda$null$79(BrokerService.java:1768) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:670) ~[?:1.8.0_292] at java.util.concurrent.CompletableFuture.uniAcceptStage(CompletableFuture.java:683) ~[?:1.8.0_292] at java.util.concurrent.CompletableFuture.thenAccept(CompletableFuture.java:2010) ~[?:1.8.0_292] at org.apache.pulsar.broker.service.BrokerService.lambda$monitorBacklogQuota$81(BrokerService.java:1766) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at java.util.Optional.ifPresent(Optional.java:159) ~[?:1.8.0_292] at org.apache.pulsar.broker.service.BrokerService.lambda$forEachTopic$78(BrokerService.java:1750) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:413) ~[org.apache.pulsar-pulsar-common-2.10.0.jar:2.10.0] at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:185) ~[org.apache.pulsar-pulsar-common-2.10.0.jar:2.10.0] at org.apache.pulsar.broker.service.BrokerService.forEachTopic(BrokerService.java:1748) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at org.apache.pulsar.broker.service.BrokerService.monitorBacklogQuota(BrokerService.java:1759) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at org.apache.bookkeeper.mledger.util.SafeRun$1.safeRun(SafeRun.java:32) [org.apache.pulsar-managed-ledger-2.10.0.jar:2.10.0] at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [org.apache.bookkeeper-bookkeeper-common-4.14.4.jar:4.14.4] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_292] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_292] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_292] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_292] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_292] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_292] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292] 2022-08-15T08:57:44,641+0800 [pulsar-backlog-quota-checker-26-1] INFO org.apache.pulsar.broker.service.BacklogQuotaManager - Backlog quota type message_age exceeded for topic [persistent://ACE/0/0_0_20483_isasync_0-partition-2]. Applying [consumer_backlog_eviction] policy 2022-08-15T08:57:44,641+0800 [pulsar-backlog-quota-checker-26-1] ERROR org.apache.pulsar.broker.service.BacklogQuotaManager - [persistent://ACE/0/0_0_20483_isasync_0-partition-2] Error resetting cursor for slowest consumer [ACE_SUBSCRIPTION_1001_10597_2511] java.lang.NullPointerException: null at org.apache.pulsar.broker.service.BacklogQuotaManager.dropBacklogForTimeLimit(BacklogQuotaManager.java:218) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at org.apache.pulsar.broker.service.BacklogQuotaManager.handleExceededBacklogQuota(BacklogQuotaManager.java:98) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at org.apache.pulsar.broker.service.BrokerService.lambda$null$79(BrokerService.java:1768) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:670) ~[?:1.8.0_292] at java.util.concurrent.CompletableFuture.uniAcceptStage(CompletableFuture.java:683) ~[?:1.8.0_292] at java.util.concurrent.CompletableFuture.thenAccept(CompletableFuture.java:2010) ~[?:1.8.0_292] at org.apache.pulsar.broker.service.BrokerService.lambda$monitorBacklogQuota$81(BrokerService.java:1766) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at java.util.Optional.ifPresent(Optional.java:159) ~[?:1.8.0_292] at org.apache.pulsar.broker.service.BrokerService.lambda$forEachTopic$78(BrokerService.java:1750) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:413) ~[org.apache.pulsar-pulsar-common-2.10.0.jar:2.10.0] at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:185) ~[org.apache.pulsar-pulsar-common-2.10.0.jar:2.10.0] at org.apache.pulsar.broker.service.BrokerService.forEachTopic(BrokerService.java:1748) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at org.apache.pulsar.broker.service.BrokerService.monitorBacklogQuota(BrokerService.java:1759) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at org.apache.bookkeeper.mledger.util.SafeRun$1.safeRun(SafeRun.java:32) [org.apache.pulsar-managed-ledger-2.10.0.jar:2.10.0] at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [org.apache.bookkeeper-bookkeeper-common-4.14.4.jar:4.14.4] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_292] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_292] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_292] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_292] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_292] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_292] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292] 2022-08-15T08:57:44,641+0800 [pulsar-backlog-quota-checker-26-1] INFO org.apache.pulsar.broker.service.BacklogQuotaManager - Backlog quota type message_age exceeded for topic [persistent://ACE/0/0_0_20483_isasync_0-partition-2]. Applying [consumer_backlog_eviction] policy 2022-08-15T08:57:44,641+0800 [pulsar-backlog-quota-checker-26-1] ERROR org.apache.pulsar.broker.service.BacklogQuotaManager - [persistent://ACE/0/0_0_20483_isasync_0-partition-2] Error resetting cursor for slowest consumer [ACE_SUBSCRIPTION_1001_10597_2511] java.lang.NullPointerException: null at org.apache.pulsar.broker.service.BacklogQuotaManager.dropBacklogForTimeLimit(BacklogQuotaManager.java:218) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at org.apache.pulsar.broker.service.BacklogQuotaManager.handleExceededBacklogQuota(BacklogQuotaManager.java:98) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at org.apache.pulsar.broker.service.BrokerService.lambda$null$79(BrokerService.java:1768) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:670) ~[?:1.8.0_292] at java.util.concurrent.CompletableFuture.uniAcceptStage(CompletableFuture.java:683) ~[?:1.8.0_292] at java.util.concurrent.CompletableFuture.thenAccept(CompletableFuture.java:2010) ~[?:1.8.0_292] at org.apache.pulsar.broker.service.BrokerService.lambda$monitorBacklogQuota$81(BrokerService.java:1766) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at java.util.Optional.ifPresent(Optional.java:159) ~[?:1.8.0_292] at org.apache.pulsar.broker.service.BrokerService.lambda$forEachTopic$78(BrokerService.java:1750) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:413) ~[org.apache.pulsar-pulsar-common-2.10.0.jar:2.10.0] at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:185) ~[org.apache.pulsar-pulsar-common-2.10.0.jar:2.10.0] at org.apache.pulsar.broker.service.BrokerService.forEachTopic(BrokerService.java:1748) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at org.apache.pulsar.broker.service.BrokerService.monitorBacklogQuota(BrokerService.java:1759) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at org.apache.bookkeeper.mledger.util.SafeRun$1.safeRun(SafeRun.java:32) [org.apache.pulsar-managed-ledger-2.10.0.jar:2.10.0] at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [org.apache.bookkeeper-bookkeeper-common-4.14.4.jar:4.14.4] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_292] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_292] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_292] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_292] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_292] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_292] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292] 2022-08-15T08:57:44,641+0800 [pulsar-backlog-quota-checker-26-1] INFO org.apache.pulsar.broker.service.BacklogQuotaManager - Backlog quota type message_age exceeded for topic [persistent://ACE/0/0_0_20483_isasync_0-partition-2]. Applying [consumer_backlog_eviction] policy 2022-08-15T08:57:44,641+0800 [pulsar-backlog-quota-checker-26-1] ERROR org.apache.pulsar.broker.service.BacklogQuotaManager - [persistent://ACE/0/0_0_20483_isasync_0-partition-2] Error resetting cursor for slowest consumer [ACE_SUBSCRIPTION_1001_10597_2511] java.lang.NullPointerException: null at org.apache.pulsar.broker.service.BacklogQuotaManager.dropBacklogForTimeLimit(BacklogQuotaManager.java:218) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at org.apache.pulsar.broker.service.BacklogQuotaManager.handleExceededBacklogQuota(BacklogQuotaManager.java:98) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at org.apache.pulsar.broker.service.BrokerService.lambda$null$79(BrokerService.java:1768) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:670) ~[?:1.8.0_292] at java.util.concurrent.CompletableFuture.uniAcceptStage(CompletableFuture.java:683) ~[?:1.8.0_292] at java.util.concurrent.CompletableFuture.thenAccept(CompletableFuture.java:2010) ~[?:1.8.0_292] at org.apache.pulsar.broker.service.BrokerService.lambda$monitorBacklogQuota$81(BrokerService.java:1766) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at java.util.Optional.ifPresent(Optional.java:159) ~[?:1.8.0_292] at org.apache.pulsar.broker.service.BrokerService.lambda$forEachTopic$78(BrokerService.java:1750) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:413) ~[org.apache.pulsar-pulsar-common-2.10.0.jar:2.10.0] at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:185) ~[org.apache.pulsar-pulsar-common-2.10.0.jar:2.10.0] at org.apache.pulsar.broker.service.BrokerService.forEachTopic(BrokerService.java:1748) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at org.apache.pulsar.broker.service.BrokerService.monitorBacklogQuota(BrokerService.java:1759) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at org.apache.bookkeeper.mledger.util.SafeRun$1.safeRun(SafeRun.java:32) [org.apache.pulsar-managed-ledger-2.10.0.jar:2.10.0] at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [org.apache.bookkeeper-bookkeeper-common-4.14.4.jar:4.14.4] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_292] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_292] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_292] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_292] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_292] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_292] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292] 2022-08-15T08:57:44,642+0800 [pulsar-backlog-quota-checker-26-1] INFO org.apache.pulsar.broker.service.BacklogQuotaManager - Backlog quota type message_age exceeded for topic [persistent://ACE/0/0_0_20483_isasync_0-partition-2]. Applying [consumer_backlog_eviction] policy 2022-08-15T08:57:44,642+0800 [pulsar-backlog-quota-checker-26-1] ERROR org.apache.pulsar.broker.service.BacklogQuotaManager - [persistent://ACE/0/0_0_20483_isasync_0-partition-2] Error resetting cursor for slowest consumer [ACE_SUBSCRIPTION_1001_10597_2511] java.lang.NullPointerException: null at org.apache.pulsar.broker.service.BacklogQuotaManager.dropBacklogForTimeLimit(BacklogQuotaManager.java:218) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at org.apache.pulsar.broker.service.BacklogQuotaManager.handleExceededBacklogQuota(BacklogQuotaManager.java:98) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at org.apache.pulsar.broker.service.BrokerService.lambda$null$79(BrokerService.java:1768) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:670) ~[?:1.8.0_292] at java.util.concurrent.CompletableFuture.uniAcceptStage(CompletableFuture.java:683) ~[?:1.8.0_292] at java.util.concurrent.CompletableFuture.thenAccept(CompletableFuture.java:2010) ~[?:1.8.0_292] at org.apache.pulsar.broker.service.BrokerService.lambda$monitorBacklogQuota$81(BrokerService.java:1766) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at java.util.Optional.ifPresent(Optional.java:159) ~[?:1.8.0_292] at org.apache.pulsar.broker.service.BrokerService.lambda$forEachTopic$78(BrokerService.java:1750) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap$Section.forEach(ConcurrentOpenHashMap.java:413) ~[org.apache.pulsar-pulsar-common-2.10.0.jar:2.10.0] at org.apache.pulsar.common.util.collections.ConcurrentOpenHashMap.forEach(ConcurrentOpenHashMap.java:185) ~[org.apache.pulsar-pulsar-common-2.10.0.jar:2.10.0] at org.apache.pulsar.broker.service.BrokerService.forEachTopic(BrokerService.java:1748) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at org.apache.pulsar.broker.service.BrokerService.monitorBacklogQuota(BrokerService.java:1759) ~[org.apache.pulsar-pulsar-broker-2.10.0.jar:2.10.0] at org.apache.bookkeeper.mledger.util.SafeRun$1.safeRun(SafeRun.java:32) [org.apache.pulsar-managed-ledger-2.10.0.jar:2.10.0] at org.apache.bookkeeper.common.util.SafeRunnable.run(SafeRunnable.java:36) [org.apache.bookkeeper-bookkeeper-common-4.14.4.jar:4.14.4] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_292] at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_292] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_292] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_292] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_292] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_292] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [io.netty-netty-common-4.1.74.Final.jar:4.1.74.Final] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_292] -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
