mattisonchao opened a new pull request #14486:
URL: https://github.com/apache/pulsar/pull/14486
### Motivation
I found a lot of stack trace logs when I use 25k consumers to consume
messages.
```java
2022-02-28T05:06:25,472+0000 [bookkeeper-ml-scheduler-OrderedScheduler-0-0]
WARN org.apache.bookkeeper.mledger.impl.ManagedCursorImpl -
[public/default/persistent/bench%252F24412][ip-10-0-0-132_bench_sub_24412_880606346]
Failed to flush mark-delete position
org.apache.bookkeeper.mledger.ManagedLedgerException:
java.lang.IllegalArgumentException: Mark deleting an already mark-deleted
position
Caused by: java.lang.IllegalArgumentException: Mark deleting an already
mark-deleted position
at
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl.setAcknowledgedPosition(ManagedCursorImpl.java:1580)
~[org.apache.pulsar-managed-ledger-2.9.1.jar:2.9.1]
at
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl.asyncMarkDelete(ManagedCursorImpl.java:1718)
~[org.apache.pulsar-managed-ledger-2.9.1.jar:2.9.1]
at
org.apache.bookkeeper.mledger.impl.ManagedCursorImpl.flush(ManagedCursorImpl.java:3020)
~[org.apache.pulsar-managed-ledger-2.9.1.jar:2.9.1]
at
org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl.lambda$flushCursors$1(ManagedLedgerFactoryImpl.java:232)
~[org.apache.pulsar-managed-ledger-2.9.1.jar:2.9.1]
at java.lang.Iterable.forEach(Iterable.java:75) ~[?:?]
at
org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl.lambda$flushCursors$2(ManagedLedgerFactoryImpl.java:232)
~[org.apache.pulsar-managed-ledger-2.9.1.jar:2.9.1]
at
java.util.concurrent.ConcurrentHashMap$ValuesView.forEach(ConcurrentHashMap.java:4772)
~[?:?]
at
org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl.flushCursors(ManagedLedgerFactoryImpl.java:228)
~[org.apache.pulsar-managed-ledger-2.9.1.jar:2.9.1]
at
org.apache.pulsar.common.util.Runnables$CatchingAndLoggingRunnable.run(Runnables.java:53)
[org.apache.pulsar-pulsar-common-2.9.1.jar:2.9.1]
at
org.apache.bookkeeper.common.util.OrderedExecutor$TimedRunnable.run(OrderedExecutor.java:203)
[org.apache.bookkeeper-bookkeeper-common-4.14.2.jar:4.14.2]
at
org.apache.bookkeeper.common.util.OrderedExecutor$TimedRunnable.run(OrderedExecutor.java:203)
[org.apache.bookkeeper-bookkeeper-common-4.14.2.jar:4.14.2]
at
com.google.common.util.concurrent.MoreExecutors$ScheduledListeningDecorator$NeverSuccessfulListenableFutureTask.run(MoreExecutors.java:705)
[com.google.guava-guava-30.1-jre.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
[?:?]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?]
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
[?:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[?:?]
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
[io.netty-netty-common-4.1.72.Final.jar:4.1.72.Final]
at java.lang.Thread.run(Thread.java:829) [?:?]
```
I think the problem is caused by
``ManagedCursorImpl#setAcknowledgedPosition`` being called in a concurrent
environment.
### Modifications
- Logging with debug level failed because the marker location was already
marked for deletion.
### Verifying this change
- [x] Make sure that the change passes the CI checks.
### Documentation
- [x] `no-need-doc`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
