*Motivation*
Following stack trace is observed in CI.
```
Error Message
java.io.IOException: java.util.concurrent.RejectedExecutionException: Task
org.apache.bookkeeper.mledger.util.SafeRun$1@7ec176da rejected from
java.util.concurrent.ThreadPoolExecutor@33d2f9d3[Terminated, pool size = 0,
active threads = 0, queued tasks = 0, completed tasks = 2377]
Stacktrace
org.apache.pulsar.broker.PulsarServerException: java.io.IOException:
java.util.concurrent.RejectedExecutionException: Task
org.apache.bookkeeper.mledger.util.SafeRun$1@7ec176da rejected from
java.util.concurrent.ThreadPoolExecutor@33d2f9d3[Terminated, pool size = 0,
active threads = 0, queued tasks = 0, completed tasks = 2377]
at org.apache.pulsar.broker.PulsarService.close(PulsarService.java:298)
at
org.apache.pulsar.functions.worker.PulsarWorkerAssignmentTest.shutdown(PulsarWorkerAssignmentTest.java:147)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:124)
at
org.testng.internal.MethodInvocationHelper.invokeMethodConsideringTimeout(MethodInvocationHelper.java:59)
at
org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:451)
at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:222)
at org.testng.internal.Invoker.invokeMethod(Invoker.java:634)
at org.testng.internal.Invoker.retryFailed(Invoker.java:829)
at org.testng.internal.Invoker.invokeTestMethods(Invoker.java:1000)
at
org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:125)
at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:109)
at org.testng.TestRunner.privateRun(TestRunner.java:648)
at org.testng.TestRunner.run(TestRunner.java:505)
at org.testng.SuiteRunner.runTest(SuiteRunner.java:455)
at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:450)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:415)
at org.testng.SuiteRunner.run(SuiteRunner.java:364)
at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:84)
at org.testng.TestNG.runSuitesSequentially(TestNG.java:1187)
at org.testng.TestNG.runSuitesLocally(TestNG.java:1116)
at org.testng.TestNG.runSuites(TestNG.java:1028)
at org.testng.TestNG.run(TestNG.java:996)
at
org.apache.maven.surefire.testng.TestNGExecutor.run(TestNGExecutor.java:135)
at
org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.executeSingleClass(TestNGDirectoryTestSuite.java:112)
at
org.apache.maven.surefire.testng.TestNGDirectoryTestSuite.execute(TestNGDirectoryTestSuite.java:99)
at
org.apache.maven.surefire.testng.TestNGProvider.invoke(TestNGProvider.java:146)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:379)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:340)
at
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:125)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:413)
Caused by: java.io.IOException:
java.util.concurrent.RejectedExecutionException: Task
org.apache.bookkeeper.mledger.util.SafeRun$1@7ec176da rejected from
java.util.concurrent.ThreadPoolExecutor@33d2f9d3[Terminated, pool size = 0,
active threads = 0, queued tasks = 0, completed tasks = 2377]
at
org.apache.pulsar.broker.ManagedLedgerClientFactory.close(ManagedLedgerClientFactory.java:70)
at org.apache.pulsar.broker.PulsarService.close(PulsarService.java:225)
... 34 more
Caused by: java.util.concurrent.RejectedExecutionException: Task
org.apache.bookkeeper.mledger.util.SafeRun$1@7ec176da rejected from
java.util.concurrent.ThreadPoolExecutor@33d2f9d3[Terminated, pool size = 0,
active threads = 0, queued tasks = 0, completed tasks = 2377]
at
java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
at
com.google.common.util.concurrent.ForwardingExecutorService.execute(ForwardingExecutorService.java:99)
at
org.apache.bookkeeper.common.util.BoundedExecutorService.execute(BoundedExecutorService.java:89)
at
org.apache.bookkeeper.common.util.OrderedExecutor.executeOrdered(OrderedExecutor.java:291)
at
org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl.lambda$initializeBookKeeper$2(ManagedLedgerImpl.java:376)
at
org.apache.bookkeeper.client.LedgerCreateOp.createComplete(LedgerCreateOp.java:240)
at
org.apache.bookkeeper.client.LedgerCreateOp.operationComplete(LedgerCreateOp.java:199)
at
org.apache.bookkeeper.client.LedgerCreateOp.operationComplete(LedgerCreateOp.java:59)
at
org.apache.bookkeeper.meta.CleanupLedgerManager.close(CleanupLedgerManager.java:232)
at org.apache.bookkeeper.client.BookKeeper.close(BookKeeper.java:1467)
at
org.apache.pulsar.broker.ManagedLedgerClientFactory.close(ManagedLedgerClientFactory.java:66)
... 35 more
```
The race condition happens when:
when closing bookkeeper client, it will error outs all pending metadata
operations.
those callbacks of those operations will be triggered, and submitted to the
scheduler
in managed ledger factory. but the managed ledger factory has been shutdown
before,
so `RejectedExecutionException` will be thrown there.
we can safely ignore this exception.
*Changes*
Catch RejectedExecutionException and ignore it
After your change, what will change.
[ Full content available at: https://github.com/apache/pulsar/pull/2646 ]
This message was relayed via gitbox.apache.org for [email protected]
