zbentley opened a new issue #12551:
URL: https://github.com/apache/pulsar/issues/12551
**Describe the bug**
When testing, I often tear down tenants/namespaces/topics and re-create
those same tenants/namespaces/topics immediately thereafter to reset test state.
Sometimes (but not always) when I create a tenant/namespace/topic and then
immediately request their deletion, I get a 500 error back from the management
API, which correlates to the below stacktrace on the broker.
This should not happen; management API deletion operations should only
return when deletion of all related resources has completed.
**To Reproduce**
This doesn't reproduce 100% of the time. It seems likely to happen more
often when it is done on a brand new broker cluster that hasn't ever hosted
topics/non-default tenants.
1. Create a tenant.
2. Create a namespace in that tenant
3. Create a persistent, partitioned topic in that namespace.
4. Create a subscription in that topic and connect at least one consumer to
that subscription (no messages need to be produced or consumed).
5. Immediately thereafter, attempt to delete the topic/namespace/tenant.
**Expected behavior**
1. No errors occur.
2. This is probably more important: all operations related to
deletion/creation of topics/namespaces/tenants conclude *before* the management
API returns success codes to the user. This bug and the other similar ones I
filed (see github links below) all seem to arise from CRUD operations with the
management API being asynchronous: i.e. when I create/delete a
tenant/topic/namespace, the actual side effects of that creation or deletion
(e.g. adding/removing ledgers in BookKeeper, updating metadata in ZK) occur
later, not during the API post. Not only is that bound to cause bugs like this,
but it's also not what users expect; I would be happy to wait seconds or
minutes for management API operations to complete in exchange for knowing that
when they successfully complete that the thing I requested has actually been
done.
**Environment**
- Kubernetes on EKS 1.18. StreamNative Platform Pulsar Broker 2.1.8.5.
**Stacktrace**
```
pulsar-blt-sn-platform-broker-2 java.util.concurrent.ExecutionException:
org.apache.bookkeeper.mledger.ManagedLedgerException$MetadataNotFoundException:
Managed ledger not found │
at
java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
~[?:?]
│
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)
~[?:?]
│
at
org.apache.pulsar.broker.admin.impl.PersistentTopicsBase.internalDeleteTopicForcefully(PersistentTopicsBase.java:304)
~[io.streamnative-pulsar-broker-2.8.1.5.jar:2.8.1.5] │
at
org.apache.pulsar.broker.admin.impl.PersistentTopicsBase.internalDeleteTopic(PersistentTopicsBase.java:946)
~[io.streamnative-pulsar-broker-2.8.1.5.jar:2.8.1.5] │
at
org.apache.pulsar.broker.admin.v2.PersistentTopics.deleteTopic(PersistentTopics.java:889)
~[io.streamnative-pulsar-broker-2.8.1.5.jar:2.8.1.5]
│
at jdk.internal.reflect.GeneratedMethodAccessor397.invoke(Unknown Source)
~[?:?]
│
at
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:?]
│
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
│
at
org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
~[org.glassfish.jersey.core-jerse │
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)
~[org.glassfish.jersey.core-jersey-server-2.34 │
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)
~[org.glassfish.jersey.core-jersey-server-2.3 │
at
org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$VoidOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:159)
~[org.glassfish.jersey.cor │
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)
~[org.glassfish.jersey.core-jersey-server-2. │
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:475)
~[org.glassfish.jersey.core-jersey-server-2.34.jar:?]
│
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:397)
~[org.glassfish.jersey.core-jersey-server-2.34.jar:?]
│
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
~[org.glassfish.jersey.core-jersey-server-2.34.jar:?]
│
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:255)
~[org.glassfish.jersey.core-jersey-server-2.34.jar:?]
│
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
│
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
│
at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
│
at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
│
at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
│
at
org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
│
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234)
~[org.glassfish.jersey.core-jersey-server-2.34.jar:?]
│
at
org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
~[org.glassfish.jersey.core-jersey-server-2.34.jar:?]
│
at
org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
~[org.glassfish.jersey.containers-jersey-container-servlet-core-2.34.jar:?]
│
at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
~[org.glassfish.jersey.containers-jersey-container-servlet-core-2.34.jar:?]
│
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
~[org.glassfish.jersey.containers-jersey-container-servlet-core-2.34.jar:?]
│
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
~[org.glassfish.jersey.containers-jersey-container-servlet-core-2.34.jar:?]
│
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
~[org.glassfish.jersey.containers-jersey-container-servlet-core-2.34.jar:?]
│
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
~[org.eclipse.jetty-jetty-servlet-9.4.43.v20210629.jar:9.4.43.v20210629]
│
at
org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)
~[org.eclipse.jetty-jetty-servlet-9.4.43.v20210629.jar:9.4.43.v20210629]
│
at
org.apache.pulsar.broker.web.ResponseHandlerFilter.doFilter(ResponseHandlerFilter.java:67)
~[io.streamnative-pulsar-broker-2.8.1.5.jar:2.8.1.5]
│
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
~[org.eclipse.jetty-jetty-servlet-9.4.43.v20210629.jar:9.4.43.v20210629]
│
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
~[org.eclipse.jetty-jetty-servlet-9.4.43.v20210629.jar:9.4.43.v20210629]
│
at
org.apache.pulsar.broker.intercept.BrokerInterceptor.onFilter(BrokerInterceptor.java:88)
~[io.streamnative-pulsar-broker-2.8.1.5.jar:2.8.1.5]
│
at
org.apache.pulsar.broker.web.ProcessHandlerFilter.doFilter(ProcessHandlerFilter.java:49)
~[io.streamnative-pulsar-broker-2.8.1.5.jar:2.8.1.5]
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
~[org.eclipse.jetty-jetty-servlet-9.4.43.v20210629.jar:9.4.43.v20210629]
│
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
~[org.eclipse.jetty-jetty-servlet-9.4.43.v20210629.jar:9.4.43.v20210629]
│
at
org.apache.pulsar.broker.web.PreInterceptFilter.doFilter(PreInterceptFilter.java:70)
~[io.streamnative-pulsar-broker-2.8.1.5.jar:2.8.1.5]
│
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
~[org.eclipse.jetty-jetty-servlet-9.4.43.v20210629.jar:9.4.43.v20210629]
│
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
~[org.eclipse.jetty-jetty-servlet-9.4.43.v20210629.jar:9.4.43.v20210629]
│
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
~[org.eclipse.jetty-jetty-servlet-9.4.43.v20210629.jar:9.4.43.v20210629]
│
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
│
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
│
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
│
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
│
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
│
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
~[org.eclipse.jetty-jetty-servlet-9.4.43.v20210629.jar:9.4.43.v20210629]
│
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
│
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
│
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
│
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
│
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:234)
~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
│
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
│
at
org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:179)
~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
│
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
│
at org.eclipse.jetty.server.Server.handle(Server.java:516)
~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
│
at
org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
│
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
│
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
│
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]
│
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
[org.eclipse.jetty-jetty-io-9.4.43.v20210629.jar:9.4.43.v20210629]
│
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
[org.eclipse.jetty-jetty-io-9.4.43.v20210629.jar:9.4.43.v20210629]
│
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
[org.eclipse.jetty-jetty-io-9.4.43.v20210629.jar:9.4.43.v20210629]
│
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
[org.eclipse.jetty-jetty-util-9.4.43.v20210629.jar:9.4.43.v20210629]
│
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
[org.eclipse.jetty-jetty-util-9.4.43.v20210629.jar:9.4.43.v20210629]
│
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
[org.eclipse.jetty-jetty-util-9.4.43.v20210629.jar:9.4.43.v20210629]
│
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
[org.eclipse.jetty-jetty-util-9.4.43.v20210629.jar:9.4.43.v20210629]
│
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:386)
[org.eclipse.jetty-jetty-util-9.4.43.v20210629.jar:9.4.43.v20210629] │
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
[?:?]
│
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
[?:?]
│
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
[io.netty-netty-common-4.1.68.Final.jar:4.1.68.Final]
│
at java.lang.Thread.run(Thread.java:829) [?:?]
│
│ pulsar-blt-sn-platform-broker-2 Caused by:
org.apache.bookkeeper.mledger.ManagedLedgerException$MetadataNotFoundException:
Managed ledger not found
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
