zbentley opened a new issue, #15433:
URL: https://github.com/apache/pulsar/issues/15433
**Describe the bug**
Using pulsar standalone, when running integration tests, I often create
tenants, namespaces, and topics, then rapidly take actions on those topics
(like publishing, termination, deletion). When I attempt to terminate a topic
right after it has been created, it sometimes fails with
"PreconditionFailedException: Failed to find ownership for topic" errors (see
stacktrace below).
This condition persists for quite awhile; if I retry once a second for up to
5 seconds, sometimes it goes away, but sometimes it's still occurring after 5
seconds.
In 2.9.1 the error occurs with a 412 response code. In 2.10 it has a 500
response code.
It shouldn't happen regardless; topics should be terminateable once the
management API reports that they've been successfully created.
**To Reproduce**
On a standalone 2.10 or 2.9 broker, in rapid succession, do the following:
1. Using the management API, create a tenant.
2. Using the management API, create a namespace.
3. Using the management API, create a persistent, partitioned topic.
4. `POST` to the `/terminate/partitions` endpoint on that topic.
Doing this repeatedly will eventually encounter an error with the below
stacktrace.
**Expected behavior**
Once created, topics should be "done", in that they're ready for use from
anywhere in the cluster.
If using autocreated topics, I can understand why this would be tricky
(autocreating a topic via the binary protocol can be more eventually
consistent, since only the thing issuing the create needs to use it right away).
But when using the management API to create a topic, the API shouldn't
return 200 for creation until all backend operations report success. That might
mean topic creation has to take a long time, or might fail if all brokers can't
be reached to confirm metadata update, but that's preferable to topics breaking
when users try to do things with them later on.
**Desktop (please complete the following information):**
- MacOS 12, Intel
- Pulsar standalone 2.9 and 2.10 via Homebrew
**Example Stacktrace**
```
E chariot.pulsar.client.exceptions.PiecemealDeletionError:
ClientResponseError: 500, message='Internal Server Error',
url=URL('http://localhost:8080/admin/v2/persistent/chariot-test-obj-ChariotSite-251-gqtb_1/chariot_namespace_integration_test_topic/chariot_topic_integration_test_topic/terminate/partitions')
E Response code: 500
E URL: POST
http://localhost:8080/admin/v2/persistent/chariot-test-obj-ChariotSite-251-gqtb_1/chariot_namespace_integration_test_topic/chariot_topic_integration_test_topic/terminate/partitions
E
E --- An unexpected error occurred in the server ---
E
E Message: Failed to find ownership for
topic:persistent://chariot-test-obj-ChariotSite-251-gqtb_1/chariot_namespace_integration_test_topic/chariot_topic_integration_test_topic-partition-0
E
E Stacktrace:
E
E
org.apache.pulsar.client.admin.PulsarAdminException$PreconditionFailedException:
Failed to find ownership for
topic:persistent://chariot-test-obj-ChariotSite-251-gqtb_1/chariot_namespace_integration_test_topic/chariot_topic_integration_test_topic-partition-0
E at
org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:240)
E at
org.apache.pulsar.client.admin.internal.TopicsImpl$8.failed(TopicsImpl.java:536)
E at
org.glassfish.jersey.client.JerseyInvocation$1.failed(JerseyInvocation.java:882)
E at
org.glassfish.jersey.client.JerseyInvocation$1.completed(JerseyInvocation.java:863)
E at
org.glassfish.jersey.client.ClientRuntime.processResponse(ClientRuntime.java:229)
E at
org.glassfish.jersey.client.ClientRuntime.access$200(ClientRuntime.java:62)
E at
org.glassfish.jersey.client.ClientRuntime$2.lambda$response$0(ClientRuntime.java:173)
E at
org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
E at
org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
E at
org.glassfish.jersey.internal.Errors.process(Errors.java:292)
E at
org.glassfish.jersey.internal.Errors.process(Errors.java:274)
E at
org.glassfish.jersey.internal.Errors.process(Errors.java:244)
E at
org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:288)
E at
org.glassfish.jersey.client.ClientRuntime$2.response(ClientRuntime.java:173)
E at
org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$apply$1(AsyncHttpConnector.java:228)
E at
java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
E at
java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
E at
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
E at
java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
E at
org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$4(AsyncHttpConnector.java:270)
E at
java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
E at
java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
E at
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
E at
java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
E at
org.asynchttpclient.netty.NettyResponseFuture.loadContent(NettyResponseFuture.java:222)
E at
org.asynchttpclient.netty.NettyResponseFuture.done(NettyResponseFuture.java:257)
E at
org.asynchttpclient.netty.handler.AsyncHttpClientHandler.finishUpdate(AsyncHttpClientHandler.java:241)
E at
org.asynchttpclient.netty.handler.HttpHandler.handleChunk(HttpHandler.java:114)
E at
org.asynchttpclient.netty.handler.HttpHandler.handleRead(HttpHandler.java:143)
E at
org.asynchttpclient.netty.handler.AsyncHttpClientHandler.channelRead(AsyncHttpClientHandler.java:78)
E at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
E at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
E at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
E at
io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
E at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
E at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
E at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
E at
io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
E at
io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327)
E at
io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299)
E at
io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
E at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
E at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
E at
io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
E at
io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
E at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
E at
io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
E at
io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
E at
io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
E at
io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
E at
io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
E at
io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
E at
io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
E at
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
E at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
E at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
E at java.base/java.lang.Thread.run(Thread.java:829)
E Caused by: javax.ws.rs.ClientErrorException: HTTP 412
Precondition Failed
E at
org.glassfish.jersey.client.JerseyInvocation.createExceptionForFamily(JerseyInvocation.java:985)
E at
org.glassfish.jersey.client.JerseyInvocation.convertToException(JerseyInvocation.java:967)
E at
org.glassfish.jersey.client.JerseyInvocation.access$700(JerseyInvocation.java:82)
E ... 54 more
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
