tomjohn9 commented on issue #5284:
URL: https://github.com/apache/pulsar/issues/5284#issuecomment-1012187931
Hello, we had have the same issue. We could not even delete the topic which
was affected by command
``` ./pulsar-admin topics delete /tenant/space/topic -f```
response:
```
[AsyncHttpClient-7-1] WARN
org.apache.pulsar.client.admin.internal.BaseResource -
[http://localhost:8080/admin/v2/persistent/tenant/space/topic?force=true&deleteSchema=false]
Failed to perform http delete request:
javax.ws.rs.InternalServerErrorException: HTTP 500 Internal Server Error
HTTP 500 Internal Server ErrorReason: HTTP 500 Internal Server
Errorjava.io.ByteArrayInputStream@5d32b342
```
also reponse for same command after all brokeres were restarted:
```
[AsyncHttpClient-7-1] WARN
org.apache.pulsar.client.admin.internal.BaseResource -
[http://localhost:8080/admin/v2/persistent/tenant/space/topic?force=true&deleteSchema=false]
Failed to perform http delete request:
java.util.concurrent.CompletionException:
org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException:
Could not complete the operation. Number of retries has been exhausted. Failed
reason: test-pulsar-broker-2.test-pulsar-broker.str-test.svc.cluster.local
nullReason: java.util.concurrent.CompletionException:
org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector$RetryException:
Could not complete the operation. Number of retries has been exhausted. Failed
reason: test-pulsar-broker-2.test-pulsar-broker.str-test.svc.cluster.local
```
On server side was observed
`
[pulsar-web-42-1] ERROR
org.apache.pulsar.broker.admin.impl.PersistentTopicsBase - [null] Failed to
delete topic forcefully
persistent://tenant/space/topic2022-01-12T10:12:22.112578917Z
java.util.concurrent.ExecutionException:
org.apache.pulsar.broker.service.BrokerServiceException$TopicFencedException:
Topic is already fenced2022-01-12T10:12:22.112584692Z at
java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
~[?:1.8.0_312]2022-01-12T10:12:22.112589692Z at
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
~[?:1.8.0_312]2022-01-12T10:12:22.112594023Z at
org.apache.pulsar.broker.admin.impl.PersistentTopicsBase.internalDeleteTopicForcefully(PersistentTopicsBase.java:381)
~[org.apache.pulsar-pulsar-broker-2.7.4.jar:2.7.4]2022-01-12T10:12:22.112597842Z
at
org.apache.pulsar.broker.admin.impl.PersistentTopicsBase.internalDeleteTopic(PersistentTopicsBase.java:928)
~[org.apache.pulsar-pulsar-broker-2.7.4.jar:2.7.4]2022-01-12T10:12:22.112
602209Z at
org.apache.pulsar.broker.admin.v2.PersistentTopics.deleteTopic(PersistentTopics.java:895)
~[org.apache.pulsar-pulsar-broker-2.7.4.jar:2.7.4]2022-01-12T10:12:22.112606269Z
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_312]2022-01-12T10:12:22.112610533Z at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_312]2022-01-12T10:12:22.112614725Z at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_312]2022-01-12T10:12:22.112618793Z at
java.lang.reflect.Method.invoke(Method.java:498)
~[?:1.8.0_312]2022-01-12T10:12:22.112624879Z at
org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
~[org.glassfish.jersey.core-jersey-server-2.31.jar:?]2022-01-12T10:12:22.112628946Z
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceM
ethodDispatcher.java:124)
~[org.glassfish.jersey.core-jersey-server-2.31.jar:?]2022-01-12T10:12:22.112632798Z
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)
~[org.glassfish.jersey.core-jersey-server-2.31.jar:?]2022-01-12T10:12:22.112637722Z
at
org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$VoidOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:159)
~[org.glassfish.jersey.core-jersey-server-2.31.jar:?]2022-01-12T10:12:22.112643032Z
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)
~[org.glassfish.jersey.core-jersey-server-2.31.jar:?]2022-01-12T10:12:22.112647454Z
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469)
~[org.glassfish.jersey.core-jersey-server-2.31.jar:?]2022-01-12T10:12:22.112651631Z
at org.glassfish.
jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391)
~[org.glassfish.jersey.core-jersey-server-2.31.jar:?]2022-01-12T10:12:22.112663041Z
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80)
~[org.glassfish.jersey.core-jersey-server-2.31.jar:?]2022-01-12T10:12:22.112668516Z
at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253)
~[org.glassfish.jersey.core-jersey-server-2.31.jar:?]2022-01-12T10:12:22.112672817Z
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
~[org.glassfish.jersey.core-jersey-common-2.31.jar:?]2022-01-12T10:12:22.112677321Z
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
~[org.glassfish.jersey.core-jersey-common-2.31.jar:?]2022-01-12T10:12:22.112681470Z
at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
~[org.glassfish.jersey.core-jersey-common-2.31.jar:?]2022-01-12T10:12:22.112685407Z
at org.glassfish.jersey.internal.Errors.pro
cess(Errors.java:274)
~[org.glassfish.jersey.core-jersey-common-2.31.jar:?]2022-01-12T10:12:22.112689357Z
at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
~[org.glassfish.jersey.core-jersey-common-2.31.jar:?]2022-01-12T10:12:22.112693780Z
at
org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
~[org.glassfish.jersey.core-jersey-common-2.31.jar:?]2022-01-12T10:12:22.112698117Z
at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)
~[org.glassfish.jersey.core-jersey-server-2.31.jar:?]2022-01-12T10:12:22.112702566Z
at
org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
~[org.glassfish.jersey.core-jersey-server-2.31.jar:?]2022-01-12T10:12:22.112706866Z
at
org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
~[org.glassfish.jersey.containers-jersey-container-servlet-core-2.31.jar:?]2022-01-12T10:12:22.112711807Z
at org.glassfish.jersey.servlet.WebComponent.se
rvice(WebComponent.java:346)
~[org.glassfish.jersey.containers-jersey-container-servlet-core-2.31.jar:?]2022-01-12T10:12:22.112717009Z
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
~[org.glassfish.jersey.containers-jersey-container-servlet-core-2.31.jar:?]2022-01-12T10:12:22.112721814Z
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
~[org.glassfish.jersey.containers-jersey-container-servlet-core-2.31.jar:?]2022-01-12T10:12:22.112726179Z
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
~[org.glassfish.jersey.containers-jersey-container-servlet-core-2.31.jar:?]2022-01-12T10:12:22.112730612Z
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
~[org.eclipse.jetty-jetty-servlet-9.4.43.v20210629.jar:9.4.43.v20210629]2022-01-12T10:12:22.112735081Z
at
org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)
~[org.eclipse.jetty-je
tty-servlet-9.4.43.v20210629.jar:9.4.43.v20210629]2022-01-12T10:12:22.112740008Z
at
org.apache.pulsar.broker.web.ResponseHandlerFilter.doFilter(ResponseHandlerFilter.java:66)
~[org.apache.pulsar-pulsar-broker-2.7.4.jar:2.7.4]2022-01-12T10:12:22.112744543Z
at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
~[org.eclipse.jetty-jetty-servlet-9.4.43.v20210629.jar:9.4.43.v20210629]2022-01-12T10:12:22.112751573Z
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
~[org.eclipse.jetty-jetty-servlet-9.4.43.v20210629.jar:9.4.43.v20210629]2022-01-12T10:12:22.112756057Z
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
~[org.eclipse.jetty-jetty-servlet-9.4.43.v20210629.jar:9.4.43.v20210629]2022-01-12T10:12:22.112760477Z
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]2022-01-12T10:12:22.112764961Z
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]2022-01-12T10:12:22.112769449Z
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]2022-01-12T10:12:22.112774433Z
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]2022-01-12T10:12:22.112778917Z
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]2022-01-12T10:12:22.112783064Z
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
~[org.eclipse.jetty-jetty-servlet-9.4.43.v20210629.jar:9.4.43.v20210629]2022-01-12T10:12:22.112787539Z
at org.eclipse.jetty.server.session.SessionHandler.doScope(Sessio
nHandler.java:1594)
~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]2022-01-12T10:12:22.112792091Z
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]2022-01-12T10:12:22.112796477Z
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]2022-01-12T10:12:22.112801031Z
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]2022-01-12T10:12:22.112805373Z
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:234)
~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]2022-01-12T10:12:22.112809970Z
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
~[org.eclipse.jetty-
jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]2022-01-12T10:12:22.112814470Z
at
org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:179)
~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]2022-01-12T10:12:22.112818967Z
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]2022-01-12T10:12:22.112823499Z
at org.eclipse.jetty.server.Server.handle(Server.java:516)
~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]2022-01-12T10:12:22.112829936Z
at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
~[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]2022-01-12T10:12:22.112834628Z
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]2022-01-12T10:12:22.112839081Z
at org.eclipse.jetty
.server.HttpChannel.handle(HttpChannel.java:380)
[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]2022-01-12T10:12:22.112843388Z
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
[org.eclipse.jetty-jetty-server-9.4.43.v20210629.jar:9.4.43.v20210629]2022-01-12T10:12:22.112847845Z
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
[org.eclipse.jetty-jetty-io-9.4.43.v20210629.jar:9.4.43.v20210629]2022-01-12T10:12:22.112852292Z
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
[org.eclipse.jetty-jetty-io-9.4.43.v20210629.jar:9.4.43.v20210629]2022-01-12T10:12:22.112856722Z
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
[org.eclipse.jetty-jetty-io-9.4.43.v20210629.jar:9.4.43.v20210629]2022-01-12T10:12:22.112861022Z
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
[org.eclipse.jetty-jetty-util-9.4.43.v2021062
9.jar:9.4.43.v20210629]2022-01-12T10:12:22.112865495Z at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
[org.eclipse.jetty-jetty-util-9.4.43.v20210629.jar:9.4.43.v20210629]2022-01-12T10:12:22.112870070Z
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
[org.eclipse.jetty-jetty-util-9.4.43.v20210629.jar:9.4.43.v20210629]2022-01-12T10:12:22.112875082Z
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
[org.eclipse.jetty-jetty-util-9.4.43.v20210629.jar:9.4.43.v20210629]2022-01-12T10:12:22.112879486Z
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:386)
[org.eclipse.jetty-jetty-util-9.4.43.v20210629.jar:9.4.43.v20210629]2022-01-12T10:12:22.112883891Z
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_312]2022-01-12T10:12:22.112887551Z at java.util.concurrent.ThreadPoolE
xecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_312]2022-01-12T10:12:22.112891123Z at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
[io.netty-netty-common-4.1.68.Final.jar:4.1.68.Final]2022-01-12T10:12:22.112894872Z
at java.lang.Thread.run(Thread.java:748)
[?:1.8.0_312]2022-01-12T10:12:22.112898812Z Caused by:
org.apache.pulsar.broker.service.BrokerServiceException$TopicFencedException:
Topic is already fenced2022-01-12T10:12:22.112902802Z at
org.apache.pulsar.broker.service.persistent.PersistentTopic.delete(PersistentTopic.java:937)
~[org.apache.pulsar-pulsar-broker-2.7.4.jar:2.7.4]2022-01-12T10:12:22.112906464Z
at
org.apache.pulsar.broker.service.persistent.PersistentTopic.deleteForcefully(PersistentTopic.java:908)
~[org.apache.pulsar-pulsar-broker-2.7.4.jar:2.7.4]2022-01-12T10:12:22.112910144Z
at
org.apache.pulsar.broker.service.BrokerService.deleteTopic(BrokerService.java:831)
~[org.apache.pulsar-pulsar-broker-2.7.4.jar:2.7.4
]2022-01-12T10:12:22.112915645Z ... 66 more
`
As solution we had to perform removing item from zookeeper by zookeeper
shell command and restart all brokers
`deleteall /managed-ledgers/tenant/namespace/persistent/topic`
Is there any better workaround for it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
