dragonls edited a comment on issue #5284:
URL: https://github.com/apache/pulsar/issues/5284#issuecomment-1077489078
We had exactly the same issue in 2.7.2.
It happens that when unloading the topic, the topic becomes fenced, and then
new producer and new consumer having error `Topic is temporarily unavailable`.
After restarting the broker, the issue resolved.
The broker log:
```
// unload
16:11:11.603 [pulsar-web-42-4] INFO
org.apache.pulsar.broker.admin.impl.PersistentTopicsBase - [admin] Unloading
topic persistent://sec/app/mid_291476475-partition-1
// disconnecting producers
16:11:11.603 [pulsar-web-42-4] INFO
org.apache.pulsar.broker.service.Producer - Disconnecting producer:
Producer{topic=PersistentTopic{topic=persistent://sec/app/mid_291476475-partition-1},
client=/9.165.193.134:55508, producerName=pulsar-bi-1406-4093, producerId=73}
16:11:11.603 [pulsar-web-42-4] INFO
org.apache.pulsar.broker.service.Producer - Disconnecting producer:
Producer{topic=PersistentTopic{topic=persistent://sec/app/mid_291476475-partition-1},
client=/9.165.243.144:44908, producerName=pulsar-bi-1406-4533, producerId=99}
...
// disconnecting consumer
16:11:11.607 [pulsar-web-42-4] INFO
org.apache.pulsar.broker.service.Consumer - Disconnecting consumer:
Consumer{subscription=PersistentSubscription{topic=persistent://sec/app/mid_291476475-partition-1,
name=mmdcfisherimport-mid_291476475-dw_safe-ods_safe_topic_291476475-669},
consumerId=1,
consumerName=mmdcfisherimport-mid_291476475-dw_safe-ods_safe_topic_291476475-669,
address=/9.165.226.119:33722}
16:11:11.607 [pulsar-web-42-4] INFO
org.apache.pulsar.broker.service.persistent.PersistentDispatcherMultipleConsumers
- Removed consumer
Consumer{subscription=PersistentSubscription{topic=persistent://sec/app/mid_291476475-partition-1,
name=mmdcfisherimport-mid_291476475-dw_safe-ods_safe_topic_291476475-669},
consumerId=1,
consumerName=mmdcfisherimport-mid_291476475-dw_safe-ods_safe_topic_291476475-669,
address=/9.165.226.119:33722} with pending 0 acks
...
16:11:11.712 [ForkJoinPool.commonPool-worker-15] WARN
org.apache.pulsar.broker.service.AbstractTopic -
[persistent://sec/app/mid_291476475-partition-1] Attempting to add producer to
a fenced topic
16:11:11.712 [pulsar-io-24-33] INFO
org.apache.pulsar.broker.service.ServerCnx -
[/9.165.217.148:59240][persistent://sec/app/mid_291476475-partition-1] Creating
producer. producerId=45
16:11:11.712 [pulsar-io-24-22] INFO
org.apache.pulsar.broker.service.ServerCnx -
[/9.165.193.122:40556][persistent://sec/app/mid_291476475-partition-1] Creating
producer. producerId=33
16:11:11.712 [ForkJoinPool.commonPool-worker-15] ERROR
org.apache.pulsar.broker.service.ServerCnx - [/9.165.193.228:45852] Failed to
add producer to topic persistent://sec/app/mid_291476475-partition-1: Topic is
temporarily unavailable
16:11:11.774 [pulsar-io-24-22] INFO
org.apache.pulsar.broker.service.ServerCnx - [/9.165.221.85:40652] Subscribing
on topic persistent://sec/app/mid_291476475-partition-1 /
mmdcfisherimport-mid_291476475-dw_safe-ods_safe_topic_291476475-669
16:11:11.774 [pulsar-io-24-22] WARN
org.apache.pulsar.broker.service.persistent.PersistentTopic -
[persistent://sec/app/mid_291476475-partition-1] Attempting to subscribe to a
fenced topic
16:11:11.774 [pulsar-io-24-22] WARN
org.apache.pulsar.broker.service.ServerCnx -
[/9.165.221.85:40652][persistent://sec/app/mid_291476475-partition-1][mmdcfisherimport-mid_291476475-dw_safe-ods_safe_topic_291476475-669]
Failed to create consumer: consumerId=1, Topic is temporarily unavailable
// secondly unload the topic
16:12:00.672 [pulsar-web-42-8] INFO
org.apache.pulsar.broker.admin.impl.PersistentTopicsBase - [admin] Unloading
topic persistent://sec/app/mid_291476475-partition-1
16:12:00.673 [pulsar-web-42-8] WARN
org.apache.pulsar.broker.service.persistent.PersistentTopic -
[persistent://sec/app/mid_291476475-partition-1] Topic is already being closed
or deleted
16:12:00.673 [pulsar-web-42-8] ERROR
org.apache.pulsar.broker.admin.impl.PersistentTopicsBase - [admin] Failed to
unload topic persistent://sec/app/mid_291476475-partition-1, Topic is already
fenced
org.apache.pulsar.broker.service.BrokerServiceException$TopicFencedException:
Topic is already fenced
at
org.apache.pulsar.broker.service.persistent.PersistentTopic.close(PersistentTopic.java:1043)
~[org.apache.pulsar-pulsar-broker-2.7.3-SNAPHOT.jar:2.7.3-SNAPHOT]
at
org.apache.pulsar.broker.admin.impl.PersistentTopicsBase.internalUnloadNonPartitionedTopic(PersistentTopicsBase.java:951)
~[org.apache.pulsar-pulsar-broker-2.7.3-SNAPHOT.jar:2.7.3-SNAPHOT]
at
org.apache.pulsar.broker.admin.impl.PersistentTopicsBase.internalUnloadTopic(PersistentTopicsBase.java:763)
~[org.apache.pulsar-pulsar-broker-2.7.3-SNAPHOT.jar:2.7.3-SNAPHOT]
at
org.apache.pulsar.broker.admin.v2.PersistentTopics.unloadTopic(PersistentTopics.java:857)
~[org.apache.pulsar-pulsar-broker-2.7.3-SNAPHOT.jar:2.7.3-SNAPHOT]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_312]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_312]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_312]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_312]
at
org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
~[org.glassfish.jersey.core-jersey-server-2.31.jar:?]
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)
~[org.glassfish.jersey.core-jersey-server-2.31.jar:?]
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)
~[org.glassfish.jersey.core-jersey-server-2.31.jar:?]
at
org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$VoidOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:159)
~[org.glassfish.jersey.core-jersey-server-2.31.jar:?]
at
org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)
~[org.glassfish.jersey.core-jersey-server-2.31.jar:?]
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469)
~[org.glassfish.jersey.core-jersey-server-2.31.jar:?]
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391)
~[org.glassfish.jersey.core-jersey-server-2.31.jar:?]
at
org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80)
~[org.glassfish.jersey.core-jersey-server-2.31.jar:?]
at
org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253)
~[org.glassfish.jersey.core-jersey-server-2.31.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
~[org.glassfish.jersey.core-jersey-common-2.31.jar:?]
at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
~[org.glassfish.jersey.core-jersey-common-2.31.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
~[org.glassfish.jersey.core-jersey-common-2.31.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
~[org.glassfish.jersey.core-jersey-common-2.31.jar:?]
at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
~[org.glassfish.jersey.core-jersey-common-2.31.jar:?]
at
org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
~[org.glassfish.jersey.core-jersey-common-2.31.jar:?]
at
org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)
~[org.glassfish.jersey.core-jersey-server-2.31.jar:?]
at
org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
~[org.glassfish.jersey.core-jersey-server-2.31.jar:?]
at
org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
~[org.glassfish.jersey.containers-jersey-container-servlet-core-2.31.jar:?]
at
org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
~[org.glassfish.jersey.containers-jersey-container-servlet-core-2.31.jar:?]
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
~[org.glassfish.jersey.containers-jersey-container-servlet-core-2.31.jar:?]
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
~[org.glassfish.jersey.containers-jersey-container-servlet-core-2.31.jar:?]
at
org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
~[org.glassfish.jersey.containers-jersey-container-servlet-core-2.31.jar:?]
at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
~[org.eclipse.jetty-jetty-servlet-9.4.42.v20210604.jar:9.4.42.v20210604]
at
org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)
~[org.eclipse.jetty-jetty-servlet-9.4.42.v20210604.jar:9.4.42.v20210604]
at
org.apache.pulsar.broker.web.ResponseHandlerFilter.doFilter(ResponseHandlerFilter.java:66)
~[org.apache.pulsar-pulsar-broker-2.7.3-SNAPHOT.jar:2.7.3-SNAPHOT]
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
~[org.eclipse.jetty-jetty-servlet-9.4.42.v20210604.jar:9.4.42.v20210604]
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
~[org.eclipse.jetty-jetty-servlet-9.4.42.v20210604.jar:9.4.42.v20210604]
at
org.apache.pulsar.broker.web.AuthenticationFilter.doFilter(AuthenticationFilter.java:82)
~[org.apache.pulsar-pulsar-broker-common-2.7.3-SNAPHOT.jar:2.7.3-SNAPHOT]
at
org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
~[org.eclipse.jetty-jetty-servlet-9.4.42.v20210604.jar:9.4.42.v20210604]
at
org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
~[org.eclipse.jetty-jetty-servlet-9.4.42.v20210604.jar:9.4.42.v20210604]
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
~[org.eclipse.jetty-jetty-servlet-9.4.42.v20210604.jar:9.4.42.v20210604]
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
~[org.eclipse.jetty-jetty-server-9.4.42.v20210604.jar:9.4.42.v20210604]
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
~[org.eclipse.jetty-jetty-server-9.4.42.v20210604.jar:9.4.42.v20210604]
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
~[org.eclipse.jetty-jetty-server-9.4.42.v20210604.jar:9.4.42.v20210604]
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435)
~[org.eclipse.jetty-jetty-server-9.4.42.v20210604.jar:9.4.42.v20210604]
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
~[org.eclipse.jetty-jetty-server-9.4.42.v20210604.jar:9.4.42.v20210604]
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
~[org.eclipse.jetty-jetty-servlet-9.4.42.v20210604.jar:9.4.42.v20210604]
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
~[org.eclipse.jetty-jetty-server-9.4.42.v20210604.jar:9.4.42.v20210604]
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
~[org.eclipse.jetty-jetty-server-9.4.42.v20210604.jar:9.4.42.v20210604]
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350)
~[org.eclipse.jetty-jetty-server-9.4.42.v20210604.jar:9.4.42.v20210604]
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
~[org.eclipse.jetty-jetty-server-9.4.42.v20210604.jar:9.4.42.v20210604]
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:234)
~[org.eclipse.jetty-jetty-server-9.4.42.v20210604.jar:9.4.42.v20210604]
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
~[org.eclipse.jetty-jetty-server-9.4.42.v20210604.jar:9.4.42.v20210604]
at
org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:179)
~[org.eclipse.jetty-jetty-server-9.4.42.v20210604.jar:9.4.42.v20210604]
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
~[org.eclipse.jetty-jetty-server-9.4.42.v20210604.jar:9.4.42.v20210604]
at org.eclipse.jetty.server.Server.handle(Server.java:516)
~[org.eclipse.jetty-jetty-server-9.4.42.v20210604.jar:9.4.42.v20210604]
at
org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
~[org.eclipse.jetty-jetty-server-9.4.42.v20210604.jar:9.4.42.v20210604]
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
[org.eclipse.jetty-jetty-server-9.4.42.v20210604.jar:9.4.42.v20210604]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
[org.eclipse.jetty-jetty-server-9.4.42.v20210604.jar:9.4.42.v20210604]
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
[org.eclipse.jetty-jetty-server-9.4.42.v20210604.jar:9.4.42.v20210604]
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
[org.eclipse.jetty-jetty-io-9.4.42.v20210604.jar:9.4.42.v20210604]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
[org.eclipse.jetty-jetty-io-9.4.42.v20210604.jar:9.4.42.v20210604]
at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
[org.eclipse.jetty-jetty-io-9.4.42.v20210604.jar:9.4.42.v20210604]
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
[org.eclipse.jetty-jetty-util-9.4.42.v20210604.jar:9.4.42.v20210604]
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
[org.eclipse.jetty-jetty-util-9.4.42.v20210604.jar:9.4.42.v20210604]
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
[org.eclipse.jetty-jetty-util-9.4.42.v20210604.jar:9.4.42.v20210604]
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
[org.eclipse.jetty-jetty-util-9.4.42.v20210604.jar:9.4.42.v20210604]
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:383)
[org.eclipse.jetty-jetty-util-9.4.42.v20210604.jar:9.4.42.v20210604]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_312]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_312]
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
[io.netty-netty-common-4.1.60.Final.jar:4.1.60.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_312]
```
There are no logs shows like `Failed to unload topic` or `Successfully
unloaded topic` of the first unload operation, which means that the unload
operation is stuck somewhere.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
