massakam opened a new pull request #8561:
URL: https://github.com/apache/pulsar/pull/8561
### Motivation
The other day, we faced a problem where a topic remained fenced and
unavailable. This topic remained unavailable until it was unloaded. The
following is the broker log at that time.
```
11:37:55.905 [bookkeeper-ml-workers-OrderedExecutor-77-0] INFO
o.a.b.mledger.impl.OpAddEntry - [tenant/ns/persistent/topic] Closing
ledger 40891546 for being full
11:37:56.208 [pulsar-ordered-OrderedExecutor-0-0-EventThread] ERROR
o.a.b.client.MetadataUpdateLoop -
UpdateLoop(ledgerId=40891546,loopId=6ce63876) Error writing metadata to store
11:37:56.209 [pulsar-ordered-OrderedExecutor-0-0-EventThread] WARN
o.a.b.mledger.impl.OpAddEntry - Error when closing ledger 40891546.
Status=Error while using ZooKeeper
11:37:56.359 [pulsar-ordered-OrderedExecutor-0-0-EventThread] ERROR
o.a.b.mledger.impl.ManagedLedgerImpl - [tenant/ns/persistent/topic] Error
creating ledger rc=-9 Error while using ZooKeeper
11:37:56.359 [pulsar-ordered-OrderedExecutor-0-0-EventThread] INFO
o.a.pulsar.broker.service.Producer - Disconnecting producer:
Producer{topic=PersistentTopic{topic=persistent://tenant/ns/topic},
client=/xxx.xxx.xxx.xxx:40646, producerName=pulsar.repl.jp-west, producerId=668}
11:37:56.360 [pulsar-ordered-OrderedExecutor-0-0-EventThread] WARN
o.a.p.b.s.persistent.PersistentTopic - [persistent://tenant/ns/topic] Failed to
persist msg in store: Error while using ZooKeeper
11:37:56.360 [pulsar-ordered-OrderedExecutor-0-0-EventThread] INFO
o.a.pulsar.broker.service.Producer - Disconnecting producer:
Producer{topic=PersistentTopic{topic=persistent://tenant/ns/topic},
client=/xxx.xxx.xxx.xxx:40646, producerName=pulsar.repl.jp-west, producerId=668}
11:37:56.360 [pulsar-ordered-OrderedExecutor-0-0-EventThread] WARN
o.a.p.b.s.persistent.PersistentTopic - [persistent://tenant/ns/topic] Failed to
persist msg in store: Error while using ZooKeeper
11:37:56.360 [pulsar-ordered-OrderedExecutor-0-0-EventThread] INFO
o.a.pulsar.broker.service.Producer - Disconnecting producer:
Producer{topic=PersistentTopic{topic=persistent://tenant/ns/topic},
client=/xxx.xxx.xxx.xxx:40646, producerName=pulsar.repl.jp-west, producerId=668}
11:37:56.360 [pulsar-ordered-OrderedExecutor-0-0-EventThread] WARN
o.a.p.b.s.persistent.PersistentTopic - [persistent://tenant/ns/topic] Failed to
persist msg in store: Error while using ZooKeeper
11:37:56.360 [pulsar-ordered-OrderedExecutor-0-0-EventThread] WARN
o.a.p.b.s.persistent.PersistentTopic - [persistent://tenant/ns/topic] Failed to
persist msg in store: Error while using ZooKeeper
11:37:56.360 [pulsar-ordered-OrderedExecutor-0-0-EventThread] WARN
o.a.p.b.s.persistent.PersistentTopic - [persistent://tenant/ns/topic] Failed to
persist msg in store: Error while using ZooKeeper
11:37:56.360 [pulsar-ordered-OrderedExecutor-0-0-EventThread] WARN
o.a.p.b.s.persistent.PersistentTopic - [persistent://tenant/ns/topic] Failed to
persist msg in store: Error while using ZooKeeper
11:37:57.495 [ForkJoinPool.commonPool-worker-51] INFO
o.a.pulsar.broker.service.ServerCnx -
[/xxx.xxx.xxx.xxx:40256][persistent://tenant/ns/topic] Creating producer.
producerId=668
11:37:58.291 [bookkeeper-ml-workers-OrderedExecutor-77-0] INFO
o.a.b.mledger.impl.ManagedLedgerImpl - [tenant/ns/persistent/topic] End
TrimConsumedLedgers. ledgers=2 totalSize=162868668
11:37:58.291 [bookkeeper-ml-workers-OrderedExecutor-77-0] INFO
o.a.b.mledger.impl.ManagedLedgerImpl - [tenant/ns/persistent/topic] Removing
ledger 40880508 - size: 82183409
11:37:58.292 [ForkJoinPool.commonPool-worker-20] INFO
o.a.pulsar.broker.service.ServerCnx - [/xxx.xxx.xxx.xxx:40256]-668
persistent://tenant/ns/topic configured with schema false
11:37:58.292 [ForkJoinPool.commonPool-worker-20] WARN
o.a.p.b.s.persistent.PersistentTopic - [persistent://tenant/ns/topic]
Attempting to add producer to a fenced topic
11:37:58.292 [ForkJoinPool.commonPool-worker-20] ERROR
o.a.pulsar.broker.service.ServerCnx - [/xxx.xxx.xxx.xxx:40256] Failed to add
producer to topic persistent://tenant/ns/topic: Topic is temporarily unavailable
11:37:58.728 [ForkJoinPool.commonPool-worker-75] INFO
o.a.pulsar.broker.service.ServerCnx -
[/xxx.xxx.xxx.xxx:40330][persistent://tenant/ns/topic] Creating producer.
producerId=668
11:37:58.729 [ForkJoinPool.commonPool-worker-75] INFO
o.a.pulsar.broker.service.ServerCnx - [/xxx.xxx.xxx.xxx:40330]-668
persistent://tenant/ns/topic configured with schema false
11:37:58.729 [ForkJoinPool.commonPool-worker-75] WARN
o.a.p.b.s.persistent.PersistentTopic - [persistent://tenant/ns/topic]
Attempting to add producer to a fenced topic
11:37:58.729 [ForkJoinPool.commonPool-worker-75] ERROR
o.a.pulsar.broker.service.ServerCnx - [/xxx.xxx.xxx.xxx:40330] Failed to add
producer to topic persistent://tenant/ns/topic: Topic is temporarily unavailable
11:37:59.489 [ForkJoinPool.commonPool-worker-106] INFO
o.a.pulsar.broker.service.ServerCnx -
[/xxx.xxx.xxx.xxx:40260][persistent://tenant/ns/topic] Creating producer.
producerId=668
11:37:59.489 [ForkJoinPool.commonPool-worker-106] INFO
o.a.pulsar.broker.service.ServerCnx - [/xxx.xxx.xxx.xxx:40260]-668
persistent://tenant/ns/topic configured with schema false
11:37:59.489 [ForkJoinPool.commonPool-worker-106] WARN
o.a.p.b.s.persistent.PersistentTopic - [persistent://tenant/ns/topic]
Attempting to add producer to a fenced topic
11:37:59.489 [ForkJoinPool.commonPool-worker-106] ERROR
o.a.pulsar.broker.service.ServerCnx - [/xxx.xxx.xxx.xxx:40260] Failed to add
producer to topic persistent://tenant/ns/topic: Topic is temporarily unavailable
11:38:01.062 [ForkJoinPool.commonPool-worker-51] INFO
o.a.pulsar.broker.service.ServerCnx -
[/xxx.xxx.xxx.xxx:40248][persistent://tenant/ns/topic] Creating producer.
producerId=668
11:38:01.062 [ForkJoinPool.commonPool-worker-51] INFO
o.a.pulsar.broker.service.ServerCnx - [/xxx.xxx.xxx.xxx:40248]-668
persistent://tenant/ns/topic configured with schema false
11:38:01.063 [ForkJoinPool.commonPool-worker-51] WARN
o.a.p.b.s.persistent.PersistentTopic - [persistent://tenant/ns/topic]
Attempting to add producer to a fenced topic
11:38:01.063 [ForkJoinPool.commonPool-worker-51] ERROR
o.a.pulsar.broker.service.ServerCnx - [/xxx.xxx.xxx.xxx:40248] Failed to add
producer to topic persistent://tenant/ns/topic: Topic is temporarily unavailable
11:38:04.103 [ForkJoinPool.commonPool-worker-90] INFO
o.a.pulsar.broker.service.ServerCnx -
[/xxx.xxx.xxx.xxx:40338][persistent://tenant/ns/topic] Creating producer.
producerId=668
11:38:04.104 [ForkJoinPool.commonPool-worker-102] INFO
o.a.pulsar.broker.service.ServerCnx - [/xxx.xxx.xxx.xxx:40338]-668
persistent://tenant/ns/topic configured with schema false
```
We were maintaining the ZooKeeper servers, so I think this phenomenon was
caused by the shutdown of some ZK servers. However, the causal relationship has
not been clarified.
### Modifications
As a workaround, close the topic if it remains fenced for a period of time.
Reconnecting from the clients will instantiate a new `PersistentTopic` topic
and the topic will back to normal.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
