rdhabalia opened a new pull request #9212:
URL: https://github.com/apache/pulsar/pull/9212
### Motivation
Sometimes schema ledger gets deleted but failed to clean up schema-locator
in zk which can make the topic unavailable and inaccessible in the broker. It
mainly happens when the broker tries to delete an inactive topic and the user
tries to connect the producer again to that topic and can't access the topic
due to the below error:
In this case, if the storage ledger doesn't exist then it's a
non-recoverable error and the broker should be resilient to clean up the broken
schema locator and allow the topic to load again.
**client-error**
```
2020-12-24 07:27:32,221 [Executor task launch worker for task 81] ERROR
org.apache.spark.executor.Executor - Exception in task 18.0 in stage 2.0 (TID
81)
org.apache.pulsar.client.api.PulsarClientException:
java.util.concurrent.ExecutionException:
org.apache.pulsar.client.api.PulsarClientException: java.io.IOException: No
such ledger exists - ledger=123456890 - operation=Failed to open ledger
at
org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:719)
at
org.apache.pulsar.client.impl.ProducerBuilderImpl.create(ProducerBuilderImpl.java:93)
Caused by: java.util.concurrent.ExecutionException:
org.apache.pulsar.client.api.PulsarClientException: java.io.IOException: No
such ledger exists - ledger=123456890 - operation=Failed to open ledger
at
java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
at
org.apache.pulsar.client.impl.ProducerBuilderImpl.create(ProducerBuilderImpl.java:91)
... 14 more
```
**Broker-error**
```
01:33:48.674 [ForkJoinPool.commonPool-worker-9] WARN
org.apache.pulsar.broker.service.persistent.PersistentTopic -
[persistent://my-prop/cluster/ns/t1] Inactive topic deletion failed
java.util.concurrent.CompletionException: java.io.IOException: No such
ledger exists - ledger=123456790 - operation=Failed to open ledger
at
java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
~[?:?]
at
java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
~[?:?]
at
java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1063)
~[?:?]
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
~[?:?]
at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
~[?:?]
at
org.apache.pulsar.broker.service.schema.BookkeeperSchemaStorage.lambda$openLedger$39(BookkeeperSchemaStorage.java:567)
~[pulsar-broker.jar:]
at
org.apache.bookkeeper.client.LedgerOpenOp.openComplete(LedgerOpenOp.java:232)
~[bookkeeper-server-4.9.4.8-yahoo.jar:4.9.4.8-yahoo]
at
org.apache.bookkeeper.client.LedgerOpenOp.lambda$initiate$0(LedgerOpenOp.java:117)
~[bookkeeper-server-4.9.4.8-yahoo.jar:4.9.4.8-yahoo]
at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
~[?:?]
at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
~[?:?]
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
~[?:?]
at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
~[?:?]
at
org.apache.bookkeeper.meta.AbstractZkLedgerManager$3.processResult(AbstractZkLedgerManager.java:396)
~[bookkeeper-server-4.9.4.8-yahoo.jar:4.9.4.8-yahoo]
at
org.apache.bookkeeper.zookeeper.ZooKeeperClient$19$1.processResult(ZooKeeperClient.java:994)
~[bookkeeper-server-4.9.4.8-yahoo.jar:4.9.4.8-yahoo]
at
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:575)
~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
at
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:508)
~[zookeeper-3.4.13.jar:3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03]
Caused by: java.io.IOException: No such ledger exists - ledger=571076459 -
operation=Failed to open ledger
at
org.apache.pulsar.broker.service.schema.BookkeeperSchemaStorage.bkException(BookkeeperSchemaStorage.java:656)
~[pulsar-broker.jar:]
... 11 more
```
### Modification
- Handle broken schema storage locator , cleanup and load the topic
successfully.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
