oneby-wang commented on issue #24879: URL: https://github.com/apache/pulsar/issues/24879#issuecomment-3440341666
> Does the GC process also use the admin api? Yes, after I read through the code, I found the partitioned-topic deletion is fired by admin api. If all partitions are deleted, the GC process will fire partitioned-topic deletion operation. https://github.com/apache/pulsar/blob/88287345d3246fe0d6ea34e06389356ada516cfa/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java#L3498-L3500 I found the following two partitions were fired nearly at the same time, and failed nearly at the same time after 5min. Maybe this is the race condition. ``` Oct 18 06:31:14 pulsar-broker-0 pulsar-broker [bookkeeper-ml-scheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [ourtenant/ourapp/persistent/ourapp.v1.725feb06-ab85-11f0-9600-c297ef652e75-partition-1] Successfully deleted managed ledger Oct 18 06:31:14 pulsar-broker-0 pulsar-broker [bookkeeper-ml-scheduler-OrderedScheduler-0-0] INFO org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://ourtenant/ourapp/ourapp.v1.725feb06-ab85-11f0-9600-c297ef652e75-partition-1] Topic deleted Oct 18 06:31:14 pulsar-broker-0 pulsar-broker [bookkeeper-ml-scheduler-OrderedScheduler-0-0] INFO org.apache.bookkeeper.mledger.impl.ManagedLedgerImpl - [ourtenant/ourapp/persistent/ourapp.v1.725feb06-ab85-11f0-9600-c297ef652e75-partition-5] Successfully deleted managed ledger Oct 18 06:31:14 pulsar-broker-0 pulsar-broker [bookkeeper-ml-scheduler-OrderedScheduler-0-0] INFO org.apache.pulsar.broker.service.persistent.PersistentTopic - [persistent://ourtenant/ourapp/ourapp.v1.725feb06-ab85-11f0-9600-c297ef652e75-partition-5] Topic deleted ``` > It must, as I'm the only administrator for this dev cluster, so that wouldn't be run manually. As mentioned above, admin api in invoked by GC process through code, you can't control this. I found many zookeeper update conflicts in logs, but they are in every small retry interval, so I'm not sure how this update operation can failed after 5min retrying. ``` Oct 18 06:31:07 pulsar-broker-2 pulsar-broker INFO [metadata-store-10-1] INFO org.apache.pulsar.metadata.cache.impl.MetadataCacheImpl - Update key /admin/partitioned-topics/ourtenant/ourapp/persistent/ourapp.v1.725feb06-ab85-11f0-9600-c297ef652e75 conflicts. Retrying in 10 ms. Mandatory stop: false. Elapsed time: 184 ms Oct 18 06:31:07 pulsar-broker-2 pulsar-broker INFO [metadata-store-10-1] INFO org.apache.pulsar.metadata.cache.impl.MetadataCacheImpl - Update key /admin/partitioned-topics/ourtenant/ourapp/persistent/ourapp.v1.725feb06-ab85-11f0-9600-c297ef652e75 conflicts. Retrying in 19 ms. Mandatory stop: false. Elapsed time: 213 ms Oct 18 06:31:14 pulsar-broker-0 pulsar-broker INFO [metadata-store-10-1] INFO org.apache.pulsar.metadata.cache.impl.MetadataCacheImpl - Update key /admin/partitioned-topics/ourtenant/ourapp/persistent/ourapp.v1.725feb06-ab85-11f0-9600-c297ef652e75 conflicts. Retrying in 5 ms. Mandatory stop: false. Elapsed time: 1760783474475 ms ``` `runWithMarkDeleteAsync` will update zookeeper path /admin/partitioned-topics/your-tenant/your-ns/your-topic. [pulsar/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/resources/NamespaceResources.java](https://github.com/apache/pulsar/blob/88287345d3246fe0d6ea34e06389356ada516cfa/pulsar-broker-common/src/main/java/org/apache/pulsar/broker/resources/NamespaceResources.java#L360-L396) Method to work around: 1. Try `pulsar-admin topics delete-partitioned-topic`, see https://pulsar.apache.org/reference/#/3.0.x/pulsar-admin/topics?id=delete-partitioned-topic. I read through the code, it should be an idempotent operation even if some(or all) partitions are deleted. 2. If the first method fails, delete zookeeper path /admin/partitioned-topics/your-tenant/your-ns/your-topic, and check all zookeeper paths /managed-ledgers/your-tenant/your-ns/your-topic-partition-n is deleted, n is your partition num. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
