eugene-polozhenkov opened a new issue, #26157: URL: https://github.com/apache/pulsar/issues/26157
### Search before reporting - [x] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar. ### Read release policy - [x] I understand that [unsupported versions](https://pulsar.apache.org/contribute/release-policy/#supported-versions) don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker. ### User environment - Broker version: Affects all versions with Oxia metadata backend support (confirmed in current master source) - Metadata backend: Oxia (metadataServiceUri=oxia://...) - BookKeeper version: 4.17.3 - Deployment: Kubernetes (GKE) via Pulsar Helm chart - Broker OS: Linux (GKE nodes) - Broker Java version: JDK 17 ### Issue Description When BookKeeper autorecovery processes an under-replicated ledger using the Oxia metadata backend, `markLedgerReplicated()` deletes the UR leaf entry (`urL<id>`) but never cleans up the 4 intermediate parent nodes that were created alongside it. These orphaned nodes accumulate indefinitely in the UR tree. There are two bugs in the same method: **Bug 1 — Parent cleanup is completely skipped for Oxia** The hierarchy cleanup in `markLedgerReplicated()` is gated on a metadata store type check that excludes `OxiaMetadataStore`: ```java if (store instanceof ZKMetadataStore || store instanceof DualMetadataStore) { // clean up the hierarchy — L4, L3, L2, L1 parent nodes // OxiaMetadataStore: this entire block is never entered ❌ } OxiaMetadataStore is not imported in PulsarLedgerUnderreplicationManager.java and not included in the check. This is particularly damaging for Oxia because OxiaMetadataStore.doStorePut() calls createParents() to explicitly create up to 4 intermediate path entries for every UR leaf written (Oxia is a flat key-value store; these parents must be created and deleted explicitly). Since markLedgerReplicated() never cleans them up, intermediate nodes are permanently orphaned after each ledger is successfully replicated. Bug 2 — Exception handling does not cover the Oxia "has children" error Even if Bug 1 is fixed by adding OxiaMetadataStore to the guard, the inner catch block still fails for Oxia: } catch (ExecutionException ee) { if (ee.getCause() instanceof MetadataStoreException && ee.getCause().getCause() instanceof KeeperException.NotEmptyException) { //do nothing. } else { log.warn().exception(ee).log("Error deleting underrepcalited ledger parent node"); } } When trying to delete a parent node that still has sibling children: - ZooKeeper throws MetadataStoreException(cause=KeeperException.NotEmptyException) → ✅ caught correctly - Oxia throws MetadataStoreException("Key '...' has children") with no inner cause → ❌ getCause().getCause() is null, falls through to log.warn() — a false positive warning logged on every successfully replicated ledger What we expected: After markLedgerReplicated() processes a ledger, the urL* leaf and all empty ancestor nodes (L4 bucket, L3 segment, L2, L1) should be removed from the UR tree. What actually happened: Only the urL* leaf is deleted. The 4 ancestor nodes remain indefinitely as orphans, growing without bound across cluster lifetime. ### Error messages ```text ``` ### Reproducing the issue The bug is visible directly in the source of PulsarLedgerUnderreplicationManager.java — the instanceof guard excludes OxiaMetadataStore. To observe the effect in a running cluster: 1. Deploy Pulsar with Oxia as the metadata backend (metadataServiceUri=oxia://...) 2. Simulate a bookie failure or rolling restart so the Auditor marks several ledgers as under-replicated (with lostBookieRecoveryDelay=0 for simplicity so that active ledgers are marked an UnderReplicated). 3. Allow the ReplicationWorker to process and replicate those ledgers (watch logs for markLedgerReplicated completions) 4. Inspect the UR tree after processing — orphaned intermediate nodes remain: # Run inside the Oxia server pod oxia client list -n bookkeeper \ -s /ledgers/underreplication/ledgers/ \ -e /ledgers/underreplication/ledgers/~ 5. Observe that intermediate bucket nodes (L4/L3/L2/L1) remain after all urL* leaf entries have been deleted. Example: # Valid entry (has urL* leaf child): /ledgers/underreplication/ledgers/0000/0000/0001/028a/urL0000197258 # Orphaned node (no urL* child, empty bucket — should have been deleted): /ledgers/underreplication/ledgers/0000/0000/0001/051f Production evidence: This issue was discovered following an upgrade of Oxia from v0.15.3 to v0.16.6 (required to pick up the fix for a notification dispatcher bug: oxia-db/oxia#1149). The Oxia upgrade triggered a rolling restart of bookie pods. With `lostBookieRecoveryDelay` at its default of 0, the Auditor fired immediately on each bookie loss event and marked a large number of ledgers as under-replicated. Once the ReplicationWorker processed those entries, the `urL*` leaves were removed but the intermediate parent nodes (L4/L3/L2/L1) were left behind as orphans. Inspecting the UR tree on one cluster after recovery showed thousands of orphaned bucket nodes with no `urL*` leaf children: Orphaned nodes (no urL* child — should have been deleted after replication): /ledgers/underreplication/ledgers/0000/0000/0007/06e0 /ledgers/underreplication/ledgers/0000/0000/0007/b51d ... Valid entry still being processed (urL* leaf present): /ledgers/underreplication/ledgers/0000/0000/0007/baf3/urL0000506611 The `oxia_server_db_lists_count_total` metric on the affected cluster was sustained at ~100–130 list ops/sec against a healthy baseline of <5/s, driven entirely by `getLedgerToRereplicateFromHierarchy()` traversing the orphaned subtrees on every replication poll. After manually removing the orphaned nodes the metric returned to baseline immediately. ### Additional information Root cause in source File: pulsar-metadata/src/main/java/org/apache/pulsar/metadata/bookkeeper/PulsarLedgerUnderreplicationManager.java Method: markLedgerReplicated(long ledgerId) OxiaMetadataStore (org.apache.pulsar.metadata.impl.oxia.OxiaMetadataStore) is not imported in this file and not referenced in the instanceof guard. Proposed fix Two changes to markLedgerReplicated(): 1. Add OxiaMetadataStore to the instanceof guard 2. Replace the ZK-specific catch with a backend-agnostic helper: ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
