Shawyeok opened a new issue, #4812:
URL: https://github.com/apache/bookkeeper/issues/4812
## Summary
When an open ledger's ensemble bookies are decommissioned and replaced with
fresh bookies at the same network addresses, the original client's subsequent
`addEntry` call **returns success** (entry ID assigned, LAC advanced), yet the
entry is **permanently lost**: the recovery process had already closed the
ledger at a lower `lastEntry` in ZK metadata, and `lh.close()` then fails with
`BKMetadataVersionException`. The client receives no `BKLedgerFencedException`;
the data loss is silent.
## Environment
| | |
|---|---|
| BookKeeper version | 4.17.3 (bundled in Apache Pulsar 4.0.10) |
| Also reproduced on | BK 4.14.x (bundled in Pulsar 3.0.x) |
| ZooKeeper | embedded in `apachepulsar/pulsar:4.0.10` |
| Java | OpenJDK 21 |
## Steps to Reproduce
**Cluster setup**
- 1 ZooKeeper node
- 3 bookies (`bk1`, `bk2`, `bk3`) on the same Docker bridge network
**Client**
Opens a ledger with `E=2, WQ=2, AQ=2`. The ensemble is assigned two of the
three bookies (say `bk2` and `bk3`); `bk1` is the spare.
Writes `entry-0`, then blocks waiting for a proceed signal.
**Decommission + replace (one at a time)**
Because E=2/WQ=2 with only 3 bookies leaves one spare, the order must be
interleaved:
1. **D1** — `docker stop bk2` → `bookkeeper shell decommissionbookie
-bookieid <bk2-ip>:3181`
The ReplicationWorker copies bk2's fragments to bk1 (the spare) and
deletes bk2's ZK cookie.
2. **E1** — Start a fresh `bk2-new` at the **exact same IP** as the original
`bk2`.
3. **D2** — `docker stop bk3` → `bookkeeper shell decommissionbookie
-bookieid <bk3-ip>:3181`
During this step the ReplicationWorker on bk1 detects ledger 0 as
under-replicated. After `openLedgerRereplicationGracePeriod` (30 s default) it
performs recovery-mode open: fences the ledger, reads up to the last confirmed
entry (`lastEntry=0`), closes it in ZK metadata. ZK version is bumped.
4. **E2** — Start a fresh `bk3-new` at the **exact same IP** as the original
`bk3`.
**Proceed signal**
After both replacements are registered, the client's proceed sentinel is
created.
```
addEntry("entry-1") → id=1, LAC=1 ← SUCCESS, no exception
lh.close() → BKMetadataVersionException: Bad ledger metadata
version
```
## Observed Behaviour
```
[client] entry-1 id=1 LAC=1
[client] ERROR lh.close(): BKMetadataVersionException: Bad ledger metadata
version
```
Log from the recovery side (during D2):
```
INFO LedgerFragmentReplicator - Updated ZK to point ledger fragments from
old bookies
to new bookies: {<bk3-ip>:3181=<bk2-new-ip>:3181}
```
And from the original client's close path:
```
ERROR Metadata conflict when closing ledger 0. Another client may have
recovered
the ledger while there were writes outstanding.
(local lastEntry:1 length:14) (metadata lastEntry:0 length:7)
```
## Expected Behaviour
The original client should receive `BKLedgerFencedException` (or equivalent)
on the `addEntry` call — **not** a success acknowledgement — because the ledger
was already fenced and closed by the recovery process before the write was
attempted.
Alternatively, if `addEntry` is allowed to succeed, `lh.close()` should not
silently discard the entry: it should either succeed (reconcile the metadata)
or propagate a clear error that signals the committed entry can no longer be
confirmed.
## Root Cause Analysis
BookKeeper's fencing protocol works by sending `fenceLedger` RPCs to the
**current ensemble** bookies. Those bookies record the fence in memory and
reject future writes with `BKLedgerFencedException`. This relies on the
**same** bookie processes being present at the time of both the fence and any
subsequent write attempt.
When a bookie is replaced by a **fresh process at the same IP and port**,
the new bookie has no in-memory fence state for any previously open ledger. The
original client — whose `LedgerHandle` still holds a valid connection to the
(now new) bookies — sends `addEntry` RPCs which the new bookies accept without
complaint, because from their perspective this ledger has never been fenced.
The result is that `addEntry` completes with a success acknowledgement from
`AQ` (=2) replicas, advancing `LAC` to 1. The entry is physically written to
the new bookies' journals. But since ZK metadata records `lastEntry=0` (set
during recovery before the writer wrote), the entry will never be visible to
any future reader and is functionally lost.
The `BKMetadataVersionException` from `lh.close()` is the only signal the
writer receives, and it arrives **after** `addEntry` has already confirmed the
write — too late for the application to react.
## Impact
- **Silent data loss**: writes that receive a successful `addEntry`
acknowledgement can be permanently dropped with no indication to the writer.
- This affects any workflow where bookies are decommissioned via `bookkeeper
shell decommissionbookie` while writers hold open `LedgerHandle` objects whose
ensembles include the decommissioned bookies — a routine operational procedure.
- Reproduced on two independent Pulsar/BookKeeper releases (3.0.x / 4.0.x),
suggesting this is a fundamental design gap rather than a version-specific
regression.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
