pvary commented on code in PR #18674:
URL: https://github.com/apache/iceberg/pull/18674#discussion_r4441954139
##########
docs/docs/flink-maintenance.md:
##########
@@ -114,9 +114,9 @@ Notes:
**State and restart**
- The PK-index worker keeps a keyed row-position index of the target branch in
Flink state and **maintains it incrementally across checkpoints**: each trigger
cycle only applies the commits added on the target branch since the last
indexed snapshot, and the resulting index is persisted with the next Flink
checkpoint. On failover, the worker resumes from the most recent checkpointed
index rather than rebuilding from scratch. Its size scales with the number of
live rows in the table for the configured equality columns; size Flink state
backend, checkpoint storage, and TaskManager memory accordingly. **RocksDB is
the preferred state backend** because the PK index can grow beyond the
available heap and RocksDB spills to local disk instead of failing with an
`OutOfMemoryError`.
-- Full (re)indexing from the target branch head is only triggered in a few
cases: cold start with no checkpoint, external commits that advance the target
past the currently-indexed snapshot (see below), when the planner detects that
the target branch has diverged from the indexed snapshot (rollback /
replace-main / expired marker), or when a staging snapshot is planned again
because the cycle that planned it did not commit.
+- Full (re)indexing from the target branch head is triggered when the index
does not reflect the head of the target branch: cold start with no checkpoint,
any commit on the target other than the conversion commit of the staging
snapshot planned last (external commits as described below, rollback /
replace-main, or a restore from a checkpoint taken before a conversion commit),
a planned staging snapshot whose conversion commit did not land, after an index
update failed, or on the first trigger after a restore from state written by an
older version. A failed update on a trigger with nothing to convert keeps the
next conversion from committing, so that conversion is planned again on a
rebuilt index, unless another rebuild finished without a failure in between.
Review Comment:
Getting a bit long, hard to read...
Shall we refactor the documentation a bit
```suggestion
- The index is **reused across triggers and restored on failover** while it
matches the head of the target branch: the head is the snapshot the index was
built from, or the converter's own commit for the last planned staging snapshot
directly on top of it. Each cycle applies its staging snapshot's changes to the
index, so consecutive conversions do not rebuild it.
- Otherwise the next trigger **rebuilds the index** from the target branch
head, reading the equality columns of every live data file. This happens:
- on cold start without a checkpoint, and on the first trigger after
restoring state written by an older Iceberg version
- after any other commit on the target branch, e.g. compaction, external
writes, rollback, or replace-main (see *Concurrent commits on the target
branch* below)
- after restoring a checkpoint that predates the latest conversion commit
- when the last planned conversion did not commit
- when an earlier rebuild did not finish
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
