pvary commented on code in PR #18674: URL: https://github.com/apache/iceberg/pull/18674#discussion_r4441971285
########## docs/docs/flink-maintenance.md: ########## @@ -114,9 +114,9 @@ Notes: **State and restart** - The PK-index worker keeps a keyed row-position index of the target branch in Flink state and **maintains it incrementally across checkpoints**: each trigger cycle only applies the commits added on the target branch since the last indexed snapshot, and the resulting index is persisted with the next Flink checkpoint. On failover, the worker resumes from the most recent checkpointed index rather than rebuilding from scratch. Its size scales with the number of live rows in the table for the configured equality columns; size Flink state backend, checkpoint storage, and TaskManager memory accordingly. **RocksDB is the preferred state backend** because the PK index can grow beyond the available heap and RocksDB spills to local disk instead of failing with an `OutOfMemoryError`. -- Full (re)indexing from the target branch head is only triggered in a few cases: cold start with no checkpoint, external commits that advance the target past the currently-indexed snapshot (see below), when the planner detects that the target branch has diverged from the indexed snapshot (rollback / replace-main / expired marker), or when a staging snapshot is planned again because the cycle that planned it did not commit. +- Full (re)indexing from the target branch head is triggered when the index does not reflect the head of the target branch: cold start with no checkpoint, any commit on the target other than the conversion commit of the staging snapshot planned last (external commits as described below, rollback / replace-main, or a restore from a checkpoint taken before a conversion commit), a planned staging snapshot whose conversion commit did not land, after an index update failed, or on the first trigger after a restore from state written by an older version. A failed update on a trigger with nothing to convert keeps the next conversion from committing, so that conversion is planned again on a rebuilt index, unless another rebuild finished without a failure in between. - The `equalityFieldColumns` set is persisted in operator state. Reconfiguring it across a restart from savepoint is **not supported** and the job will fail fast on restore. Restart from a clean state (no savepoint) if the equality columns change. -- The committer is intentionally stateless. On restart, the planner rediscovers its position by walking the target branch for the marker property `equality-convert-staging-snapshot` written by the committer. If that marker is no longer reachable on the target branch (target was rolled back, replace-main'ed, or the marker snapshot was expired) the planner fails and requires manual intervention. +- The committer keeps no state besides the index generation of the last failed cycle, which keeps the next conversion commit on that generation from committing. On restart, the planner rediscovers its position by walking the target branch for the marker property `equality-convert-staging-snapshot` written by the committer. If that marker is no longer reachable on the target branch (target was rolled back, replace-main'ed, or the marker snapshot was expired) the planner fails and requires manual intervention. Review Comment: Might just remove the first sentence after the changes above. ```suggestion - On restart, the planner rediscovers its position by walking the target branch for the marker property `equality-convert-staging-snapshot` written by the committer. If that marker is no longer reachable on the target branch (target was rolled back, replace-main'ed, or the marker snapshot was expired) the planner fails and requires manual intervention. ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
