mxm commented on code in PR #18717: URL: https://github.com/apache/iceberg/pull/18717#discussion_r4492941058
########## docs/docs/flink-maintenance.md: ########## @@ -114,23 +114,25 @@ Notes: **State and restart** - The PK-index worker keeps a keyed row-position index of the target branch in Flink state, persisted with each Flink checkpoint. Its size scales with the number of live rows in the table for the configured equality columns; size Flink state backend, checkpoint storage, and TaskManager memory accordingly. **RocksDB is the preferred state backend** because the PK index can grow beyond the available heap and RocksDB spills to local disk instead of failing with an `OutOfMemoryError`. -- The index is **reused across triggers and restored on failover** while it matches the head of the target branch: the head is the snapshot the index was built from, or the converter's own commit for the last planned staging snapshot directly on top of it. Each cycle applies its staging snapshot's changes to the index, so consecutive conversions do not rebuild it. -- Otherwise the next trigger **rebuilds the index** from the target branch head, reading the equality columns of every live data file. This happens: +- The index is **reused across triggers and restored on failover** while it matches the head of the target branch, or while it can be reached from the latest head of the target branch. We refresh the index based on the delta between the index snapshot and the latest snapshot, so it corresponds to the latest table state. Review Comment: Ack, I like to use the royal we, but it's not for everybody 😄 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
