pvary commented on code in PR #18717: URL: https://github.com/apache/iceberg/pull/18717#discussion_r4468496337
########## docs/docs/flink-maintenance.md: ########## @@ -115,12 +115,14 @@ Notes: - The PK-index worker keeps a keyed row-position index of the target branch in Flink state, persisted with each Flink checkpoint. Its size scales with the number of live rows in the table for the configured equality columns; size Flink state backend, checkpoint storage, and TaskManager memory accordingly. **RocksDB is the preferred state backend** because the PK index can grow beyond the available heap and RocksDB spills to local disk instead of failing with an `OutOfMemoryError`. - The index is **reused across triggers and restored on failover** while it matches the head of the target branch: the head is the snapshot the index was built from, or the converter's own commit for the last planned staging snapshot directly on top of it. Each cycle applies its staging snapshot's changes to the index, so consecutive conversions do not rebuild it. Review Comment: Do we need to fix this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
