kinolaev opened a new pull request, #19062:
URL: https://github.com/apache/iceberg/pull/19062
`ConvertEqualityDeletes` keeps every live row of the target branch in its PK
index, so state grows with the table even when updates reach only recent
partitions. This PR adds `partitionIndexIdleTimeout` (off by default): the
planner indexes a partition when its first equality delete arrives and evicts
it after the timeout passes without one. An unpartitioned delete indexes every
partition. A partition that receives rows in the evicting cycle is kept,
because the worker could apply those rows after the eviction. Changing the
setting rebuilds the index on restore.
Open question: lazy loading without eviction never indexes or reads more
than the full index, at the cost of planning the snapshot when deletes first
reach a partition. Should it become the only mode, with the idle timeout
controlling only eviction?
Tests:
- Planner:
- a partition is loaded on its first delete (every partition for an
unpartitioned delete) and not again while it stays indexed
- each delete restarts the timeout of the partitions it applies to, and a
partition is evicted once its timeout passes, unless the cycle sends rows to it
- after an eviction or a rebuild, the next delete that reaches a partition
loads it again
- staging and external files of unindexed partitions are skipped
- on restore, the index is rebuilt when the index mode changes
- Worker: eviction removes only the evicted partitions' rows and leaves no
state for emptied keys.
- End to end: a partition evicted and reloaded with earlier DVs gives
correct results.
@mxm
---
**AI Disclosure**
- Model: Claude Opus 5.5
- Platform/Tool: Claude Code
- Human Oversight: fully reviewed
- Prompt Summary: Walk me through ConvertEqualityDeletes, then help me
design a partition-scoped lazy PK index for it and build a test plan.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]