kinolaev opened a new pull request, #19062:
URL: https://github.com/apache/iceberg/pull/19062

   `ConvertEqualityDeletes` keeps every live row of the target branch in its PK 
index, so state grows with the table even when updates reach only recent 
partitions. This PR adds `partitionIndexIdleTimeout` (off by default): the 
planner indexes a partition when its first equality delete arrives and evicts 
it after the timeout passes without one. An unpartitioned delete indexes every 
partition. A partition that receives rows in the evicting cycle is kept, 
because the worker could apply those rows after the eviction. Changing the 
setting rebuilds the index on restore.
   
   Open question: lazy loading without eviction never indexes or reads more 
than the full index, at the cost of planning the snapshot when deletes first 
reach a partition. Should it become the only mode, with the idle timeout 
controlling only eviction?
   
   Tests:
   - Planner:
     - a partition is loaded on its first delete (every partition for an 
unpartitioned delete) and not again while it stays indexed
     - each delete restarts the timeout of the partitions it applies to, and a 
partition is evicted once its timeout passes, unless the cycle sends rows to it
     - after an eviction or a rebuild, the next delete that reaches a partition 
loads it again
     - staging and external files of unindexed partitions are skipped
     - on restore, the index is rebuilt when the index mode changes
   - Worker: eviction removes only the evicted partitions' rows and leaves no 
state for emptied keys.
   - End to end: a partition evicted and reloaded with earlier DVs gives 
correct results.
   
   @mxm
   
   ---
   **AI Disclosure**
   - Model: Claude Opus 5.5
   - Platform/Tool: Claude Code
   - Human Oversight: fully reviewed
   - Prompt Summary: Walk me through ConvertEqualityDeletes, then help me 
design a partition-scoped lazy PK index for it and build a test plan.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to