mxm opened a new pull request, #16889: URL: https://github.com/apache/iceberg/pull/16889
This adds EqualityConvertPlanner, the operator that drives the equality delete conversion pipeline. For every trigger it picks the next unconverted staging snapshot and emits the read commands and plan metadata that drive the rest of the pipeline. Equality-delete resolution is only correct against an index that reflects the current target branch, so each trigger reconciles the worker index before resolving. It builds the index on the first run and rebuilds it when external commits (compaction, concurrent writes) have moved the branch, evicting entries the rebuild won't overwrite. Read commands are split into watermark-separated phases because the readers run in parallel: without an explicit ordering the worker could resolve a delete against a stale or half-built index. Idempotency is ensured by the planner, not the committer. It recognizes already-converted snapshots from the committer's marker on the target branch, so a restart re-derives its position and never reprocesses a committed snapshot. The planner fails the cycle on inputs it cannot convert correctly rather than dropping them silently: data-file rewrites, V2 positional deletes, and equality deletes whose field IDs don't match the configured set. When there is nothing to convert it still emits a no-op plan so the maintenance task run completes. This PR is factored out of #15996. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
