goutamadwant opened a new pull request, #2456: URL: https://github.com/apache/auron/pull/2456
# Which issue does this PR close? Closes #2431 # Rationale for this change Iceberg changelog metadata values are constant for each changelog task. Auron currently plans every supported task in the changelog range even when a predicate on `_change_type`, `_commit_snapshot_id`, or `_change_ordinal` proves that a task cannot match. # What changes are included in this PR? - Associate supported metadata-only filters with an unambiguous Iceberg changelog scan before adaptive query stages are created. - Prune validated changelog tasks for `=` and `IN` predicates combined with `AND`. - Keep `OR`, `NOT`, complex expressions, mixed data-column predicates, and ambiguous scan matches on the existing post-scan filter path. - Preserve existing fallback validation for unsupported changelog tasks and mixed file formats. - Add regression coverage for matching, nonmatching, conjunction, and unsupported-expression cases. # Are there any user-facing changes? Yes. Eligible Iceberg changelog queries plan and read fewer native scan files. There are no public API or configuration changes. # How was this patch tested? - `AuronIcebergIntegrationSuite`: 43 tests passed. - Focused changelog metadata pruning regression: passed. - `./dev/reformat --check`: passed across the configured Spark 3.0-3.5 and 4.0-4.1 profiles. - `./build/mvn -B spotless:check test-compile -pl thirdparty/auron-iceberg -am -Ppre -Pscala-2.12 -Pspark-3.5 -Piceberg-1.10.1 -DskipBuildNative -DskipTests`: passed. # Was this patch authored or co-authored using generative AI tooling? - [ ] Yes - [ ] No Codex was used to Review and understand the codebase <!-- Select one option. If yes, add the repository-required Generated-by line before submitting. --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
