anuragmantri opened a new pull request, #18615:
URL: https://github.com/apache/iceberg/pull/18615
Follow-up to #17956, which returns every opaque partition predicate to Spark
for re-evaluation. After this PR, a predicate is returned only when some spec
in the table cannot evaluate it, using the same per-spec check the scan uses to
prune tasks.
This PR adds two guards to keep results correct.
- The changelog builder still returns these predicates because its scan does
not evaluate them.
- Aggregate pushdown is skipped when partition predicates are present, since
it ignores them and `count(*)` returns the unfiltered count otherwise.
The check covers all table specs rather than only scanned ones, which is
conservative. Narrowing it is possible as a follow-up.
Test plan:
- builder tests cover the dropped, kept and spec-evolved cases, including a
field whose ordinal differs across specs, plus the changelog builder.
- SQL tests cover count(*) and results on an evolved spec.
---
AI Disclosure
- Model: Claude Sonnet 5.5
- Platform/Tool: Claude Code
- Human Oversight: fully reviewed by me.
- Prompt Summary: Implement the TODO in BaseSparkScanBuilder.pushPredicates
left by #17956
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
