anuragmantri opened a new pull request, #58979: URL: https://github.com/apache/spark/pull/58979
### What changes were proposed in this pull request? `V2ExpressionUtils.toCatalystOpt` and `toCatalystTransformOpt` should gracefully return `None` when a reference can't be resolved. This is similar to how `V2ScanPartitioningAndOrdering`'s `partitioning()` rule relies on to degrade a `KeyGroupedPartitioning` to `UnknownPartitioning` instead of crashing when a partition key isn't resolvable (SPARK-59248) This PR adds a `resolveRefOpt`, and makes `resolveRef` a thin wrapper around it. ### Why are the changes needed? Found while reviewing SPARK-58111 ([PR #55518](https://github.com/apache/spark/pull/55518)). This is a general, pre-existing bug in `V2ExpressionUtils`, not something introduced by #55518. ### Does this PR introduce _any_ user-facing change? Yes. A query against a data source whose scan reports a `KeyGroupedPartitioning` with key that can't be resolved previously failed outright. it now succeeds, with partitioning degraded to `unknown` instead. One narrow side effect: `V2ExpressionUtils.toCatalystOrdering` still fails when a reported ordering key can't resolve, but the error message changes from the specific "Unable to resolve X given [...]") to the generic ("X is not currently supported"). ### How was this patch tested? New tests in `V2ExpressionUtilsSuite`: - `toCatalystOpt` returns `None` (not throw) for an unresolvable `FieldReference`, and still resolves an existing one. - `toCatalystTransformOpt` returns `None` for an unresolvable `IdentityTransform` ref, `BucketTransform` ref, and `NamedTransform` arg. - `toCatalystOrdering` still fails on an unresolvable sort key, asserting the new generic error condition. Also ran the full `KeyGroupedPartitioningSuite` to confirm no regression in the partitioning/SPJ paths this change touches. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Sonnet 5) Verified manually by me. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
