peter-toth opened a new pull request, #59251:
URL: https://github.com/apache/spark/pull/59251
### What changes were proposed in this pull request?
`TransformExpression.doGenCode` now generates the code of the bound function
call. That is the same call `eval` runs (`resolvedFunction`). It fails the same
way `eval` does when there is no such call:
- the bound function is not a `ScalarFunction`;
- a join reduced the keys, so the call no longer computes them;
- the function declares neither `invoke` nor `produceResult` in its own
class, so building the call throws `SCALAR_FUNCTION_NOT_FULLY_IMPLEMENTED`.
### Why are the changes needed?
With `spark.sql.sources.v2.bucketing.preserveOrderingOnCoalesce.enabled` on,
`GroupPartitionsExec` can coalesce partitions with a k-way merge over its
child's ordering (SPARK-55715). The merge compares rows with generated code
(`LazyCodeGenOrdering`). When the ordering contained a partition transform,
such as `years(arrive_time)`, the task failed:
```
[INTERNAL_ERROR] Cannot generate code for expression:
transformexpression(org.apache.spark.sql.connector.catalog.functions.YearsFunction$@...,
input[2, timestamp, true], None) SQLSTATE: XX000
```
The ordering can contain a transform in two ways:
- the source reports it via `SupportsReportOrdering`, e.g. `[id, name,
years(arrive_time)]`;
- the scan derives it from its partition keys (SPARK-56241), e.g. `[id,
years(arrive_time)]`.
The other consumers of a transform did not need its code. The keyed shuffle
calls `eval`, and the write path replaces the transform with its bound function
call before planning.
### Does this PR introduce _any_ user-facing change?
Yes. Such a query used to fail with the error above. Now it returns its
rows, and the merge stays, as long as Spark can build the call.
### How was this patch tested?
- New tests in `TransformExpressionSuite`:
- "the generated code computes what eval does" checks a function for each
call Spark builds: an `invoke` method, a static `invoke` method and
`produceResult`;
- "a transform whose keys a join reduced generates no code".
- New tests in `KeyGroupedPartitioningSuite`, one for each way above. Each
checks the answer and that the plan k-way merges over the transform. They fail
on master with the error above.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
