metanil commented on PR #59760: URL: https://github.com/apache/spark/pull/59760#issuecomment-6547622792
@peter-toth pushed the SPARK-60027 part as f82b578b1ea, and updated the title and description. Quick summary: - `evaluableFunction` is the call `eval` runs now. It requires the arguments, bucket count included, to suit `inputTypes()`. Values stored the same way go as they are: the same physical type, nested types by position ignoring names and nullability, a UDT as its `sqlType`, a decimal at the same scale. A narrower integral gets an explicit `Cast`, so your Iceberg `TINYINT`/`SMALLINT` case keeps its plan, and it works for `produceResult` functions too, not just magic `invoke` ones. Anything else is refused and shuffles, including `DATE` vs `TIMESTAMP` because of the time-zone cast (SPARK-60539). - Your `years` over `DATE` example: a one-side shuffle now returns all 3 rows, instead of 1 of 3. - It all goes through `canEvaluate`, so it backports with SPARK-60522. - For #55885: I'd land this one first, then rebase #55885 onto it. That drops its column check from both gates, since it's stricter than this one and would bring back the `SMALLINT` shuffle and the cogroup pairing from your finding 27. #55885 keeps only the literal-parameter checks the generalized reducer needs, and its migration note shrinks to that part. This is also inline with what you've mentioned previously. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
