metanil commented on PR #59760: URL: https://github.com/apache/spark/pull/59760#issuecomment-6374963577
@peter-toth @dongjoon-hyun I ran into this while working on #55885 (SPARK-50593). Only after I'd opened this did I notice Dongjoon had already flagged it in [item 2 on #58943](https://github.com/apache/spark/pull/58943#issuecomment-5763022823), the `cannotEvaluateExpressionError` when the transform isn't a `ScalarFunction`. So this is basically a follow-up to that. It turned out the one-side shuffle has the same problem (since SPARK-48012), not just the identity reduce. With `shuffle.enabled` it fails in a task instead of at planning. So this just refuses those transforms in both gates and lets the join shuffle. One thing you might want to look at: `canEvaluate` catches the `AnalysisException` that resolving a `ScalarFunction` can throw, e.g. when `produceResult` is inherited from a base class. Without it, a candidate that never gets picked can still fail planning. That's what the third test covers. A plain type check would also work, but then those functions would still blow up when their pair does get picked. Let me know if you'd rather go that way. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
