tlfua opened a new issue, #7585: URL: https://github.com/apache/datafusion-comet/issues/7585
### What is the problem the feature request solves? Comet's Scala side has more than one answer to "can evaluating this expression raise an error?": - `cannotThrow` in `arrays.scala` admits only a `Literal`, `Attribute` or `BoundReference`. `CometArrayAppend` uses it to decide whether the item can be evaluated for a row Spark would not evaluate it on: https://github.com/apache/datafusion-comet/blob/8f4ce5e643e9c10803369dae6a1646195f36cb72/spark/src/main/scala/org/apache/comet/serde/arrays.scala#L86-L89 - `QueryPlanSerde.cannotFail`, added in https://github.com/apache/datafusion-comet/pull/7457 , answers the same question for TopK's later sort keys with a wider list (non-ANSI arithmetic, some string functions, casts through `CometCast.cannotFail`). - https://github.com/apache/datafusion-comet/pull/7322 adds a third list, `NativeLambdaSerde.mayRaise`, for native lambdas, and https://github.com/apache/datafusion-comet/issues/7535 already tracks a case where it is stricter than it needs to be. Each copy has to be widened or tightened on its own, so a fix to one (for example a new cast that cannot fail, or a kernel found to raise where Spark does not) is easy to miss in the others. ### Describe the potential solution Keep one Scala notion of an expression that cannot fail, `QueryPlanSerde.cannotFail`, and have `arrays.scala` (and `NativeLambdaSerde.mayRaise`, if https://github.com/apache/datafusion-comet/pull/7322 lands first) call it instead of keeping its own list. This widens when `array_append` runs natively, so the change needs its own tests for the newly admitted item expressions, including ones that fail under ANSI. ### Additional context Raised in the review of https://github.com/apache/datafusion-comet/pull/7457 : - https://github.com/apache/datafusion-comet/pull/7457#discussion_r4460105722 - https://github.com/apache/datafusion-comet/pull/7457#pullrequestreview-5831312676 ### Willingness to contribute I can contribute a fix for this bug independently -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
