andygrove opened a new issue, #384: URL: https://github.com/apache/datafusion-spark/issues/384
## Is your feature request related to a problem or challenge? Some Spark expressions require an argument to be foldable, that is a constant expression, and reject anything else in `checkInputDataTypes` with `DATATYPE_MISMATCH.NON_FOLDABLE_INPUT`. In this crate these include the `scale` of `round` and `bround` (`RoundBase`), `n` in `stack`, and `fmt` in `to_binary`. `to_json`'s options and the `from_json`/`from_csv` schemas also must be constants. Each function here handles this differently, and none of them match Spark: - `stack` reads `scalar_arguments` in `return_field_from_args`, so it only accepts a literal, not a constant expression like `3 - 1` (its docs note this difference). Its error message is not Spark's. - `bround` reads `scalar_arguments` and falls back to the input's scale when the argument is a constant that is not yet a literal, such as `1::bigint`. It relies on the optimizer folding it later. - `to_binary` checks for `ColumnarValue::Scalar` in `invoke_with_args`, at execution time, with its own error message. - `round` only rejects a column `scale` when it is evaluated, with its own error message. ## Describe the solution you'd like Let `SparkInputTypes` mark an argument as foldable. In `return_field_from_args` it would: - raise Spark's `DATATYPE_MISMATCH.NON_FOLDABLE_INPUT` error for an argument that is not a constant - return the constant's value, so that the function can use it to compute its return type `return_field_from_args` only sees literals in `scalar_arguments`, so the helper must decide what to do with a constant expression that is not yet folded, such as `CAST(1 AS BIGINT)` or `3 - 1`. One option is to fold it with DataFusion's `ConstEvaluator` before the check. ## Describe alternatives you've considered Keep the checks in each function. They would stay inconsistent, and each new function that needs one would add another variant. ## Additional context Follows on from #346. Related: #40 (`trunc` and `date_trunc` with non-literal formats). Spark 4.1 functions that use `NON_FOLDABLE_INPUT` include `RoundBase`, `Stack`, `ToBinary`, `RegExpReplace` (`position`), `SortArray` (`ascendingOrder`), `ToCharacter`/`ToNumberBase`, `PercentileBase` and `ApproximatePercentile`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
