andygrove opened a new issue, #384:
URL: https://github.com/apache/datafusion-spark/issues/384

   ## Is your feature request related to a problem or challenge?
   
   Some Spark expressions require an argument to be foldable, that is a 
constant expression, and reject anything else in `checkInputDataTypes` with 
`DATATYPE_MISMATCH.NON_FOLDABLE_INPUT`. In this crate these include the `scale` 
of `round` and `bround` (`RoundBase`), `n` in `stack`, and `fmt` in 
`to_binary`. `to_json`'s options and the `from_json`/`from_csv` schemas also 
must be constants.
   
   Each function here handles this differently, and none of them match Spark:
   
   - `stack` reads `scalar_arguments` in `return_field_from_args`, so it only 
accepts a literal, not a constant expression like `3 - 1` (its docs note this 
difference). Its error message is not Spark's.
   - `bround` reads `scalar_arguments` and falls back to the input's scale when 
the argument is a constant that is not yet a literal, such as `1::bigint`. It 
relies on the optimizer folding it later.
   - `to_binary` checks for `ColumnarValue::Scalar` in `invoke_with_args`, at 
execution time, with its own error message.
   - `round` only rejects a column `scale` when it is evaluated, with its own 
error message.
   
   ## Describe the solution you'd like
   
   Let `SparkInputTypes` mark an argument as foldable. In 
`return_field_from_args` it would:
   
   - raise Spark's `DATATYPE_MISMATCH.NON_FOLDABLE_INPUT` error for an argument 
that is not a constant
   - return the constant's value, so that the function can use it to compute 
its return type
   
   `return_field_from_args` only sees literals in `scalar_arguments`, so the 
helper must decide what to do with a constant expression that is not yet 
folded, such as `CAST(1 AS BIGINT)` or `3 - 1`. One option is to fold it with 
DataFusion's `ConstEvaluator` before the check.
   
   ## Describe alternatives you've considered
   
   Keep the checks in each function. They would stay inconsistent, and each new 
function that needs one would add another variant.
   
   ## Additional context
   
   Follows on from #346. Related: #40 (`trunc` and `date_trunc` with 
non-literal formats). Spark 4.1 functions that use `NON_FOLDABLE_INPUT` include 
`RoundBase`, `Stack`, `ToBinary`, `RegExpReplace` (`position`), `SortArray` 
(`ascendingOrder`), `ToCharacter`/`ToNumberBase`, `PercentileBase` and 
`ApproximatePercentile`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to