andygrove opened a new issue, #346:
URL: https://github.com/apache/datafusion-spark/issues/346

   ## Is your feature request related to a problem or challenge?
   
   Spark casts a function's arguments to its declared input types in one place. 
Most expressions mix in `ImplicitCastInputTypes` and declare `inputTypes: 
Seq[AbstractDataType]`, for example `DoubleType`, `NumericType` or 
`TypeCollection(StringType, BinaryType)`. One rule, `implicitCast(from, 
expected)` in `TypeCoercion` and `AnsiTypeCoercion`, then inserts the casts, 
and `checkInputDataTypes` raises `DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE` for 
the rest.
   
   Here each function writes a DataFusion `Signature` by hand (`Exact`, 
`Coercible`, or `user_defined` with `coerce_types`), and picks which source 
types to accept. That leads to the same bugs again and again:
   
   - Arguments that Spark casts are rejected: #26 (`pmod`), #29 (`factorial`), 
#41 (`last_day`, `monthname`, `weekday`, `round`, `unbase64`), and string 
arguments to `ln` and `log` in #334.
   - A rejected type gives DataFusion's `coercion from Utf8 to the signature 
Exact(Float64) failed` instead of Spark's `UNEXPECTED_INPUT_TYPE`.
   - ANSI mode reaches a function's coercion only when that function opts in 
through `with_updated_config`.
   - Nothing checks a function's accepted types against Spark's `implicitCast` 
table, so reviewers check each one by hand.
   
   Finding a common type for several arguments (`greatest`, `array_position` 
and the other array functions) already has one shared copy of Spark's rules in 
`function/type_coercion.rs` (#345). This issue is about the other half: casting 
to declared input types.
   
   ## Describe the solution you'd like
   
   Let a function declare its input types the way Spark does, and coerce them 
in one place:
   
   1. A `SparkInputType` enum in `function/type_coercion.rs`, mirroring the 
`AbstractDataType`s that Spark functions use: `Double`, `Integer`, `Long`, 
`Numeric`, `Integral`, `Decimal`, `String`, `Binary`, `Date`, `Timestamp`, 
`Boolean`, `Any`, `OneOf(Vec<SparkInputType>)`, and array and map forms.
   2. `spark_coerce_inputs(name, arg_types, &[SparkInputType], ansi_mode) -> 
Result<Vec<DataType>>`, a port of `implicitCast` for both modes, that returns 
Spark's `UNEXPECTED_INPUT_TYPE` error, with the SQLSTATE, when an argument 
cannot be cast.
   3. A small trait or macro that implements `ScalarUDFImpl::coerce_types` from 
the declared types, with `Signature::user_defined`, and passes ANSI mode 
through `with_updated_config`.
   4. For functions that cast all their arguments to one common type 
(`greatest`, `least`, `coalesce`, the array functions), a declared form that 
uses the existing common-type rules.
   
   Then each function states what Spark declares, and the casts, the ANSI 
difference and the error text come out the same everywhere.
   
   Points to settle:
   
   - `user_defined` signatures lose DataFusion's signature-based documentation 
and `information_schema` output. The declared types could generate that text 
instead.
   - The cast that runs is still DataFusion's cast, so differences such as a 
malformed string returning NULL with ANSI mode off (#88) stay with the cast 
work in #88.
   - Comet calls the physical expressions directly and does its coercion in 
Spark's plan, so this mainly affects DataFusion SQL users.
   
   ## Describe alternatives you've considered
   
   Keep per-function `Signature`s and fix each gap as it is found, as in #26, 
#29 and #41. That works, but every new function repeats the work and the review.
   
   ## Additional context
   
   I suggest starting with a few functions (`ln` and `log` from #334, `pmod` 
for #26, `factorial` for #29) before migrating the rest.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to