andygrove opened a new issue, #346: URL: https://github.com/apache/datafusion-spark/issues/346
## Is your feature request related to a problem or challenge? Spark casts a function's arguments to its declared input types in one place. Most expressions mix in `ImplicitCastInputTypes` and declare `inputTypes: Seq[AbstractDataType]`, for example `DoubleType`, `NumericType` or `TypeCollection(StringType, BinaryType)`. One rule, `implicitCast(from, expected)` in `TypeCoercion` and `AnsiTypeCoercion`, then inserts the casts, and `checkInputDataTypes` raises `DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE` for the rest. Here each function writes a DataFusion `Signature` by hand (`Exact`, `Coercible`, or `user_defined` with `coerce_types`), and picks which source types to accept. That leads to the same bugs again and again: - Arguments that Spark casts are rejected: #26 (`pmod`), #29 (`factorial`), #41 (`last_day`, `monthname`, `weekday`, `round`, `unbase64`), and string arguments to `ln` and `log` in #334. - A rejected type gives DataFusion's `coercion from Utf8 to the signature Exact(Float64) failed` instead of Spark's `UNEXPECTED_INPUT_TYPE`. - ANSI mode reaches a function's coercion only when that function opts in through `with_updated_config`. - Nothing checks a function's accepted types against Spark's `implicitCast` table, so reviewers check each one by hand. Finding a common type for several arguments (`greatest`, `array_position` and the other array functions) already has one shared copy of Spark's rules in `function/type_coercion.rs` (#345). This issue is about the other half: casting to declared input types. ## Describe the solution you'd like Let a function declare its input types the way Spark does, and coerce them in one place: 1. A `SparkInputType` enum in `function/type_coercion.rs`, mirroring the `AbstractDataType`s that Spark functions use: `Double`, `Integer`, `Long`, `Numeric`, `Integral`, `Decimal`, `String`, `Binary`, `Date`, `Timestamp`, `Boolean`, `Any`, `OneOf(Vec<SparkInputType>)`, and array and map forms. 2. `spark_coerce_inputs(name, arg_types, &[SparkInputType], ansi_mode) -> Result<Vec<DataType>>`, a port of `implicitCast` for both modes, that returns Spark's `UNEXPECTED_INPUT_TYPE` error, with the SQLSTATE, when an argument cannot be cast. 3. A small trait or macro that implements `ScalarUDFImpl::coerce_types` from the declared types, with `Signature::user_defined`, and passes ANSI mode through `with_updated_config`. 4. For functions that cast all their arguments to one common type (`greatest`, `least`, `coalesce`, the array functions), a declared form that uses the existing common-type rules. Then each function states what Spark declares, and the casts, the ANSI difference and the error text come out the same everywhere. Points to settle: - `user_defined` signatures lose DataFusion's signature-based documentation and `information_schema` output. The declared types could generate that text instead. - The cast that runs is still DataFusion's cast, so differences such as a malformed string returning NULL with ANSI mode off (#88) stay with the cast work in #88. - Comet calls the physical expressions directly and does its coercion in Spark's plan, so this mainly affects DataFusion SQL users. ## Describe alternatives you've considered Keep per-function `Signature`s and fix each gap as it is found, as in #26, #29 and #41. That works, but every new function repeats the work and the review. ## Additional context I suggest starting with a few functions (`ln` and `log` from #334, `pmod` for #26, `factorial` for #29) before migrating the rest. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
