andygrove opened a new pull request, #350:
URL: https://github.com/apache/datafusion-spark/pull/350
## Which issue does this PR close?
- Part of #346.
## Rationale for this change
Spark casts a function's arguments to the input types the function declares,
in one place. Here each function picks its accepted types in a hand-written
`Signature`, so some arguments that Spark casts are rejected, and a rejected
type gives DataFusion's error instead of Spark's.
`sha2` shows both problems. Spark declares `(BINARY, INT)`:
| Query | Before | Spark 4.2.0 and this PR |
|---|---|---|
| `sha2('Spark', '256')` | `Function 'sha2' requires Int32, but received
String` | the SHA-256 hash |
| `sha2('Spark', 256.7::DOUBLE)` | `requires Int32, but received Float64` |
the SHA-256 hash |
| `sha2('Spark', 256.0::DECIMAL(10, 2))` | `requires Int32, but received
Decimal(10, 2)` | the SHA-256 hash |
| `sha2(1::INT, 256)` | `requires Binary, but received Int32` | ``
[DATATYPE_MISMATCH.UNEXPECTED_INPUT_TYPE] The first parameter of `sha2`
requires the "BINARY" type, however it has the type "INT". SQLSTATE: 42K09 `` |
| `sha2('Spark')` | `Function 'sha2' expects 2 arguments but received 1` |
`` [WRONG_NUM_ARGS.WITHOUT_SUGGESTION] The `sha2` requires 2 parameters but the
actual number is 1. ... `` |
Spark's message also names the argument's SQL text (`"1"`). `coerce_types`
sees only the types, so the message here says "it has the type" instead.
## What changes are included in this PR?
- New `function/implicit_cast.rs`:
- `SparkInputType` mirrors the `AbstractDataType`s Spark functions
declare: the concrete scalar types, `Numeric`, `Integral`, `Decimal`, `Array`
and `OneOf` (Spark's `TypeCollection`).
- `implicit_cast` ports the private `implicitCast(DataType,
AbstractDataType)` from both `TypeCoercion` (ANSI mode off) and
`AnsiTypeCoercion` (ANSI mode on), with `Cast.canANSIStoreAssign` and
`DecimalType.forType`. The two differ, for example a string becomes a BOOLEAN
only with ANSI mode on.
- `SparkInputTypes` holds the declared types and the ANSI flag. A function
calls `coerce` from `coerce_types` and `with_config` from
`with_updated_config`, so each function needs two short methods. `coerce`
returns Spark's `UNEXPECTED_INPUT_TYPE` and `WRONG_NUM_ARGS` errors with the
SQLSTATE.
- A string cast to BINARY keeps its layout (`Utf8View` becomes
`BinaryView`). A dictionary argument has the type of its values, as in #345
(#349 covers encodings more broadly).
- `sha2` uses `Signature::user_defined` with Spark's `(BINARY, INT)`
declaration.
Left for later PRs:
- DATE and TIMESTAMP targets need the session time zone. They can come with
the first function that declares them. Map targets are left out until a
function declares one.
- The module is in its own file so it does not conflict with #345, which
adds `function/type_coercion.rs`. `spark_sql_type` is copied from #345, and
whichever PR lands second should use the other's copy. The two files could
merge once both land.
- The cast that runs is DataFusion's. With ANSI mode off Spark returns NULL
for `sha2('Spark', '256x')` and wraps `sha2('Spark', 4294967552)` to 256, and
both raise a cast error here. That is part of the CAST work in #88.
- Most `SparkInputType` variants are unused until more functions move over,
so the enum has a scoped `allow(dead_code)`.
## Are these changes tested?
Yes. Unit tests in `implicit_cast.rs` cover each rule in both modes,
`OneOf`, arrays, dictionaries and the error text. New records in
`hash/sha2.slt` cover the casts, the errors and ANSI mode. I checked `sha2.slt`
against Spark 4.2.0 with the PySpark runner, and all 43 records match. 17 of
the 22 new records fail without this change.
## Are there any user-facing changes?
`sha2` accepts STRING, FLOAT, DOUBLE and DECIMAL bit lengths, and its errors
for other types and for the wrong number of arguments now match Spark's. There
are no public API changes, since the new module is `pub(crate)`.
This PR was prepared with AI assistance (Claude Code) and reviewed by me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]