andygrove opened a new issue, #88:
URL: https://github.com/apache/datafusion-spark/issues/88
Tracks porting Comet's Spark-compatible `CAST`
(`native/spark-expr/src/conversion_funcs`, about 11.6k lines including about
150 Rust unit tests) into `datafusion-spark`. Once it is here, Comet can call
it instead of its own copy, as it already does for about 34 other expressions.
The crate already has a small `spark_cast(value, 'timestamp')` UDF that
covers integer and float to timestamp. This epic replaces it with the full
implementation.
## Design decisions (settle before porting starts)
- [ ] **Public API.** Proposal: a kernel `spark_cast(ColumnarValue,
&DataType, &SparkCastOptions) -> Result<ColumnarValue>` for engines that build
physical plans (Comet), and the `spark_cast(expr, '<spark type>')` ScalarUDF
built on it for SQL and SLT. Comet's `Cast` `PhysicalExpr`, with its `expr_id`
and `QueryContext`, stays in Comet as a thin wrapper.
- [ ] **Eval mode.** `ConfigOptions` only has `enable_ansi_mode`. Proposal:
a public `EvalMode { Legacy, Ansi, Try }` in `SparkCastOptions`. The UDF reads
`enable_ansi_mode`, and a separate `try_cast` UDF selects `Try`.
- [ ] **Errors.** Comet maps a structured `SparkError` (`CastInvalidValue`,
`InvalidInputInCastToDatetime`, `CastOverFlow`, `NumericValueOutOfRange`) to
JVM exceptions. This crate uses formatted strings, as in `abs.rs`. Cast needs a
typed error, wrapped in `DataFusionError::External`, that renders Spark's
message (`[CAST_INVALID_INPUT] ...`) and that Comet can downcast. This decision
will apply to other functions too.
- [ ] **Spark-version options.** Comet has `is_spark4_plus`,
`decimal_zero_any_exponent` and `decimal_skip_integral_digit_limit`. Proposal:
keep them as options that default to Spark 4.2.0 behavior. SLT tests the
defaults, and Rust unit tests cover the other values (#16,
apache/datafusion#23887).
- [ ] **Options that stay in Comet.** `is_adapting_schema` and
`allow_cast_unsigned_ints` exist for Comet's Parquet schema adapter, not
Spark's `Cast`. Proposal: leave them out. Comet handles those cases before it
calls the kernel.
- [ ] **Target-type syntax for the UDF.** Parse Spark's type grammar (`int`,
`decimal(10,2)`, `timestamp_ntz`, `array<int>`, `map<string,int>`,
`struct<a:int,b:string>`, `binary`, ...), and reject `Dictionary` and other
types Spark can't name.
## Foundation
- [ ] Cast framework: `SparkCastOptions`, `EvalMode`, the `cast_array`
dispatcher with an explicit error for an unsupported cast, and the
`is_datafusion_spark_compatible` allowlist that falls through to Arrow's cast.
Port `utils.rs` (`spark_cast_postprocess`, `timestamp_micros_with_tz`,
`cast_overflow`) and `trim.rs`, plus the crate-level helpers cast needs from
Comet's `utils.rs` (`array_with_timezone`, `resolve_local_datetime`,
`timestamp_ntz_to_timestamp`, `cast_timestamp_to_ntz`).
- [ ] Typed cast error and Spark-formatted messages
- [ ] Rework the `spark_cast` UDF: parse the target type, compute the return
field from it in `return_field_from_args`, and read `enable_ansi_mode` and
`time_zone` from the config. Keep `cast_int_to_timestamp.slt` and
`cast_float_to_timestamp.slt` passing, and reconcile the current int and float
to timestamp code with Comet's.
- [ ] SLT conventions for cast: a `conversion/cast/` directory with one file
per source type family, and Legacy, ANSI (`set
datafusion.execution.enable_ansi_mode = true`) and session time zone sections
in each
- [ ] Port Comet's criterion benchmarks
(`native/spark-expr/benches/cast_*.rs`) alongside each family below
## Porting by source type
Each item ports the code and its Rust unit tests, adds SLT cases checked
against PySpark (harvesting Comet's
`spark/src/test/resources/sql-tests/expressions/cast/*.sql` and the Spark
`CastSuite*`), and runs the `audit-datafusion-spark-expression` checks for that
slice.
**Boolean and numeric**
- [ ] Boolean to integral, float, decimal, string and timestamp
(`boolean.rs`)
- [ ] Integral to integral, with Legacy wrapping and ANSI `CAST_OVERFLOW`
(`spark_cast_int_to_int`)
- [ ] Integral to float, boolean and string (Arrow fallthrough, confirm
compatibility)
- [ ] Integral to decimal (`cast_int_to_decimal128`)
- [ ] Integral to binary, Legacy only (`cast_whole_num_to_binary!`)
- [ ] Integral, float, decimal and boolean to timestamp
- [ ] Float and double to integral
(`spark_cast_nonintegral_numeric_to_integral`)
- [ ] Float and double to decimal (shortest round-trip via `ryu`, matching
`Double.toString`)
- [ ] Float and double to string (`JavaFloatString`, Java's `Float.toString`
and `Double.toString` formatting)
- [ ] Decimal to decimal, with ANSI `NUMERIC_VALUE_OUT_OF_RANGE`
- [ ] Decimal to integral, float (single rounding), boolean and string
(Legacy scientific notation such as `0E-18`)
**String**
- [ ] String to boolean
- [ ] String to integral (`cast_string_to_int`), including Java whitespace
trimming
- [ ] String to float and double
- [ ] String to decimal (exponents, Unicode digits, the integral-digit limit)
- [ ] String to date
- [ ] String to timestamp. This is the largest part of `string.rs`, so it
may need to split into date-time formats, time-only `T` strings, and zone IDs
and offsets.
- [ ] String to timestamp_ntz
- [ ] `LargeUtf8` and `Utf8View` input for every string cast above
**Binary**
- [ ] Binary to string, including `binaryOutputStyle` and invalid UTF-8
(`decode_utf8_spark_lossy` from `datafusion-comet-common`)
- [ ] String to binary
**Temporal**
- [ ] Date to timestamp and timestamp_ntz, and date to int
- [ ] Timestamp to timestamp_ntz and back (session time zone)
- [ ] Date, timestamp and timestamp_ntz to string, and timestamp to date,
long and double
**Complex and special inputs**
- [ ] Struct to struct (`castStruct` semantics, duplicate field names)
- [ ] Struct, array and map to string, Legacy and non-Legacy formats, using
the `null_string` option
- [ ] Array to array, recursively, including Spark's null elements for
`array<date>` to `array<int>`
- [ ] Map to map
- [ ] Dictionary-encoded string and binary input, and `Null` input
## TRY mode
- [ ] `try_cast` UDF using `EvalMode::Try`, with SLT coverage for each
family above
## Adoption
- [ ] Document cast behavior and its known divergences from Spark (#55)
- [ ] Comet: replace `conversion_funcs` with `datafusion-spark`'s cast, keep
`Cast` as a wrapper, run `CometCastSuite` and the cast SQL tests on every
supported Spark version, then delete Comet's copy (tracked in
apache/datafusion-comet)
## Out of scope
- Planning SQL `CAST(x AS t)` to `spark_cast` automatically. DataFusion has
no `ExprPlanner` hook for cast, and Spark's implicit coercions differ from
DataFusion's. Open a follow-up if needed.
- Casts that Comet's native cast does not implement. Comet falls back to
Spark for these.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
