andygrove opened a new pull request, #317:
URL: https://github.com/apache/datafusion-spark/pull/317
## Which issue does this PR close?
Part of #88: the "Rework the `spark_cast` UDF" item. Builds on #304, which
builds on #89. Until those merge, this PR also shows their commits. Only the
last commit, `feat(spark): parse Spark type names in spark_cast and call the
cast kernel`, is new here.
## Rationale for this change
`spark_cast(expr, 'timestamp')` only supported a `timestamp` target and
integer or float input, with its own conversion code. To test the
Spark-compatible cast kernel from SQL and SLT as each family is ported, the
function needs to accept any Spark type name and call the kernel.
## What changes are included in this PR?
- **Target type parser** (`cast/data_type.rs`): parses the second argument
with Spark's `dataType` grammar from `SqlBaseParser.g4`, case-insensitively.
- **Primitive types:** `boolean`, `tinyint`/`byte`, `smallint`/`short`,
`int`/`integer`, `bigint`/`long`, `float`/`real`, `double`, `date`, `binary`
and `string`.
- **Decimals:** `decimal`/`dec`/`numeric`, with an optional precision and
scale. A bare `decimal` is `decimal(10, 0)`, and precision and scale are
validated.
- **Character types:** `char(n)` and `varchar(n)` become `STRING`, which
is what Spark casts them to by default.
- **Timestamps:**
- `timestamp`, `timestamp_ltz` and `timestamp with local time zone` give
a `TIMESTAMP` tagged with the session time zone. That matches Spark's default
`spark.sql.timestampType`.
- `timestamp_ntz` and `timestamp without time zone` give a
`TIMESTAMP_NTZ`.
- **Complex types:** `array<…>`, `map<…, …>` (with Spark's `ArrowUtils`
field names `entries`, `key` and `value`), and `struct<…>`. Struct fields can
be backquoted, `NOT NULL` or have a `COMMENT`, and duplicate field names are
allowed, as in Spark.
- **Unsupported types:** intervals, `TIME`, `VARIANT`, `VOID`, geospatial
types and collated strings get an "unsupported" error. Malformed type names get
a parse error that names the problem.
- **`spark_cast` UDF:**
- accepts any input type
- requires the target type to be a string literal
- computes its return field from the parsed type
- calls the kernel with `EvalMode::from_config` and the session time zone
as a `SparkTimeZone`
The old conversion code in the UDF is removed.
- **Integer and float to `TIMESTAMP`:** ported into the kernel
(`cast/numeric.rs`). Spark has no cast from a number to `TIMESTAMP_NTZ`, so
that cast is rejected.
Float input follows Spark 4.2.0's `doubleToTimestamp` and
`doubleToTimestampAnsi`:
| Mode | NaN or infinite | Out of range |
|---|---|---|
| Legacy | NULL | saturates, like Scala's `toLong` |
| ANSI | `CAST_INVALID_INPUT` | `CAST_OVERFLOW`, such as `1.0E25D` from
`DOUBLE` to `BIGINT` |
| TRY | NULL | NULL |
The previous UDF already saturated in Legacy mode. Comet returns NULL
there, which differs from Spark.
## Are these changes tested?
Yes.
- **Unit tests:**
- the type parser: every primitive and complex type, invalid names and
unsupported types
- integer and float to timestamp in each eval mode, including saturation,
a NaN under a null, `FLOAT` widening and the error messages
- the UDF: return fields, a non-literal target type, array and scalar
input, and reading ANSI mode from the config
- **New SLT file** `conversion/cast_target_type.slt`: return types from
`arrow_typeof`, the session time zone on `timestamp`, casts that already work
through Arrow, invalid and unsupported targets, and casts that are not
implemented yet.
- **Existing SLT files:** `cast_int_to_timestamp.slt` and
`cast_float_to_timestamp.slt` still pass, with these changes:
- The old "unsupported target" test used `spark_cast(100, 'string')`,
which now works, so it now uses `interval` instead.
- It also now checks that a number can't be cast to `timestamp_ntz`.
- The ANSI errors are checked against Spark's messages.
- It has a Legacy saturation test that casts the result back to `bigint`.
The old file said this couldn't be shown in SLT.
## Are there any user-facing changes?
Yes. `spark_cast` accepts any Spark type name as its target and any input
type.
- **Newly working:** casts that are already Spark-compatible through Arrow
now succeed. One example is `spark_cast(42, 'string')`.
- **Errors for casts not ported yet:** those return an explicit "not
supported" or "not implemented yet" error rather than a result that differs
from Spark's. Integer narrowing is one example. The unsuffixed literal `1` is a
`BIGINT`, so even `spark_cast(1, 'int')` needs that port.
- **Error message format:** in ANSI mode, errors for NaN and out-of-range
input now use Spark's `[CAST_INVALID_INPUT]` and `[CAST_OVERFLOW]` messages.
_This PR was written by an LLM (Claude) from the Comet and Spark sources,
and reviewed before opening._
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]