andygrove opened a new issue, #353:
URL: https://github.com/apache/datafusion-spark/issues/353

   ## Is your feature request related to a problem or challenge?
   
   There are now shared helpers for casts and type coercion: the cast kernel 
`spark_cast` (#304), the common-type rules in `function/type_coercion.rs` 
(#345), and `SparkInputTypes` in `function/implicit_cast.rs` (#350). Many 
functions were written before these existed, so they still have their own 
copies of the same logic or use DataFusion's rules where Spark's apply. This 
issue tracks moving them onto the shared code. It follows on from #346.
   
   ## Describe the solution you'd like
   
   ### Duplicated code
   
   - [ ] **Array and element coercion.** `array_position` and `array_remove` 
each have a `coerce_array_and_element`, and `array_insert` does the same steps 
inline: unwrap the list, call `tightest_common_type(.., false)`, check 
`struct_field_types_match` and `is_orderable`, then rebuild the 
`List`/`LargeList`. They differ only in how they report a non-array first 
argument and whether they check ordering. Move one version into 
`type_coercion.rs`, with the `ARRAY_FUNCTION_DIFF_TYPES` and 
`INVALID_ORDERING_TYPE` errors built there. `array_contains`, which still uses 
`Signature::array_and_element`, can then use it too.
   - [ ] **String and binary layout widening in `contains`.** `unify` in 
`string/contains.rs` repeats `wider_physical_type` in `type_coercion.rs`, and 
`classify` repeats the dictionary and `FixedSizeBinary` unwrapping in 
`implicit_cast.rs`.
   
   ### Time zone conversion
   
   - [ ] **`to_utc_timestamp`.** `adjust_to_utc_time` parses the zone with 
chrono's `Tz` and computes the offset itself, while `from_utc_timestamp` uses 
`TimeZoneRules`. It looks up the offset with `offset_from_utc_datetime` on a 
value that is a local time, so the result can be wrong near daylight saving 
changes. `Tz` also does not accept Spark zone IDs such as `+05:30`. Use 
`TimeZoneRules::local_to_utc`, the inverse of what `from_utc_timestamp` uses.
   
   ### Functions to move onto `SparkInputTypes`
   
   These need no new `SparkInputType` variants:
   
   - [ ] `sha1`, `crc32`: `[Binary]`, like `sha2`
   - [ ] `repeat` (`array_repeat`): `[Any, Integer]`. The count is cast to 
BIGINT now, and Spark uses INT.
   
   These need new variants first:
   
   - [ ] A concrete `Decimal(p, s)`: `make_time` (`[Integer, Integer, 
Decimal(16, 6)]`), `make_interval`, `make_dt_interval`
   - [ ] `Date` and `Timestamp`: `weekday`, `dayname`/`monthname` 
(`date_part_name.rs`), `datediff`, `next_day`, `to_utc_timestamp`, 
`unix_date`/`unix_*`
   - [ ] Day-time and year-month intervals: `positive`
   - [ ] Variadic arguments: `elt` (Spark casts the index to INT, while it is 
cast to BIGINT here, and it is checked with Arrow's `can_cast_types`), 
`concat_ws`
   
   These need to be checked against Spark first: `try_sum`, `encode`.
   
   ### Casts at execution time
   
   - [ ] `bround` and `positive` cast a string argument to DOUBLE in 
`invoke_with_args`, so that a malformed string is NULL with ANSI mode off. 
Moving them onto `SparkInputTypes` now would leave the cast to the planner's 
DataFusion cast, which raises an error instead. Once string to number casts are 
ported in #88, use `spark_cast` there, or move them onto `SparkInputTypes` 
(`[Numeric, Integer]` for `bround`).
   
   ### Common type rules
   
   - [ ] `if` uses DataFusion's `try_type_union_resolution`, `array` uses 
`make_array`'s `coerce_types_inner`, and `equal_null` uses 
`comparison_coercion`. Spark uses `findWiderCommonType` for `if` and `array`, 
and its rules for binary comparisons for `<=>`. `type_coercion.rs` only has 
`findTightestCommonType`, so this needs `findWiderCommonType` ported first.
   
   ## Describe alternatives you've considered
   
   Leave the existing functions as they are and use the shared code only in new 
ones. The copies would then drift apart, and the gaps that #346 describes would 
remain in the older functions.
   
   ## Additional context
   
   Checked and left alone: Spark type names in errors all come from 
`spark_sql_type`. The other Arrow casts at execution time change the layout or 
width of values (`pmod`, `shuffle`, `encode`'s charset), where Arrow's cast is 
correct. `to_json` formats timestamps in Spark's JSON format, not the CAST 
format.
   
   I suggest the `to_utc_timestamp` fix first, since it is a bug, then the 
array and element helper, then `sha1`, `crc32` and `repeat`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to