andygrove opened a new issue, #353: URL: https://github.com/apache/datafusion-spark/issues/353
## Is your feature request related to a problem or challenge? There are now shared helpers for casts and type coercion: the cast kernel `spark_cast` (#304), the common-type rules in `function/type_coercion.rs` (#345), and `SparkInputTypes` in `function/implicit_cast.rs` (#350). Many functions were written before these existed, so they still have their own copies of the same logic or use DataFusion's rules where Spark's apply. This issue tracks moving them onto the shared code. It follows on from #346. ## Describe the solution you'd like ### Duplicated code - [ ] **Array and element coercion.** `array_position` and `array_remove` each have a `coerce_array_and_element`, and `array_insert` does the same steps inline: unwrap the list, call `tightest_common_type(.., false)`, check `struct_field_types_match` and `is_orderable`, then rebuild the `List`/`LargeList`. They differ only in how they report a non-array first argument and whether they check ordering. Move one version into `type_coercion.rs`, with the `ARRAY_FUNCTION_DIFF_TYPES` and `INVALID_ORDERING_TYPE` errors built there. `array_contains`, which still uses `Signature::array_and_element`, can then use it too. - [ ] **String and binary layout widening in `contains`.** `unify` in `string/contains.rs` repeats `wider_physical_type` in `type_coercion.rs`, and `classify` repeats the dictionary and `FixedSizeBinary` unwrapping in `implicit_cast.rs`. ### Time zone conversion - [ ] **`to_utc_timestamp`.** `adjust_to_utc_time` parses the zone with chrono's `Tz` and computes the offset itself, while `from_utc_timestamp` uses `TimeZoneRules`. It looks up the offset with `offset_from_utc_datetime` on a value that is a local time, so the result can be wrong near daylight saving changes. `Tz` also does not accept Spark zone IDs such as `+05:30`. Use `TimeZoneRules::local_to_utc`, the inverse of what `from_utc_timestamp` uses. ### Functions to move onto `SparkInputTypes` These need no new `SparkInputType` variants: - [ ] `sha1`, `crc32`: `[Binary]`, like `sha2` - [ ] `repeat` (`array_repeat`): `[Any, Integer]`. The count is cast to BIGINT now, and Spark uses INT. These need new variants first: - [ ] A concrete `Decimal(p, s)`: `make_time` (`[Integer, Integer, Decimal(16, 6)]`), `make_interval`, `make_dt_interval` - [ ] `Date` and `Timestamp`: `weekday`, `dayname`/`monthname` (`date_part_name.rs`), `datediff`, `next_day`, `to_utc_timestamp`, `unix_date`/`unix_*` - [ ] Day-time and year-month intervals: `positive` - [ ] Variadic arguments: `elt` (Spark casts the index to INT, while it is cast to BIGINT here, and it is checked with Arrow's `can_cast_types`), `concat_ws` These need to be checked against Spark first: `try_sum`, `encode`. ### Casts at execution time - [ ] `bround` and `positive` cast a string argument to DOUBLE in `invoke_with_args`, so that a malformed string is NULL with ANSI mode off. Moving them onto `SparkInputTypes` now would leave the cast to the planner's DataFusion cast, which raises an error instead. Once string to number casts are ported in #88, use `spark_cast` there, or move them onto `SparkInputTypes` (`[Numeric, Integer]` for `bround`). ### Common type rules - [ ] `if` uses DataFusion's `try_type_union_resolution`, `array` uses `make_array`'s `coerce_types_inner`, and `equal_null` uses `comparison_coercion`. Spark uses `findWiderCommonType` for `if` and `array`, and its rules for binary comparisons for `<=>`. `type_coercion.rs` only has `findTightestCommonType`, so this needs `findWiderCommonType` ported first. ## Describe alternatives you've considered Leave the existing functions as they are and use the shared code only in new ones. The copies would then drift apart, and the gaps that #346 describes would remain in the older functions. ## Additional context Checked and left alone: Spark type names in errors all come from `spark_sql_type`. The other Arrow casts at execution time change the layout or width of values (`pmod`, `shuffle`, `encode`'s charset), where Arrow's cast is correct. `to_json` formats timestamps in Spark's JSON format, not the CAST format. I suggest the `to_utc_timestamp` fix first, since it is a bug, then the array and element helper, then `sha1`, `crc32` and `repeat`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
