andygrove opened a new issue, #5090: URL: https://github.com/apache/datafusion-comet/issues/5090
### What is the problem the feature request solves? Several hand-rolled per-row loops implement temporal unit conversions that arrow's `cast` kernel performs bit-identically (verified against the arrow-cast 58.4.0 source): - `native/core/src/parquet/cast_column.rs` (`cast_timestamp_micros_to_millis_array` / `_scalar`): `unary(|v| v / 1000)` plus manual timezone re-attachment. Arrow's Timestamp-to-Timestamp cast performs the same truncating division and attaches the target tz. - `native/spark-expr/src/conversion_funcs/temporal.rs` (NTZ arm of `cast_date_to_timestamp`): per-row `(d as i64) * 86_400 * 1_000_000` into a builder. Arrow's `(Date32, Timestamp(Microsecond, _))` cast computes exactly `(x as i64) * MICROSECONDS_IN_DAY`. - `native/spark-expr/src/utils.rs` (millis-to-micros): `arity::unary(|v| v * 1000)`. Arrow's timestamp-unit cast performs the same multiply. - `native/spark-expr/src/datetime_funcs/date_from_unix_date.rs`: manual Int32-to-Date32 reinterpretation plus a scalar match arm. Arrow dispatches `(Int32, Date32)` to a zero-copy reinterpret cast; the scalar path can use `ScalarValue::cast_to`. ### Describe the potential solution Replace each loop with `arrow::compute::cast_with_options` to the target type. ### Additional context Caveats to preserve: - micros-to-millis: safe when source and target tz are both Some or both None. If the source is NTZ and the target has a tz, arrow wall-clock-adjusts values, so that combination must keep the current relabel behavior. - Date32-to-Timestamp: only the `target_tz.is_none()` branch is replaceable. The session-tz branch implements Spark's DST ambiguity/gap rules (earlier occurrence, pre-transition offset), while arrow's adjustment returns null on both, so it must stay. Found during an audit of native code that replicates existing arrow-rs kernels. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
