andygrove opened a new issue, #5095: URL: https://github.com/apache/datafusion-comet/issues/5095
### What is the problem the feature request solves? The numeric-to-decimal cast arms in `native/spark-expr/src/conversion_funcs/` re-implement arrow's cast algorithms (verified line-for-line against arrow-cast 58.4.0): - `numeric.rs` `cast_int_to_decimal128_internal`: per-row `checked_mul(10^scale)` plus precision validation, null on overflow (Legacy/Try) or Spark error (ANSI). Arrow's `cast_integer_to_decimal` with `safe: true` runs the identical algorithm. - `numeric.rs` `cast_floating_point_to_decimal128`: the fast path (`(f * mul).round()` through f64, filter on precision) matches arrow's `cast_floating_point_to_decimal` safe path exactly, including going through f64 for Float32 and half-away-from-zero rounding. Both share the `10_f64.powi` imprecision tracked in #1371, so results stay bit-identical. - `boolean.rs` `cast_boolean_to_decimal`: per-row map to `10^scale` or 0. Composable as arrow `cast(Boolean -> Int8)` then `cast(Int8 -> Decimal128(p, s))`. ### Describe the potential solution Replace the vectorized passes with `cast_with_options(safe: true)` and keep the existing thin Spark wrappers: - For ANSI, keep the existing rescan-on-error pattern (already used by the float path: compare output null count against input null count, rescan to find the offending value, and raise `SparkError::NumericValueOutOfRange` with the original unscaled value). Arrow's `safe: false` error text is not Spark-shaped, so error remap-by-parsing is not viable. - Boolean-to-decimal currently errors on overflow in all eval modes, so use `safe: false` plus remap, or keep the existing precision pre-check. ### Additional context Negative scale is impossible from Spark here, so arrow's negative-scale division branch never fires. Found during an audit of native code that replicates existing arrow-rs kernels. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
