peterxcli opened a new issue, #5517:
URL: https://github.com/apache/datafusion-comet/issues/5517
## Context
#5177 makes the Parquet reader's `TIMESTAMP_MILLIS -> microseconds`
conversion checked: overflow now raises an error instead of silently wrapping
(`parquet_convert_array` in `native/core/src/parquet/parquet_support.rs` uses
`try_unary` + `mul_checked`).
The resulting error is a raw `ArrowError::ComputeError("Overflow happened
on: <v> * 1000")`, which surfaces to users as a `CometNativeException` rather
than what Spark throws on the same input.
## What Spark throws
Spark's vectorized Parquet reader calls `SparkDateTimeUtils.millisToMicros`,
which is `Math.multiplyExact`, so it throws an **untyped**
`java.lang.ArithmeticException("long overflow")` — independent of ANSI mode:
-
https://github.com/apache/spark/blob/v4.2.0/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetVectorUpdaterFactory.java#L817-L833
-
https://github.com/apache/spark/blob/v4.2.0/sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkDateTimeUtils.scala#L103-L108
## Why the existing typed variant does not fit
Mapping this to the existing `SparkError::ArithmeticOverflow` would render:
> [ARITHMETIC_OVERFLOW] ... If necessary set "spark.sql.ansi.enabled" to
"false" to bypass this error.
That advice is wrong here: this overflow is ANSI-independent (the regression
test added in #5177, `ParquetReadSuite."TIMESTAMP_MILLIS overflow fails in
native scan"`, asserts the error with ANSI both on and off).
## Proposed work
Add a native error variant that converts to a plain
`ArithmeticException("long overflow")` (or the closest Spark-faithful
equivalent), with the corresponding `ShimSparkErrorConverter` mappings for each
supported Spark version. This aligns with the error-fidelity work in #5169.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
