peterxcli opened a new issue, #5279: URL: https://github.com/apache/datafusion-comet/issues/5279
## Describe the bug Comet maps Spark's `CalendarIntervalType` representation (`months: i32`, `days: i32`, `microseconds: i64`) to Arrow `IntervalMonthDayNano` (`months: i32`, `days: i32`, `nanoseconds: i64`). The JVM codegen-dispatch output path therefore multiplies `CalendarInterval.microseconds` by 1,000 before writing it to an `IntervalMonthDayNanoVector`. That conversion limits otherwise valid Spark intervals to approximately 292 years of elapsed time. For larger values, the default dispatch path aborts with an uncaught `ArithmeticException: long overflow`, even when ANSI mode is disabled. The native `make_interval` kernel has the same nanosecond range limitation, in addition to the `Decimal(18,6)` to `Float64` precision issue tracked by #5131. ## Steps to reproduce Run without `spark.comet.expression.MakeInterval.allowIncompatible` so `MakeInterval` uses the default JVM codegen-dispatch path: ```sql CREATE TABLE test_make_interval_dispatch(hours INT) USING parquet; INSERT INTO test_make_interval_dispatch VALUES (2562048); SELECT make_interval(0, 0, 0, 0, hours) FROM test_make_interval_dispatch; ``` Spark returns a valid `CalendarInterval`. Comet aborts while writing the result to `IntervalMonthDayNanoVector` because `Math.multiplyExact(interval.microseconds, 1000L)` overflows. ## Expected behavior Represent `CalendarIntervalType` losslessly across Comet boundaries using the Spark-compatible components: ```text months: i32 days: i32 microseconds: i64 ``` A tagged Arrow struct is one possible representation. The implementation must keep calendar days separate from elapsed microseconds because a calendar day can be 23, 24, or 25 hours across DST transitions. Once this representation is available: - make native `make_interval` preserve `Decimal(18,6)` microseconds exactly; - support the full Spark interval range in native and JVM dispatch paths; - remove `spark.comet.expression.MakeInterval.allowIncompatible` and the related ignored tests; - cover FFI, serde, JVM/native writers and readers, codegen dispatch, native shuffle, nulls, ANSI behavior, and `try_make_interval`. ## Additional context - PR: #5039 - Default dispatch range-limit review: https://github.com/apache/datafusion-comet/pull/5039#discussion_r3722969753 - Native precision/range issue: #5131 - Spark precedent: https://github.com/apache/spark/commit/5ca6b1062887664b16b11f2bfcc015c9e616dc49 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
