peterxcli opened a new pull request, #4898: URL: https://github.com/apache/datafusion-comet/pull/4898
## Which issue does this PR close? Part of https://github.com/apache/datafusion-comet/issues/4540. ## Rationale for this change Comet does not currently support Spark's `CalendarIntervalType`, causing queries carrying calendar interval columns through Comet operators to fall back to Spark. This change adds the foundational Arrow and codegen support needed to preserve calendar intervals—months, days, and microseconds—across the Spark, Arrow, and native execution boundaries. ## What changes are included in this PR? - Map Spark `CalendarIntervalType` to Arrow `Interval(MonthDayNano)`. - Add `CALENDAR_INTERVAL` to the serialized data type protocol and native Arrow type conversion. - Support reading and writing calendar intervals through Arrow vectors, including microsecond-to-nanosecond conversion. - Add `CalendarIntervalType` support to Comet batch-kernel code generation for scalar and nested values. - Allow calendar interval columns through Comet’s data type support checks and query plan serialization. - Add SQL and unit-test coverage for nullable and non-null calendar interval values. ## How are these changes tested? Added coverage for: - Round-tripping `CalendarIntervalType` through the Arrow writer and `CometPlainVector`. - Round-tripping calendar intervals through the batch-kernel codegen path. - SQL queries containing positive, negative, mixed-component, and null calendar interval values with native local table scans enabled. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
