sunchao commented on PR #5457:
URL:
https://github.com/apache/datafusion-comet/pull/5457#issuecomment-5982845983
Reworked the branch against current main to address the broad-fallback
concern. DATE-to-NTZ Parquet adaptation now uses a narrow checked conversion
through CometCastColumnExpr, with main's nested visibility masks and existing
filtered-scan safe-conversion policy. The top-level adapter route matters: the
new end-to-end test caught and fixed the generic Cast path bypassing those
reader options. Genuine NTZ scans remain native, and the blanket
scan/Arrow-bridge and ordinary DATE-to-NTZ fallback guards are gone. Main's
original top-level NTZ fixtures and fuzz helpers are restored.
For the temporal rebase, I deliberately retained checked day arithmetic to
preserve plain ArithmeticException("long overflow") instead of the generic
Arrow error. Region zones still dispatch, and TRY nullability guards remain.
The nested-cast performance entry relative to an earlier PR head was removed;
no performance gain is claimed against current main.
Validation: matching native/JNI build; 125 native conversion-function tests;
the new checked-reader visibility unit test; 35 Spark 4.1 tests (22 cast, 12
error-converter and the reader regression); Rust formatting, Spotless,
Scalastyle and generated documentation all passed. The reader test verifies
native physical-DATE adaptation, genuine NTZ storage, unfiltered overflow and a
filtered LIMIT skipping a later overflowing value. Other profiles and full
suites remain for hosted CI; the Spark 4.1 and all-profile labels are applied.
No new benchmarks were run.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
