sunchao commented on code in PR #5457:
URL: https://github.com/apache/datafusion-comet/pull/5457#discussion_r4178978702
##########
native/core/src/parquet/schema_adapter.rs:
##########
@@ -1457,6 +1457,10 @@ impl SparkPhysicalExprAdapter {
| (DataType::Map(_, _), DataType::Map(_, _))
| (DataType::Timestamp(_, _), DataType::Timestamp(_, _))
| (DataType::Timestamp(_, _), DataType::Int64)
+ | (
+ DataType::Date32,
Review Comment:
[P2] [P2] Preserve overflow errors for rows retained by scan filters
This new adapter route applies unchecked reader conversion to top-level
DATE-to-NTZ fields whenever a data filter exists. Using the added 6,000-row
Parquet fixture, collect `read().filter("id = 5000")`: the selected row
contains epoch day `106751992`. Spark throws `ArithmeticException("long
overflow")`, but this route produces NULL because `parquet_exec.rs` clears
`checked_timestamp_overflow` and conversion falls through to
`PARQUET_OPTIONS.safe = true`. The base’s top-level `Cast` route throws
instead. This silently substitutes NULL for an invalid selected value. Could
overflow validation be preserved for surviving rows, with a narrow fallback
where that cannot be guaranteed?
Evidence: The disposable harness reproduced NULL for visible days
`106751992` and `-106751992` using the head’s reader conversion with
`checked_timestamp_overflow=false`. Checked conversion and the base NTZ
kernel’s `safe=false` Arrow conversion both threw. Spark 4.1.3’s actual
`DateToTimestampNTZUpdater.decodeSingleDictionaryId` threw plain
`ArithmeticException("long overflow")` for both values. The filter-to-option
route is established by `CometNativeScan.scala:262`, `planner.rs:1807` and
`parquet_exec.rs:111`. Validation covered the conversion boundary and source
routing, not a complete Comet query.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
