rdtr opened a new pull request, #13590:
URL: https://github.com/apache/gluten/pull/13590

   ## What changes are proposed in this pull request?
   
   Depends on #13520. Part of #13517. Only the last commit belongs to this PR: 
https://github.com/rdtr/incubator-gluten/commit/386f99f4208266b8cdbe19714736e1f2e04f7248
   
   Spark 4.3 adds `TimestampNTZNanosType` and `TimestampLTZNanosType` 
(SPARK-57455, SPARK-57102). Gluten already falls back plans whose schema has 
these types: `Schema / data type not supported: TimestampNTZNanosType(9)`. Two 
scans still reach Velox because the requested type is an older one, and the 
file holds the nanos data.
   
   ORC: Spark 4.3 rejects reading a nanos column as `timestamp_ntz` with 
`UNSUPPORTED_FEATURE.ORC_TYPE_CAST`. The Velox scan has no such check. It read 
`1970-01-01T00:00:00.000000001` as `1970-01-01T08:00`. A plain `timestamp_ntz` 
column written by Spark fails in Velox instead, since Spark stores it as ORC 
LONG: `Schema mismatch, From Kind: BIGINT, To Kind: TIMESTAMP`. `VeloxBackend` 
now falls back ORC scans with `TimestampNTZType` at any nesting level, next to 
the existing `TimestampType` restriction. These scans only produced errors or 
shifted values, so no working offload is lost. The check is in `VeloxBackend` 
and applies to every Spark version, not only 4.3.
   
   Parquet: with `spark.sql.legacy.parquet.nanosAsLong` (since Spark 3.2.4) 
Spark reads `TIMESTAMP(NANOS)` as `LongType`. Velox fails with `Converted type 
TIMESTAMP is not allowed for requested type BIGINT for file column 'ts'`. 
`VeloxBackend` now falls back Parquet scans that read a `LongType` column while 
this config is on. The scan can't tell which `LongType` columns are nanos 
without the file footers, so the check covers all of them. The config is 
internal and off by default.
   
   Two more changes are test-only. `GlutenTestsTrait.canConvertToDataFrame` now 
skips rows holding a `TimestampNanosVal`. The row doesn't carry the type or 
precision, and `convertInternalRowToDataFrame` built an Integer column for it, 
which failed with a `ClassCastException`. The `BoundReference` cases in 
`HashExpression supports nanosecond timestamp types` are skipped for that 
reason and the `Literal` cases still run. `checkEvaluation roundtrip for nanos 
timestamp Literal` has a Gluten variant because `collect()` truncates to the 
type's precision, and Spark's test puts 999 nanos in an LTZ(7) literal.
   
   Native execution of the nanos types is out of scope. Those plans keep 
falling back.
   
   ## How was this patch tested?
   
   Re-enabled on Spark 4.3:
   
   - `GlutenHashExpressionsSuite`: `HashExpression supports nanosecond 
timestamp types`
   - `GlutenTimestampNanosRowSuite`: `checkEvaluation roundtrip for nanos 
timestamp Literal`, as a `testGluten` variant
   - `GlutenQueryExecutionErrorsSuite`: `UNSUPPORTED_FEATURE - SPARK-57455: 
can't read nanos timestamp as micros timestamp` and `UNSUPPORTED_FEATURE - 
SPARK-57455: nanos timestamp mismatch nested in array is caught`
   - `GlutenParquetTimestampNanosSuite`: `SPARK-57102: legacy nanosAsLong reads 
a foreign TIMESTAMP(NANOS) file as LongType`
   - `GlutenQueryExecutionErrorsSuite`: `UNSUPPORTED_FEATURE - SPARK-36346: 
can't read Timestamp as TimestampNTZ`
   - `GlutenOrcSourceSuite`: `SPARK-60400: ORC TIMESTAMP is inferred as 
TIMESTAMP_LTZ regardless of spark.sql.timestampType`
   
   The first 5 fail on #13520 once their excludes are removed. With this change 
they pass, and the last 2 pass with their excludes removed too.
   
   On Spark 4.3, `GlutenHashExpressionsSuite`, `GlutenTimestampNanosRowSuite` 
and `GlutenParquetTimestampNanosSuite` pass. `GlutenQueryExecutionErrorsSuite` 
plus the ORC query, source, filter, read schema, partition discovery and schema 
pruning suites: 832 succeeded, 0 failed. On Spark 4.2, 
`GlutenQueryExecutionErrorsSuite` and the same ORC suites: 656 succeeded, 0 
failed. The 4.2 run shows no regression and doesn't exercise the new fallback, 
since the SPARK-36346 exclude stays on versions before 4.3. `gluten-ut` 
compiles on Spark 3.5.
   
   ## Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: Claude Code (Claude Opus 5.5)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

  • [PR] [GLUTEN-13517][VL] Fall ... rdtr (via GitHub) via commits

Reply via email to