rdtr opened a new pull request, #13590: URL: https://github.com/apache/gluten/pull/13590
## What changes are proposed in this pull request? Depends on #13520. Part of #13517. Only the last commit belongs to this PR: https://github.com/rdtr/incubator-gluten/commit/386f99f4208266b8cdbe19714736e1f2e04f7248 Spark 4.3 adds `TimestampNTZNanosType` and `TimestampLTZNanosType` (SPARK-57455, SPARK-57102). Gluten already falls back plans whose schema has these types: `Schema / data type not supported: TimestampNTZNanosType(9)`. Two scans still reach Velox because the requested type is an older one, and the file holds the nanos data. ORC: Spark 4.3 rejects reading a nanos column as `timestamp_ntz` with `UNSUPPORTED_FEATURE.ORC_TYPE_CAST`. The Velox scan has no such check. It read `1970-01-01T00:00:00.000000001` as `1970-01-01T08:00`. A plain `timestamp_ntz` column written by Spark fails in Velox instead, since Spark stores it as ORC LONG: `Schema mismatch, From Kind: BIGINT, To Kind: TIMESTAMP`. `VeloxBackend` now falls back ORC scans with `TimestampNTZType` at any nesting level, next to the existing `TimestampType` restriction. These scans only produced errors or shifted values, so no working offload is lost. The check is in `VeloxBackend` and applies to every Spark version, not only 4.3. Parquet: with `spark.sql.legacy.parquet.nanosAsLong` (since Spark 3.2.4) Spark reads `TIMESTAMP(NANOS)` as `LongType`. Velox fails with `Converted type TIMESTAMP is not allowed for requested type BIGINT for file column 'ts'`. `VeloxBackend` now falls back Parquet scans that read a `LongType` column while this config is on. The scan can't tell which `LongType` columns are nanos without the file footers, so the check covers all of them. The config is internal and off by default. Two more changes are test-only. `GlutenTestsTrait.canConvertToDataFrame` now skips rows holding a `TimestampNanosVal`. The row doesn't carry the type or precision, and `convertInternalRowToDataFrame` built an Integer column for it, which failed with a `ClassCastException`. The `BoundReference` cases in `HashExpression supports nanosecond timestamp types` are skipped for that reason and the `Literal` cases still run. `checkEvaluation roundtrip for nanos timestamp Literal` has a Gluten variant because `collect()` truncates to the type's precision, and Spark's test puts 999 nanos in an LTZ(7) literal. Native execution of the nanos types is out of scope. Those plans keep falling back. ## How was this patch tested? Re-enabled on Spark 4.3: - `GlutenHashExpressionsSuite`: `HashExpression supports nanosecond timestamp types` - `GlutenTimestampNanosRowSuite`: `checkEvaluation roundtrip for nanos timestamp Literal`, as a `testGluten` variant - `GlutenQueryExecutionErrorsSuite`: `UNSUPPORTED_FEATURE - SPARK-57455: can't read nanos timestamp as micros timestamp` and `UNSUPPORTED_FEATURE - SPARK-57455: nanos timestamp mismatch nested in array is caught` - `GlutenParquetTimestampNanosSuite`: `SPARK-57102: legacy nanosAsLong reads a foreign TIMESTAMP(NANOS) file as LongType` - `GlutenQueryExecutionErrorsSuite`: `UNSUPPORTED_FEATURE - SPARK-36346: can't read Timestamp as TimestampNTZ` - `GlutenOrcSourceSuite`: `SPARK-60400: ORC TIMESTAMP is inferred as TIMESTAMP_LTZ regardless of spark.sql.timestampType` The first 5 fail on #13520 once their excludes are removed. With this change they pass, and the last 2 pass with their excludes removed too. On Spark 4.3, `GlutenHashExpressionsSuite`, `GlutenTimestampNanosRowSuite` and `GlutenParquetTimestampNanosSuite` pass. `GlutenQueryExecutionErrorsSuite` plus the ORC query, source, filter, read schema, partition discovery and schema pruning suites: 832 succeeded, 0 failed. On Spark 4.2, `GlutenQueryExecutionErrorsSuite` and the same ORC suites: 656 succeeded, 0 failed. The 4.2 run shows no regression and doesn't exercise the new fallback, since the SPARK-36346 exclude stays on versions before 4.3. `gluten-ut` compiles on Spark 3.5. ## Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
