rdtr opened a new pull request, #13398:
URL: https://github.com/apache/gluten/pull/13398
## What changes are proposed in this pull request?
Fixes #11914.
Spark 4.1 added `returnNullStructIfAllFieldsMissing` (SPARK-53535) and new
tests for it in `ParquetIOSuite`. Gluten excluded all of them for Spark 4.1 and
4.2. As @rui-mo mentioned in #11914, Velox was expected to support this
behavior with facebookincubator/velox#17555, which is now in the Velox version
Gluten uses. With the current main, most of these tests pass without any change
to Gluten code.
This PR:
- Enables the 3 tests that now pass:
- `vectorized reader: missing all struct fields`
- `SPARK-53535: ... struct with map field only`
- `SPARK-53535: ... struct with cheap map and more expensive array field`
- Keeps 2 tests excluded and adds a Gluten version of each:
- `SPARK-53535: ... struct with complex fields` (new Gluten version in
this PR)
- `SPARK-54220: ... struct with NullType only` (a Gluten version already
exists)
Why 2 tests stay excluded: the Spark tests take the scan node out of the
plan and run it alone with a test helper, `VerifyNoAdditionalScanOutputExec`.
In Gluten, this node is `FileSourceScanExecTransformer`. When it runs alone,
outside a Gluten whole stage, it uses Spark's scan code. With the Parquet-mr
reader, that code returns rows, so the helper fails with `ClassCastException:
UnsafeRow cannot be cast to ColumnarBatch`. This is a problem with the test
setup, not with the result. The Gluten versions run the same queries and check
only the result.
The same change is made for Spark 4.1 and 4.2, because the tests and the
excludes are the same.
## How was this patch tested?
Ran `GlutenParquetIOSuite` locally:
- Spark 4.1 (4.1.1 source as `spark.test.home`): 83 passed, 0 failed.
- Spark 4.2 (4.2.0 source as `spark.test.home`): 85 passed, 0 failed.
I also checked locally (not in this PR) that:
- the new Gluten test reads the file with `FileSourceScanExecTransformer`,
so it tests Velox and not vanilla Spark
- the new Gluten test fails when the expected answers are swapped
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5.5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
