ashokraminedi opened a new pull request, #20193:
URL: https://github.com/apache/hudi/pull/20193
### Describe the issue this Pull Request addresses
Closes #20140
`HoodieFileGroupReaderBasedFileFormat` computes `supportVectorizedRead` for
the current read, but also writes that query-local decision into the
session-level `spark.sql.parquet.enableVectorizedReader` configuration.
For reads where vectorization is not supported, such as Spark 4.1 VARIANT
reads, this can leave vectorized Parquet reading disabled in the Spark session
after the query completes. Subsequent queries in the same session can therefore
inherit a decision that was intended only for the previous read.
The session mutation was introduced in #14161 and has remained in this path
since then. However, the current reader path already passes the same decision
directly to the base-file reader:
```scala
buildBaseFileReader(
spark,
options,
augmentedStorageConf.unwrap(),
dataStructType,
supportVectorizedRead)
```
The Spark 4.1 reader path consumes this explicit `supportVectorizedRead`
value when constructing the Parquet reader. Therefore, removing the session
mutation does not remove or change the per-read vectorization decision. It only
removes the additional session-level side effect.
Before this change, the decision was propagated through two paths:
```text
supportVectorizedRead
├── base-file reader argument <- query/read-local
└── Spark session configuration <- persists beyond the read
```
After this change, the reader continues receiving the same value directly,
while the Spark session configuration remains unchanged.
### Summary and Changelog
- Remove the mutation of `spark.sql.parquet.enableVectorizedReader` from
`HoodieFileGroupReaderBasedFileFormat`.
- Keep the vectorization decision query-local by using the existing
`supportVectorizedRead` argument passed directly to `buildBaseFileReader`.
- Add a regression test that starts with
`spark.sql.parquet.enableVectorizedReader=true`, performs a VARIANT read that
requires the non-vectorized path, and verifies that the session configuration
remains unchanged afterward.
- Remove the existing `count(*)` assertion that was unintentionally relying
on the leaked session configuration. With the session leak fixed, it exposes a
separate pre-existing issue tracked by #20139, which is outside the scope of
this PR.
### Impact
No public API or configuration changes.
The current read continues to use the same `supportVectorizedRead` decision
when constructing its base-file reader.
The behavioral change is limited to preventing that read-local decision from
overwriting the Spark session's `spark.sql.parquet.enableVectorizedReader`
configuration and affecting subsequent queries.
### Risk Level
low
The change removes a session-level side effect without changing how the
vectorization decision is supplied to the reader.
The Spark 4.1 / Scala 2.13 `TestVariantShreddingMixedLayouts` suite was run
after the change:
```text
Total number of tests run: 36
Suites: completed 1, aborted 0
Tests: succeeded 36, failed 0, canceled 0, ignored 0, pending 0
All tests passed.
```
The regression test specifically verifies that a read requiring the fallback
path does not mutate the session-level vectorized-reader configuration.
### Documentation Update
none
- The config description must be updated if new configs are added or the
default value of the configs are changed.
- Any new feature or user-facing change requires updating the Hudi website.
Please follow the
[instruction](https://hudi.apache.org/contribute/developer-setup#website)
to make changes to the website. -->
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
