yihua opened a new pull request, #20102: URL: https://github.com/apache/hudi/pull/20102
### Describe the issue this Pull Request addresses closes #20100 part of #20064 Stacked on #20079, review that first: until it merges, this PR's diff against master also shows its commits. For every parquet base file, the Spark readers copy the Hadoop conf three times and render the requested schema to JSON three times, although those keys depend only on the requested schema. Every task also converts the table schema for timestamp repair even when repair is off, the Spark 3.x readers walk every footer for shredded variants, and the vectorized reader allocates a full batch even for small files. ### Summary and Changelog `SparkParquetReaderBase` prepares a `ParquetReadConf` (a `JobConf` with the requested schema keys) once per scan and requested schema when the caller passes a `SharedScanStorageConfiguration`, as the file group reader format now does for base files; other callers get one copy per file instead of three. `ParquetSchemaEvolutionUtils.getFileReadConf` (from #20079) copies the conf only when the file needs its own requested schema or a pushed filter, so the shared conf is never modified. The Spark 3.x readers skip the variant walk when no requested struct can be an unshredded variant, vectorized batches on Spark 3.5+ are sized to the split's rows, and the timestamp repair schema is converted only for tables with a timestamp-millis field. New tests check that a COW scan reads all base files with one conf and that shared and per-file confs return the same rows, including with pushed filters, type changes and 8 concurrent threads. ### Impact Lower per-file CPU on Spark parquet base file reads. No API, config or output change. ### Risk Level low. The files of a scan share a read-only `JobConf`, as vanilla Spark already does for footer reads; per-file keys go on a copy, and a concurrent test checks the shared conf stays unchanged. The shared conf is safe because `getFileReadConf` never writes to it: a file's own requested schema or pushed filter goes on a copy. `TestSparkParquetReaderBase#testAttemptConfIsTheSharedConfUnlessTheFileNeedsItsOwnKeys` and `TestSharedScanParquetReadConf` fail if a per-file key reaches the shared conf. This PR also conflicts with #20090 (`ParquetSchemaEvolutionUtils`, the readers, `TestBasicSchemaEvolution`) and #20077 (`HoodieFileGroupReaderBasedFileFormat`, whose two edits move to `HoodieFileGroupReaderFunction`); whichever merges second rebases. ### Documentation Update none ### Contributor's checklist - [ ] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [ ] Enough context is provided in the sections above - [ ] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
