dwsmith1983 opened a new pull request, #59287: URL: https://github.com/apache/spark/pull/59287
### What changes were proposed in this pull request? `SpecificParquetRecordReaderBase.initialize` creates a `ParquetReadSupport`, calls its `init` (which parses the requested Spark schema JSON from the `SPARK_ROW_REQUESTED_SCHEMA` conf value), and then parsed the same JSON again with `StructType.fromString`. This PR adds a package-private accessor for the schema `ParquetReadSupport.init` parsed and reuses it. For any other read support class, the reader still parses the conf value as before. ### Why are the changes needed? Performance on wide reads. The parse runs once per file split and its cost grows with the number of requested columns: about 0.6 ms per call for 1000 columns. On 400 Parquet files with 1000 INT columns (local[1], JDK 17, two interleaved before/after pairs, best time): | | Before | After | |---|---|---| | `SELECT *` | 5031 / 4987 ms | 4745 / 4629 ms | | `SELECT c0` | 811 / 808 ms | 800 / 813 ms | `SELECT *` is about 4-7% faster. `SELECT c0` requests one column, so its parse is trivial and it does not change. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? New test in `ParquetIOSuite` checking that `VectorizedParquetRecordReader` uses the same schema instance its read support parsed (it fails without this change because the reader parses a second copy). Existing suites pass: `ParquetIOSuite`, `ParquetV1QuerySuite`, `ParquetV2QuerySuite`, `ParquetSchemaSuite`, `ParquetVectorizedSuite`, `ParquetColumnIndexSuite`, `ParquetEncodingSuite`, `ParquetStorageFilterSuite`, `ParquetFileFormatV1Suite` and `ParquetFileFormatV2Suite`. The numbers above come from a local benchmark run before and after the change on the same commit. ### Was this patch authored or co-authored using generative AI tooling? No -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
