nssalian opened a new pull request, #19039: URL: https://github.com/apache/iceberg/pull/19039
Vectorized Parquet reads throw for struct columns, so Spark uses the row reader for any scan that projects a struct. This adds a struct reader to the Arrow vectorized reader. It reads each child column and derives the struct's own nulls from the definition levels of a child leaf. When no projected child is stored in the file, it reads only the levels of a leaf under the struct, as the row reader does. Struct support is behind a `supportsStructs()` builder hook that is off by default. Spark 4.2 turns it on, adds a `StructColumnVector`, and plans struct scans as columnar unless the struct contains a variant, which still uses the row reader. Spark 3.5, 4.0 and 4.1 follow in a separate PR. Part of https://github.com/apache/iceberg/issues/18669, and splits https://github.com/apache/iceberg/pull/18121. `ColumnIterator.newIterator` and `ParquetSchemaUtil.selectPresenceColumn` become public so the Arrow reader can share the row reader's presence column selection. `TestParquetVectorizedReads.generateData` now passes `nullPercentage` through, so inherited tests run at their stated null rate. Test plan: - `TestVectorizedStructReader`: struct readers are built only when the hook is on, children are assembled in order, a nested struct reuses a descendant's presence reader, and struct nulls are read from an unprojected boolean leaf on v2 pages and a geometry leaf, which have no vectorized value reader. - Spark 4.2 `TestParquetVectorizedReads` and its dictionary-encoded and plain-fallback subclasses: vectorized reads match the written records for nullable, three-level and all-null nested structs, a dictionary-encoded struct, reordered and defaulted fields, a promoted child, and a projection of only an added field (presence from a plain and from a repeated leaf). The plain-fallback subclass skips the all-null case because its generator ignores the null rate. - `TestSparkVectorizedFormatModel`: the shared format model tests now cover structs for Parquet. - `TestSparkReadProjection`: a required inner struct under a present parent reads as present. `TestSparkReaderDeletes`: position deletes on a struct column with null structs, read columnar when vectorization is on. - `TestSparkVariantRead`: a struct scan is columnar unless the struct holds a variant. --- **AI Disclosure** - Model: Claude Opus 5.5 - Platform/Tool: Claude Code - Human Oversight: fully reviewed with tests run locally - Prompt Summary: Add vectorized Parquet reads for struct columns in Arrow and Spark 4.2, behind a default-off builder hook, with Arrow and Spark tests. The code changes and tests are generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
