nssalian opened a new pull request, #18121: URL: https://github.com/apache/iceberg/pull/18121
### Rationale Struct columns can't be read through the vectorized Parquet path today (`testNestedStruct` asserted "not supported yet"), forcing struct-projecting scans onto the row reader. This adds vectorized struct reads to the Spark 4.1 Parquet path: a struct is built from its child vectors, and null structs are detected from a file-backed child leaf's definition level, matching the row reader. A struct projected with only a field absent from the file (e.g. a field added after the file was written) resolves per-row - rows where the struct is present read the struct with the added field null, rows where the struct was null read null - by deriving presence from a retained file leaf, mirroring the Parquet row reader. ORC and Avro reads are unaffected. The standalone arrow-native reader (`ArrowReader.SUPPORTED_TYPES`) is intentionally unchanged - the Spark consumer reads presence from a `NullabilityHolder`, whereas the arrow-native `VectorSchemaRoot` contract needs the physical Arrow `StructVect or` fully materialized for all consumers, so that is a focused follow-up. Related to #2485; prerequisite for a shredded-Variant vectorized-read follow-up. The identical change for Spark 4.0 will follow in a separate PR to keep this one to a single concern. ### Changes - `VectorizedArrowReader.StructReader` (+ `VectorHolder.StructVectorHolder`, `VectorizedReaderBuilder.struct`): build a struct from child readers. When a struct projects no file-backed child (e.g. only a field added after the file was written), `StructReader` derives per-row presence from a retained presence leaf (`ParquetSchemaUtil.selectPresenceColumn`, the shallowest real file leaf under the struct), so null struct rows read null and present rows read the struct with the added field null - mirroring the Parquet row reader. `VectorizedReaderBuilder.struct` returns null (falls back) rather than throwing when a struct lacks a field ID, matching `primitive()`. - `VectorizedColumnIterator` / `VectorizedPageIterator` / `VectorizedParquetDefinitionLevelReader`: definition-level struct-presence hook (RLE - run-length-encoded - and bit-packed runs); early-returns for non-struct reads; `setBatchSize` clears prior-pass presence registrations so a reused reader model doesn't accumulate duplicates. - Spark 4.1: `StructColumnVector`, `ColumnVectorBuilder`, `SparkBatch` batch-read eligibility. The existing variant lower-bounds check now recurses into struct fields (via `TypeUtil.indexById`) so an unshredded variant with stored bounds nested in a now-vectorizable struct still falls back to the row reader. - `RandomGenericData`: null-percentage generate overload (test support). - New `iceberg-parquet` public API: `ParquetSchemaUtil.selectPresenceColumn` (promoted from package-private) and `ParquetSchemaUtil.presenceField`, used by the arrow module to mirror the row reader's presence-leaf selection. ### Tests - arrow `TestVectorizedStructReader`: struct assembles child holders in order; optional and required structs with no file-backed child stay present; a presence column absent from the row group reads present (partition-constant fallback); a nested struct reuses its descendant's presence reader instead of re-reading the column; `VectorizedVariantReader.fileBackedLeaf()` resolves through the metadata reader. - Spark 4.1 `TestReadProjection`: struct projecting only an added field reads present with that field null; required inner struct under a present parent reads present (Parquet-scoped, vectorized + row). - Spark 4.1 `TestParquetVectorizedReads`: nested / nullable / 2- and 3-level / dict-encoded child / all-null struct; struct projecting only an added field resolves per-row null (single-level and deeply-nested, mixed present/null rows); `testNestedStruct` now round-trips. - Spark 4.1 `TestSparkReaderDeletes`: struct under position deletes + deletion vectors. --- **AI Disclosure** - Model: Claude Opus 4.8 - Platform/Tool: Claude Code - Human Oversight: partially reviewed - Prompt Summary: vectorized struct reads on the Spark 4.1 path (reader + presence plumbing + column vector + batch-read eligibility), no-file-backed-child fallback, arrow + Spark v4.1 tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
