nssalian opened a new pull request, #19009:
URL: https://github.com/apache/iceberg/pull/19009

   #16292 added plan-time checks in `SparkBatch` because the vectorized reader 
could not decode shredded VARIANT `typed_value`: a scan goes to the row reader 
when the table sets `write.parquet.shred-variants=true`, the variant column 
uses `none` or `counts` metrics, or a file has variant lower bounds, which 
`SparkScanBuilder` loads stats for. These checks guess the file layout and miss 
both ways: shredded files without the property and without variant bounds, such 
as arrays, failed (#18601), and unshredded files in such tables never 
vectorize. #18851 makes the reader handle shredded files per file, so this 
removes the checks and the stats loading. Those scans follow the same 
vectorization rules as unshredded variants, and disabling vectorization still 
selects the row reader.
   
   Part of #18669.
   
   Test plan: Spark 4.2 `TestSparkVariantRead` asserts the scan is columnar 
only when vectorization is enabled. These fail without this change: 
`readShreddedEnabledByTableProperty` (property), 
`testReadShreddedWithMetricsDisabled` (`none` and `counts`, with the property 
also set), `readMixedObjectShreddedAndUnshreddedFiles` and the vectorized case 
of `testReadShreddedAfterPropertyToggled` (variant bounds, batch scan), 
`readShreddedIncrementalAppendScan` (incremental append scan) and 
`readShreddedVariantAfterDelete` (merge-on-read delete). The new tests compare 
vectorized and row-reader bytes.
   
   ---
   **AI Disclosure**
   - Model: Claude Opus 5.5
   - Platform/Tool: Claude Code
   - Human Oversight: fully reviewed with tests run locally
   - Prompt Summary: Remove the Spark plan-time shredded variant checks that 
the per-file vectorized reader makes redundant, with Spark tests asserting 
columnar scans. The code changes and tests are generated.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to