nssalian opened a new pull request, #19009: URL: https://github.com/apache/iceberg/pull/19009
#16292 added plan-time checks in `SparkBatch` because the vectorized reader could not decode shredded VARIANT `typed_value`: a scan goes to the row reader when the table sets `write.parquet.shred-variants=true`, the variant column uses `none` or `counts` metrics, or a file has variant lower bounds, which `SparkScanBuilder` loads stats for. These checks guess the file layout and miss both ways: shredded files without the property and without variant bounds, such as arrays, failed (#18601), and unshredded files in such tables never vectorize. #18851 makes the reader handle shredded files per file, so this removes the checks and the stats loading. Those scans follow the same vectorization rules as unshredded variants, and disabling vectorization still selects the row reader. Part of #18669. Test plan: Spark 4.2 `TestSparkVariantRead` asserts the scan is columnar only when vectorization is enabled. These fail without this change: `readShreddedEnabledByTableProperty` (property), `testReadShreddedWithMetricsDisabled` (`none` and `counts`, with the property also set), `readMixedObjectShreddedAndUnshreddedFiles` and the vectorized case of `testReadShreddedAfterPropertyToggled` (variant bounds, batch scan), `readShreddedIncrementalAppendScan` (incremental append scan) and `readShreddedVariantAfterDelete` (merge-on-read delete). The new tests compare vectorized and row-reader bytes. --- **AI Disclosure** - Model: Claude Opus 5.5 - Platform/Tool: Claude Code - Human Oversight: fully reviewed with tests run locally - Prompt Summary: Remove the Spark plan-time shredded variant checks that the per-file vectorized reader makes redundant, with Spark tests asserting columnar scans. The code changes and tests are generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
