nssalian commented on issue #18601: URL: https://github.com/apache/iceberg/issues/18601#issuecomment-6326057137
@kevinjqliu @randreucetti Thanks for filing this. I took some time to get to the bottom of this since I originally observed it and didn't have a great solution for it back then. This is what is causing the problem: Whether a file is shredded is recorded only in its Parquet footer. I didn't want to open every file's footer at planning time, so `SparkBatch` guesses from the `write.parquet.shred-variants` table property, the table's current metrics mode, and the file's variant lower bounds. All three can miss: shredding can be enabled through the session conf or a write option, and shredded arrays never get variant bounds. When they miss, the shredded file reaches the vectorized reader, which throws on `typed_value`. What I think is a good short-term fix: Decide per file at read time, from the footer the reader already opens. A shredded variant column is explicitly read through the existing row-based variant reader, while the other columns and unshredded files stay on the vectorized path. That lets us drop the plan-time guesses in `SparkBatch` for 4.0, 4.1, and 4.2. It also covers the unprojected case in #18430. In the long-term, fully vectorized reader that reconstructs shredded variants, replacing the row-based reader at the same point. Already have that in the works, but that needs a few PRs of work to get in. I'll open a PR shortly for the short term fix if this sounds good. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
