yangshangqing95 commented on PR #17597: URL: https://github.com/apache/iceberg/pull/17597#issuecomment-5699574000
> @yangshangqing95 — I’m interested in this work as a foundation for position-selective reads. While evaluating your POC I tested with nested data and unequal page boundaries. > > I found a case where `_pos` and the scalar ID are correct, but nested values come from a different physical row. For example, row 1001 contains a nested number from row 932 where it should contain null. The read is correct without page pruning; enabling it produces the wrong result with both V1 and V2 pages. > > I put together [a small branch based directly on your POC](https://github.com/erikwright/iceberg/compare/101a48da43db62ea00e2ea6dedf07c275c11b882...erik/parquet-poc-column-sync) with the reproduction and a proposed fix. It adapts Parquet’s existing ColumnReadStoreImpl into Iceberg’s record construction so the data columns follow the same selected positions, retaining native decoding for unfiltered groups. The Parquet module checks, data-reader tests and Revapi pass on the current dependencies. > > There’s a separate issue in Parquet’s synchronizer: it can stop before reaching the final selected row’s page. [Parquet #3748](https://github.com/apache/parquet-java/pull/3748) fixes that upstream, but isn’t in the dependency this POC uses. These particular regressions pass without it, but other valid selections still need that fix before production enablement. I’ve asked about getting it into a release. > > Feel free to merge or cherry-pick the branch into your POC if this approach fits your planned follow-ons. Hi @erikwright Thanks for pointed out this issue! But it looks like the community isn't actively moving this POC forward at the moment. I've submitted a few PRs but haven't received any feedback yet, so let's wait and see what the next steps are. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
