anoopj commented on PR #3058: URL: https://github.com/apache/iceberg-rust/pull/3058#issuecomment-5397746386
> It's impossible to compute _row_id in iceberg layer, give the filter pushed to parquet arrow reader and row group pruning. @blackmwk We can move the `_row_id` synthesis into a reader-level step over the parquet stream. That is a probably the right layer. But I believe the current code is correct. This is because we don't compute the position in the Iceberg code. We consume the `_pos` / `RowNumber` virtual column that the parquet reader produces, and that column is the true global file position, resilient to all the pushdown paths. This includes row group pruning, page index pruning etc. Code [pointer](https://github.com/apache/arrow-rs/blob/e1e71ea89b211758e20254296008f96e744ed138/parquet/src/arrow/array_reader/row_number.rs#L42-L71). To summarize `first_row_id + pos` is built on the reader's authoritative position, so it should be safe. Is there a pushdown case where RowNumber wouldn't yield the correct absolute position? If so I'd like to add a test for it. Andrei's suggested multi-row-group test (and a positional-delete case) will exercise this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
