LiuYC3213687107 commented on issue #18543: URL: https://github.com/apache/iceberg/issues/18543#issuecomment-6387124342
I'd like to pick this up, building on the approach in #17635 (closed as stale). Plan: in `VectorizedSparkOrcReaders.StructConverter`, handle `_row_id` and `_last_updated_sequence_number` before the generic `idToConstant` branch, like `OrcValueReaders.StructReader` does: return the stored value when present, otherwise `first_row_id + position` / the file's sequence number. I'll locate the stored column by field ID rather than by position, and use the physical row index when ORC selection is in use. I'll add tests to `TestVectorizedOrcDataReader` covering stored values, nulls and files without the columns, starting with Spark 4.2 and backporting to the other Spark versions in follow-up PRs. cc @pvary @Guosmilesmile, who were asked to review #17635. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
