JakeDern commented on PR #8001: URL: https://github.com/apache/arrow-rs/pull/8001#issuecomment-3185902178
@albertlockett Not sure why I can't reply directly to [this](https://github.com/apache/arrow-rs/pull/8001#discussion_r2273576278) comment, but it's a good observation and I actually spent a fair bit of time thinking about it and pulling on various threads here but the required change stack got too deep. There are some chicken and egg problems if you look into how we use RecordBatchReader and the state required including the dictionaries and schema. Definitely want to continue iterating here over time and also look at the FileReader which has some of it's own logic I wasn't able to account for either. I can remove the fields that aren't read if you think it's worth it, but the computation to produce those fields (including schema) is already done so I thought it would be somewhat harmless to leave it so the ability is advertised to others who might leverage it. In the future I hope we can fully consolidate logic for interpreting IPC messages to one place. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
