professor-moody opened a new issue, #55369: URL: https://github.com/apache/arrow/issues/55369
### Describe the bug, including details regarding any error messages, version, and platform. When `IpcReadOptions::included_fields` is set, `RecordBatchFileReader` reads only the byte ranges of a record batch body needed by the selected fields. `ReadFieldsSubset` in `cpp/src/arrow/ipc/message.cc` determines those ranges by running `LoadFieldsSubset` (`cpp/src/arrow/ipc/reader.cc`) against an `IoRecordedRandomAccessFile`, which assumes the default V5 metadata buffer layout. The buffer layouts differ for union arrays between V4 and V5 metadata: in V4 metadata a union array includes a validity bitmap buffer slot, whereas in V5 it does not. For an IPC message written with V4 metadata that contains a union column (including Feather V2 files written with V4 metadata when reading a column subset), the byte ranges recorded by `ReadFieldsSubset` correspond to the wrong buffer offsets. When the batch is subsequently loaded using the message's V4 metadata version, it reads from unfilled regions of the body buffer and returns corrupted values for the selected union column. ### Expected behavior Reading a subset of fields via `IpcReadOptions::included_fields` from a V4 metadata IPC stream/file containing sparse or dense union columns should return the same array values as reading the full record batch and selecting those columns (for example, by falling back to reading the full message body in `ReadFieldsSubset` when `metadata_version() < MetadataVersion::V5` while keeping selective reads for V5). ### Component(s) C++ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
