professor-moody opened a new pull request, #55370: URL: https://github.com/apache/arrow/pull/55370
# GH-55369: [C++][IPC] Map V4 union child fields when subsetting ### Rationale for this change When `IpcReadOptions::included_fields` is set, RecordBatchFileReader reads only the parts of a record batch body that the selected fields need. `ReadFieldsSubset` in `cpp/src/arrow/ipc/message.cc` obtains those ranges by running the fields loader against an `IoRecordedRandomAccessFile`, and that pass (LoadFieldsSubset in reader.cc) uses the default V5 buffer layout. The two layouts differ for unions: with V4 metadata a union array has a validity buffer slot, with V5 it does not. For a V4 message that contains a union column, the recorded ranges therefore belong to the wrong buffers. The batch is then loaded with the message's own V4 layout, from parts of the body buffer that `ReadFieldsSubset` did not fill, so the selected union column does not hold the values stored in the file. This also affects Feather V2 files, since the Feather reader reads a column subset through `included_fields`. ### What changes are included in this PR? In `ReadFieldsSubset`, when the message metadata version is older than V5, the whole body is read into the body buffer after the fields loader has run, instead of only the recorded ranges. V5 messages keep the selective reads. ### Are these changes tested? Yes. `cpp/src/arrow/ipc/read_write_test.cc` adds `ReadFieldSubsetWithUnionsV4RoundTrip` and `ReadFieldSubsetWithUnionsV5RoundTrip` for the stream reader, the file reader and the two file generator fixtures. Each writes a batch with a sparse union, a dense union and an `int32` column using the given metadata version, reads several field subsets, checks `ValidateFull()` and compares the result with the expected columns. Without this change `TestFileFormat.ReadFieldSubsetWithUnionsV4RoundTrip` fails and the V5 tests pass. ### Are there any user-facing changes? Reading a subset of fields from a V4 IPC file (or a Feather V2 file written with V4 metadata) that contains union columns now returns the stored values. For messages with pre-V5 metadata a subset read now reads the full message body. **This PR contains a "Critical Fix".** Reading a subset of fields from such a file could return values that are not in the file. ### AI usage This change, its test and this description were drafted with an AI coding assistant. The change was built and the IPC and Parquet test suites were run against it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
