jiangxt2 opened a new pull request, #55350:
URL: https://github.com/apache/arrow/pull/55350
### Rationale for this change
ORC Dataset scans can confuse a literal top-level field such as `a.b` with
the nested path `a` / `b`. This can replace stored values with nulls and cause
filters to discard matching rows.
Fixes #55326.
### What changes are included in this PR?
- Pass per-file physical top-level field indices from Dataset to the ORC
batch reader.
- Add `GetRecordBatchReaderByIndices`, reusing integer selection and
streaming reader construction while preserving the existing name-based API and
its empty-initializer-list calls.
- Add adapter and Dataset regressions covering colliding field paths,
projection and filter dependencies, field reordering, missing columns, batch
boundaries, empty inputs, and memory-backed and on-disk files.
### Are these changes tested?
Validated with matching C++ and Python builds on macOS arm64, using CPython
3.13.7 and liborc 2.2.1.
- `cpp/build/dev/release/arrow-orc-adapter-test`: 41 passed; one existing
timezone-database check is skipped because it applies only to ORC versions
before 2.0.
- `cpp/build/dev/release/arrow-dataset-file-orc-test`: 27 passed.
- The following complete Python ORC collection: 39 passed, no skips.
```bash
python -m pytest -q -ra --enable-orc --enable-dataset --enable-pandas
--enable-snappy \
python/pyarrow/tests/test_orc.py \
python/pyarrow/tests/test_dataset.py::test_orc_format \
python/pyarrow/tests/test_dataset.py::test_orc_dotted_field_identity \
python/pyarrow/tests/test_dataset.py::test_orc_dotted_field_schema_evolution \
python/pyarrow/tests/test_dataset.py::test_orc_scan_options \
python/pyarrow/tests/test_dataset.py::test_orc_format_not_supported \
python/pyarrow/tests/test_dataset.py::test_orc_writer_not_implemented_for_dataset
```
clang-format, cpplint, autopep8, Flake8, and `git diff --check` passed.
### Are there any user-facing changes?
ORC Dataset full scans, projections, and filters preserve the correct fields
when literal dotted names collide with nested paths, regardless of field order.
Existing name-based ORCFile selector semantics are preserved.
**This PR contains a "Critical Fix".** It corrects silent null substitution
and dropped rows in query results.
### Was AI used for this PR?
Codex generated the code, tests, and PR description and performed local
review and verification.
**PR code and description written by:**
- [x] Human
- [x] AI
**Reviewed before submission by:**
- [x] Human
- [x] AI
- [ ] Not reviewed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
