jiangxt2 opened a new issue, #55326:
URL: https://github.com/apache/arrow/issues/55326
### Describe the bug, including details regarding any error messages,
version, and platform.
An ORC Dataset scan can silently replace stored values with nulls when the
file contains both a literal top-level field named `a.b` and a struct field `a`
containing `b`.
This occurs during a full scan without a `columns` selector. Filtering the
affected field can subsequently discard matching rows. Reading the same file
through `ORCFile.read()` **without a columns selector** preserves both fields
correctly.
The issue was found while investigating ORC reads used by Ray Data. The
reproducer requires only PyArrow.
**Reproducer**
```python
import pyarrow as pa
import pyarrow.dataset as ds
import pyarrow.orc as orc
table = pa.table({
"a.b": [1, 2],
"a": [{"b": 10}, {"b": 20}],
})
sink = pa.BufferOutputStream()
orc.write_table(table, sink)
data = sink.getvalue()
def scan(**kwargs):
fragment = ds.OrcFileFormat().make_fragment(pa.BufferReader(data))
return fragment.to_table(**kwargs).to_pydict()
reader = orc.ORCFile(pa.BufferReader(data))
print("ORCFile:", reader.read().to_pydict())
print("Dataset:", scan())
print("Dataset filter:", scan(
columns=["a.b"],
filter=ds.field("a.b") > 1,
))
```
**Actual output**
```text
ORCFile: {'a.b': [1, 2], 'a': [{'b': 10}, {'b': 20}]}
Dataset: {'a.b': [None, None], 'a': [{'b': 10}, {'b': 20}]}
Dataset filter: {'a.b': []}
```
No exception is raised. The Dataset result also passes
`validate(full=True)`. The file itself retains the correct values; the
incorrect nulls are produced during reading.
**Expected behavior**
The full Dataset scan should preserve both fields and their stored values.
The filtered result should be `{'a.b': [2]}`, matching an in-memory filter of
the correctly read table.
This report concerns full-scan correctness and does not require redefining
precedence between ambiguous string selectors.
**Additional observations**
- Reversing the top-level field order to `["a", "a.b"]` makes these scans
return correct results.
- The corresponding cases without a field-name collision work.
- The issue also reproduces with an on-disk ORC file, so it is not specific
to `BufferReader`.
- For this reproducer, `ORCFile.read()` without a columns selector, followed
by in-memory selection or filtering, is a workaround, but requires whole-file
materialization.
**Likely cause / Source analysis**
The source suggests the following chain:
- liborc’s
[`ColumnSelector`](https://github.com/apache/orc/blob/v2.2.1/c%2B%2B/src/Reader.cc#L151)
maps both field paths to the same dotted string, with the later assignment
replacing the earlier one.
- The [Arrow ORC Dataset
reader](https://github.com/apache/arrow/blob/9b9fad6810850c3b884e14617356500143ef4aeb/cpp/src/arrow/dataset/file_orc.cc#L77)
passes field names to this selection path.
-
[`MakeExecBatch()`](https://github.com/apache/arrow/blob/9b9fad6810850c3b884e14617356500143ef4aeb/cpp/src/arrow/compute/expression.cc#L668-L693)
supplies nulls for an expected field missing from the returned batch.
This source-based explanation is consistent with the observed behavior, but
has not been confirmed through runtime debugging.
**Environment**
- macOS arm64, Python 3.10.18, released PyArrow 23.0.1.
- Additional independent reproduction: Linux x86_64, Python 3.12.14, with
PyArrow 23.0.1 and 17.0.0, including the on-disk file and control cases above.
**Related work**
[#52997](https://github.com/apache/arrow/issues/52997) /
[#53010](https://github.com/apache/arrow/pull/53010) discuss related
dotted-name selection behavior in Parquet. They do not cover this ORC Dataset
full-scan correctness issue. Searches found no existing ORC Dataset report or
merged fix covering this behavior.
Investigation, local reproduction and report drafting used AI assistance.
### Component(s)
C++, Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
