Mohit-25-tech opened a new pull request, #53010:
URL: https://github.com/apache/arrow/pull/53010
## What changes are included in this PR?
Fixes #52997
This PR fixes an inconsistent column-projection behavior in
`pyarrow.parquet.ParquetFile` when a literal top-level column name collides
with a nested field path.
Previously, requesting `columns=["a.b"]` could unexpectedly return both the
literal column `a.b` and a struct column `a` containing nested field `b`.
The high-level `pq.read_table()` API returns only the literal column in this
scenario.
### Root cause
`ParquetFile._build_nested_paths()` builds projection lookup keys by joining
physical path components with `"."`.
This makes two distinct paths share the same lookup key:
```text
Literal top-level field: ["a.b"] -> "a.b"
Nested field: ["a", "b"] -> "a.b"
```
As a result, `_get_column_indices()` can select physical columns belonging
to both fields.
### Changes
- Track exact top-level column names separately from nested-path prefixes.
- Prioritize an exact top-level column match when resolving requested
projections.
- Preserve existing nested-field selection when no exact top-level match
exists.
- Update the relevant `ParquetFile` method documentation to explain the
precedence rule.
- Add regression tests covering:
- Literal dotted names colliding with nested paths.
- `read()`, `read_row_group()`, and `iter_batches()`.
- Deeper nested prefixes.
- Non-conflicting dotted names.
- Duplicate column selections.
- Pandas index metadata handling.
### Expected behavior
For a file containing a literal field `a.b` and a struct `a` with child `b`:
```python
file.read(columns=["a.b"])
```
Now selects the literal top-level field `a.b`.
The nested field remains accessible by selecting its parent `a`.
This intentionally clarifies the precedence of ambiguous dotted names and
aligns the result with `pq.read_table()` in the reproduced case.
### Testing
**Original bug reproduction**
- Independently reproduced with PyArrow 26.0.0 in Kaggle/Linux.
- Also observed with locally installed PyArrow 16.1.0 and 24.0.0.
**Proposed fix validation**
- 8 independent Kaggle checks passed using an adapted Python monkeypatch on
PyArrow 26.0.0.
- 4 focused pytest tests passed using the modified Python methods and
locally installed PyArrow 24.0.0 binary reader.
- Flake8 passed.
- autopep8 diff check passed.
- Python syntax compilation passed.
- `git diff --check` passed.
**Limitations:** The full repository test suite could not run locally
because the checkout does not contain a built `pyarrow.lib`. Pre-commit was
unavailable. Full checkout and cross-platform validation remain pending in CI.
### Compatibility considerations
This change intentionally modifies the behavior of ambiguous projections
where a dotted top-level column name collides with a nested path.
Existing callers relying on both fields being selected for one ambiguous
name may observe a different output schema.
Maintainer feedback on the proposed exact-name precedence is welcome.
### AI assistance
OpenAI Codex assisted with investigation, implementation, regression-test
development, and code review.
I independently reproduced the original issue in Kaggle, validated the
proposed resolution using additional checks, and reviewed the changes and test
results.
### Related issue
Fixes #ISSUE_NUMBER
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
