FadyYosry77 opened a new pull request, #51691:
URL: https://github.com/apache/arrow/pull/51691
### Rationale for this change
ProjectNode creates output column definitions with the requested output name
and the type of the bound expression for that column. Nullability information
is available from the input schema but is not copied during output column
creation; thus, output columns are nullable by default.
As a result, projecting a nonnullable column from the input schema produces
a nullable column in the output schema, even when only that column is
projected. An identity projection may therefore produce a schema incompatible
with its input.
GH-51231 restricts this change to top-level input fields.
### What changes does this PR contain?
For direct projections of top-level fields, the ProjectNode class now uses
the nullability of the input field to determine the nullability of the output
field.
The projected field name and type of the bound expression remain unchanged.
Computed fields and nested field references continue to have their current
nullable behavior. This pull request does not include full expression
nullability inference.
A regression test covers:
- direct nonnullable-nullable field correspondence
- reordered projections
- renaming a required field
- a calculated control with nonnullable input and nullable output as before
### Are these changes tested?
Yes.
The regression test checks the schemas created. It failed with the old
implementation and passed with the new implementation.
Local validation included:
- targeted GH-51231 regression: 1/1 passed
- full arrow-acero-plan-test: 63/63 passed
- registered plan CTest: 1/1 passed
- full arrow_acero CTest run: 14/14 tests passed
- clang-format v18.1.8 passed
- cpplint 1.6.1 passed
- output of git diff --check was clean
The targeted regression analysis was conducted with:
```bash
./cpp/build-gh51231/debug/arrow-acero-plan-test \
--gtest_filter=ExecPlanExecution.ProjectPreservesDirectFieldNullability
```
The larger Acero test set was evaluated with:
```bash
PYTHON=python3 ctest --test-dir cpp/build-gh51231 \
-L arrow_acero \
--output-on-failure \
-j2
```
Substrait integration was not built locally. The regression test is defined
at the native Acero ProjectNode level.
### Are there any visible user impacts?
Yes. Top-level field projections copy the nullable attribute from input
fields to output fields.
This change only alters the projected schema. Projected values remain the
same.
There are no changes to public API or ABI signatures.
### Was AI involved in this PR?
Yes. AI tools were used to assist with the investigation, implementation and
test drafting, review, and preparation of this description.
I reproduced the issue locally and reviewed the patch, verifying that the
test case failed without the patch and passed with the patch, and ran the
listed C++ tests and style checks.
**PR code and description by:**
- [x] Human
- [x] AI
**Reviewed before submission by:**
- [x] Human
- [x] AI
- [ ] Not reviewed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
