huan233usc opened a new pull request, #942:
URL: https://github.com/apache/iceberg-cpp/pull/942
## Summary
Writing an Arrow batch with an older schema currently requires callers to
materialize every column added by schema evolution. This PR lets the data
writers align an input batch to the current table schema and populate omitted
columns with Iceberg `write-default` values.
The end-to-end behavior is intentionally demonstrated with different
defaults:
```text
input schema: required int id
write schema: required int id, required int added
initial-default: 42
write-default: 7
input rows: [[1], [2]]
stored rows: [[1, 7], [2, 7]]
```
Because the file is written after `added` exists, the missing value must be
`write-default` (`7`), not `initial-default` (`42`). The test writes and reads
the file through the public `DataWriter` and reader registry for both Parquet
and Avro.
- Add optional input-schema metadata to writer options; leaving it null
preserves the existing exact-schema behavior.
- Align fields by Iceberg field ID before handing a batch to the Parquet or
Avro implementation.
- Materialize missing fields from `write-default`, allow missing optional
fields as null, and recurse through nested structs.
- Keep `initial-default` out of the write path; it remains a read-time rule
for older files.
This is a self-contained E2E presentation of the work in #941 and includes
the alignment implementation plus its writer integration. It addresses the
write-path portion of #637.
### How to review
1. Start with `WriteMissingColumnUsesWriteDefault` in `data_writer_test.cc`
for the observable contract.
2. Follow `DataWriterOptions::input_schema` into the Parquet and Avro
writers.
3. Review `AlignBatchForWrite` for field-ID matching, default
materialization, and nested-struct handling.
### Risk / scope
Alignment is opt-in through `input_schema`, so existing callers continue to
use the current zero-conversion path. This PR covers write-time defaults;
applying `initial-default` while projecting older files is separate read-path
behavior.
## Test Plan
- [x] Compiled all changed production translation units with `-Werror`.
- [x] Ran `WriteMissingColumnUsesWriteDefault` for Parquet and Avro: 2/2
passed.
- [x] Ran all parameterized `DataWriterFormatTest` cases: 6/6 passed.
- [ ] CI full build and test suite (local CMake regeneration is blocked by
an existing Homebrew Arrow/utf8proc target collision).
## Verification Commands
```bash
cmake --build build --target data_test
build/src/iceberg/test/data_test \
--gtest_filter='FormatTypes/DataWriterFormatTest.WriteMissingColumnUsesWriteDefault/*'
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
