fairytale0828 opened a new pull request, #1044:
URL: https://github.com/apache/iceberg-cpp/pull/1044
## Summary
Parquet files without field IDs (for example, files added by table
migration) currently fail with `NotImplemented` in `BuildProjection`. This PR
resolves the TODO by applying the name mapping from
`ReaderOptions::name_mapping`, which the Avro reader already supports.
- If the Parquet schema has no field IDs, `ApplyNameMapping` walks the
`SchemaManifest` and adds `PARQUET:field_id` metadata to each field found in
the mapping. The existing projection then runs unchanged.
- Names follow the Iceberg name mapping spec: list elements use `element`,
and map entries use `key` and `value`, regardless of the physical names in the
file. Both `list` and `large_list` are handled.
- Fields not in the mapping, or mapped without a field ID, get no ID and
follow the existing missing-column rules (initial default, null, or an error
for required fields). This matches the Java Parquet reader.
- Field IDs in the file take precedence. The mapping is applied only when
the file has no field IDs at all.
- A file without field IDs and without a name mapping now returns
`InvalidSchema`, consistent with Avro.
Known limitations, not changed in this PR:
- `NameMapping` builds its lookup indexes lazily on first use, so concurrent
readers sharing one mapping can race on that first lookup. This already affects
Avro and can be fixed separately.
- Field names containing `.` cannot be told apart from nested paths, because
`NameMapping::Find` joins path segments with `.`.
## Test Plan
- `ParquetNameMappingTest` in `parquet_schema_test.cc`: assigning IDs by
column name, matching an alias, nested struct/list/map fields with non-standard
physical names (`item`, `key_value`), and unmapped fields or fields mapped
without an ID.
- New cases in `ParquetReaderTest` in `parquet_test.cc`:
- `ReadWithoutFieldIdsOrNameMapping`: returns `InvalidSchema`.
- `ReadWithNameMapping`: alias, column reordering, type promotion, and a
missing optional column.
- `ReadWithFieldIdsAndNameMapping`: file field IDs win over a conflicting
mapping.
- `ReadNestedFieldsWithNameMapping`: end-to-end read of struct,
`list<struct>`, and `map<string, struct>`, including null and empty values,
with both `list` and `large_list`.
- `pre-commit run` passes on the changed files.
- `ctest -R parquet_test`: 140 passed, 1 skipped.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
