ump45nose opened a new pull request, #1055: URL: https://github.com/apache/iceberg-cpp/pull/1055
Closes #514. ## What `ReaderOptions::projection` is no longer required for the Avro and Parquet readers. When it is not set, the reader derives the read schema from the file schema and reads all fields. - Avro: add `FromAvroSchema()`, which converts an Avro schema written by Iceberg (with `field-id`, `element-id`, `key-id`/`value-id` and the `map` logical type for non-string keys) back into an Iceberg `Schema`. Sanitized field names are restored from `iceberg-field-name`. - Parquet: add `FromParquetSchema()`, which builds the schema from the `SchemaManifest`. Since it runs after `ApplyNameMapping`, files without field IDs work when a name mapping is provided. Fields that still have no ID (or lists/maps whose element/key/value has no ID) are skipped, because they can't be read by ID anyway. A file with neither field IDs nor a name mapping still fails as before. - `ParquetReader` now builds the `SchemaManifest` once in `Open()` and uses it for both the schema derivation and the projection. ## Tests - `FromAvroSchemaTest`: primitive types, nested struct/list/map (string and non-string keys), sanitized names, missing field ID and unsupported types. - `ParquetSchemaConversionTest`: round trip through `ToParquetSchema` for list and large_list, and conversion with a partial name mapping. - Reader tests for Avro and Parquet without a projection, Parquet with a name mapping, and Parquet without field IDs or name mapping. `avro_test`, `parquet_test`, `schema_test`, `data_test`, `scan_test`, `manifest_test`, `table_test` and `arrow_test` pass locally. This PR was written with the help of an AI coding assistant. I reviewed the changes and tests myself. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
