erensanlier opened a new issue, #936:
URL: https://github.com/apache/incubator-xtable/issues/936
Translating an Iceberg schema with field IDs to Delta through the Kernel
target produces a table that delta-rs 1.6.3 cannot open:
```
Kernel error: Schema error: Column mapping is not enabled but field
'block_number' is annotated with delta.columnMapping.id
```
`DeltaKernelSchemaExtractor.getFieldMetadata()` copies
`InternalField.fieldId` into `delta.columnMapping.id`. The target does not
enable `delta.columnMapping.mode`, and the resulting protocol remains reader 1
/ writer 2. The same emission is present on current upstream `main`.
To reproduce:
1. Sync an Iceberg snapshot whose schema contains a `long` field named
`block_number` with field ID 1 to a fresh Delta Kernel target.
2. Inspect the first Delta commit: `schemaString` contains
`"metadata":{"delta.columnMapping.id":1}`, configuration has no
`delta.columnMapping.mode`, and the protocol is
`{"minReaderVersion":1,"minWriterVersion":2}`.
3. Open the output using Python `deltalake` 1.6.3: `DeltaTable(path)`. It
fails with the error above before reading data.
Expected: omit column-mapping annotations when the target does not enable
column mapping. Keep reader 1 / writer 2 for these tables. Source-side
extraction of IDs from column-mapped Delta tables should continue to work.
Locally, removing the five-line ID propagation block lets delta-rs 1.6.3
open the translated table and read both rows from a synthetic Parquet fixture,
with the protocol unchanged. Regression checks cover nested structs, arrays,
maps, preserved UUID/comment metadata, and the committed protocol. The current
Spark target already uses `SparkSchemaExtractor`, which does not emit these
annotations.
Reference: [Delta column-mapping
protocol](https://github.com/delta-io/delta/blob/master/PROTOCOL.md#column-mapping).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
