emilie-wang commented on issue #624:
URL:
https://github.com/apache/incubator-xtable/issues/624#issuecomment-2611161449
A quick update,, to answer this `" if Iceberg will allow us to set the ID of
the element level higher than the value's ID."`, I tested by using same Schema
without enabling column mappings. This is the schema of the converted Iceberg
table, the ids are set from Xtable, while array field `info`'s id is `3` and
`element-id` is `4`, which has no issue.
``` "schemas" : [ {
"type" : "struct",
"schema-id" : 0,
"fields" : [ {
"id" : 1,
"name" : "id",
"required" : false,
"type" : "int"
}, {
"id" : 2,
"name" : "name",
"required" : false,
"type" : "string"
}, {
"id" : 3,
"name" : "info",
"required" : false,
"type" : {
"type" : "list",
"element-id" : 4,
"element" : "string",
"element-required" : false
}
} ]
} ],
```
However, back to original issue for a Delta table with enabling column
mapping, when the schema evolves and for example a new column is added,
actually the column name written in the underlying parquet file is different:
```
{
"name": "type",
"type": "string",
"nullable": true,
"metadata": {
"delta.columnMapping.id": 4,
"delta.columnMapping.physicalName":
"col-1ee2c1d5-0785-4787-be84-a44a114cfa7d"
}
},
```
The added field name is `type`, but the physicalName in parquet file is
`col-1ee2c1d5-0785-4787-be84-a44a114cfa7d`. If the field name `type` is carried
to Iceberg schema, Iceberg can't read successfully from the parquet file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@xtable.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
