the-other-tim-brown commented on code in PR #908:
URL: https://github.com/apache/incubator-xtable/pull/908#discussion_r4067697568
##########
xtable-core/src/test/java/org/apache/xtable/delta/TestDeltaHelper.java:
##########
@@ -169,8 +180,17 @@ public void createTable(SparkSession sparkSession, String
tableName, String base
}
if (enableColumnMapping) {
tableBuilder.property("delta.minReaderVersion", "2");
- tableBuilder.property("delta.minWriterVersion", "5");
tableBuilder.property("delta.columnMapping.mode", "name");
+ // Delta 3.x writes parquet field ids for the top level columns but
assigns none to a map
+ // key, a map value or a list element. Iceberg reads a file that carries
any id by id
+ // alone and ignores the name mapping, so those children become
unresolvable.
+ // IcebergCompatV2 is what makes Delta assign and record them.
+ if (enableIcebergCompatV2) {
+ tableBuilder.property("delta.minWriterVersion", "7");
Review Comment:
In the case where `parquet.field.id` is stripped, there is a default mapping
that Iceberg can use to read the table still if I am remembering correctly. It
applies that mapping to files written without field IDs. Not all engines
support this Iceberg feature though.
I think if we are going to produce something that may not be able to be
read, we should log out an error. If it definitely will not be able to be read,
we should throw an exception.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
