felixYyu commented on code in PR #4602:
URL: https://github.com/apache/iceberg/pull/4602#discussion_r855092966
##########
api/src/main/java/org/apache/iceberg/PartitionSpec.java:
##########
@@ -539,6 +539,9 @@ PartitionSpec buildUnchecked() {
static void checkCompatibility(PartitionSpec spec, Schema schema) {
for (PartitionField field : spec.fields) {
Type sourceType = schema.findType(field.sourceId());
+ if (sourceType == null &&
field.transform().toString().equalsIgnoreCase("void")) {
Review Comment:
Create table with TBLPROPERTIES ('format-version' = 1) , run
testUnboundPartitionSpecFormatVersion1 ValidationException log:
```
Cannot find source column for partition field: 1000: data_trunc_4: void(3)
org.apache.iceberg.exceptions.ValidationException: Cannot find source column
for partition field: 1000: data_trunc_4: void(3)
at
org.apache.iceberg.exceptions.ValidationException.check(ValidationException.java:50)
at
org.apache.iceberg.PartitionSpec.checkCompatibility(PartitionSpec.java:542)
at
org.apache.iceberg.TableMetadata$Builder.build(TableMetadata.java:1172)
at org.apache.iceberg.TableMetadata.updateSchema(TableMetadata.java:497)
at org.apache.iceberg.SchemaUpdate.commit(SchemaUpdate.java:387)
at
org.apache.iceberg.spark.SparkCatalog.commitChanges(SparkCatalog.java:503)
at
org.apache.iceberg.spark.SparkCatalog.alterTable(SparkCatalog.java:244)
at
org.apache.iceberg.spark.SparkCatalog.alterTable(SparkCatalog.java:98)
at
org.apache.iceberg.spark.SparkSessionCatalog.alterTable(SparkSessionCatalog.java:227)
at
org.apache.spark.sql.execution.datasources.v2.AlterTableExec.run(AlterTableExec.scala:37)
```
##########
api/src/main/java/org/apache/iceberg/UnboundPartitionSpec.java:
##########
@@ -52,10 +53,13 @@ private PartitionSpec.Builder copyToBuilder(Schema schema) {
PartitionSpec.Builder builder =
PartitionSpec.builderFor(schema).withSpecId(specId);
for (UnboundPartitionField field : fields) {
- if (field.partitionId != null) {
- builder.add(field.sourceId, field.partitionId, field.name,
field.transformAsString);
- } else {
- builder.add(field.sourceId, field.name, field.transformAsString);
+ Types.NestedField column = schema.findField(field.sourceId);
Review Comment:
Create table with TBLPROPERTIES ('format-version' = 2) , run
testUnboundPartitionSpecFormatVersion2 ValidationException log:
```
Cannot find source column: 3
java.lang.NullPointerException: Cannot find source column: 3
at
org.apache.iceberg.relocated.com.google.common.base.Preconditions.checkNotNull(Preconditions.java:963)
at org.apache.iceberg.PartitionSpec$Builder.add(PartitionSpec.java:517)
at
org.apache.iceberg.UnboundPartitionSpec.copyToBuilder(UnboundPartitionSpec.java:58)
at
org.apache.iceberg.UnboundPartitionSpec.bind(UnboundPartitionSpec.java:44)
at
org.apache.iceberg.PartitionSpecParser.fromJson(PartitionSpecParser.java:87)
at
org.apache.iceberg.TableMetadataParser.fromJson(TableMetadataParser.java:370)
at
org.apache.iceberg.TableMetadataParser.fromJson(TableMetadataParser.java:301)
at
org.apache.iceberg.TableMetadataParser.read(TableMetadataParser.java:264)
at
org.apache.iceberg.TableMetadataParser.read(TableMetadataParser.java:258)
at
org.apache.iceberg.BaseMetastoreTableOperations.lambda$refreshFromMetadataLocation$0(BaseMetastoreTableOperations.java:177)
at
org.apache.iceberg.BaseMetastoreTableOperations.lambda$refreshFromMetadataLocation$1(BaseMetastoreTableOperations.java:191)
at
org.apache.iceberg.util.Tasks$Builder.runTaskWithRetry(Tasks.java:404)
at
org.apache.iceberg.util.Tasks$Builder.runSingleThreaded(Tasks.java:214)
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:198)
at org.apache.iceberg.util.Tasks$Builder.run(Tasks.java:190)
at
org.apache.iceberg.BaseMetastoreTableOperations.refreshFromMetadataLocation(BaseMetastoreTableOperations.java:191)
at
org.apache.iceberg.BaseMetastoreTableOperations.refreshFromMetadataLocation(BaseMetastoreTableOperations.java:176)
at
org.apache.iceberg.BaseMetastoreTableOperations.refreshFromMetadataLocation(BaseMetastoreTableOperations.java:171)
at
org.apache.iceberg.hive.HiveTableOperations.doRefresh(HiveTableOperations.java:208)
at
org.apache.iceberg.BaseMetastoreTableOperations.refresh(BaseMetastoreTableOperations.java:96)
at
org.apache.iceberg.BaseMetastoreTableOperations.current(BaseMetastoreTableOperations.java:79)
at
org.apache.iceberg.BaseMetastoreCatalog.loadTable(BaseMetastoreCatalog.java:42)
at org.apache.iceberg.spark.SparkCatalog.load(SparkCatalog.java:526)
at
org.apache.iceberg.spark.SparkCatalog.loadTable(SparkCatalog.java:141)
at org.apache.iceberg.spark.SparkCatalog.loadTable(SparkCatalog.java:98)
at
org.apache.iceberg.spark.SparkSessionCatalog.loadTable(SparkSessionCatalog.java:118)
at
org.apache.spark.sql.connector.catalog.TableCatalog.tableExists(TableCatalog.java:119)
at
org.apache.spark.sql.execution.datasources.v2.CreateTableExec.run(CreateTableExec.scala:40)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
