anuragmantri commented on code in PR #16977:
URL: https://github.com/apache/iceberg/pull/16977#discussion_r3489458026
##########
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/SparkTableUtil.java:
##########
@@ -968,6 +982,33 @@ public static List<SparkPartition> filterPartitions(
}
}
+ private static void verifyRequiredField(Schema schema, DataFile dataFile) {
Review Comment:
```suggestion
private static void verifyRequiredFields(Schema schema, DataFile dataFile)
{
```
##########
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/SparkTableUtil.java:
##########
@@ -968,6 +982,33 @@ public static List<SparkPartition> filterPartitions(
}
}
+ private static void verifyRequiredField(Schema schema, DataFile dataFile) {
+ Set<Integer> requiredFieldIds =
+ TypeUtil.indexById(schema.asStruct()).entrySet().stream()
+ .filter(entry -> entry.getValue().isRequired())
+ .map(Map.Entry::getKey)
+ .collect(Collectors.toSet());
Review Comment:
Can computation of `requiredFieldIds` be extracted out? Wouldn't the schema
be same for all data files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
