amogh-jahagirdar commented on code in PR #18252:
URL: https://github.com/apache/iceberg/pull/18252#discussion_r4457758883
##########
api/src/main/java/org/apache/iceberg/types/Types.java:
##########
@@ -1155,6 +1155,136 @@ private Map<Integer, NestedField> lazyFieldsById() {
}
}
+ public static final class FileType extends NestedType {
+ public static final String NAME = "file";
+ public static final int NUM_NESTED_FIELDS = 6;
+
+ private static final String URI = "uri";
+ private static final String OFFSET = "offset";
+ private static final String SIZE = "size";
+ private static final String CONTENT_TYPE = "content_type";
+ private static final String CHECKSUM = "checksum";
+ private static final String INLINE = "inline";
+
+ public static FileType of(int baseId) {
Review Comment:
Something feels a bit conceptually off to have to publicly expose this. The
nested IDs are fully determined by the holding field's ID, so callers shouldn't
be supplying it imo. We can run into situations where the two disagree. For
example, supporting `addColumn("photo", FileType.of(?))` would mean the caller
guesses an ID before one is assigned, and we'd need special-cased file handling
in ID assignment to correct it.
I see part 2 adds validation for the mismatch, but having to validate it
suggests the API exposes a value callers shouldn't choose.
Have we considered an ID-free `FileType.get()`, then binding the base ID
internally when the `NestedField` (or list/map) is built? That way there's
nothing to mismatch, and `applyAssignedId` goes away.Sorry if this was
discussed before, is feel free to just point me to any context...
##########
api/src/main/java/org/apache/iceberg/types/TypeUtil.java:
##########
@@ -664,31 +729,62 @@ public static GetID reassignConflictingIds(Set<Integer>
conflictingIds, Set<Inte
private static class ReassignConflictingIds implements GetID {
private final Set<Integer> conflictingIds;
private final Set<Integer> allUsedIds;
+ private final Set<Integer> claimedIds;
private final AtomicInteger nextId;
private ReassignConflictingIds(Set<Integer> conflictingIds, Set<Integer>
allUsedIds) {
this.conflictingIds = conflictingIds;
this.allUsedIds = allUsedIds;
+ this.claimedIds = Sets.newHashSet();
Review Comment:
I'd prefer to not add this until we know it's needed.
`ReassignConflictingIds` is for remapping colliding IDs when combining ID
spaces into a read projection (e.g. the `_partition` metadata column in Spark,
whose fields are transform results and so can't even be a file). It doesn't
allocate persistent IDs, so I don't think file block reservation means much
here. Could we leave it unchanged? Without an override, the `GetID` default
already throws for file types, and we can add support if a caller ever needs it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
