amogh-jahagirdar commented on code in PR #18795:
URL: https://github.com/apache/iceberg/pull/18795#discussion_r4485764478
##########
core/src/main/java/org/apache/iceberg/V4ManifestReader.java:
##########
@@ -150,20 +145,27 @@ public CloseableIterator<TrackedFile> iterator() {
return CloseableIterable.transform(files, this::copyResolved).iterator();
}
- private TrackedFile applyInheritance(TrackedFile file) {
- // the reader uses TrackingStruct to read tracking so this cast is safe
- TrackingStruct tracking = (TrackingStruct) file.tracking();
- if (isUncommitted) {
- // uncommitted files cannot have a sequence number or assign first row ID
- tracking.inherit(manifest.snapshotId());
- } else {
- tracking.inherit(manifest.snapshotId(), manifest.sequenceNumber());
- if (tracking.assignFirstRowId(nextRowId)) {
- this.nextRowId += file.recordCount();
- }
- }
+ private Function<TrackedFile, TrackedFile> applyInheritance() {
Review Comment:
Minor: could we split out the uncommitted case before building the function,
like `ManifestReader.idAssigner` does? `isUncommitted` doesn't change for the
reader, so there's no reason to check it per file or create a counter that an
uncommitted reader never uses. Something like:
```java
private Function<TrackedFile, TrackedFile> applyInheritance() {
if (isUncommitted) {
// uncommitted files cannot have a sequence number or assign first row ID
return file -> {
((TrackingStruct) file.tracking()).inherit(manifest.snapshotId());
return file;
};
}
return new Function<>() {
private Long nextRowId = manifest.firstRowId();
...
};
}
```
##########
core/src/main/java/org/apache/iceberg/V4ManifestReader.java:
##########
@@ -150,20 +145,27 @@ public CloseableIterator<TrackedFile> iterator() {
return CloseableIterable.transform(files, this::copyResolved).iterator();
}
- private TrackedFile applyInheritance(TrackedFile file) {
- // the reader uses TrackingStruct to read tracking so this cast is safe
- TrackingStruct tracking = (TrackingStruct) file.tracking();
- if (isUncommitted) {
- // uncommitted files cannot have a sequence number or assign first row ID
- tracking.inherit(manifest.snapshotId());
- } else {
- tracking.inherit(manifest.snapshotId(), manifest.sequenceNumber());
- if (tracking.assignFirstRowId(nextRowId)) {
- this.nextRowId += file.recordCount();
- }
- }
+ private Function<TrackedFile, TrackedFile> applyInheritance() {
+ return new Function<>() {
+ private Long nextRowId = manifest.firstRowId();
Review Comment:
Minor: this is boxed, so every `+=` allocates on the hot path when reading
entries. `idAssigner` avoids that by handling the null `first_row_id` case
before building the function, so it can keep a primitive `long`. Could we do
the same here? If the manifest's `first_row_id` is null, we'd just return a
function that does `assignFirstRowId(null)` for each file.
##########
core/src/test/java/org/apache/iceberg/TestV4ManifestReader.java:
##########
@@ -663,6 +664,59 @@ public void
inheritanceFirstRowIdIncrementsByRecordCount(FileFormat format) thro
.isEqualTo(expected);
}
+ @ParameterizedTest
+ @FieldSource("MANIFEST_FORMATS")
+ public void inheritanceFirstRowIdIsRepeatableAcrossIterations(FileFormat
format)
Review Comment:
I think the interleaved iterators test already covers this. It's effectively
testing the same behavior, and the interleaved one is the stronger check. Can
we drop this one?
##########
core/src/test/java/org/apache/iceberg/TestV4ManifestReader.java:
##########
@@ -663,6 +664,59 @@ public void
inheritanceFirstRowIdIncrementsByRecordCount(FileFormat format) thro
.isEqualTo(expected);
}
+ @ParameterizedTest
+ @FieldSource("MANIFEST_FORMATS")
+ public void inheritanceFirstRowIdIsRepeatableAcrossIterations(FileFormat
format)
+ throws IOException {
+ TrackedFile fileA = dataFile(ADDED_TRACKING,
"s3://bucket/table/file-a.parquet");
+ TrackedFile fileB = dataFile(ADDED_TRACKING,
"s3://bucket/table/file-b.parquet");
+
+ ManifestFile manifest =
+ writeManifest(format, UNPARTITIONED_TYPE, ImmutableList.of(fileA,
fileB));
+ when(manifest.firstRowId()).thenReturn(10_000L);
+
+ V4ManifestReader.Builder builder =
+ V4ManifestReader.builder(manifest, IO, TABLE_SCHEMA,
ID_PARTITIONING_SPECS)
+ .metricsConfig(METRICS_CONFIG);
+
+ try (V4ManifestReader reader = builder.build()) {
+ assertThat(Lists.newArrayList(reader))
+ .extracting(file -> file.tracking().firstRowId())
+ .containsExactly(10_000L, 10_000L + RECORD_COUNT);
+ assertThat(Lists.newArrayList(reader))
+ .as("a second iteration assigns the same first row IDs")
+ .extracting(file -> file.tracking().firstRowId())
+ .containsExactly(10_000L, 10_000L + RECORD_COUNT);
+ }
+ }
+
+ @ParameterizedTest
+ @FieldSource("MANIFEST_FORMATS")
+ public void inheritanceFirstRowIdIsIndependentBetweenIterators(FileFormat
format)
Review Comment:
Nit: new tests should be package private.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
