stevenzwu commented on code in PR #19019:
URL: https://github.com/apache/iceberg/pull/19019#discussion_r4609683581
##########
core/src/main/java/org/apache/iceberg/TrackingStruct.java:
##########
@@ -62,6 +63,7 @@ class TrackingStruct extends SupportsIndexProjection
implements Tracking, Serial
private Long firstRowId = null;
private byte[] deletedPositions = null;
private byte[] replacedPositions = null;
+ private Long snapshotTimestampMicros = null;
Review Comment:
Note that `inherit(long, long)` below doesn't fill `snapshotTimestampMicros`
yet. That's intentional: this PR is limited to the schema, interface, and
struct changes.
Read inheritance comes in the next stacked PR. There, `inherit` takes the
manifest's snapshot timestamp as a third argument and fills it under the same
condition as the data sequence number, and the root manifest writer persists
the commit timestamp for added and modified entries.
##########
core/src/test/java/org/apache/iceberg/TestV4ManifestReader.java:
##########
@@ -341,19 +347,31 @@ public void statusFilter(FileFormat format) throws
IOException {
// inherits seq number and assigned first row ID
dataFile(
new TrackingStruct(
- EntryStatus.ADDED, 42L, MANIFEST_SEQ, MANIFEST_SEQ, null,
10_000L, null, null),
+ EntryStatus.ADDED,
+ 42L, // snapshotId
+ MANIFEST_SEQ,
+ MANIFEST_SEQ,
+ null, // modifiedSnapshotId
+ 10_000L, // firstRowId
+ null, // deletedPositions
+ null, // replacedPositions
+ null), // snapshotTimestampMicros
Review Comment:
`null` because timestamp inheritance on read isn't in this PR (the sequence
number and first row ID here are inherited). The next stacked PR changes this
to the manifest's timestamp. Same for the added entry in `mdvFilter`.
##########
core/src/test/java/org/apache/iceberg/TestV4ManifestReader.java:
##########
@@ -585,12 +635,20 @@ public void inheritanceAddedSequenceNumbers(FileFormat
format) throws IOExceptio
// inherits sequence numbers
dataFile(
new TrackingStruct(
- EntryStatus.ADDED, SNAPSHOT_ID, 5L, 5L, null, 5_000L,
null, null),
+ EntryStatus.ADDED, SNAPSHOT_ID, 5L, 5L, null, 5_000L,
null, null, null),
Review Comment:
`null` because timestamp inheritance on read isn't in this PR. The next
stacked PR changes this to the manifest's timestamp.
##########
core/src/test/java/org/apache/iceberg/TestV4ManifestReader.java:
##########
@@ -516,11 +557,12 @@ public void
inheritanceModifiedSnapshotIdNotInherited(FileFormat format) throws
TrackedFile withModifiedSnapshotId =
dataFile(
new TrackingStruct(
- EntryStatus.ADDED, SNAPSHOT_ID, 5L, 5L, 1234567L, 5_000L,
null, null),
+ EntryStatus.ADDED, SNAPSHOT_ID, 5L, 5L, 1234567L, 5_000L,
null, null, SNAPSHOT_TS),
Review Comment:
Unrelated to this PR, but noticed while filling in timestamps:
`withModifiedSnapshotId` here is an ADDED entry with a non-null modified
snapshot ID. Since #18984, `TrackingBuilder.dvUpdated()` no longer sets a
modified snapshot ID on added entries, so this fixture describes a state the
builder can't produce. The test still passes because it only checks that the
modified snapshot ID isn't inherited. A carried-forward EXISTING entry that was
modified earlier would be a realistic replacement. I left it alone to keep this
PR focused, and can follow up separately.
##########
core/src/test/java/org/apache/iceberg/TestV4ManifestReader.java:
##########
@@ -639,11 +697,13 @@ public void
inheritanceModifiedDataSequenceNumber(FileFormat format) throws IOEx
public void inheritanceFirstRowId(FileFormat format) throws IOException {
TrackedFile withoutFirstRowId =
dataFile(
- new TrackingStruct(EntryStatus.ADDED, SNAPSHOT_ID, 5L, 5L, null,
null, null, null),
+ new TrackingStruct(
+ EntryStatus.ADDED, SNAPSHOT_ID, 5L, 5L, null, null, null,
null, SNAPSHOT_TS),
Review Comment:
Note that `withoutFirstRowId` isn't a realistic production state: an added
entry wouldn't have a explicit file sequence number and a snapshot timestamp
without a first row ID. In practice the file sequence number, first row ID, and
snapshot timestamp are either all assigned by the root writer at commit time or
all inherited together on read. The fixture keeps the other fields explicit on
purpose, so first row ID is the only value this test exercises.
##########
core/src/main/java/org/apache/iceberg/V4Metadata.java:
##########
@@ -993,28 +995,20 @@ public <T> void set(int pos, T value) {
@Override
public <T> T get(int pos, Class<T> javaClass) {
- return javaClass.cast(get(pos));
+ return javaClass.cast(getInternal(pos));
}
- private Object get(int pos) {
Review Comment:
The tracking wrapper is split into leaf and root `getInternal`
implementations because the two write schemas no longer share a prefix: leaf
tracking omits `deleted_positions` and `replaced_positions`, so the appended
`snapshot_timestamp` is at position 6 in leaf manifests and 8 in root
manifests.
The shared switch only worked before because the leaf tracking schema
happened to be a prefix of the root schema. One switch per schema doesn't rely
on that, which should be more robust as fields are added, and it follows the
same pattern as `LeafTrackedFileWrapper` and `RootTrackedFileWrapper`.
##########
core/src/test/java/org/apache/iceberg/TestV4ManifestWriter.java:
##########
@@ -272,7 +281,8 @@ public void rootDataFileAppend() throws IOException {
null, // not modified
COMMIT_FIRST_ROW_ID,
null, // no change bitmaps
- null));
+ null,
+ null)); // snapshot timestamp
Review Comment:
`null` even though `FILE_A` has a timestamp: `append` starts fresh from
`TrackingBuilder.added`, and the root writer doesn't write the commit timestamp
in this PR. The next stacked PR changes this to `COMMIT_SNAPSHOT_TS`. Same for
the other append tests in this file.
##########
core/src/test/java/org/apache/iceberg/TestV4ManifestReader.java:
##########
@@ -625,7 +683,7 @@ public void
inheritanceModifiedDataSequenceNumber(FileFormat format) throws IOEx
List.of(
dataFileWithColumnFiles(
new TrackingStruct(
- EntryStatus.MODIFIED, SNAPSHOT_ID, 5L, 3L, 34L, 5_000L,
null, null),
+ EntryStatus.MODIFIED, SNAPSHOT_ID, 5L, 3L, 34L, 5_000L,
null, null, null),
Review Comment:
`null` because timestamp inheritance on read isn't in this PR. Like the data
sequence number, the timestamp is reset when column files are updated, so the
next stacked PR changes this to the manifest's timestamp.
##########
core/src/test/java/org/apache/iceberg/TestV4ManifestReader.java:
##########
@@ -706,23 +776,31 @@ public void
inheritanceFirstRowIdIncrementsByRecordCount(FileFormat format) thro
List.of(
dataFile(
new TrackingStruct(
- EntryStatus.ADDED, SNAPSHOT_ID, 5L, 5L, null, 10_000L,
null, null),
+ EntryStatus.ADDED, SNAPSHOT_ID, 5L, 5L, null, 10_000L,
null, null, null),
Review Comment:
`null` because timestamp inheritance on read isn't in this PR. The next
stacked PR changes these to the manifest's timestamp.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]