wombatu-kun commented on code in PR #17144:
URL: https://github.com/apache/iceberg/pull/17144#discussion_r3555997670
##########
core/src/test/java/org/apache/iceberg/TestTrackedFileAdapters.java:
##########
@@ -59,40 +59,41 @@ class TestTrackedFileAdapters {
.build();
private static final PartitionData PARTITION = partition("books");
- // Tracking field ordinals, looked up from the schema so the tests do not
hard-code offsets.
- private static final int DATA_SEQUENCE_NUMBER_ORDINAL =
- ordinalOf(Tracking.schema(), "sequence_number");
- private static final int FILE_SEQUENCE_NUMBER_ORDINAL =
- ordinalOf(Tracking.schema(), "file_sequence_number");
- private static final int FIRST_ROW_ID_ORDINAL = ordinalOf(Tracking.schema(),
"first_row_id");
- // manifestPos is appended after the tracking schema fields by the manifest
reader.
private static final int MANIFEST_POS_ORDINAL =
Tracking.schema().fields().size();
- // TrackedFile optional field ordinals, looked up from the schema.
- private static final Types.StructType TRACKED_FILE_SCHEMA =
- TrackedFile.schemaWithContentStats(Types.StructType.of(),
Types.StructType.of());
- private static final int CONTENT_TYPE_ORDINAL =
ordinalOf(TRACKED_FILE_SCHEMA, "content_type");
- private static final int SPEC_ID_ORDINAL = ordinalOf(TRACKED_FILE_SCHEMA,
"spec_id");
- private static final int DELETION_VECTOR_ORDINAL =
- ordinalOf(TRACKED_FILE_SCHEMA, "deletion_vector");
-
@Test
void testDataFileAdapterDelegation() {
+ TrackingStruct tracking =
+ new TrackingStruct(
+ EntryStatus.ADDED,
+ 42L,
+ DATA_SEQUENCE_NUMBER,
+ FILE_SEQUENCE_NUMBER,
+ null,
+ FIRST_ROW_ID,
+ null,
+ null);
+ tracking.setManifestLocation(MANIFEST_LOCATION);
+ tracking.set(MANIFEST_POS_ORDINAL, MANIFEST_POS);
Review Comment:
The description says ordinal setters were removed from this file, but
`tracking.set(MANIFEST_POS_ORDINAL, MANIFEST_POS)` stays at three call sites
(here, line 153, line 238). That is correct - `_pos` has no setter and is not
in `Tracking.schema()`, which is what @rdblue concluded on #17041 - but the
description reads as if it were gone.
What did get removed is `// manifestPos is appended after the tracking
schema fields by the manifest reader.`, the only line explaining why
`Tracking.schema().fields().size()` is the right index for it. @anoopj pointed
at this file as the reference for that pattern on #17041; worth keeping the
comment on line 62.
##########
core/src/test/java/org/apache/iceberg/TestTrackedFileAdapters.java:
##########
@@ -59,40 +59,41 @@ class TestTrackedFileAdapters {
.build();
private static final PartitionData PARTITION = partition("books");
- // Tracking field ordinals, looked up from the schema so the tests do not
hard-code offsets.
- private static final int DATA_SEQUENCE_NUMBER_ORDINAL =
- ordinalOf(Tracking.schema(), "sequence_number");
- private static final int FILE_SEQUENCE_NUMBER_ORDINAL =
- ordinalOf(Tracking.schema(), "file_sequence_number");
- private static final int FIRST_ROW_ID_ORDINAL = ordinalOf(Tracking.schema(),
"first_row_id");
- // manifestPos is appended after the tracking schema fields by the manifest
reader.
private static final int MANIFEST_POS_ORDINAL =
Tracking.schema().fields().size();
- // TrackedFile optional field ordinals, looked up from the schema.
- private static final Types.StructType TRACKED_FILE_SCHEMA =
- TrackedFile.schemaWithContentStats(Types.StructType.of(),
Types.StructType.of());
- private static final int CONTENT_TYPE_ORDINAL =
ordinalOf(TRACKED_FILE_SCHEMA, "content_type");
- private static final int SPEC_ID_ORDINAL = ordinalOf(TRACKED_FILE_SCHEMA,
"spec_id");
- private static final int DELETION_VECTOR_ORDINAL =
- ordinalOf(TRACKED_FILE_SCHEMA, "deletion_vector");
-
@Test
void testDataFileAdapterDelegation() {
+ TrackingStruct tracking =
+ new TrackingStruct(
Review Comment:
After this PR `TrackingBuilder` has no caller in `core/src/main`: its only
production call sites are inside `TrackedFileBuilder` (`build()`, `deleted()`,
`replaced()`), all deleted here. The only reference left anywhere is
`TestTrackingBuilder`, which is the same "only tests use it" position that
motivates dropping `TrackedFileBuilder`, and @rdblue's note on #16964 covers it
too ("stop using the builders that are already present").
Either drop `TrackingBuilder` and `TestTrackingBuilder` in this PR, or say
in the description that they follow separately - #17145 is currently investing
in `TestTrackingBuilder` in the opposite direction.
##########
core/src/test/java/org/apache/iceberg/TestTrackedFileAdapters.java:
##########
@@ -337,8 +388,24 @@ void testNullSpecIdThrowsWhenNoUnpartitionedSpec() {
@Test
void testUnknownSpecIdThrows() {
- TrackedFileStruct file = dummyTrackedFile(FileContent.DATA);
- file.set(SPEC_ID_ORDINAL, 99);
+ TrackedFileStruct file =
+ new TrackedFileStruct(
Review Comment:
`dummyTrackedFile` two methods below (line 472) already builds exactly this
file, and it is still used by the rejection and spec-resolution tests. This
call site, `testSpecIdMismatchThrows` and
`testNullTrackingReturnsNullTrackingFields` now hand-roll the 16-arg
constructor with eleven `null` arguments to vary a single field, so the same
fixture is expressed two different ways in one file.
An overload such as `dummyTrackedFile(FileContent contentType, Integer
specId)` plus one for the deletion-vector case keeps these back to a one-liner
without reintroducing a builder. The "values visible at the call site" argument
@amogh-jahagirdar made on #17035 applies to `TestTrackedFileStruct`, where the
struct's own CRUD is under test; here the file is plumbing for the adapter and
the eleven visible values are all `null`.
##########
core/src/test/java/org/apache/iceberg/TestTrackedFileAdapters.java:
##########
@@ -59,40 +59,41 @@ class TestTrackedFileAdapters {
.build();
private static final PartitionData PARTITION = partition("books");
- // Tracking field ordinals, looked up from the schema so the tests do not
hard-code offsets.
- private static final int DATA_SEQUENCE_NUMBER_ORDINAL =
- ordinalOf(Tracking.schema(), "sequence_number");
- private static final int FILE_SEQUENCE_NUMBER_ORDINAL =
- ordinalOf(Tracking.schema(), "file_sequence_number");
- private static final int FIRST_ROW_ID_ORDINAL = ordinalOf(Tracking.schema(),
"first_row_id");
- // manifestPos is appended after the tracking schema fields by the manifest
reader.
private static final int MANIFEST_POS_ORDINAL =
Tracking.schema().fields().size();
- // TrackedFile optional field ordinals, looked up from the schema.
- private static final Types.StructType TRACKED_FILE_SCHEMA =
- TrackedFile.schemaWithContentStats(Types.StructType.of(),
Types.StructType.of());
- private static final int CONTENT_TYPE_ORDINAL =
ordinalOf(TRACKED_FILE_SCHEMA, "content_type");
- private static final int SPEC_ID_ORDINAL = ordinalOf(TRACKED_FILE_SCHEMA,
"spec_id");
- private static final int DELETION_VECTOR_ORDINAL =
- ordinalOf(TRACKED_FILE_SCHEMA, "deletion_vector");
-
@Test
void testDataFileAdapterDelegation() {
+ TrackingStruct tracking =
+ new TrackingStruct(
+ EntryStatus.ADDED,
+ 42L,
+ DATA_SEQUENCE_NUMBER,
+ FILE_SEQUENCE_NUMBER,
+ null,
+ FIRST_ROW_ID,
+ null,
+ null);
+ tracking.setManifestLocation(MANIFEST_LOCATION);
+ tracking.set(MANIFEST_POS_ORDINAL, MANIFEST_POS);
+
TrackedFile file =
- TrackedFileBuilder.data(42L)
- .formatVersion(FORMAT_VERSION_V4)
- .location(DATA_FILE_LOCATION)
- .fileFormat(FileFormat.PARQUET)
- .partition(PARTITION)
- .recordCount(100L)
- .fileSizeInBytes(1024L)
- .specId(PARTITIONED_SPEC_ID)
- .contentStats(createContentStats())
- .sortOrderId(3)
- .keyMetadata(ByteBuffer.wrap(new byte[] {1, 2, 3}))
- .splitOffsets(ImmutableList.of(50L, 100L))
- .build();
- populateTrackingFields(file);
+ new TrackedFileStruct(
Review Comment:
`TrackedFileBuilder` still has open consumers. #16936 ("V4 write direction
wrappers") calls `TrackedFileBuilder.data(...)`, `.equalityDelete(...)` and
`.explicitTracking(...)` from its writer wrappers, and it adds the
`explicitTracking` factory to this class, so it extends `TrackedFileBuilder`
rather than only consuming it. #16867 builds its manifest-adapter test fixtures
on `.dataManifest(...)` / `.deleteManifest(...)`.
Both PRs were updated after @rdblue's "stop using the builders that are
already present" note on #16964, so neither is adapted yet. Worth cross-linking
this PR from #16936 and #16867 so the rework happens before this lands - for
#16936 that is a rewrite, not a rebase.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
