anoopj commented on code in PR #18108:
URL: https://github.com/apache/iceberg/pull/18108#discussion_r4020391754
##########
core/src/test/java/org/apache/iceberg/TestV4ManifestReader.java:
##########
@@ -684,6 +684,38 @@ public void
filterMatchesFilesAcrossDisjointSpecs(FileFormat format) throws IOEx
}
}
+ @ParameterizedTest
+ @FieldSource("MANIFEST_FORMATS")
+ public void narrowPartitionProjectionReadsFullUnionTuple(FileFormat format)
throws IOException {
+ PartitionSpec idSpec =
+ PartitionSpec.builderFor(TABLE_SCHEMA)
+ .withSpecId(0)
+ .add(1, 1000, "id", Transforms.identity())
+ .build();
+ PartitionSpec dataSpec =
+ PartitionSpec.builderFor(TABLE_SCHEMA)
+ .withSpecId(1)
+ .add(2, 1001, "data", Transforms.identity())
+ .build();
+ Map<Integer, PartitionSpec> specsById =
+ ImmutableMap.of(idSpec.specId(), idSpec, dataSpec.specId(), dataSpec);
+ Types.StructType unionType =
Partitioning.unionPartitionTypes(specsById.values());
+
+ TrackedFile file =
+ dataFile("by-data.parquet", dataSpec.specId(),
unionPartition(unionType, null, "x"));
+ ManifestFile manifest = writeManifest(format, unionType,
ImmutableList.of(file));
+
+ // select id, which this file's spec does not use; the reader must still
read the whole union
+ // tuple (including data) so partition() can project it onto the file's
data spec
+ try (V4ManifestReader reader =
+ V4ManifestReader.builder(manifest, io, specsById, TABLE_LOCATION)
+ .select("partition.id")
+ .build()) {
+ TrackedFile actual = Iterables.getOnlyElement(reader);
+ assertThat(actual.partition().get(0,
CharSequence.class)).hasToString("x");
+ }
+ }
+
Review Comment:
We do have coverage in the scan planner here:
https://github.com/apache/iceberg/pull/17541 You can see that there is a test
failure (which will be red until this PR is merged)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
