pvary commented on code in PR #17320:
URL: https://github.com/apache/iceberg/pull/17320#discussion_r3932787500
##########
parquet/src/main/java/org/apache/iceberg/parquet/ParquetSchemaUtil.java:
##########
@@ -129,12 +131,94 @@ public static Type fieldType(GroupType group, String
name) {
public static MessageType pruneColumns(MessageType fileSchema, Schema
expectedSchema) {
Review Comment:
This method is called from ParquetReadSupport.init which is used by
parquet-mr. There the returned reader context returns the newly projected extra
columns and these columns are returned if no createReaderFunc has been set.
If this is a valid, supported scenario (parquet-mr, no createReaderFunc),
then we should fix it as the returned values will change.
This could help to repro in TestGenericData:
```
private static final Schema NESTED_MULTI_LEAF_SCHEMA =
new Schema(
Types.NestedField.required(1, "id", Types.LongType.get()),
Types.NestedField.optional(
2,
"nested",
Types.StructType.of(
Types.NestedField.required(3, "big",
Types.StringType.get()),
Types.NestedField.required(4, "inner",
Types.StringType.get()))));
private static final Schema NESTED_DEFAULT_ONLY_READ_SCHEMA =
new Schema(
Types.NestedField.required(1, "id", Types.LongType.get()),
Types.NestedField.optional("nested")
.withId(2)
.ofType(
Types.StructType.of(
Types.NestedField.optional("added")
.withId(9)
.ofType(Types.StringType.get())
.withInitialDefault(Literal.of("US"))
.build()))
.build());
@Test
public void testGenericReadDoesNotMaterializeUnprojectedColumns() throws
IOException {
Types.StructType nestedType =
NESTED_MULTI_LEAF_SCHEMA.findField("nested").type().asStructType();
Record present = GenericRecord.create(NESTED_MULTI_LEAF_SCHEMA);
present.setField("id", 1L);
Record presentNested = GenericRecord.create(nestedType);
presentNested.setField("big", "b");
presentNested.setField("inner", "a");
present.setField("nested", presentNested);
Record nullNested = GenericRecord.create(NESTED_MULTI_LEAF_SCHEMA);
nullNested.setField("id", 2L);
nullNested.setField("nested", null);
OutputFile output = new InMemoryOutputFile();
try (FileAppender<Record> appender =
Parquet.write(output)
.schema(NESTED_MULTI_LEAF_SCHEMA)
.createWriterFunc(GenericParquetWriter::create)
.build()) {
appender.add(present);
appender.add(nullNested);
}
// the generic read path has no reader function and materializes the
pruned projection directly
List<org.apache.avro.generic.GenericRecord> rows;
try (CloseableIterable<org.apache.avro.generic.GenericRecord> reader =
Parquet.read(output.toInputFile()).project(NESTED_DEFAULT_ONLY_READ_SCHEMA).build())
{
rows = Lists.newArrayList(reader);
}
assertThat(rows).hasSize(2);
for (org.apache.avro.generic.GenericRecord row : rows) {
org.apache.avro.generic.GenericRecord nested =
(org.apache.avro.generic.GenericRecord) row.get("nested");
if (nested != null) {
assertThat(nested.getSchema().getFields())
.extracting(org.apache.avro.Schema.Field::name)
.as("read records must not expose columns outside the
projection")
.doesNotContain("big", "inner");
}
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
