fs-alecbg opened a new issue, #11332:
URL: https://github.com/apache/iceberg/issues/11332
### Apache Iceberg version
1.6.0
### Query engine
None
### Please describe the bug 🐞
I was reading from an Iceberg table that had the following schema
```java
new Schema(
List.of(
Types.NestedField.of(1, false, "a1", Types.StringType.get()),
Types.NestedField.of(2, false, "a2", Types.StringType.get()),
Types.NestedField.of(3, false, "a3", Types.StructType.of(
Types.NestedField.of(4, false, "b1",
Types.StringType.get()),
Types.NestedField.of(5, false, "b2",
Types.StringType.get())
)),
Types.NestedField.of(6, false, "a4", Types.StructType.of(
Types.NestedField.of(7, false, "b1",
Types.StructType.of(
Types.NestedField.of(8, false, "c1",
Types.StringType.get()),
Types.NestedField.of(9, false, "c2",
Types.StringType.get())
)),
Types.NestedField.of(10, false, "b2",
Types.StructType.of(
Types.NestedField.of(11, false, "c1",
Types.StringType.get()),
Types.NestedField.of(12, false, "c2",
Types.StringType.get())
))
))
),
Set.of(1)
)
```
and found that while I could successfully do a read that projected only to
the columns `a1`, or `a3.b1`, I was unable to read the column `a1.b1.c1`
without also reading `a1.b1.c2` when using
`org.apache.iceberg.parquet.Parquet.ReadBuilder` without specifying a
`readerFunc`.
I have traced this issue to the implementation of
`org.apache.iceberg.parquet.PruneColumns` which is used to calculated the
schema used in the read. Specifically in the implementation of `struct` at this
[line](https://github.com/apache/iceberg/blob/5e279c868f8e2087f88ae779d8cc8474768bc5e2/parquet/src/main/java/org/apache/iceberg/parquet/PruneColumns.java#L93).
Instead of `filteredFields.add(originalField);`, I claim it should be
`filteredFields.add(java.util.Objects.requireNonNullElse(field,
originalField));`.
The issue can be reproduced simply with
```java
static final Schema ICEBERG_SCHEMA = new Schema(
List.of(
org.apache.iceberg.types.Types.NestedField.of(1, false,
"a1", org.apache.iceberg.types.Types.StringType.get()),
org.apache.iceberg.types.Types.NestedField.of(2, false,
"a2", org.apache.iceberg.types.Types.StringType.get()),
org.apache.iceberg.types.Types.NestedField.of(3, false,
"a3", org.apache.iceberg.types.Types.StructType.of(
org.apache.iceberg.types.Types.NestedField.of(4,
false, "b1", org.apache.iceberg.types.Types.StringType.get()),
org.apache.iceberg.types.Types.NestedField.of(5,
false, "b2", org.apache.iceberg.types.Types.StringType.get())
)),
org.apache.iceberg.types.Types.NestedField.of(6, false,
"a4", org.apache.iceberg.types.Types.StructType.of(
org.apache.iceberg.types.Types.NestedField.of(7,
false, "b1", org.apache.iceberg.types.Types.StructType.of(
org.apache.iceberg.types.Types.NestedField.of(8, false, "c1",
org.apache.iceberg.types.Types.StringType.get()),
org.apache.iceberg.types.Types.NestedField.of(9, false, "c2",
org.apache.iceberg.types.Types.StringType.get())
)),
org.apache.iceberg.types.Types.NestedField.of(10, false, "b2",
org.apache.iceberg.types.Types.StructType.of(
org.apache.iceberg.types.Types.NestedField.of(11, false, "c1",
org.apache.iceberg.types.Types.StringType.get()),
org.apache.iceberg.types.Types.NestedField.of(12, false, "c2",
org.apache.iceberg.types.Types.StringType.get())
))
))
),
Set.of(1)
);
public static void main(String[] args) {
MessageType messageType = ParquetSchemaUtil.convert(ICEBERG_SCHEMA,
"whatever");
Type visit = ParquetTypeVisitor.visit(messageType, new
PruneColumns(Set.of(6, 7, 8)));
System.out.println(visit); // Will include column 9 even though we
are not selecting it
}
```
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [X] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
