GGraziadei opened a new issue, #18164:
URL: https://github.com/apache/iceberg/issues/18164
### Apache Iceberg version
1.11.0 (latest release) — also reproduced on the 1.12.0 RC0 staging
artifacts and present on `main`
### Query engine
Other (plain `iceberg-core` + `iceberg-parquet` + `iceberg-data`; any engine
that ships commons-lang3 < 3.13 is affected)
### Please describe the bug 🐞
Reading a Parquet file that contains a variant column fails with
```
java.lang.NoSuchMethodError: 'java.util.stream.Stream
org.apache.commons.lang3.stream.Streams.of(java.lang.Iterable)'
at
org.apache.iceberg.parquet.ParquetVariantReaders.children(ParquetVariantReaders.java:488)
at
org.apache.iceberg.parquet.ParquetVariantReaders.children(ParquetVariantReaders.java:481)
at
org.apache.iceberg.parquet.ParquetVariantReaders$ShreddedVariantReader.<init>(ParquetVariantReaders.java:237)
at
org.apache.iceberg.parquet.ParquetVariantReaders.shredded(ParquetVariantReaders.java:78)
at
org.apache.iceberg.parquet.VariantReaderBuilder.value(VariantReaderBuilder.java:140)
```
whenever commons-lang3 older than 3.13.0 is on the classpath.
**Root cause**
`ParquetVariantReaders.children(Iterable)` calls
`org.apache.commons.lang3.stream.Streams.of(Iterable)`, which was added in
commons-lang3 3.13.0:
https://github.com/apache/iceberg/blob/apache-iceberg-1.11.0/parquet/src/main/java/org/apache/iceberg/parquet/ParquetVariantReaders.java#L484-L490
`iceberg-parquet` does not declare commons-lang3 as a dependency (it is not
in `gradle/libs.versions.toml` nor in the `iceberg-parquet` block of
`build.gradle`), so at runtime the class resolves against whatever version an
engine or another transitive dependency happens to provide. In the Iceberg
Gradle build the test classpath resolves to a recent version, so the unit tests
pass, but many consumers get 3.12.0:
- Hadoop 3.3.x and 3.4.x (`hadoop-client-runtime`) ship commons-lang3 3.12.0
- `orc-core` 1.9.x (pulled by `iceberg-data`) depends on commons-lang3 3.12.0
- Spark 3.5 and Flink 1.20 ship commons-lang3 3.12.0
The usage was introduced by #12139 (Parquet variant readers, 1.9.0), so this
affects 1.9.0 through 1.11.0 and the 1.12.0 RC. It is not a regression in
1.12.0.
**Reproducer**
Minimal Maven project with a JUnit test: (attach
`iceberg-variant-lang3-repro.zip`, or paste the test below). The test creates a
v3 table with a variant column, writes one Parquet file with
`GenericParquetWriter` and reads it back with `IcebergGenerics`. The variant
values are null on purpose: the failure happens while building the reader,
before any value is decoded.
| Iceberg | commons-lang3 on classpath | Result |
|---|---|---|
| 1.11.0 | 3.12.0 | `NoSuchMethodError` |
| 1.12.0 RC0 | 3.12.0 | `NoSuchMethodError` |
| 1.11.0 | 3.18.0 | passes |
| 1.12.0 RC0 | 3.18.0 | passes |
```
mvn -q test # fails (commons-lang3 3.12.0)
mvn -q test -Dlang3.version=3.18.0 # passes
```
**Suggested fix**
Drop the commons-lang3 usage in `ParquetVariantReaders.children`; the
bundled Guava already covers it:
```java
private static List<TripleIterator<?>>
children(Iterable<ParquetValueReader<?>> readers) {
return ImmutableList.copyOf(
Iterables.concat(
Iterables.transform(
Iterables.filter(readers, Objects::nonNull),
ParquetValueReader::columns)));
}
```
Alternatively declare commons-lang3 explicitly in `iceberg-parquet`, but
that adds a new runtime dependency and still leaves engines that pin 3.12.0 in
the classpath with a conflict, so removing the usage seems preferable. I can
open a PR for either.
### Willingness to contribute
- [x] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from
the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
