[
https://issues.apache.org/jira/browse/PARQUET-2035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17333072#comment-17333072
]
Gabor Szadovszky commented on PARQUET-2035:
-------------------------------------------
bq. Use unique names for shaded packages, e.g.
shaded.parquet.avro.it.unimi.dsi.fastutil,
shaded.parquet.column.it.unimi.dsi.fastutil. Probably not an option, as it may
break application logic due to presence of multiple instances of the same
classes.
I think it should work however, it would not be a nice solution. The classes
wouldn't be the same because their package would be different.
bq. Get rid of dependency on fastutil in any two out of three modules that are
currently shading it. That may require significant code refactoring and may be
not feasible. At least in parquet-hadoop it seems to be used in a single place
for some performance optimisation.
As you've said we require fastutil for performance. Dropping fastutil from any
place we use it would result in performance drawback.
bq. Achieve consistency of fastutil across hadoop projects. Would be ideal, but
probably even less feasible solution.
I would agree this is not possible. There are a lot of components in the
ecosystem and much smaller efforts on similar issues were died already.
bq. Create a new module, e.g. parquet-fastutil that would contain only the
shaded library. Add this module to transitive non shaded dependencies that have
dependency on fastutil: parquet-avro, parquet-column, parquet-hadoop.
I think this is our best option. We already have a separate module for jackson
for the same purpose. What do you think about instead of creating a new module
for fastutil we would rename the existing parquet-jackson module to
parquet-3rdparty (or whatever better name) and it would contain all our
dependencies that we would like to shade? The only issue with this approach is
that we cannot minimize the jar during shading and fastutil is quite big (19M).
> Java module import error due to shaded package
> shaded.parquet.it.unimi.dsi.fastutil
> -----------------------------------------------------------------------------------
>
> Key: PARQUET-2035
> URL: https://issues.apache.org/jira/browse/PARQUET-2035
> Project: Parquet
> Issue Type: Bug
> Components: parquet-avro
> Affects Versions: 1.12.0, 1.11.1
> Reporter: Maxim Kolesnikov
> Priority: Major
> Attachments: parquet-example.zip
>
>
> *Description:*
> Due to collision of shaded packages
> {code:java}
> shaded.parquet.it.unimi.dsi.fastutil{code}
> in
> {code:java}
> org.apache.parquet:parquet-avro{code}
> and
> {code:java}
> org.apache.parquet:parquet-column{code}
> it is not possible to use both these dependencies within a modularized java
> project at the same time.
>
>
> *How to reproduce:*
> * create a maven project with dependency
> org.apache.parquet:parquet-avro:1.11.1
> * declare java module that requires both parquet.avro and parquet.column
> * run
> {code:java}
> mvn compile{code}
>
> *Expected behaviour:*
> Project should compile without errors.
>
> *Actual behaviour:*
> Project fails with compilation errors:
>
> {code:java}
> [ERROR] the unnamed module reads package shaded.parquet.it.unimi.dsi.fastutil
> from both parquet.column and parquet.avro
> ...{code}
>
>
> *Reproducible example* (same code as in the attached zip file):
> https://github.com/xCASx/parquet-example
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
