No, I don’t think so, since iirc hadoop dependencies intentionally were made “provided” to support different Hadoop versions and make it "up to user" to finally decide which version to use.
— Alexey > On 26 Apr 2023, at 05:51, Evan Galpin <egal...@apache.org> wrote: > > The root cause was actually "java.lang.ClassNotFoundException: > org.apache.hadoop.io.Writable" which I eventually fixed by including > hadoop-common as a dep for my pipeline (below). Should hadoop-common be > listed as a dep of ParquetIO the beam repo itself? > > implementation "org.apache.hadoop:hadoop-common:3.2.4" > > On Fri, Apr 21, 2023 at 10:38 AM Evan Galpin <egal...@apache.org > <mailto:egal...@apache.org>> wrote: >> Oops, I was looking at the "bootleg" mvnrepository search engine, which >> shows `compileOnly` in the copy-pastable dependency installation prompts[1]. >> When I received the "ClassNotFound" error, my thought was that the dep >> should be installed in "implementation" mode. When I tried that, I get >> other more strange errors when I try to run my pipeline: >> "java.lang.NoClassDefFoundError: Could not initialize class >> org.apache.beam.sdk.coders.CoderRegistry". >> >> My deps are like so: >> implementation "org.apache.beam:beam-sdks-java-core:${beamVersion}" >> implementation "org.apache.beam:beam-sdks-java-io-parquet:${beamVersion}" >> ... >> >> Not sure why the CoderRegistry error comes up at runtime when both of the >> above deps are included. >> >> [1] >> https://mvnrepository.com/artifact/org.apache.beam/beam-sdks-java-io-parquet/2.46.0 >> >> On Fri, Apr 21, 2023 at 2:34 AM Alexey Romanenko <aromanenko....@gmail.com >> <mailto:aromanenko....@gmail.com>> wrote: >>> Just curious. where it was documented like this? >>> >>> I briefly checked it on Maven Central [1] and the provided code snippet for >>> Gradle uses “implementation” scope. >>> >>> — >>> Alexey >>> >>> [1] >>> https://search.maven.org/artifact/org.apache.beam/beam-sdks-java-io-parquet/2.46.0/jar >>> >>> > On 21 Apr 2023, at 01:52, Evan Galpin <egal...@apache.org >>> > <mailto:egal...@apache.org>> wrote: >>> > >>> > Hi all, >>> > >>> > I'm trying to make use of ParquetIO. Based on what's documented in maven >>> > central, I'm including the artifact in "compileOnly" mode (or in maven >>> > parlance, 'provided' scope). I can successfully compile my pipeline, but >>> > when I run it I (intuitively?) am met with a ClassNotFound exception for >>> > ParquetIO. >>> > >>> > Is 'compileOnly' still the desired way to include ParquetIO as a pipeline >>> > dependency? >>> > >>> > Thanks, >>> > Evan >>>
