No, I don’t think so, since iirc hadoop dependencies intentionally were made
“provided” to support different Hadoop versions and make it "up to user" to
finally decide which version to use.
—
Alexey
> On 26 Apr 2023, at 05:51, Evan Galpin wrote:
>
> The root cause was actually
The root cause was actually "java.lang.ClassNotFoundException:
org.apache.hadoop.io.Writable" which I eventually fixed by including
hadoop-common as a dep for my pipeline (below). Should hadoop-common be
listed as a dep of ParquetIO the beam repo itself?
implementation
Oops, I was looking at the "bootleg" mvnrepository search engine, which
shows `compileOnly` in the copy-pastable dependency installation
prompts[1]. When I received the "ClassNotFound" error, my thought was that
the dep should be installed in "implementation" mode. When I tried that, I
get other
Just curious. where it was documented like this?
I briefly checked it on Maven Central [1] and the provided code snippet for
Gradle uses “implementation” scope.
—
Alexey
[1]
https://search.maven.org/artifact/org.apache.beam/beam-sdks-java-io-parquet/2.46.0/jar
> On 21 Apr 2023, at 01:52,
Hi Evan,
Just to have full knowledge:
- "provided" should be used when You expect the target cluster on
environment to have the package of interest installed so you do not have
to include it in the pipeline jar (this is to have it more lightweight
and easier to maintain coherent target jre
Hi Evan,
Not sure why maven suggests using “compileOnly”.
That’s certainly wrong, make sure to use “implementation” in your case.
Cheers, Moritz
On 21.04.23, 01:52, "Evan Galpin" wrote:
Hi all, I'm trying to make use of ParquetIO. Based on what's documented in
maven central, I'm including
Hi all,
I'm trying to make use of ParquetIO. Based on what's documented in maven
central, I'm including the artifact in "compileOnly" mode (or in maven
parlance, 'provided' scope). I can successfully compile my pipeline, but
when I run it I (intuitively?) am met with a ClassNotFound exception
