Short-term I'd try relocating the okio/okhttp dependencies in your jar.
I'm not too keen on adding more relocations to the hadoop jar; I can't
gauge the possible side-effects.
On 27.02.2019 14:54, Austin Cawley-Edwards wrote:
Following up to add more info, I am building my app with maven based
on the sample Flink pom.xml
My shade plugin config is:
<groupId>org.apache.maven.plugins</groupId> <artifactId>maven-shade-plugin</artifactId> <version>3.0.0</version> <executions> <!-- Run shade goal on package phase -->
<execution> <phase>package</phase> <goals> <goal>shade</goal> </goals> <configuration> <artifactSet> <excludes> <exclude>org.apache.flink:force-shading</exclude> <exclude>com.google.code.findbugs:jsr305</exclude> <exclude>org.slf4j:*</exclude> <exclude>log4j:*</exclude> </excludes> </artifactSet> <filters> <filter> <!-- Do not
copy the signatures in the META-INF folder. Otherwise, this might
cause SecurityExceptions when using the JAR. --> <artifact>*:*</artifact> <excludes> <exclude>META-INF/*.SF</exclude> <exclude>META-INF/*.DSA</exclude> <exclude>META-INF/*.RSA</exclude> </excludes> </filter> </filters> <transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>${main.class}</mainClass> </transformer> </transformers> </configuration>
</execution> </executions>
Thanks again and let me know if there is more information I can provide.
Austin
On Tue, Feb 26, 2019 at 10:59 PM Austin Cawley-Edwards
<austin.caw...@gmail.com <mailto:austin.caw...@gmail.com>> wrote:
Hi,
I recently experienced versioning clashes with the okio and
okhttp when trying to deploy a Flink 1.6.0 app to AWS EMR on
Hadoop 2.8.4. After investigating and talking to the okio team
(see this issue) <https://github.com/square/okio/issues/559>, I
found that both okio and okhttp exist in the Flink uber jar with
versions 1.4.0 and 2.4.0, respectively, whereas I'm including
versions 2.2.2 and 3.13.1 in my shaded jar. The okio team
suggested that Flink should shade the uber jar to fix the issue,
but I'm wondering if there is something I can do on my end to have
all versions exist simultaneously.
From the issue, here are the okio contents of the uber jar:
*jar -tf flink-shaded-hadoop2-uber-1.6.0.jar | grep okio*
META-INF/maven/com.squareup.okio/
META-INF/maven/com.squareup.okio/okio/
META-INF/maven/com.squareup.okio/okio/pom.properties
META-INF/maven/com.squareup.okio/okio/pom.xml
okio/
okio/AsyncTimeout$1.class
okio/AsyncTimeout$2.class
okio/AsyncTimeout$Watchdog.class
okio/AsyncTimeout.class
okio/Base64.class
okio/Buffer$1.class
okio/Buffer$2.class
okio/Buffer.class
okio/BufferedSink.class
okio/BufferedSource.class
okio/ByteString.class
okio/DeflaterSink.class
okio/ForwardingSink.class
okio/ForwardingSource.class
okio/ForwardingTimeout.class
okio/GzipSink.class
okio/GzipSource.class
okio/InflaterSource.class
okio/Okio$1.class
okio/Okio$2.class
okio/Okio$3.class
okio/Okio.class
okio/RealBufferedSink$1.class
okio/RealBufferedSink.class
okio/RealBufferedSource$1.class
okio/RealBufferedSource.class
okio/Segment.class
okio/SegmentPool.class
okio/SegmentedByteString.class
okio/Sink.class
okio/Source.class
okio/Timeout$1.class
okio/Timeout.class
okio/Util.class
Thank you,
Austin Cawley-Edwards
