Re: Error with Avro/Kyro after upgrade to Flink 1.4

2018-01-24 Thread Edward
Thanks, Stephan. You're correct that there was in fact no shading issue in the official flink-dist_2.11-1.4.0.jar. We are using the jar in the flink docker image, but I mis-spoke when I said ObjectMapper appeared there unshaded. It turned out the issue was really a version conflict in our job's ube

Re: Error with Avro/Kyro after upgrade to Flink 1.4

2018-01-24 Thread Stephan Ewen
I would assume that once you build Flink with correct shading and add all the required Jackson dependencies to your application jar, all works fine. On Jan 24, 2018 10:23, "Stephan Ewen" wrote: > Hi! > > I have a patch coming up for Verify Error > > Concerning the Jackson Error - did you build F

Re: Error with Avro/Kyro after upgrade to Flink 1.4

2018-01-24 Thread Stephan Ewen
Hi! I have a patch coming up for Verify Error Concerning the Jackson Error - did you build Flink yourself? It looks like flink-dist was not built properly, the shading is incorrect. The class "com.fasterxml.jackson.databind.ObjectMapper" should not be in the jar in an unshaded manner. My first g

Re: Error with Avro/Kyro after upgrade to Flink 1.4

2018-01-23 Thread Edward
Thanks for the follow-up Stephan. I have been running this job from a built jar file which was submitted to an existing Flink 1.4 cluster, not from within the IDE. Interestingly, I am now getting the same error when any of the following 3 conditions are true: 1. I run the job on a local cluster fr

Re: Error with Avro/Kyro after upgrade to Flink 1.4

2018-01-23 Thread Stephan Ewen
Hi! Thanks for posting this. Ugly error. This does indeed look like either a dependency conflict, or a mix of incompatible class files. Some suggestions to diagnose this further: - Is this the setup of your POM to build the JAR file for your application, or for running it from within the IDE?

Re: Error with Avro/Kyro after upgrade to Flink 1.4

2018-01-22 Thread Edward
Also, I'm not sure if this would cause the uninitialized error, but I did notice that in the maven dependency tree there are 2 different versions of kyro listed as Flink dependencies: flink-java 1.4 requires kyro 2.24, but flink-streaming-java_2.11 requires kyro 2.21: [INFO] +- org.apache.flink:f