Re: AWS exception serialization problem

2017-03-11 Thread Robert Metzger
erializer for Throwables in Kryo, but I’m not sure if we should actually > do this, or just wait for the Kryo fix to be released. > > - Gordon > > > On March 11, 2017 at 9:54:03 AM, Shannon Carey (sca...@expedia.com) wrote: > > Here ya go (see attached). > > > From

Re: AWS exception serialization problem

2017-03-11 Thread Tzu-Li (Gordon) Tai
apache.org" Subject: Re: AWS exception serialization problem Can one of you guys provide us with a minimal example to reproduce the issue? (Ideally locally, not using EMR?) I think once we can reproduce the issue its easy to fix. On Thu, Mar 9, 2017 at 1:24 AM, Bruno Aranda wrote: Hi

Re: AWS exception serialization problem

2017-03-11 Thread Tzu-Li (Gordon) Tai
Here ya go (see attached). From: Robert Metzger Date: Friday, March 10, 2017 at 1:18 PM To: "user@flink.apache.org" Subject: Re: AWS exception serialization problem Can one of you guys provide us with a minimal example to reproduce the issue? (Ideally locally, not using EMR?) I th

Re: AWS exception serialization problem

2017-03-10 Thread Robert Metzger
Can one of you guys provide us with a minimal example to reproduce the issue? (Ideally locally, not using EMR?) I think once we can reproduce the issue its easy to fix. On Thu, Mar 9, 2017 at 1:24 AM, Bruno Aranda wrote: > Hi Stephan, we are running Flink 1.2.0 on Yarn (AWS EMR cluster) > > On W

Re: AWS exception serialization problem

2017-03-08 Thread Bruno Aranda
Hi Stephan, we are running Flink 1.2.0 on Yarn (AWS EMR cluster) On Wed, 8 Mar 2017, 21:41 Stephan Ewen, wrote: > @Bruno: How are you running Flink? On yarn, standalone, mesos, docker? > > On Wed, Mar 8, 2017 at 2:13 PM, Bruno Aranda > wrote: > > Hi, > > We have seen something similar in Flink

Re: AWS exception serialization problem

2017-03-08 Thread Stephan Ewen
@Bruno: How are you running Flink? On yarn, standalone, mesos, docker? On Wed, Mar 8, 2017 at 2:13 PM, Bruno Aranda wrote: > Hi, > > We have seen something similar in Flink 1.2. We have an operation that > parses some JSON, and when it fails to parse it, we can see the > ClassNotFoundException f

Re: AWS exception serialization problem

2017-03-08 Thread Bruno Aranda
Hi, We have seen something similar in Flink 1.2. We have an operation that parses some JSON, and when it fails to parse it, we can see the ClassNotFoundException for the relevant exception (in our case JsResultException from the play-json library). The library is indeed in the shaded JAR, otherwis

Re: AWS exception serialization problem

2017-03-08 Thread Tzu-Li (Gordon) Tai
Hi Shannon, Just to clarify: From the error trace, it seems like that the messages fetched from Kafka are serialized `AmazonS3Exception`s, and you’re emitting a stream of `AmazonS3Exception` as records from FlinkKafkaConsumer? Is this correct? If so, I think we should just make sure that the `

Re: AWS exception serialization problem

2017-03-07 Thread Shannon Carey
> is there some shading logic involved in the dependencies, concerning the AWS > libraries? Not that I am aware of. The AWS code is included in the job's fat jar as-is.

Re: AWS exception serialization problem

2017-03-07 Thread Stephan Ewen
Or somewhere that Kryo is being >> configured? >> >> Thanks, >> Shannon >> >> >> From: Shannon Carey >> Date: Monday, March 6, 2017 at 7:09 PM >> To: "user@flink.apache.org" >> Subject: Re: AWS exception serialization problem >&

Re: AWS exception serialization problem

2017-03-07 Thread Tzu-Li (Gordon) Tai
ate: Monday, March 6, 2017 at 7:09 PM To: "user@flink.apache.org" Subject: Re: AWS exception serialization problem This happened when running Flink with bin/run-local.sh I notice that there only appears to be one Java process. The job manager and the task manager run in the same JVM, right?

Re: AWS exception serialization problem

2017-03-07 Thread Stephan Ewen
figured? > > Thanks, > Shannon > > > From: Shannon Carey > Date: Monday, March 6, 2017 at 7:09 PM > To: "user@flink.apache.org" > Subject: Re: AWS exception serialization problem > > This happened when running Flink with bin/run-local.sh I notice that ther

Re: AWS exception serialization problem

2017-03-06 Thread Shannon Carey
er@flink.apache.org>" mailto:user@flink.apache.org>> Subject: Re: AWS exception serialization problem This happened when running Flink with bin/run-local.sh I notice that there only appears to be one Java process. The job manager and the task manager run in the same JVM, right?

Re: AWS exception serialization problem

2017-03-06 Thread Shannon Carey
FlinkUserCodeClassLoader objects pointing to the two different JARs? From: Shannon Carey mailto:sca...@expedia.com>> Date: Monday, March 6, 2017 at 6:39 PM To: "user@flink.apache.org<mailto:user@flink.apache.org>" mailto:user@flink.apache.org>> Subject: AWS exception seria

AWS exception serialization problem

2017-03-06 Thread Shannon Carey
Has anyone encountered this or know what might be causing it? java.lang.RuntimeException: Could not forward element to next operator at org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:394) at org.apache.flink.streaming.run