Re: Unable to write snapshots to S3 on EMR

Hi, I can't spot anything obviously wrong with your config. I've been quite busy lately with traveling and release preparations. I will get back to this and try it myself, though, once I have time again. Best, Aljoscha > On 12. Oct 2017, at 17:07, Andy M. wrote: > > Hi Ajoscha, > > That did

Re: Unable to write snapshots to S3 on EMR

Hi Ajoscha, That didn't seem to do the trick either. Do the following look correct? I see 5.9.0 is released with Flink 1.3.2, so I tried that, and got the same problem, all I did was upload my Scala .jar to the master, updated my flink-conf.yaml, set my env variables, and ran it with the followi

Re: Unable to write snapshots to S3 on EMR

Hi Andy, I remember that I was testing a job with almost exactly the same setup as part of the Flink 1.3.2 release testing. The command I used to start my job is roughly this: HADOOP_CONF_DIR=/etc/hadoop/conf bin/flink run -c my.main.Class -m yarn-cluster -yn 5 -ys 2 ... i.e. I export the pro

Re: Unable to write snapshots to S3 on EMR

Hello, Bowen: Unless I am missing something, it says there needs to be no setup on EMR, Each topic says: "You don’t have to configure this manually if you are running Flink on EMR." S3 access from CLI works fine on my clusters. Chen: Thank you for this, I will look into this if I am unable to g

Re: Unable to write snapshots to S3 on EMR

Hi! It looks like multiple Hadoop versions are in the classpath. Flink's hadoop jar and the EMR Hadoop jars. I would simply drop Flink's own Hadoop dependency and only use the EMR Hadoop jars. Delete the 'flink-shaded-h‌​adoop2-uber' jar from Flink, and make sure the setup is such that the Hadoop

Re: Unable to write snapshots to S3 on EMR

Attached my side project verified working to deploy jobmanager and taskmanager as stateless service(non yarn/mesos), configuration here https://github.com/chenqin/flink-jar/tree/master/config/hadoop more detail here https://github.com/chenqin/flink-jar/blob/master/src/main/java/FlinkBootstrap.jav

Re: Unable to write snapshots to S3 on EMR

Hi Andy, I believe it's because you didn't set your s3 impl correctly. Try to set your core-site.xml by following https://ci.apache.org/ projects/flink/flink-docs-release-1.4/ops/deployment/aws.html#s3afilesystem- recommended Bowen On Fri, Oct 6, 2017 at 7:59 AM, Andy M. wrote: > Hi Till, > >

Re: Unable to write snapshots to S3 on EMR

Hi Till, Seems like everything is in line there. hadoop-common.jar -> hadoop-common-2.7.3-amzn-3.jar And when i decompiled that jar I see public void addResource(Configuration conf) in org/apache/hadoop/conf/Configuration.java I agree that an incorrect version of the jar is probably being run,

Re: Unable to write snapshots to S3 on EMR

Hi Andy, could you check which Hadoop version this jar /usr/lib/hadoop/hadoop-common.jar is? Maybe also checking whether the contained hadoop Configuration class has the method Configuration.addResource(Lorg/apache/hadoop/conf/Configuration;)V. Maybe this jar is the culprit because it comes from a

Re: Unable to write snapshots to S3 on EMR

Hi Till, I believe this is what you are looking for, classpath is much bigger for the task manager. I can also post the whole log file if needed: 2017-10-05 14:17:53,038 INFO org.apache.flink.yarn.YarnTaskManagerRunner - Classpath: flink-consumer.jar:lib/flink-dist_2.11-1.3.2.

Re: Unable to write snapshots to S3 on EMR

Hi Andy, the CliFrontend is not executed via Yarn, thus, it is not affected by dependencies which are added due to the underlying Yarn cluster. Therefore, it would be helpful to look at the TaskManager logs. Either you have enabled log aggregation on your Yarn cluster, then you can obtain the logs

Re: Unable to write snapshots to S3 on EMR

Hi Till, That is actually the classpath used by the flink bash script(that launches the jar using the java command). I changed the execute to an echo, and grabbed that for the CLI arguments. I believe this is the class path from the log file(although it might not be the taskmanager log, is that

Re: Unable to write snapshots to S3 on EMR

Hi Andy, this looks to me indeed like a dependency problem. I assume that EMR or something else is pulling in an incompatible version of Hadoop. The classpath you've posted, is this the one logged in the log files (TaskManager log) or did you compile it yourself? In the latter case, it would also

Re: Unable to write snapshots to S3 on EMR

Hi Fabian, 1) I have looked at the linked docs, and from what I can tell no setup should really need to be done to get Flink working(Other than downloading the correct binaries, which I believe I did) 2) I have downloaded the Flink 1.3.2 binaries(flink-1.3.2-bin- hadoop27-scala_2.11.tgz

Re: Unable to write snapshots to S3 on EMR

Hi Andy, I'm not an AWS expert, so I'll just check on some common issues. I guess you already had a look at the Flink docs for AWS/EMR but I'll post the link just be to sure [1]. Since you are using Flink 1.3.2 (EMR 5.8.0 comes with Flink 1.3.1) did you built Flink yourself or did you download t

Re: Unable to write snapshots to S3 on EMR

Hi Fabian, Sorry, I just realized I forgot to include that part. The error returned is: java.lang.NoSuchMethodError: org.apache.hadoop.conf.Configuration.addResource(Lorg/apache/hadoop/conf/Configuration;)V at com.amazon.ws.emr.hadoop.fs.EmrFileSystem.initialize(EmrFileSystem.java:93) a

Re: Unable to write snapshots to S3 on EMR

Hi Andy, can you describe in more detail what exactly isn't working? Do you see error messages in the log files or on the console? Thanks, Fabian 2017-10-02 15:52 GMT+02:00 Andy M. : > Hello, > > I am about to deploy my first Flink projects to production, but I am > running into a very big hur

Unable to write snapshots to S3 on EMR

Hello, I am about to deploy my first Flink projects to production, but I am running into a very big hurdle. I am unable to launch my project so it can write to an S3 bucket. My project is running on an EMR cluster, where I have installed Flink 1.3.2. I am using Yarn to launch the application,