Hi Ajoscha,
That didn't seem to do the trick either. Do the following look correct?
I see 5.9.0 is released with Flink 1.3.2, so I tried that, and got the same
problem, all I did was upload my Scala .jar to the master, updated my
flink-conf.yaml, set my env variables, and ran it with the
Hi Andy,
I remember that I was testing a job with almost exactly the same setup as part
of the Flink 1.3.2 release testing. The command I used to start my job is
roughly this:
HADOOP_CONF_DIR=/etc/hadoop/conf bin/flink run -c my.main.Class -m yarn-cluster
-yn 5 -ys 2 ...
i.e. I export the
Hello,
Bowen: Unless I am missing something, it says there needs to be no setup
on EMR, Each topic says: "You don’t have to configure this manually if you
are running Flink on EMR." S3 access from CLI works fine on my clusters.
Chen: Thank you for this, I will look into this if I am unable to
Hi!
It looks like multiple Hadoop versions are in the classpath. Flink's hadoop
jar and the EMR Hadoop jars.
I would simply drop Flink's own Hadoop dependency and only use the EMR
Hadoop jars.
Delete the 'flink-shaded-hadoop2-uber' jar from Flink, and make sure the
setup is such that the
Attached my side project verified working to deploy jobmanager and
taskmanager as stateless service(non yarn/mesos), configuration here
https://github.com/chenqin/flink-jar/tree/master/config/hadoop
more detail here
Hi Andy,
I believe it's because you didn't set your s3 impl correctly. Try to set
your core-site.xml by following https://ci.apache.org/
projects/flink/flink-docs-release-1.4/ops/deployment/aws.html#s3afilesystem-
recommended
Bowen
On Fri, Oct 6, 2017 at 7:59 AM, Andy M.
Hi Till,
Seems like everything is in line there. hadoop-common.jar ->
hadoop-common-2.7.3-amzn-3.jar
And when i decompiled that jar I see public void addResource(Configuration
conf) in org/apache/hadoop/conf/Configuration.java
I agree that an incorrect version of the jar is probably being
Hi Andy,
could you check which Hadoop version this jar
/usr/lib/hadoop/hadoop-common.jar is? Maybe also checking whether the
contained hadoop Configuration class has the method
Configuration.addResource(Lorg/apache/hadoop/conf/Configuration;)V. Maybe
this jar is the culprit because it comes from
Hi Till,
I believe this is what you are looking for, classpath is much bigger for
the task manager. I can also post the whole log file if needed:
2017-10-05 14:17:53,038 INFO org.apache.flink.yarn.YarnTaskManagerRunner
- Classpath:
Hi Andy,
the CliFrontend is not executed via Yarn, thus, it is not affected by
dependencies which are added due to the underlying Yarn cluster. Therefore,
it would be helpful to look at the TaskManager logs. Either you have
enabled log aggregation on your Yarn cluster, then you can obtain the
Hi Andy,
this looks to me indeed like a dependency problem. I assume that EMR or
something else is pulling in an incompatible version of Hadoop.
The classpath you've posted, is this the one logged in the log files
(TaskManager log) or did you compile it yourself? In the latter case, it
would
Hi Fabian,
1) I have looked at the linked docs, and from what I can tell no setup
should really need to be done to get Flink working(Other than downloading
the correct binaries, which I believe I did)
2) I have downloaded the Flink 1.3.2 binaries(flink-1.3.2-bin-
hadoop27-scala_2.11.tgz
Hi Andy,
I'm not an AWS expert, so I'll just check on some common issues.
I guess you already had a look at the Flink docs for AWS/EMR but I'll post
the link just be to sure [1].
Since you are using Flink 1.3.2 (EMR 5.8.0 comes with Flink 1.3.1) did you
built Flink yourself or did you download
Hi Fabian,
Sorry, I just realized I forgot to include that part. The error returned
is:
java.lang.NoSuchMethodError:
org.apache.hadoop.conf.Configuration.addResource(Lorg/apache/hadoop/conf/Configuration;)V
at
com.amazon.ws.emr.hadoop.fs.EmrFileSystem.initialize(EmrFileSystem.java:93)
Hi Andy,
can you describe in more detail what exactly isn't working?
Do you see error messages in the log files or on the console?
Thanks, Fabian
2017-10-02 15:52 GMT+02:00 Andy M. :
> Hello,
>
> I am about to deploy my first Flink projects to production, but I am
> running
Hello,
I am about to deploy my first Flink projects to production, but I am
running into a very big hurdle. I am unable to launch my project so it can
write to an S3 bucket. My project is running on an EMR cluster, where I
have installed Flink 1.3.2. I am using Yarn to launch the application,
16 matches
Mail list logo