Hi, Having looked at how easy it is to use EMR, I reckon you may be right, especially if using Java 8 is no more difficult with that than with spark-ec2 (where I had to install it on the master and slaves and edit the spark-env.sh).
I'm now curious as to why the Spark documentation ( http://spark.apache.org/docs/latest/index.html) mentions EC2 but not EMR. Regards, James On 19 February 2016 at 14:25, Daniel Siegmann <daniel.siegm...@teamaol.com> wrote: > With EMR supporting Spark, I don't see much reason to use the spark-ec2 > script unless it is important for you to be able to launch clusters using > the bleeding edge version of Spark. EMR does seem to do a pretty decent job > of keeping up to date - the latest version (4.3.0) supports the latest > Spark version (1.6.0). > > So I'd flip the question around and ask: is there any reason to continue > using the spark-ec2 script rather than EMR? > > On Thu, Feb 18, 2016 at 11:39 AM, James Hammerton <ja...@gluru.co> wrote: > >> I have now... So far I think the issues I've had are not related to >> this, but I wanted to be sure in case it should be something that needs to >> be patched. I've had some jobs run successfully but this warning appears in >> the logs. >> >> Regards, >> >> James >> >> On 18 February 2016 at 12:23, Ted Yu <yuzhih...@gmail.com> wrote: >> >>> Have you seen this ? >>> >>> HADOOP-10988 >>> >>> Cheers >>> >>> On Thu, Feb 18, 2016 at 3:39 AM, James Hammerton <ja...@gluru.co> wrote: >>> >>>> HI, >>>> >>>> I am seeing warnings like this in the logs when I run Spark jobs: >>>> >>>> OpenJDK 64-Bit Server VM warning: You have loaded library >>>> /root/ephemeral-hdfs/lib/native/libhadoop.so.1.0.0 which might have >>>> disabled stack guard. The VM will try to fix the stack guard now. >>>> It's highly recommended that you fix the library with 'execstack -c >>>> <libfile>', or link it with '-z noexecstack'. >>>> >>>> >>>> I used spark-ec2 to launch the cluster with the default AMI, Spark >>>> 1.5.2, hadoop major version 2.4. I altered the jdk to be openjdk 8 as I'd >>>> written some jobs in Java 8. The 6 workers nodes are m4.2xlarge and master >>>> is m4.large. >>>> >>>> Could this contribute to any problems running the jobs? >>>> >>>> Regards, >>>> >>>> James >>>> >>> >>> >> >
