The docs mention spark-ec2 because it is part of the Spark project. There are many, many alternatives to spark-ec2 out there like EMR, but it's probably not the place of the official docs to promote any one of those third-party solutions.
On Fri, Feb 19, 2016 at 11:05 AM James Hammerton <[email protected]> wrote: > Hi, > > Having looked at how easy it is to use EMR, I reckon you may be right, > especially if using Java 8 is no more difficult with that than with > spark-ec2 (where I had to install it on the master and slaves and edit the > spark-env.sh). > > I'm now curious as to why the Spark documentation ( > http://spark.apache.org/docs/latest/index.html) mentions EC2 but not EMR. > > Regards, > > James > > > On 19 February 2016 at 14:25, Daniel Siegmann <[email protected] > > wrote: > >> With EMR supporting Spark, I don't see much reason to use the spark-ec2 >> script unless it is important for you to be able to launch clusters using >> the bleeding edge version of Spark. EMR does seem to do a pretty decent job >> of keeping up to date - the latest version (4.3.0) supports the latest >> Spark version (1.6.0). >> >> So I'd flip the question around and ask: is there any reason to continue >> using the spark-ec2 script rather than EMR? >> >> On Thu, Feb 18, 2016 at 11:39 AM, James Hammerton <[email protected]> wrote: >> >>> I have now... So far I think the issues I've had are not related to >>> this, but I wanted to be sure in case it should be something that needs to >>> be patched. I've had some jobs run successfully but this warning appears in >>> the logs. >>> >>> Regards, >>> >>> James >>> >>> On 18 February 2016 at 12:23, Ted Yu <[email protected]> wrote: >>> >>>> Have you seen this ? >>>> >>>> HADOOP-10988 >>>> >>>> Cheers >>>> >>>> On Thu, Feb 18, 2016 at 3:39 AM, James Hammerton <[email protected]> >>>> wrote: >>>> >>>>> HI, >>>>> >>>>> I am seeing warnings like this in the logs when I run Spark jobs: >>>>> >>>>> OpenJDK 64-Bit Server VM warning: You have loaded library >>>>> /root/ephemeral-hdfs/lib/native/libhadoop.so.1.0.0 which might have >>>>> disabled stack guard. The VM will try to fix the stack guard now. >>>>> It's highly recommended that you fix the library with 'execstack -c >>>>> <libfile>', or link it with '-z noexecstack'. >>>>> >>>>> >>>>> I used spark-ec2 to launch the cluster with the default AMI, Spark >>>>> 1.5.2, hadoop major version 2.4. I altered the jdk to be openjdk 8 as I'd >>>>> written some jobs in Java 8. The 6 workers nodes are m4.2xlarge and master >>>>> is m4.large. >>>>> >>>>> Could this contribute to any problems running the jobs? >>>>> >>>>> Regards, >>>>> >>>>> James >>>>> >>>> >>>> >>> >> >
