Hi Nathan, On Thu, May 21, 2015 at 7:30 PM, Nathan Kronenfeld < nkronenfeld@uncharted.software> wrote:
> > >> In researching and discussing these issues with Cloudera and others, >> we've been told that only one mechanism is supported for starting Spark >> jobs: the *spark-submit* scripts. >> > > Is this new? We've been submitting jobs directly from a programatically > created spark context (instead of through spark-submit) from the beginning > (from 0.7.x to 1.2) - to a local cluster. > > In moving to 1.3 on Yarn cluster recently, we've had no end of problems > trying to switch this over (though I think we're almost there). > > Instantiating SparkContext directly works. Well, sorta: it has limitations. For example, see discussions about Spark not really liking multiple contexts in the same JVM. It also does not work in "cluster" deploy mode. -- Marcelo
