On Thu, Jan 9, 2014 at 5:01 AM, Matei Zaharia <[email protected]>wrote:
> Just follow the docs at > http://spark.incubator.apache.org/docs/latest/quick-start.html#a-standalone-app-in-scalafor > how to run an application. Spark is designed so that you can simply run > your application *without* any scripts whatsoever, and submit your JAR to > the SparkContext constructor, which will distribute it. You can launch your > application with “scala”, “java”, or whatever tool you’d prefer. > I'm afraid what you said about 'simply run your application *without* any scripts whatsoever' does not apply to spark at the moment, and it simply does not work. Try the simple Pi calculation this on a standard spark-ec2 instance: java -cp /root/spark/examples/target/spark-examples_2.9.3-0.8.1-incubating.jar:/root/spark/assembltarget/scala-2.9.3/spark-assembly_2.9.3-0.8.1-incubating-hadoop1.0.4.jar org.apache.spark.examples.SparkPi `cat spark-ec2/cluster-url` And you'll get the error: WARN cluster.ClusterScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory While the script way works: spark/run-example org.apache.spark.examples.SparkPi `cat spark-ec2/cluster-url` What am I missing in the above java command? > > Matei > > On Jan 8, 2014, at 8:26 PM, Aureliano Buendia <[email protected]> > wrote: > > > > > On Thu, Jan 9, 2014 at 4:11 AM, Matei Zaharia <[email protected]>wrote: > >> Oh, you shouldn’t use spark-class for your own classes. Just build your >> job separately and submit it by running it with “java” and creating a >> SparkContext in it. spark-class is designed to run classes internal to the >> Spark project. >> > > Really? Apparently Eugen runs his jobs by: > > $SPARK_HOME/spark-class SPARK_CLASSPATH=PathToYour.jar com.myproject.MyJob > > , as he instructed me > here<http://mail-archives.apache.org/mod_mbox/spark-user/201401.mbox/browser>to > do this. > > I have to say while spark documentation is not sparse, it does not address > enough, and as you can see the community is confused. > > Are the spark users supposed to create something like run-example for > their own jobs? > > >> >> Matei >> >> On Jan 8, 2014, at 8:06 PM, Aureliano Buendia <[email protected]> >> wrote: >> >> >> >> >> On Thu, Jan 9, 2014 at 3:59 AM, Matei Zaharia <[email protected]>wrote: >> >>> Have you looked at the cluster UI, and do you see any workers registered >>> there, and your application under running applications? Maybe you typed in >>> the wrong master URL or something like that. >>> >> >> No, it's automated: cat spark-ec2/cluster-url >> >> I think the problem might be caused by spark-class script. It seems to >> assign too much memory. >> >> I forgot the fact that run-example doesn't use spark-class. >> >> >>> >>> Matei >>> >>> On Jan 8, 2014, at 7:07 PM, Aureliano Buendia <[email protected]> >>> wrote: >>> >>> The strange thing is that spark examples work fine, but when I include a >>> spark example in my jar and deploy it, I get this error for the very same >>> example: >>> >>> WARN ClusterScheduler: Initial job has not accepted any resources; check >>> your cluster UI to ensure that workers are registered and have sufficient >>> memory >>> >>> My jar is deployed to master and then to workers by spark-ec2/copy-dir. >>> Why would including the example in my jar cause this error? >>> >>> >>> >>> On Thu, Jan 9, 2014 at 12:41 AM, Aureliano Buendia <[email protected] >>> > wrote: >>> >>>> Could someone explain how SPARK_MEM, SPARK_WORKER_MEMORY and >>>> spark.executor.memory should be related so that this non helpful error >>>> doesn't occur? >>>> >>>> Maybe there are more env and java config variable about memory that I'm >>>> missing. >>>> >>>> By the way, that bit of the error asking to check the web UI, it's just >>>> redundant. The UI is of no help. >>>> >>>> >>>> On Wed, Jan 8, 2014 at 4:31 PM, Aureliano Buendia <[email protected] >>>> > wrote: >>>> >>>>> Hi, >>>>> >>>>> >>>>> My spark cluster is not able to run a job due to this warning: >>>>> >>>>> WARN ClusterScheduler: Initial job has not accepted any resources; >>>>> check your cluster UI to ensure that workers are registered and have >>>>> sufficient memory >>>>> >>>>> The workers have these status: >>>>> >>>>> ALIVE 2 (0 Used)6.3 GB (0.0 B Used) So there must be plenty of memory >>>>> available despite the warning message. I'm using default spark config, is >>>>> there a config parameter that needs changing for this to work? >>>>> >>>> >>>> >>> >>> >> >> > >
