The strange thing is that spark examples work fine, but when I include a spark example in my jar and deploy it, I get this error for the very same example:
WARN ClusterScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory My jar is deployed to master and then to workers by spark-ec2/copy-dir. Why would including the example in my jar cause this error? On Thu, Jan 9, 2014 at 12:41 AM, Aureliano Buendia <[email protected]>wrote: > Could someone explain how SPARK_MEM, SPARK_WORKER_MEMORY and > spark.executor.memory should be related so that this non helpful error > doesn't occur? > > Maybe there are more env and java config variable about memory that I'm > missing. > > By the way, that bit of the error asking to check the web UI, it's just > redundant. The UI is of no help. > > > On Wed, Jan 8, 2014 at 4:31 PM, Aureliano Buendia <[email protected]>wrote: > >> Hi, >> >> >> My spark cluster is not able to run a job due to this warning: >> >> WARN ClusterScheduler: Initial job has not accepted any resources; check >> your cluster UI to ensure that workers are registered and have sufficient >> memory >> >> The workers have these status: >> >> ALIVE 2 (0 Used)6.3 GB (0.0 B Used) So there must be plenty of memory >> available despite the warning message. I'm using default spark config, is >> there a config parameter that needs changing for this to work? >> > >
