If you are running spark in local mode, executor parameters are not used as there is no executor. You should try to set corresponding driver parameter to effect it.
On Mon, Jan 19, 2015, 00:21 Sean Owen <[email protected]> wrote: > OK. Are you sure the executor has the memory you think? "-Xmx24g" in > its command line? It may be that for some reason your job is reserving > an exceptionally large amount of non-heap memory. I am not sure that's > to be expected with the ALS job though. Even if the settings work, > considering using the explicit command line configuration. > > On Sat, Jan 17, 2015 at 12:49 PM, Antony Mayi <[email protected]> > wrote: > > the values are for sure applied as expected - confirmed using the spark > UI > > environment page... > > > > it comes from my defaults configured using > > 'spark.yarn.executor.memoryOverhead=8192' (yes, now increased even > more) in > > /etc/spark/conf/spark-defaults.conf and 'export > SPARK_EXECUTOR_MEMORY=24G' > > in /etc/spark/conf/spark-env.sh > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
