confirmed. spark.executor.memory doesn’t change.
@Moon ? > On Dec 10, 2015, at 4:52 PM, Florian Leitner > <florian.leit...@seleritycorp.com> wrote: > > Hi there, > > Yes, I restart Zeppelin each time. But the environment panel only shows 8g > for the executors, no matter what I define in the settings as that value. As > I have OOM errors and am actually trying to set it to *less*, who cares, I > guess, although it's annoying that my settings seem to not be "accepted". > > However, that means the heap-space (and, sometimes GC) OutOfMemory errors > that occur (only) with Zeppelin the moment I try to run SQL queries on my > DataFrames remain a mystery to me. I don't get them using the exact same > code/data/content with any other Scala/Spark notebook, so its something > specific about the Zeppelin default settings that creates the problem. > > I tried setting the defaults defaults except for executor (using 4g instead), > as that works for me with the IBM-Spark Jupyter kernel and the Spark-Notebook: > > $ grep -v "^#" conf/zeppelin-env.sh | grep -v "^$" > export ZEPPELIN_JAVA_OPTS="-Dspark.storage.memoryFraction=0.6 > -Dspark.executor.memory=4g -Dspark.driver.memory=1g > -Dspark.driver.maxResultSize=1g -Dspark.cores.max=4" > export ZEPPELIN_MEM="-Xmx2g" > export SPARK_SUBMIT_OPTIONS="--driver-memory 1g --executor-memory 4g" > > (Note that I'm using Java 8, so MaxPermSize is no longer relevant) > I checked both logs for any issues about the config settings, but there are > no suspicious messages, either. > So I'm at a complete loss why Zeppelin errors out on a simple dataset where > all other notebooks work fine (and, my dataset is tiny). > Any ideas what defaults in Zeppelin are different from other notebooks that I > am not aware of or affecting with these settings? > > Regards, > Florian > > On Thu, Dec 10, 2015 at 6:30 AM, Fengdong Yu <fengdo...@everstring.com > <mailto:fengdo...@everstring.com>> wrote: > > Did you restart Zeppelin after you export OPTS in zeppelin-env.sh? > > > > > > On Dec 10, 2015, at 7:42 AM, Florian Leitner > > <florian.leit...@seleritycorp.com > > <mailto:florian.leit...@seleritycorp.com>> wrote: > > > > Hi there, I need some help, please. > > > > I'm using Zeppelin 0.5.5 (locally), and am trying to increase my Executors' > > Memory sizes. They only get the default 1G according to the web panel. In > > the conf/zeppelin-env.sh file, I've configured as follows: > > > > export ZEPPELIN_JAVA_OPTS="-Dspark.executor.memory=10g > > -Dspark.driver.memory=5g -Dspark.cores.max=8" > > > > However, if, in a notebook, I then run sc.getConf.toDebugString I only see > > the driver memory change to whatever I set here. Neither the spark cores or > > the max. number of cores changes with this setting. Also, if I look at > > running Executors on the web panel, they are always provisioned with 1g. So > > downstream, as soon as I do DataFrame work even on just modestly sized > > datasets, I immediately run into java.lang.OutOfMemoryError: Java heap > > space errors... > > > > How/where do I then correctly define how much memory Spark Executors get > > when running via a local Zeppelin instance? > > > > Regards, > > Florian > >
