Hi there, Yes, I restart Zeppelin each time. But the environment panel only shows 8g for the executors, no matter what I define in the settings as that value. As I have OOM errors and am actually trying to set it to *less*, who cares, I guess, although it's annoying that my settings seem to not be "accepted".
However, that means the heap-space (and, sometimes GC) OutOfMemory errors that occur (only) with Zeppelin the moment I try to run SQL queries on my DataFrames remain a mystery to me. I don't get them using the exact same code/data/content with any other Scala/Spark notebook, so its something specific about the Zeppelin default settings that creates the problem. I tried setting the defaults defaults except for executor (using 4g instead), as that works for me with the IBM-Spark Jupyter kernel and the Spark-Notebook: $ grep -v "^#" conf/zeppelin-env.sh | grep -v "^$" export ZEPPELIN_JAVA_OPTS="-Dspark.storage.memoryFraction=0.6 -Dspark.executor.memory=4g -Dspark.driver.memory=1g -Dspark.driver.maxResultSize=1g -Dspark.cores.max=4" export ZEPPELIN_MEM="-Xmx2g" export SPARK_SUBMIT_OPTIONS="--driver-memory 1g --executor-memory 4g" (Note that I'm using Java 8, so MaxPermSize is no longer relevant) I checked both logs for any issues about the config settings, but there are no suspicious messages, either. So I'm at a complete loss why Zeppelin errors out on a simple dataset where all other notebooks work fine (and, my dataset is tiny). Any ideas what defaults in Zeppelin are different from other notebooks that I am not aware of or affecting with these settings? Regards, Florian On Thu, Dec 10, 2015 at 6:30 AM, Fengdong Yu <fengdo...@everstring.com> wrote: > > Did you restart Zeppelin after you export OPTS in zeppelin-env.sh? > > > > > > On Dec 10, 2015, at 7:42 AM, Florian Leitner < > florian.leit...@seleritycorp.com> wrote: > > > > Hi there, I need some help, please. > > > > I'm using Zeppelin 0.5.5 (locally), and am trying to increase my > Executors' Memory sizes. They only get the default 1G according to the web > panel. In the conf/zeppelin-env.sh file, I've configured as follows: > > > > export ZEPPELIN_JAVA_OPTS="-Dspark.executor.memory=10g > -Dspark.driver.memory=5g -Dspark.cores.max=8" > > > > However, if, in a notebook, I then run sc.getConf.toDebugString I only > see the driver memory change to whatever I set here. Neither the spark > cores or the max. number of cores changes with this setting. Also, if I > look at running Executors on the web panel, they are always provisioned > with 1g. So downstream, as soon as I do DataFrame work even on just > modestly sized datasets, I immediately run into java.lang.OutOfMemoryError: > Java heap space errors... > > > > How/where do I then correctly define how much memory Spark Executors > get when running via a local Zeppelin instance? > > > > Regards, > > Florian > >
