Re: JAVA heap space issue

I have lot of joint SQL operations, which is blocking me write data and unresisted the data, if not useful. On Oct 24, 2016 7:50 PM, "Mich Talebzadeh" wrote: > OK so you are disabling broadcasting although it is not obvious how this > helps in this case! > > Dr Mich

Re: JAVA heap space issue

OK so you are disabling broadcasting although it is not obvious how this helps in this case! Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw *

Re: JAVA heap space issue

sc <- sparkR.session(master = "spark://ip-172-31-6-116:7077",sparkConfig=list(spark.executor.memory="10g", spark.app.name="Testing",spark.driver.memory="14g",spark.executor.extraJavaOption="-Xms2g -Xmx5g -XX:-UseGCOverheadLimit",spark.driver.extraJavaOption="-Xms2g -Xmx5g

Re: JAVA heap space issue

OK so what is your full launch code now? I mean equivalent to spark-submit Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw *

Re: JAVA heap space issue

Hi Mich, I am able to write the files to storage after adding extra parameter. FYI.. This one I used. spark.sql.autoBroadcastJoinThreshold="-1" On Mon, Oct 24, 2016 at 7:22 PM, Mich Talebzadeh wrote: > Rather strange as you have plenty free memory there. > >

Re: JAVA heap space issue

Rather strange as you have plenty free memory there. Try reducing driver memory to 2GB and executer memory to 2GB and run it again ${SPARK_HOME}/bin/spark-submit \ --driver-memory 2G \ --num-executors 2 \ --executor-cores 1 \

Re: JAVA heap space issue

Hi Mich, Yes, I am using standalone mode cluster, We have two executors with 10G memory each. We have two workers. FYI.. On Mon, Oct 24, 2016 at 5:22 PM, Mich Talebzadeh wrote: > Sounds like you are running in standalone mode. > > Have you checked the UI on port

Re: JAVA heap space issue

Sounds like you are running in standalone mode. Have you checked the UI on port 4040 (default) to see where memory is going. Why do you need executor memory of 10GB? How many executors are running and plus how many slaves started? In standalone mode executors run on workers (UI 8080) HTH Dr