Re: Re: how to tuning spark shuffle

2016-07-18 Thread lizhenm...@163.com
Hi, Can you print out the environment tab on your UI. By default spark-sql runs on local mode which means that you only have one driver and one executor in one jvm. Do you increase the executor memory through SET spark.executor.memory=xG And adjust it and run the SQL again. HTH Dr Mich

Re: how to tuning spark shuffle

2016-07-18 Thread Mich Talebzadeh
Hi, Can you print out the environment tab on your UI. By default spark-sql runs on local mode which means that you only have one driver and one executor in one jvm. Do you increase the executor memory through SET spark.executor.memory=xG And adjust it and run the SQL again. HTH Dr Mich