when you start hive on spark do you set any parameters for the submitted job (or read them from init file)?
set spark.master=yarn; set spark.deploy.mode=client; set spark.executor.memory=3g; set spark.driver.memory=3g; set spark.executor.instances=2; set spark.ui.port=7777; Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com *Disclaimer:* Use it at your own risk. Any and all responsibility for any loss, damage or destruction of data or any other property which may arise from relying on this email's technical content is explicitly disclaimed. The author will in no case be liable for any monetary damages arising from such loss, damage or destruction. On 9 September 2016 at 09:30, ?? ? <qiuff...@hotmail.com> wrote: > Hi there, > > > I encountered a problem that makes hive on spark with a very low > performance. > > I'm using spark 1.6.2 and hive 2.1.0, I specified > > > spark.shuffle.service.enabled true > spark.dynamicAllocation.enabled true > > in my spark-default.conf file (the file is in both spark and hive conf > folder) to make spark job to get executors dynamically. > The configuration works correctly when I run spark jobs, but when I use > hive on spark, it only started a few executors although there are more > enough cores and memories to start more executors. > For example, for the same SQL query, if I run on sparkSQL, it can start > more than 20 executors, but with hive on spark, only 3. > > How can I improve the performance on hive on spark? Any suggestions please. > > Thanks, > Minghao Feng > >
