[
https://issues.apache.org/jira/browse/HIVE-9342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14275180#comment-14275180
]
Chengxiang Li commented on HIVE-9342:
-------------------------------------
Thanks for verification, [~fangxi.yin]. From you previous finding, instead of
transfer executor cores/instances/memory configuration into spark-submit
command line options, it seems more make sense to make Spark support executor
cores/instances/memory configuration on yarn-cluster mode as well if possible.
If we try to resolve this in Hive, here is a few suggestions about the patch:
# use spark.executor.memory/cores/instances instead of
spark.yarn.executor.memory/cores/instances, as previous configurations already
exists in spark, we'd better keep consistent with them.
# add condition check of spark.master, as we only need to transfer them into
spark-submit command line options on yarn-cluster mode.
# the patch is not well formatted.
> add num-executors / executor-cores / executor-memory option support for hive
> on spark in Yarn mode
> --------------------------------------------------------------------------------------------------
>
> Key: HIVE-9342
> URL: https://issues.apache.org/jira/browse/HIVE-9342
> Project: Hive
> Issue Type: Improvement
> Components: spark-branch
> Affects Versions: spark-branch
> Reporter: Pierre Yin
> Priority: Minor
> Labels: spark
> Fix For: spark-branch
>
> Attachments: HIVE-9342.1-spark.patch
>
>
> When I run hive on spark with Yarn mode, I want to control some yarn option,
> such as --num-executors, --executor-cores, --executor-memory.
> We can append these options into argv in SparkClientImpl.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
