[jira] [Commented] (HIVE-9342) add num-executors / executor-cores / executor-memory option support for hive on spark in Yarn mode [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14285061#comment-14285061 ] Chengxiang Li commented on HIVE-9342: - Wow! this is the first time i see all checks pass from Hive QA, what an exciting message! add num-executors / executor-cores / executor-memory option support for hive on spark in Yarn mode [Spark Branch] - Key: HIVE-9342 URL: https://issues.apache.org/jira/browse/HIVE-9342 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Pierre Yin Priority: Minor Labels: spark Fix For: spark-branch Attachments: HIVE-9342.1-spark.patch, HIVE-9342.2-spark.patch, HIVE-9342.3-spark.patch When I run hive on spark with Yarn mode, I want to control some yarn option, such as --num-executors, --executor-cores, --executor-memory. We can append these options into argv in SparkClientImpl. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9342) add num-executors / executor-cores / executor-memory option support for hive on spark in Yarn mode [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14283478#comment-14283478
]
Hive QA commented on HIVE-9342:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12693212/HIVE-9342.3-spark.patch
{color:green}SUCCESS:{color} +1 7308 tests passed
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/654/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/654/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-654/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12693212 - PreCommit-HIVE-SPARK-Build
add num-executors / executor-cores / executor-memory option support for hive
on spark in Yarn mode [Spark Branch]
-
Key: HIVE-9342
URL: https://issues.apache.org/jira/browse/HIVE-9342
Project: Hive
Issue Type: Sub-task
Components: Spark
Affects Versions: spark-branch
Reporter: Pierre Yin
Priority: Minor
Labels: spark
Fix For: spark-branch
Attachments: HIVE-9342.1-spark.patch, HIVE-9342.2-spark.patch,
HIVE-9342.3-spark.patch
When I run hive on spark with Yarn mode, I want to control some yarn option,
such as --num-executors, --executor-cores, --executor-memory.
We can append these options into argv in SparkClientImpl.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9342) add num-executors / executor-cores / executor-memory option support for hive on spark in Yarn mode [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14283425#comment-14283425 ] Xuefu Zhang commented on HIVE-9342: --- Sure. Sorry for missing this one. add num-executors / executor-cores / executor-memory option support for hive on spark in Yarn mode [Spark Branch] - Key: HIVE-9342 URL: https://issues.apache.org/jira/browse/HIVE-9342 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Pierre Yin Priority: Minor Labels: spark Fix For: spark-branch Attachments: HIVE-9342.1-spark.patch, HIVE-9342.2-spark.patch, HIVE-9342.3-spark.patch When I run hive on spark with Yarn mode, I want to control some yarn option, such as --num-executors, --executor-cores, --executor-memory. We can append these options into argv in SparkClientImpl. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9342) add num-executors / executor-cores / executor-memory option support for hive on spark in Yarn mode
[ https://issues.apache.org/jira/browse/HIVE-9342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14277900#comment-14277900 ] Xuefu Zhang commented on HIVE-9342: --- +1 add num-executors / executor-cores / executor-memory option support for hive on spark in Yarn mode -- Key: HIVE-9342 URL: https://issues.apache.org/jira/browse/HIVE-9342 Project: Hive Issue Type: Improvement Components: spark-branch Affects Versions: spark-branch Reporter: Pierre Yin Priority: Minor Labels: spark Fix For: spark-branch Attachments: HIVE-9342.1-spark.patch, HIVE-9342.2-spark.patch When I run hive on spark with Yarn mode, I want to control some yarn option, such as --num-executors, --executor-cores, --executor-memory. We can append these options into argv in SparkClientImpl. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9342) add num-executors / executor-cores / executor-memory option support for hive on spark in Yarn mode
[ https://issues.apache.org/jira/browse/HIVE-9342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14276643#comment-14276643 ] Pierre Yin commented on HIVE-9342: -- [~Chengxiang Li], Thanks for your suggestions. I append another patch with your suggestions. add num-executors / executor-cores / executor-memory option support for hive on spark in Yarn mode -- Key: HIVE-9342 URL: https://issues.apache.org/jira/browse/HIVE-9342 Project: Hive Issue Type: Improvement Components: spark-branch Affects Versions: spark-branch Reporter: Pierre Yin Priority: Minor Labels: spark Fix For: spark-branch Attachments: HIVE-9342.1-spark.patch, HIVE-9342.2-spark.patch When I run hive on spark with Yarn mode, I want to control some yarn option, such as --num-executors, --executor-cores, --executor-memory. We can append these options into argv in SparkClientImpl. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9342) add num-executors / executor-cores / executor-memory option support for hive on spark in Yarn mode
[ https://issues.apache.org/jira/browse/HIVE-9342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14276646#comment-14276646 ] Chengxiang Li commented on HIVE-9342: - Thanks, [~fangxi.yin]. The patch looks good to me. [~xuefuz], you may take a look at this either. add num-executors / executor-cores / executor-memory option support for hive on spark in Yarn mode -- Key: HIVE-9342 URL: https://issues.apache.org/jira/browse/HIVE-9342 Project: Hive Issue Type: Improvement Components: spark-branch Affects Versions: spark-branch Reporter: Pierre Yin Priority: Minor Labels: spark Fix For: spark-branch Attachments: HIVE-9342.1-spark.patch, HIVE-9342.2-spark.patch When I run hive on spark with Yarn mode, I want to control some yarn option, such as --num-executors, --executor-cores, --executor-memory. We can append these options into argv in SparkClientImpl. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9342) add num-executors / executor-cores / executor-memory option support for hive on spark in Yarn mode
[
https://issues.apache.org/jira/browse/HIVE-9342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14276730#comment-14276730
]
Hive QA commented on HIVE-9342:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12692178/HIVE-9342.2-spark.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 7307 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_mapjoin_memcheck
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_optimize_nullscan
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/641/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/641/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-641/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12692178 - PreCommit-HIVE-SPARK-Build
add num-executors / executor-cores / executor-memory option support for hive
on spark in Yarn mode
--
Key: HIVE-9342
URL: https://issues.apache.org/jira/browse/HIVE-9342
Project: Hive
Issue Type: Improvement
Components: spark-branch
Affects Versions: spark-branch
Reporter: Pierre Yin
Priority: Minor
Labels: spark
Fix For: spark-branch
Attachments: HIVE-9342.1-spark.patch, HIVE-9342.2-spark.patch
When I run hive on spark with Yarn mode, I want to control some yarn option,
such as --num-executors, --executor-cores, --executor-memory.
We can append these options into argv in SparkClientImpl.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9342) add num-executors / executor-cores / executor-memory option support for hive on spark in Yarn mode
[ https://issues.apache.org/jira/browse/HIVE-9342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14274947#comment-14274947 ] Pierre Yin commented on HIVE-9342: -- Thank you for your information. I will try it. add num-executors / executor-cores / executor-memory option support for hive on spark in Yarn mode -- Key: HIVE-9342 URL: https://issues.apache.org/jira/browse/HIVE-9342 Project: Hive Issue Type: Improvement Components: spark-branch Affects Versions: spark-branch Reporter: Pierre Yin Priority: Minor Labels: spark Fix For: spark-branch Attachments: HIVE-9342.1-spark.patch When I run hive on spark with Yarn mode, I want to control some yarn option, such as --num-executors, --executor-cores, --executor-memory. We can append these options into argv in SparkClientImpl. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9342) add num-executors / executor-cores / executor-memory option support for hive on spark in Yarn mode
[ https://issues.apache.org/jira/browse/HIVE-9342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14274948#comment-14274948 ] Pierre Yin commented on HIVE-9342: -- Thank you for your information. I will try it. add num-executors / executor-cores / executor-memory option support for hive on spark in Yarn mode -- Key: HIVE-9342 URL: https://issues.apache.org/jira/browse/HIVE-9342 Project: Hive Issue Type: Improvement Components: spark-branch Affects Versions: spark-branch Reporter: Pierre Yin Priority: Minor Labels: spark Fix For: spark-branch Attachments: HIVE-9342.1-spark.patch When I run hive on spark with Yarn mode, I want to control some yarn option, such as --num-executors, --executor-cores, --executor-memory. We can append these options into argv in SparkClientImpl. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9342) add num-executors / executor-cores / executor-memory option support for hive on spark in Yarn mode
[ https://issues.apache.org/jira/browse/HIVE-9342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14274972#comment-14274972 ] Pierre Yin commented on HIVE-9342: -- I have verified with the new options ( spark.executor.instances / cores / memory) in my cluster by command line . In yarn-client mode, it worked as what I expected. But in yarn-cluster mode, these options doesn't work. I will try to find it why yarn-cluster can't support these options. add num-executors / executor-cores / executor-memory option support for hive on spark in Yarn mode -- Key: HIVE-9342 URL: https://issues.apache.org/jira/browse/HIVE-9342 Project: Hive Issue Type: Improvement Components: spark-branch Affects Versions: spark-branch Reporter: Pierre Yin Priority: Minor Labels: spark Fix For: spark-branch Attachments: HIVE-9342.1-spark.patch When I run hive on spark with Yarn mode, I want to control some yarn option, such as --num-executors, --executor-cores, --executor-memory. We can append these options into argv in SparkClientImpl. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9342) add num-executors / executor-cores / executor-memory option support for hive on spark in Yarn mode
[ https://issues.apache.org/jira/browse/HIVE-9342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14274984#comment-14274984 ] Pierre Yin commented on HIVE-9342: -- Another finding is: If I use the options together with my patch (spark.yarn.executor.memory / cores / memory), yarn-cluster mode also can work as what I expected. spark.executor.instances is one properties in spark. It seems to be one issue in spark. It seems that SparkSubmit doesn't pick up spark.executor.instances but it can pick up --num-executors option. add num-executors / executor-cores / executor-memory option support for hive on spark in Yarn mode -- Key: HIVE-9342 URL: https://issues.apache.org/jira/browse/HIVE-9342 Project: Hive Issue Type: Improvement Components: spark-branch Affects Versions: spark-branch Reporter: Pierre Yin Priority: Minor Labels: spark Fix For: spark-branch Attachments: HIVE-9342.1-spark.patch When I run hive on spark with Yarn mode, I want to control some yarn option, such as --num-executors, --executor-cores, --executor-memory. We can append these options into argv in SparkClientImpl. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9342) add num-executors / executor-cores / executor-memory option support for hive on spark in Yarn mode
[ https://issues.apache.org/jira/browse/HIVE-9342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14275180#comment-14275180 ] Chengxiang Li commented on HIVE-9342: - Thanks for verification, [~fangxi.yin]. From you previous finding, instead of transfer executor cores/instances/memory configuration into spark-submit command line options, it seems more make sense to make Spark support executor cores/instances/memory configuration on yarn-cluster mode as well if possible. If we try to resolve this in Hive, here is a few suggestions about the patch: # use spark.executor.memory/cores/instances instead of spark.yarn.executor.memory/cores/instances, as previous configurations already exists in spark, we'd better keep consistent with them. # add condition check of spark.master, as we only need to transfer them into spark-submit command line options on yarn-cluster mode. # the patch is not well formatted. add num-executors / executor-cores / executor-memory option support for hive on spark in Yarn mode -- Key: HIVE-9342 URL: https://issues.apache.org/jira/browse/HIVE-9342 Project: Hive Issue Type: Improvement Components: spark-branch Affects Versions: spark-branch Reporter: Pierre Yin Priority: Minor Labels: spark Fix For: spark-branch Attachments: HIVE-9342.1-spark.patch When I run hive on spark with Yarn mode, I want to control some yarn option, such as --num-executors, --executor-cores, --executor-memory. We can append these options into argv in SparkClientImpl. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9342) add num-executors / executor-cores / executor-memory option support for hive on spark in Yarn mode
[
https://issues.apache.org/jira/browse/HIVE-9342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14273393#comment-14273393
]
Hive QA commented on HIVE-9342:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12691598/HIVE-9342.1-spark.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 7299 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_windowing
org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/632/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/632/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-632/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12691598 - PreCommit-HIVE-SPARK-Build
add num-executors / executor-cores / executor-memory option support for hive
on spark in Yarn mode
--
Key: HIVE-9342
URL: https://issues.apache.org/jira/browse/HIVE-9342
Project: Hive
Issue Type: Improvement
Components: spark-branch
Affects Versions: spark-branch
Reporter: Pierre Yin
Priority: Minor
Labels: spark
Fix For: spark-branch
Attachments: HIVE-9342.1-spark.patch
When I run hive on spark with Yarn mode, I want to control some yarn option,
such as --num-executors, --executor-cores, --executor-memory.
We can append these options into argv in SparkClientImpl.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9342) add num-executors / executor-cores / executor-memory option support for hive on spark in Yarn mode
[ https://issues.apache.org/jira/browse/HIVE-9342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14274074#comment-14274074 ] Xuefu Zhang commented on HIVE-9342: --- [~fangxi.yin], thanks for working on this. [~chengxiang li], could you please take a look the proposed change, especially in light of Spark dynamic executor scaling? Also note that Spark standalone mode is also supported by Hive. add num-executors / executor-cores / executor-memory option support for hive on spark in Yarn mode -- Key: HIVE-9342 URL: https://issues.apache.org/jira/browse/HIVE-9342 Project: Hive Issue Type: Improvement Components: spark-branch Affects Versions: spark-branch Reporter: Pierre Yin Priority: Minor Labels: spark Fix For: spark-branch Attachments: HIVE-9342.1-spark.patch When I run hive on spark with Yarn mode, I want to control some yarn option, such as --num-executors, --executor-cores, --executor-memory. We can append these options into argv in SparkClientImpl. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9342) add num-executors / executor-cores / executor-memory option support for hive on spark in Yarn mode
[ https://issues.apache.org/jira/browse/HIVE-9342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14274831#comment-14274831 ] Chengxiang Li commented on HIVE-9342: - Thanks, [~fangxi.yin] bring this up. While we launch spark client with spark-submit script, it load configurations from 2 ways: spark configuration file and command line options. Hive on Spark actually write all spark related configurations into a property file and add it to spark-submit's --properties-file option. For the 3 executor options you mentioned, there should already corresponding configurations, like: # --num-executors - spark.executor.instances # --executor-cores - spark.executor.cores # --executor-memory - spark.executor.memory So theoreticly, you can configure these properties through hive configuration file or CLI, while it's possible that these configuration does not work in certain deploy mode due to spark implementation. I think we shoud verify if it works in yarn-client or yarn-cluster mode first. add num-executors / executor-cores / executor-memory option support for hive on spark in Yarn mode -- Key: HIVE-9342 URL: https://issues.apache.org/jira/browse/HIVE-9342 Project: Hive Issue Type: Improvement Components: spark-branch Affects Versions: spark-branch Reporter: Pierre Yin Priority: Minor Labels: spark Fix For: spark-branch Attachments: HIVE-9342.1-spark.patch When I run hive on spark with Yarn mode, I want to control some yarn option, such as --num-executors, --executor-cores, --executor-memory. We can append these options into argv in SparkClientImpl. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
