[jira] [Commented] (HIVE-16484) Investigate SparkLauncher for HoS as alternative to bin/spark-submit

2018-01-09 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319187#comment-16319187 ] Xuefu Zhang commented on HIVE-16484: Cool. {quote} Once that gets started I'll work on using

[jira] [Commented] (HIVE-16484) Investigate SparkLauncher for HoS as alternative to bin/spark-submit

2018-01-09 Thread Sahil Takiar (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318872#comment-16318872 ] Sahil Takiar commented on HIVE-16484: - Sounds good to me. I think release for Spark 2.3.0 is just

[jira] [Commented] (HIVE-16484) Investigate SparkLauncher for HoS as alternative to bin/spark-submit

2018-01-08 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16317692#comment-16317692 ] Rui Li commented on HIVE-16484: --- bq. Hive wouldn't need a separate Spark installation to be able to launch

[jira] [Commented] (HIVE-16484) Investigate SparkLauncher for HoS as alternative to bin/spark-submit

2018-01-08 Thread Sahil Takiar (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16317380#comment-16317380 ] Sahil Takiar commented on HIVE-16484: - [~xuefuz] makes sense. In that case, maybe making this change

[jira] [Commented] (HIVE-16484) Investigate SparkLauncher for HoS as alternative to bin/spark-submit

2018-01-08 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16317336#comment-16317336 ] Xuefu Zhang commented on HIVE-16484: [~stakiar], I'm not denying the potential benefits we might get,

[jira] [Commented] (HIVE-16484) Investigate SparkLauncher for HoS as alternative to bin/spark-submit

2018-01-08 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16317318#comment-16317318 ] Marcelo Vanzin commented on HIVE-16484: --- Just amending my previous response: if the concern with

[jira] [Commented] (HIVE-16484) Investigate SparkLauncher for HoS as alternative to bin/spark-submit

2018-01-08 Thread Sahil Takiar (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16317066#comment-16317066 ] Sahil Takiar commented on HIVE-16484: - [~xuefuz] thanks for voicing your concern, I see a few benefits

[jira] [Commented] (HIVE-16484) Investigate SparkLauncher for HoS as alternative to bin/spark-submit

2018-01-08 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16317034#comment-16317034 ] Xuefu Zhang commented on HIVE-16484: I'd echo with [~lirui], wondering the benefits the proposal

[jira] [Commented] (HIVE-16484) Investigate SparkLauncher for HoS as alternative to bin/spark-submit

2018-01-08 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16316894#comment-16316894 ] Marcelo Vanzin commented on HIVE-16484: --- Yes, that's what the launcher does internally, but also,

[jira] [Commented] (HIVE-16484) Investigate SparkLauncher for HoS as alternative to bin/spark-submit

2018-01-08 Thread Sahil Takiar (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16316869#comment-16316869 ] Sahil Takiar commented on HIVE-16484: - [~vanzin] thanks for the info. What about just directly calling

[jira] [Commented] (HIVE-16484) Investigate SparkLauncher for HoS as alternative to bin/spark-submit

2018-01-08 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16316759#comment-16316759 ] Marcelo Vanzin commented on HIVE-16484: --- {{InProcessLauncher}} also uses the launcher server. So

[jira] [Commented] (HIVE-16484) Investigate SparkLauncher for HoS as alternative to bin/spark-submit

2018-01-08 Thread Sahil Takiar (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16316747#comment-16316747 ] Sahil Takiar commented on HIVE-16484: - Hey [~lirui], thanks for taking a look. You bring up a good

[jira] [Commented] (HIVE-16484) Investigate SparkLauncher for HoS as alternative to bin/spark-submit

2018-01-08 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16316130#comment-16316130 ] Rui Li commented on HIVE-16484: --- Hi [~stakiar], got a higher level question: what's the main advantage of

[jira] [Commented] (HIVE-16484) Investigate SparkLauncher for HoS as alternative to bin/spark-submit

2018-01-06 Thread Hive QA (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314683#comment-16314683 ] Hive QA commented on HIVE-16484: Here are the results of testing the latest attachment:

[jira] [Commented] (HIVE-16484) Investigate SparkLauncher for HoS as alternative to bin/spark-submit

2018-01-06 Thread Hive QA (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16314564#comment-16314564 ] Hive QA commented on HIVE-16484: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem ||

[jira] [Commented] (HIVE-16484) Investigate SparkLauncher for HoS as alternative to bin/spark-submit

2018-01-05 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16313162#comment-16313162 ] Rui Li commented on HIVE-16484: --- Thanks [~stakiar] for the work! I had a quick glance at the code and left

[jira] [Commented] (HIVE-16484) Investigate SparkLauncher for HoS as alternative to bin/spark-submit

2018-01-04 Thread Sahil Takiar (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16312178#comment-16312178 ] Sahil Takiar commented on HIVE-16484: - Test failures are un-related. I updated the RB and added a few

[jira] [Commented] (HIVE-16484) Investigate SparkLauncher for HoS as alternative to bin/spark-submit

2018-01-03 Thread Hive QA (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310899#comment-16310899 ] Hive QA commented on HIVE-16484: Here are the results of testing the latest attachment:

[jira] [Commented] (HIVE-16484) Investigate SparkLauncher for HoS as alternative to bin/spark-submit

2018-01-03 Thread Hive QA (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310820#comment-16310820 ] Hive QA commented on HIVE-16484: | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem ||

[jira] [Commented] (HIVE-16484) Investigate SparkLauncher for HoS as alternative to bin/spark-submit

2018-01-03 Thread Hive QA (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16310290#comment-16310290 ] Hive QA commented on HIVE-16484: Here are the results of testing the latest attachment:

[jira] [Commented] (HIVE-16484) Investigate SparkLauncher for HoS as alternative to bin/spark-submit

2018-01-02 Thread Sahil Takiar (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308835#comment-16308835 ] Sahil Takiar commented on HIVE-16484: - Attaching updated patch. Basically, just rebased the old patch

[jira] [Commented] (HIVE-16484) Investigate SparkLauncher for HoS as alternative to bin/spark-submit

2018-01-02 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308515#comment-16308515 ] Marcelo Vanzin commented on HIVE-16484: --- That's correct. Using it only with deploy mode == cluster

[jira] [Commented] (HIVE-16484) Investigate SparkLauncher for HoS as alternative to bin/spark-submit

2018-01-02 Thread Sahil Takiar (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308510#comment-16308510 ] Sahil Takiar commented on HIVE-16484: - Thanks [~vanzin]. I think I was confused about Spark's

[jira] [Commented] (HIVE-16484) Investigate SparkLauncher for HoS as alternative to bin/spark-submit

2018-01-02 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308460#comment-16308460 ] Marcelo Vanzin commented on HIVE-16484: --- {{InProcessLauncher}} works for client mode, but I wouldn't

[jira] [Commented] (HIVE-16484) Investigate SparkLauncher for HoS as alternative to bin/spark-submit

2018-01-02 Thread Hive QA (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308400#comment-16308400 ] Hive QA commented on HIVE-16484: Here are the results of testing the latest attachment:

[jira] [Commented] (HIVE-16484) Investigate SparkLauncher for HoS as alternative to bin/spark-submit

2018-01-02 Thread Sahil Takiar (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16308312#comment-16308312 ] Sahil Takiar commented on HIVE-16484: - [~vanzin] it looks like SPARK-11035 has been complete. Will the

[jira] [Commented] (HIVE-16484) Investigate SparkLauncher for HoS as alternative to bin/spark-submit

2017-05-08 Thread Sahil Takiar (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16000924#comment-16000924 ] Sahil Takiar commented on HIVE-16484: - [~xuefuz] could you take a look?

[jira] [Commented] (HIVE-16484) Investigate SparkLauncher for HoS as alternative to bin/spark-submit

2017-04-29 Thread Hive QA (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989874#comment-15989874 ] Hive QA commented on HIVE-16484: Here are the results of testing the latest attachment:

[jira] [Commented] (HIVE-16484) Investigate SparkLauncher for HoS as alternative to bin/spark-submit

2017-04-28 Thread Hive QA (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15989180#comment-15989180 ] Hive QA commented on HIVE-16484: Here are the results of testing the latest attachment:

[jira] [Commented] (HIVE-16484) Investigate SparkLauncher for HoS as alternative to bin/spark-submit

2017-04-22 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15980148#comment-15980148 ] Marcelo Vanzin commented on HIVE-16484: --- It was mostly added for testing. > Investigate

[jira] [Commented] (HIVE-16484) Investigate SparkLauncher for HoS as alternative to bin/spark-submit

2017-04-22 Thread Sahil Takiar (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15980076#comment-15980076 ] Sahil Takiar commented on HIVE-16484: - Only related failure is {{TestSparkClient.testRemoteClient}}

[jira] [Commented] (HIVE-16484) Investigate SparkLauncher for HoS as alternative to bin/spark-submit

2017-04-22 Thread Hive QA (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15979964#comment-15979964 ] Hive QA commented on HIVE-16484: Here are the results of testing the latest attachment:

[jira] [Commented] (HIVE-16484) Investigate SparkLauncher for HoS as alternative to bin/spark-submit

2017-04-21 Thread Hive QA (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15979501#comment-15979501 ] Hive QA commented on HIVE-16484: Here are the results of testing the latest attachment:

[jira] [Commented] (HIVE-16484) Investigate SparkLauncher for HoS as alternative to bin/spark-submit

2017-04-19 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15975696#comment-15975696 ] Marcelo Vanzin commented on HIVE-16484: --- {{SparkLauncher}} is just a wrapper around spark-submit,

[jira] [Commented] (HIVE-16484) Investigate SparkLauncher for HoS as alternative to bin/spark-submit

2017-04-19 Thread Xuefu Zhang (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15975693#comment-15975693 ] Xuefu Zhang commented on HIVE-16484: Hi [~stakiar], thanks for bringing it up. I think it would be

[jira] [Commented] (HIVE-16484) Investigate SparkLauncher for HoS as alternative to bin/spark-submit

2017-04-19 Thread Sahil Takiar (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15975647#comment-15975647 ] Sahil Takiar commented on HIVE-16484: - [~lirui], [~xuefuz] not sure if this was something that the