[jira] [Updated] (HIVE-13525) HoS hangs when job is empty

2016-04-15 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-13525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-13525: -- Status: Patch Available (was: Open) > HoS hangs when job is empty > --- > >

[jira] [Updated] (HIVE-13525) HoS hangs when job is empty

2016-04-15 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-13525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-13525: -- Attachment: HIVE-13525.1.patch I think the reason is that we rely on JobStart/JobEnd events to determine if

[jira] [Commented] (HIVE-13293) Query occurs performance degradation after enabling parallel order by for Hive on Spark

2016-04-12 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-13293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15238556#comment-15238556 ] Rui Li commented on HIVE-13293: --- Thanks [~xuefuz] for the review. I mean it can work with queries that have

[jira] [Updated] (HIVE-13293) Query occurs performance degradation after enabling parallel order by for Hive on Spark

2016-04-10 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-13293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-13293: -- Attachment: HIVE-13293.1.patch I have tried both splitting the task and caching the RDD and chose the latter

[jira] [Updated] (HIVE-13293) Query occurs performance degradation after enabling parallel order by for Hive on Spark

2016-04-10 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-13293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-13293: -- Status: Patch Available (was: Open) > Query occurs performance degradation after enabling parallel order by

[jira] [Updated] (HIVE-12650) Improve error messages for Hive on Spark in case the cluster has no resources available

2016-04-01 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-12650: -- Resolution: Fixed Fix Version/s: 2.1.0 Status: Resolved (was: Patch Available) Committed to

[jira] [Updated] (HIVE-12650) Improve error messages for Hive on Spark in case the cluster has no resources available

2016-04-01 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-12650: -- Summary: Improve error messages for Hive on Spark in case the cluster has no resources available (was: Improve

[jira] [Updated] (HIVE-12650) Improve error messages in case the cluster has no resources available

2016-04-01 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-12650: -- Summary: Improve error messages in case the cluster has no resources available (was: Spark-submit is killed

[jira] [Commented] (HIVE-12650) Spark-submit is killed when Hive times out. Killing spark-submit doesn't cancel AM request. When AM is finally launched, it tries to connect back to Hive and gets refus

2016-03-31 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15221049#comment-15221049 ] Rui Li commented on HIVE-12650: --- I tried several failed tests locally and they were not reproduced.

[jira] [Commented] (HIVE-13376) HoS emits too many logs with application state

2016-03-30 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-13376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15219163#comment-15219163 ] Rui Li commented on HIVE-13376: --- Thanks [~szehon] for the update. +1. > HoS emits too many logs with

[jira] [Commented] (HIVE-13376) HoS emits too many logs with application state

2016-03-29 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-13376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15217289#comment-15217289 ] Rui Li commented on HIVE-13376: --- Thanks [~szehon] for the fix! I found the config in spark code but not in

[jira] [Updated] (HIVE-12650) Spark-submit is killed when Hive times out. Killing spark-submit doesn't cancel AM request. When AM is finally launched, it tries to connect back to Hive and gets refused

2016-03-28 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-12650: -- Status: Patch Available (was: Open) > Spark-submit is killed when Hive times out. Killing spark-submit doesn't

[jira] [Updated] (HIVE-12650) Spark-submit is killed when Hive times out. Killing spark-submit doesn't cancel AM request. When AM is finally launched, it tries to connect back to Hive and gets refused

2016-03-28 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-12650: -- Attachment: HIVE-12650.1.patch Assigned this to me and upload a patch. The main change in the patch is that we

[jira] [Commented] (HIVE-13293) Query occurs performance degradation after enabling parallel order by for Hive on Spark

2016-03-25 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-13293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15211748#comment-15211748 ] Rui Li commented on HIVE-13293: --- Just did some research about this. Actually the overhead is not so big as I

[jira] [Commented] (HIVE-12650) Spark-submit is killed when Hive times out. Killing spark-submit doesn't cancel AM request. When AM is finally launched, it tries to connect back to Hive and gets refus

2016-03-24 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15211335#comment-15211335 ] Rui Li commented on HIVE-12650: --- I think the difficult part is that we really don't know the possible

[jira] [Commented] (HIVE-12650) Spark-submit is killed when Hive times out. Killing spark-submit doesn't cancel AM request. When AM is finally launched, it tries to connect back to Hive and gets refus

2016-03-21 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15205603#comment-15205603 ] Rui Li commented on HIVE-12650: --- Regarding better error message, do you think we can throw a timeout

[jira] [Commented] (HIVE-13277) Exception "Unable to create serializer 'org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer' " occurred during query execution on spark engine when ve

2016-03-21 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-13277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15205580#comment-15205580 ] Rui Li commented on HIVE-13277: --- Yes I'm using ORC table. Pinging [~xhao1] regarding whether there're other

[jira] [Updated] (HIVE-7292) Hive on Spark

2016-03-19 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-7292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-7292: - Assignee: Xuefu Zhang (was: heywood) > Hive on Spark > - > > Key: HIVE-7292 >

[jira] [Commented] (HIVE-12650) Spark-submit is killed when Hive times out. Killing spark-submit doesn't cancel AM request. When AM is finally launched, it tries to connect back to Hive and gets refus

2016-03-19 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15201029#comment-15201029 ] Rui Li commented on HIVE-12650: --- Here're my findings so far (for yarn-client mode). # If the cluster has no

[jira] [Commented] (HIVE-13277) Exception "Unable to create serializer 'org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer' " occurred during query execution on spark engine when ve

2016-03-19 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-13277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15200785#comment-15200785 ] Rui Li commented on HIVE-13277: --- Not sure about it. I'll do some investigation to see if we have a

[jira] [Assigned] (HIVE-13293) Query occurs performance degradation after enabling parallel order by for Hive on sprak

2016-03-19 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-13293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li reassigned HIVE-13293: - Assignee: Rui Li > Query occurs performance degradation after enabling parallel order by for > Hive on

[jira] [Commented] (HIVE-13293) Query occurs performance degradation after enabling parallel order by for Hive on sprak

2016-03-18 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-13293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15198571#comment-15198571 ] Rui Li commented on HIVE-13293: --- My understanding is that to do the sampling, we need to compute the RDD,

[jira] [Updated] (HIVE-7292) Hive on Spark

2016-03-18 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-7292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-7292: - Issue Type: Improvement (was: Wish) > Hive on Spark > - > > Key: HIVE-7292 >

[jira] [Commented] (HIVE-13277) Exception "Unable to create serializer 'org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer' " occurred during query execution on spark engine when ve

2016-03-15 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-13277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15196806#comment-15196806 ] Rui Li commented on HIVE-13277: --- Pinging [~xuefuz] > Exception "Unable to create serializer >

[jira] [Commented] (HIVE-13277) Exception "Unable to create serializer 'org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer' " occurred during query execution on spark engine when ve

2016-03-14 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-13277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15194645#comment-15194645 ] Rui Li commented on HIVE-13277: --- I built a local snapshot of kryo with latest code and verified the query

[jira] [Commented] (HIVE-13277) Exception "Unable to create serializer 'org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer' " occurred during query execution on spark engine when ve

2016-03-14 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-13277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15193220#comment-15193220 ] Rui Li commented on HIVE-13277: --- I managed to reproduce the issue and I found {{StackOverflowError}} in the

[jira] [Commented] (HIVE-12650) Spark-submit is killed when Hive times out. Killing spark-submit doesn't cancel AM request. When AM is finally launched, it tries to connect back to Hive and gets refus

2016-03-14 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15193174#comment-15193174 ] Rui Li commented on HIVE-12650: --- The timeout is necessary in case the RSC crashes due to some errors. But

[jira] [Updated] (HIVE-13277) Exception "Unable to create serializer 'org.apache.hive.com.esotericsoftware.kryo.serializers.FieldSerializer' " occurred during query execution on spark engine when vect

2016-03-14 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-13277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-13277: -- Description: Found when executing TPCx-BB query2 for Hive on Spark engine, and switch on : Found during TPCx-BB

[jira] [Updated] (HIVE-13066) Hive on Spark gives incorrect results when speculation is on

2016-02-21 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-13066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-13066: -- Attachment: HIVE-13066.1.patch Trigger tests. > Hive on Spark gives incorrect results when speculation is on >

[jira] [Updated] (HIVE-13066) Hive on Spark gives incorrect results when speculation is on

2016-02-21 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-13066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-13066: -- Status: Patch Available (was: Open) > Hive on Spark gives incorrect results when speculation is on >

[jira] [Commented] (HIVE-13066) Hive on Spark gives incorrect results when speculation is on

2016-02-18 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-13066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15152252#comment-15152252 ] Rui Li commented on HIVE-13066: --- I'm not able to reproduce the issue. But I tried to make the task fail if

[jira] [Commented] (HIVE-12951) Reduce Spark executor prewarm timeout to 5s

2016-02-03 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15131538#comment-15131538 ] Rui Li commented on HIVE-12951: --- +1. > Reduce Spark executor prewarm timeout to 5s >

[jira] [Commented] (HIVE-12650) Increase default value of hive.spark.client.server.connect.timeout to exceeds spark.yarn.am.waitTime

2016-02-02 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15129595#comment-15129595 ] Rui Li commented on HIVE-12650: --- bq. Regarding your last question, I tried submitting application when no

[jira] [Commented] (HIVE-12650) Increase default value of hive.spark.client.server.connect.timeout to exceeds spark.yarn.am.waitTime

2016-02-02 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15129682#comment-15129682 ] Rui Li commented on HIVE-12650: --- Thanks Xuefu. Yeah I tried again and found the application is served (AM

[jira] [Commented] (HIVE-12650) Increase default value of hive.spark.client.server.connect.timeout to exceeds spark.yarn.am.waitTime

2016-02-01 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127492#comment-15127492 ] Rui Li commented on HIVE-12650: --- Thanks guys for your inputs. My understanding is that

[jira] [Commented] (HIVE-12650) Increase default value of hive.spark.client.server.connect.timeout to exceeds spark.yarn.am.waitTime

2016-02-01 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15126219#comment-15126219 ] Rui Li commented on HIVE-12650: --- Hi [~vanzin], any idea on this? > Increase default value of

[jira] [Commented] (HIVE-12650) Increase default value of hive.spark.client.server.connect.timeout to exceeds spark.yarn.am.waitTime

2016-02-01 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127747#comment-15127747 ] Rui Li commented on HIVE-12650: --- Hi [~xuefuz], the exception you posted doesn't seem to be a timeout, at

[jira] [Updated] (HIVE-12828) Update Spark version to 1.6

2016-01-28 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-12828: -- Fix Version/s: 2.1.0 > Update Spark version to 1.6 > --- > > Key:

[jira] [Updated] (HIVE-12708) Hive on Spark doesn't work with Kerboresed HBase [Spark Branch]

2016-01-28 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-12708: -- Fix Version/s: 2.1.0 > Hive on Spark doesn't work with Kerboresed HBase [Spark Branch] >

[jira] [Updated] (HIVE-12568) Provide an option to specify network interface used by Spark remote client [Spark Branch]

2016-01-28 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-12568: -- Fix Version/s: 2.1.0 > Provide an option to specify network interface used by Spark remote client > [Spark

[jira] [Updated] (HIVE-12515) Clean the SparkCounters related code after remove counter based stats collection[Spark Branch]

2016-01-28 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-12515: -- Fix Version/s: 2.1.0 > Clean the SparkCounters related code after remove counter based stats >

[jira] [Updated] (HIVE-9774) Print yarn application id to console [Spark Branch]

2016-01-28 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-9774: - Fix Version/s: 2.1.0 > Print yarn application id to console [Spark Branch] >

[jira] [Updated] (HIVE-12811) Name yarn application name more meaning than just "Hive on Spark"

2016-01-28 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-12811: -- Fix Version/s: 2.1.0 > Name yarn application name more meaning than just "Hive on Spark" >

[jira] [Commented] (HIVE-9774) Print yarn application id to console [Spark Branch]

2016-01-28 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15121207#comment-15121207 ] Rui Li commented on HIVE-9774: -- It's for internal use only, just like {{hadoop.bin.path}}. > Print yarn

[jira] [Updated] (HIVE-12466) SparkCounter not initialized error

2016-01-28 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-12466: -- Fix Version/s: 2.1.0 > SparkCounter not initialized error > -- > >

[jira] [Updated] (HIVE-12554) Fix Spark branch build after merge [Spark Branch]

2016-01-28 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-12554: -- Fix Version/s: 2.1.0 > Fix Spark branch build after merge [Spark Branch] >

[jira] [Updated] (HIVE-12611) Make sure spark.yarn.queue is effective and takes the value from mapreduce.job.queuename if given [Spark Branch]

2016-01-28 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-12611: -- Fix Version/s: 2.1.0 > Make sure spark.yarn.queue is effective and takes the value from >

[jira] [Commented] (HIVE-12940) Cherry pick spark branch to master

2016-01-28 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15121218#comment-15121218 ] Rui Li commented on HIVE-12940: --- Thanks [~leftylev]! I just updated the issues. > Cherry pick spark branch

[jira] [Commented] (HIVE-12951) Reduce Spark executor prewarm timeout to 5s

2016-01-27 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120600#comment-15120600 ] Rui Li commented on HIVE-12951: --- Hi [~xuefuz], Spark has configurations

[jira] [Commented] (HIVE-12951) Reduce Spark executor prewarm timeout to 5s

2016-01-27 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120651#comment-15120651 ] Rui Li commented on HIVE-12951: --- Hi [~xuefuz], I just thought more about this. Maybe we should use the

[jira] [Commented] (HIVE-12940) Cherry pick spark branch to master

2016-01-27 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120850#comment-15120850 ] Rui Li commented on HIVE-12940: --- OK. I'll do it. > Cherry pick spark branch to master >

[jira] [Commented] (HIVE-12951) Reduce Spark executor prewarm timeout to 5s

2016-01-27 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120795#comment-15120795 ] Rui Li commented on HIVE-12951: --- Generally speaking, I think we have a better chance to get more reducers

[jira] [Commented] (HIVE-12940) Cherry pick spark branch to master

2016-01-27 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120800#comment-15120800 ] Rui Li commented on HIVE-12940: --- Hi [~xuefuz], I think the failures are not related. To maintain the

[jira] [Updated] (HIVE-12940) Cherry pick spark branch to master

2016-01-27 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-12940: -- Description: We need to cherry-pick the patches that on spark branch to master, and probably discard the spark

[jira] [Commented] (HIVE-12940) Cherry pick spark branch to master

2016-01-27 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15118844#comment-15118844 ] Rui Li commented on HIVE-12940: --- cc [~xuefuz] > Cherry pick spark branch to master >

[jira] [Updated] (HIVE-12940) Cherry pick spark branch to master

2016-01-27 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-12940: -- Summary: Cherry pick spark branch to master (was: Merge master into spark [Spark Branch]) > Cherry pick spark

[jira] [Updated] (HIVE-12940) Merge master into spark [Spark Branch]

2016-01-27 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-12940: -- Attachment: HIVE-12940.1.patch Run tests. > Merge master into spark [Spark Branch] >

[jira] [Commented] (HIVE-12940) Cherry pick spark branch to master

2016-01-27 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15118860#comment-15118860 ] Rui Li commented on HIVE-12940: --- Cherry-picked patches are: HIVE-12045, HIVE-12466, HIVE-12554, HIVE-12515,

[jira] [Updated] (HIVE-9774) Print yarn application id to console [Spark Branch]

2016-01-17 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-9774: - Attachment: HIVE-9774.1-spark.patch The patch uses {{SparkContext::applicationId}}, which is the YARN app ID when

[jira] [Assigned] (HIVE-9774) Print yarn application id to console [Spark Branch]

2016-01-17 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li reassigned HIVE-9774: Assignee: Rui Li (was: Chinna Rao Lalam) > Print yarn application id to console [Spark Branch] >

[jira] [Commented] (HIVE-9774) Print yarn application id to console [Spark Branch]

2016-01-17 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15104091#comment-15104091 ] Rui Li commented on HIVE-9774: -- OK, assigned this to me. > Print yarn application id to console [Spark

[jira] [Commented] (HIVE-12828) Update Spark version to 1.6

2016-01-14 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15101125#comment-15101125 ] Rui Li commented on HIVE-12828: --- Looked at the log and error is {noformat} 2016-01-14T14:38:11,889 -

[jira] [Commented] (HIVE-12828) Update Spark version to 1.6

2016-01-14 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15101204#comment-15101204 ] Rui Li commented on HIVE-12828: --- [~xuefuz], do we need to make parquet_join pass here? > Update Spark

[jira] [Commented] (HIVE-12828) Update Spark version to 1.6

2016-01-13 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15097621#comment-15097621 ] Rui Li commented on HIVE-12828: --- OK. Thanks for taking care of this, Xuefu. > Update Spark version to 1.6 >

[jira] [Commented] (HIVE-12828) Update Spark version to 1.6

2016-01-13 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15096325#comment-15096325 ] Rui Li commented on HIVE-12828: --- The parquet_join passes on my machine with a locally built tar ball.

[jira] [Updated] (HIVE-12811) Name yarn application name more meaning than just "Hive on Spark"

2016-01-12 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-12811: -- Attachment: HIVE-12811.1-spark.patch Make the app name settable, and avoid re-creating session if user just

[jira] [Commented] (HIVE-12828) Update Spark version to 1.6

2016-01-12 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15094077#comment-15094077 ] Rui Li commented on HIVE-12828: --- We found the profile is needed when we updated to spark 1.5. I have also

[jira] [Updated] (HIVE-12828) Update Spark version to 1.6

2016-01-12 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-12828: -- Attachment: HIVE-12828.2-spark.patch Thanks Xuefu. Run tests again. > Update Spark version to 1.6 >

[jira] [Commented] (HIVE-12828) Update Spark version to 1.6

2016-01-11 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093090#comment-15093090 ] Rui Li commented on HIVE-12828: --- Thanks Xuefu for the patch. I'll try it out. > Update Spark version to 1.6

[jira] [Commented] (HIVE-12811) Name yarn application name more meaning than just "Hive on Spark"

2016-01-11 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15093120#comment-15093120 ] Rui Li commented on HIVE-12811: --- Thanks Xuefu for the suggestions. Do you think we have to restart the spark

[jira] [Updated] (HIVE-12828) Update Spark version to 1.6

2016-01-11 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-12828: -- Attachment: HIVE-12828.2-spark.patch Integrated Xuefu's patch. I changed the mem overheads to 0 because

[jira] [Updated] (HIVE-12828) Update Spark version to 1.6

2016-01-11 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-12828: -- Attachment: HIVE-12828.1-spark.patch It just works out of box for simple queries locally. [~xuefuz], please

[jira] [Assigned] (HIVE-12828) Update Spark version to 1.6

2016-01-10 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li reassigned HIVE-12828: - Assignee: Rui Li (was: Xuefu Zhang) > Update Spark version to 1.6 > --- > >

[jira] [Commented] (HIVE-12811) Name yarn application name more meaning than just "Hive on Spark"

2016-01-10 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15091322#comment-15091322 ] Rui Li commented on HIVE-12811: --- [~xuefuz], one quick question: after we launch a spark app, we may submit

[jira] [Assigned] (HIVE-12811) Name yarn application name more meaning than just "Hive on Spark"

2016-01-10 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li reassigned HIVE-12811: - Assignee: Rui Li (was: Xuefu Zhang) > Name yarn application name more meaning than just "Hive on Spark"

[jira] [Assigned] (HIVE-12611) Make sure spark.yarn.queue is effective and takes the value from mapreduce.job.queuename if given [Spark Branch]

2016-01-10 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li reassigned HIVE-12611: - Assignee: Rui Li (was: Xuefu Zhang) > Make sure spark.yarn.queue is effective and takes the value from

[jira] [Commented] (HIVE-12515) Clean the SparkCounters related code after remove counter based stats collection[Spark Branch]

2015-12-02 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035483#comment-15035483 ] Rui Li commented on HIVE-12515: --- {{mapjoin_memcheck}} also passes on my side, so doesn't seem related. >

[jira] [Commented] (HIVE-12569) Excessive console message from SparkClientImpl [Spark Branch]

2015-12-02 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037439#comment-15037439 ] Rui Li commented on HIVE-12569: --- I also hit the issue before. Changing log4j conf to make the logs go to RFA

[jira] [Commented] (HIVE-12515) Clean the SparkCounters related code after remove counter based stats collection[Spark Branch]

2015-12-02 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15037417#comment-15037417 ] Rui Li commented on HIVE-12515: --- Thanks guys. I'll commit this shortly. > Clean the SparkCounters related

[jira] [Commented] (HIVE-12554) Fix Spark branch build after merge [Spark Branch]

2015-12-01 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15035069#comment-15035069 ] Rui Li commented on HIVE-12554: --- Thanks Xuefu for taking care of this. > Fix Spark branch build after merge

[jira] [Assigned] (HIVE-12515) Clean the SparkCounters related code after remove counter based stats collection[Spark Branch]

2015-12-01 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li reassigned HIVE-12515: - Assignee: Rui Li (was: Xuefu Zhang) > Clean the SparkCounters related code after remove counter based

[jira] [Updated] (HIVE-12515) Clean the SparkCounters related code after remove counter based stats collection[Spark Branch]

2015-12-01 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-12515: -- Attachment: HIVE-12515.1-spark.patch As I mentioned above, SparkCounters is not removed so that we can use it

[jira] [Commented] (HIVE-12515) Clean the SparkCounters related code after remove counter based stats collection[Spark Branch]

2015-11-27 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15029694#comment-15029694 ] Rui Li commented on HIVE-12515: --- [~chengxiang li] - If we want to do this in spark branch, how about first

[jira] [Commented] (HIVE-12515) Clean the SparkCounters related code after remove counter based stats collection[Spark Branch]

2015-11-26 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15029519#comment-15029519 ] Rui Li commented on HIVE-12515: --- Shall we target this to master or wait until HIVE-12411 gets merged to

[jira] [Commented] (HIVE-12515) Clean the SparkCounters related code after remove counter based stats collection[Spark Branch]

2015-11-26 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15029546#comment-15029546 ] Rui Li commented on HIVE-12515: --- That class is already removed in HIVE-12411. So we should do this task in

[jira] [Commented] (HIVE-12466) SparkCounter not initialized error

2015-11-24 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15026017#comment-15026017 ] Rui Li commented on HIVE-12466: --- If spark counter is removed, does HoS support other methods to collect

[jira] [Updated] (HIVE-12466) SparkCounter not initialized error

2015-11-23 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-12466: -- Attachment: HIVE-12466.1-spark.patch > SparkCounter not initialized error > --

[jira] [Commented] (HIVE-12466) SparkCounter not initialized error

2015-11-23 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15023551#comment-15023551 ] Rui Li commented on HIVE-12466: --- Yeah they're appending suffix to the counter name now. Maybe we can

[jira] [Assigned] (HIVE-12466) SparkCounter not initialized error

2015-11-23 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li reassigned HIVE-12466: - Assignee: Rui Li (was: Xuefu Zhang) > SparkCounter not initialized error >

[jira] [Commented] (HIVE-12466) SparkCounter not initialized error

2015-11-23 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15023582#comment-15023582 ] Rui Li commented on HIVE-12466: --- Sure. Assigned this to me. > SparkCounter not initialized error >

[jira] [Commented] (HIVE-11180) Enable native vectorized map join for spark [Spark Branch]

2015-11-23 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-11180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15023718#comment-15023718 ] Rui Li commented on HIVE-11180: --- updated

[jira] [Commented] (HIVE-11180) Enable native vectorized map join for spark [Spark Branch]

2015-11-22 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-11180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15021419#comment-15021419 ] Rui Li commented on HIVE-11180: --- We should update the doc after release, right? > Enable native vectorized

[jira] [Updated] (HIVE-12493) HIVE-11180 didn't merge cleanly to branch-1

2015-11-22 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-12493: -- Attachment: HIVE-12493.1.patch HIVE-12461 is also fixing build of branch-1. Let's trigger tests after it's

[jira] [Commented] (HIVE-11180) Enable native vectorized map join for spark [Spark Branch]

2015-11-22 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-11180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15021403#comment-15021403 ] Rui Li commented on HIVE-11180: --- Thanks [~vikram.dixit]. Just filed HIVE-12493 for that. > Enable native

[jira] [Updated] (HIVE-12045) ClassNotFound for GenericUDF in "select distinct..." query (Hive on Spark)

2015-11-19 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-12045: -- Attachment: HIVE-12045.3-spark.patch The latest patch mainly fixes two things: 1. When we start the RSC, we

[jira] [Updated] (HIVE-12045) ClassNotFound for GenericUDF in "select distinct..." query (Hive on Spark)

2015-11-19 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-12045: -- Attachment: (was: HIVE-12045.2-spark.patch) > ClassNotFound for GenericUDF in "select distinct..." query

[jira] [Commented] (HIVE-12466) SparkCounter not initialized error

2015-11-19 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15015100#comment-15015100 ] Rui Li commented on HIVE-12466: --- [~xuefuz] - Seems this JIRA gets assigned to you automatically :) >

[jira] [Updated] (HIVE-12045) ClassNotFound for GenericUDF in "select distinct..." query (Hive on Spark)

2015-11-19 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-12045: -- Attachment: HIVE-12045.4-spark.patch [~xuefuz] - The failed test is because SessionState is not available

[jira] [Commented] (HIVE-12045) ClassNotFound for GenericUDF in "select distinct..." query (Hive on Spark)

2015-11-19 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15015276#comment-15015276 ] Rui Li commented on HIVE-12045: --- Latest failures are not related. > ClassNotFound for GenericUDF in "select

[jira] [Commented] (HIVE-12045) ClassNotFound for GenericUDF in "select distinct..." query (Hive on Spark)

2015-11-16 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15007774#comment-15007774 ] Rui Li commented on HIVE-12045: --- Thanks Xuefu. I'll try with master. > ClassNotFound for GenericUDF in

<    8   9   10   11   12   13   14   15   >