[jira] [Commented] (HIVE-15202) Concurrent compactions for the same partition may generate malformed folder structure

2016-11-21 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15685412#comment-15685412 ] Rui Li commented on HIVE-15202: --- Hi [~ekoifman], looking at your suggestions in HIVE-14980, is the idea to

[jira] [Assigned] (HIVE-15237) Propagate Spark job failure to Hive

2016-11-17 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li reassigned HIVE-15237: - Assignee: Rui Li > Propagate Spark job failure to Hive > --- > >

[jira] [Commented] (HIVE-15237) Propagate Spark job failure to Hive

2016-11-17 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15675299#comment-15675299 ] Rui Li commented on HIVE-15237: --- [~xuefuz], thanks for raising the issue. I'll take a look at this. >

[jira] [Commented] (HIVE-15202) Concurrent compactions for the same partition may generate malformed folder structure

2016-11-16 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15672903#comment-15672903 ] Rui Li commented on HIVE-15202: --- Thank you [~ekoifman]. That's great to know. > Concurrent compactions for

[jira] [Commented] (HIVE-15202) Concurrent compactions for the same partition may generate malformed folder structure

2016-11-15 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15669690#comment-15669690 ] Rui Li commented on HIVE-15202: --- [~ekoifman], is there any plan to implement this on Hive side? Or do you

[jira] [Commented] (HIVE-15202) Concurrent compactions for the same partition may generate malformed folder structure

2016-11-15 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15669292#comment-15669292 ] Rui Li commented on HIVE-15202: --- Also pinging [~ekoifman]. Seems you're quite knowledgeable about

[jira] [Commented] (HIVE-15202) Concurrent compactions for the same partition may generate malformed folder structure

2016-11-14 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15666351#comment-15666351 ] Rui Li commented on HIVE-15202: --- [~ashutoshc], would you mind share your ideas on this? Thank you! >

[jira] [Commented] (HIVE-15202) Concurrent compactions for the same partition may generate malformed folder structure

2016-11-14 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15666323#comment-15666323 ] Rui Li commented on HIVE-15202: --- Prior to HIVE-13040, selecting on such a table fails with NPE in split

[jira] [Commented] (HIVE-15101) Spark client can be stuck in RUNNING state

2016-11-09 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15653187#comment-15653187 ] Rui Li commented on HIVE-15101: --- Hi [~tzenmyo], thanks for your input. I'll try your scenario and see what I

[jira] [Updated] (HIVE-15139) HoS local mode fails with NumberFormatException

2016-11-08 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15139: -- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Committed to

[jira] [Commented] (HIVE-15139) HoS local mode fails with NumberFormatException

2016-11-07 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15646692#comment-15646692 ] Rui Li commented on HIVE-15139: --- [~ctang.ma] - no problem. [~xuefuz], I'll commit this if you have no

[jira] [Updated] (HIVE-14825) Figure out the minimum set of required jars for Hive on Spark after bumping up to Spark 2.0.0

2016-11-07 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-14825: -- Fix Version/s: 2.2.0 > Figure out the minimum set of required jars for Hive on Spark after bumping > up to

[jira] [Resolved] (HIVE-14825) Figure out the minimum set of required jars for Hive on Spark after bumping up to Spark 2.0.0

2016-11-07 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li resolved HIVE-14825. --- Resolution: Resolved Assignee: Rui Li Updated the wiki. > Figure out the minimum set of required jars

[jira] [Updated] (HIVE-14825) Figure out the minimum set of required jars for Hive on Spark after bumping up to Spark 2.0.0

2016-11-07 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-14825: -- Component/s: Documentation > Figure out the minimum set of required jars for Hive on Spark after bumping > up

[jira] [Updated] (HIVE-14825) Figure out the minimum set of required jars for Hive on Spark after bumping up to Spark 2.0.0

2016-11-07 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-14825: -- Issue Type: Task (was: Bug) > Figure out the minimum set of required jars for Hive on Spark after bumping >

[jira] [Commented] (HIVE-15139) HoS local mode fails with NumberFormatException

2016-11-07 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15646403#comment-15646403 ] Rui Li commented on HIVE-15139: --- [~ctang.ma], usually we use yarn mode (either yarn-client or yarn-cluster).

[jira] [Commented] (HIVE-15139) HoS local mode fails with NumberFormatException

2016-11-07 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15646385#comment-15646385 ] Rui Li commented on HIVE-15139: --- [~xuefuz], no it's actually related to HIVE-12205. It only happens with

[jira] [Commented] (HIVE-15139) HoS local mode fails with NumberFormatException

2016-11-07 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15646213#comment-15646213 ] Rui Li commented on HIVE-15139: --- Hi [~ctang.ma], thanks for the review. Yeah your solution also works, like

[jira] [Commented] (HIVE-15101) Spark client can be stuck in RUNNING state

2016-11-07 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15644300#comment-15644300 ] Rui Li commented on HIVE-15101: --- The comments state that RPC server will take care of timeout. If it's not

[jira] [Updated] (HIVE-14910) Flaky test: TestSparkClient.testJobSubmission

2016-11-07 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-14910: -- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Committed to

[jira] [Updated] (HIVE-15139) HoS local mode fails with NumberFormatException

2016-11-07 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15139: -- Attachment: HIVE-15139.1.patch I think we can only store the stageId in the map, because that's what we need

[jira] [Updated] (HIVE-15139) HoS local mode fails with NumberFormatException

2016-11-07 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15139: -- Status: Patch Available (was: Open) > HoS local mode fails with NumberFormatException >

[jira] [Commented] (HIVE-14825) Figure out the minimum set of required jars for Hive on Spark after bumping up to Spark 2.0.0

2016-11-07 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15643542#comment-15643542 ] Rui Li commented on HIVE-14825: --- For yarn, I can run some simple queries with the following jars: {noformat}

[jira] [Commented] (HIVE-14919) Improve the performance of Hive on Spark 2.0.0

2016-11-06 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15643352#comment-15643352 ] Rui Li commented on HIVE-14919: --- One thing I noted is the Xms flag was removed from the executor's options

[jira] [Updated] (HIVE-13066) Hive on Spark gives incorrect results when speculation is on

2016-11-06 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-13066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-13066: -- Resolution: Duplicate Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Fixed via

[jira] [Updated] (HIVE-15054) Hive insertion query execution fails on Hive on Spark

2016-11-06 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15054: -- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Committed to

[jira] [Commented] (HIVE-15104) Hive on Spark generate more shuffle data than hive on mr

2016-11-03 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15634914#comment-15634914 ] Rui Li commented on HIVE-15104: --- [~xuefuz], [~aihuaxu], both MR and Spark need HiveKey.hashCode to compute

[jira] [Commented] (HIVE-15104) Hive on Spark generate more shuffle data than hive on mr

2016-11-03 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15632244#comment-15632244 ] Rui Li commented on HIVE-15104: --- [~xuefuz], here's what I find so far. Firstly, MR uses HiveKey as the [key

[jira] [Updated] (HIVE-15039) A better job monitor console output for HoS

2016-11-02 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15039: -- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Committed to

[jira] [Updated] (HIVE-15039) A better job monitor console output for HoS

2016-11-02 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15039: -- Labels: TODOC2.2 (was: ) > A better job monitor console output for HoS >

[jira] [Commented] (HIVE-14910) Flaky test: TestSparkClient.testJobSubmission

2016-11-02 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15631077#comment-15631077 ] Rui Li commented on HIVE-14910: --- +1 > Flaky test: TestSparkClient.testJobSubmission >

[jira] [Commented] (HIVE-15101) Spark client can be stuck in RUNNING state

2016-11-02 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15631069#comment-15631069 ] Rui Li commented on HIVE-15101: --- Maybe RemoteDriver's log is more useful. For yarn-client, you should find

[jira] [Commented] (HIVE-15104) Hive on Spark generate more shuffle data than hive on mr

2016-11-02 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15631053#comment-15631053 ] Rui Li commented on HIVE-15104: --- We need to use HiveKey because it holds the proper hash code to be used for

[jira] [Commented] (HIVE-15104) Hive on Spark generate more shuffle data than hive on mr

2016-11-02 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15628605#comment-15628605 ] Rui Li commented on HIVE-15104: --- Seems MR can just serialize the key as BytesWritable instead of HiveKey. We

[jira] [Commented] (HIVE-15039) A better job monitor console output for HoS

2016-11-02 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15628317#comment-15628317 ] Rui Li commented on HIVE-15039: --- Failed tests cannot be reproduced. I'll commit this shortly if no one has

[jira] [Commented] (HIVE-15054) Hive insertion query execution fails on Hive on Spark

2016-11-01 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627560#comment-15627560 ] Rui Li commented on HIVE-15054: --- Thanks for updating [~aihuaxu]. +1 > Hive insertion query execution fails

[jira] [Updated] (HIVE-15039) A better job monitor console output for HoS

2016-11-01 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15039: -- Attachment: HIVE-15039.2.patch Thanks [~Ferd] and [~sseth]. Upload the same patch to try again. > A better job

[jira] [Updated] (HIVE-15039) A better job monitor console output for HoS

2016-11-01 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15039: -- Attachment: (was: HIVE-15039.2.patch) > A better job monitor console output for HoS >

[jira] [Commented] (HIVE-15039) A better job monitor console output for HoS

2016-10-31 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15622333#comment-15622333 ] Rui Li commented on HIVE-15039: --- Thanks [~Ferd]. Do you know how to restart it? > A better job monitor

[jira] [Commented] (HIVE-14910) Flaky test: TestSparkClient.testJobSubmission

2016-10-31 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15622330#comment-15622330 ] Rui Li commented on HIVE-14910: --- It makes sense to me. Have a question: is the Listener only used for tests?

[jira] [Updated] (HIVE-15039) A better job monitor console output for HoS

2016-10-31 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15039: -- Attachment: (was: HIVE-15039.2.patch) > A better job monitor console output for HoS >

[jira] [Updated] (HIVE-15039) A better job monitor console output for HoS

2016-10-31 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15039: -- Attachment: HIVE-15039.2.patch > A better job monitor console output for HoS >

[jira] [Updated] (HIVE-15081) RetryingMetaStoreClient.getProxy(HiveConf, Boolean) doesn't match constructor of HiveMetaStoreClient

2016-10-31 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15081: -- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Committed to

[jira] [Commented] (HIVE-15054) Hive insertion query execution fails on Hive on Spark

2016-10-31 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15621481#comment-15621481 ] Rui Li commented on HIVE-15054: --- Thanks [~aihuaxu] for the investigation and update! The patch looks good.

[jira] [Commented] (HIVE-15054) Hive insertion query execution fails on Hive on Spark

2016-10-28 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15614491#comment-15614491 ] Rui Li commented on HIVE-15054: --- This is how MR generates the attempt Id:

[jira] [Commented] (HIVE-15054) Hive insertion query execution fails on Hive on Spark

2016-10-28 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15614437#comment-15614437 ] Rui Li commented on HIVE-15054: --- Hi [~aihuaxu], I don't know for sure either. My hunch is that since

[jira] [Commented] (HIVE-15081) RetryingMetaStoreClient.getProxy(HiveConf, Boolean) doesn't match constructor of HiveMetaStoreClient

2016-10-27 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15614392#comment-15614392 ] Rui Li commented on HIVE-15081: --- The failed test can't be reproduced. [~sershe], would you mind take a look?

[jira] [Updated] (HIVE-15039) A better job monitor console output for HoS

2016-10-27 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15039: -- Attachment: HIVE-15039.2.patch Update patch to address [~Ferd]'s comments. {{InPlaceUpdates}} is general enough

[jira] [Updated] (HIVE-15081) RetryingMetaStoreClient.getProxy(HiveConf, Boolean) doesn't match constructor of HiveMetaStoreClient

2016-10-26 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15081: -- Status: Patch Available (was: Open) > RetryingMetaStoreClient.getProxy(HiveConf, Boolean) doesn't match

[jira] [Updated] (HIVE-15081) RetryingMetaStoreClient.getProxy(HiveConf, Boolean) doesn't match constructor of HiveMetaStoreClient

2016-10-26 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15081: -- Attachment: HIVE-15081.1.patch > RetryingMetaStoreClient.getProxy(HiveConf, Boolean) doesn't match constructor

[jira] [Updated] (HIVE-15039) A better job monitor console output for HoS

2016-10-26 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15039: -- Attachment: HIVE-15039.1.patch Thanks Xuefu. None of the failures can be reproduced. Try again. > A better job

[jira] [Updated] (HIVE-15039) A better job monitor console output for HoS

2016-10-26 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15039: -- Attachment: (was: HIVE-15039.1.patch) > A better job monitor console output for HoS >

[jira] [Commented] (HIVE-15054) Hive insertion query execution fails on Hive on Spark

2016-10-26 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15610276#comment-15610276 ] Rui Li commented on HIVE-15054: --- I could be wrong, but from what I remember, partitionId_attemptNumber is

[jira] [Commented] (HIVE-15054) Hive insertion query execution fails on Hive on Spark

2016-10-26 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15608577#comment-15608577 ] Rui Li commented on HIVE-15054: --- Thanks for the explanations Aihua. I'm not sure which format is better:

[jira] [Updated] (HIVE-15039) A better job monitor console output for HoS

2016-10-26 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15039: -- Attachment: HIVE-15039.1.patch > A better job monitor console output for HoS >

[jira] [Updated] (HIVE-15039) A better job monitor console output for HoS

2016-10-26 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15039: -- Attachment: (was: HIVE-15039.1.patch) > A better job monitor console output for HoS >

[jira] [Updated] (HIVE-15039) A better job monitor console output for HoS

2016-10-26 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15039: -- Attachment: with patch.png > A better job monitor console output for HoS >

[jira] [Updated] (HIVE-15039) A better job monitor console output for HoS

2016-10-26 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15039: -- Status: Patch Available (was: Open) > A better job monitor console output for HoS >

[jira] [Updated] (HIVE-15039) A better job monitor console output for HoS

2016-10-26 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15039: -- Attachment: HIVE-15039.1.patch I basically just copied code from {{TezJobMonitor}}. So the output looks quite

[jira] [Commented] (HIVE-15054) Hive insertion query execution fails on Hive on Spark

2016-10-25 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15607064#comment-15607064 ] Rui Li commented on HIVE-15054: --- Yeah I tried something similar in HIVE-13066. I'll mark it as dup and let's

[jira] [Commented] (HIVE-12951) Reduce Spark executor prewarm timeout to 5s

2016-10-25 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-12951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15605111#comment-15605111 ] Rui Li commented on HIVE-12951: --- Hi [~richard_xin], pre-warm is intended to wait until some containers are

[jira] [Updated] (HIVE-15039) A better job monitor console output for HoS

2016-10-23 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15039: -- Attachment: example screenshot.png > A better job monitor console output for HoS >

[jira] [Updated] (HIVE-15039) A better job monitor console output for HoS

2016-10-23 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-15039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-15039: -- Description: When there're many stages, it's very difficult to read the console output of job progress of HoS.

[jira] [Commented] (HIVE-14797) reducer number estimating may lead to data skew

2016-10-21 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15597040#comment-15597040 ] Rui Li commented on HIVE-14797: --- [~roncenzhao], would you mind update the patch as Xuefu suggested? Or let

[jira] [Commented] (HIVE-14797) reducer number estimating may lead to data skew

2016-10-20 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15591149#comment-15591149 ] Rui Li commented on HIVE-14797: --- [~xuefuz] - I see. Thanks for the information. In that case, I think we

[jira] [Commented] (HIVE-14797) reducer number estimating may lead to data skew

2016-10-18 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15587451#comment-15587451 ] Rui Li commented on HIVE-14797: --- Hi [~xuefuz], for the example in the description, B is skewed but (A, B)

[jira] [Commented] (HIVE-14029) Update Spark version to 2.0.0

2016-10-17 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15584325#comment-15584325 ] Rui Li commented on HIVE-14029: --- Hmm even with a shim layer, it's difficult to support different Spark

[jira] [Commented] (HIVE-14797) reducer number estimating may lead to data skew

2016-10-17 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15584307#comment-15584307 ] Rui Li commented on HIVE-14797: --- Thanks for the update [~roncenzhao]. I have one more question.

[jira] [Commented] (HIVE-9583) Rolling upgrade of Hive MetaStore Server

2016-10-16 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-9583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15581026#comment-15581026 ] Rui Li commented on HIVE-9583: -- The feature has been implemented for over a year. Is it ready to be used? Any

[jira] [Commented] (HIVE-14797) reducer number estimating may lead to data skew

2016-10-12 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15568152#comment-15568152 ] Rui Li commented on HIVE-14797: --- Seems for MR, we need to get #reducers from hconf, but for Spark/Tez, we

[jira] [Commented] (HIVE-14797) reducer number estimating may lead to data skew

2016-10-11 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15565084#comment-15565084 ] Rui Li commented on HIVE-14797: --- I did some tests locally. It turns out

[jira] [Commented] (HIVE-14412) Add a timezone-aware timestamp

2016-10-11 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15564689#comment-15564689 ] Rui Li commented on HIVE-14412: --- Thanks [~alangates] and [~jdere] for your inputs. I'll investigate how to

[jira] [Commented] (HIVE-14735) Build Infra: Spark artifacts download takes a long time

2016-09-28 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15531584#comment-15531584 ] Rui Li commented on HIVE-14735: --- I think that means you have to install md5sum. Maybe we should document

[jira] [Commented] (HIVE-14029) Update Spark version to 2.0.0

2016-09-28 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15528537#comment-15528537 ] Rui Li commented on HIVE-14029: --- Since different versions of Spark are not binary compatible, it'd be good

[jira] [Commented] (HIVE-14412) Add a timezone-aware timestamp

2016-09-27 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15528517#comment-15528517 ] Rui Li commented on HIVE-14412: --- [~jdere], thank you for the comments and trying this out. bq. Is there a

[jira] [Resolved] (HIVE-14687) Metastore API - Retrieving PrincipalPrivilegeSet Object using getPrivileges() method inside metastore Table class give Null value

2016-09-27 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li resolved HIVE-14687. --- Resolution: Not A Problem Fix Version/s: (was: 1.2.1) > Metastore API - Retrieving

[jira] [Commented] (HIVE-14687) Metastore API - Retrieving PrincipalPrivilegeSet Object using getPrivileges() method inside metastore Table class give Null value

2016-09-27 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15525595#comment-15525595 ] Rui Li commented on HIVE-14687: --- We have {{IMetaStoreClient::list_privileges}} for this purpose. Closing

[jira] [Commented] (HIVE-14029) Update Spark version to 2.0.0

2016-09-27 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15525573#comment-15525573 ] Rui Li commented on HIVE-14029: --- Yeah I'll update HIVE-14825 once I have identified the minimum set for

[jira] [Commented] (HIVE-14029) Update Spark version to 2.0.0

2016-09-27 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15525416#comment-15525416 ] Rui Li commented on HIVE-14029: --- I tried the 5th patch locally. The jersey 2 stuff won't be pulled to lib,

[jira] [Commented] (HIVE-14029) Update Spark version to 2.0.0

2016-09-24 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15519134#comment-15519134 ] Rui Li commented on HIVE-14029: --- [~Ferd], you mean unit test will fail right? I don't see failed spark qtest

[jira] [Commented] (HIVE-14029) Update Spark version to 2.0.0

2016-09-23 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515964#comment-15515964 ] Rui Li commented on HIVE-14029: --- The offending jar comes as a dependency of jersey-client: {noformat} [INFO]

[jira] [Updated] (HIVE-14412) Add a timezone-aware timestamp

2016-09-23 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-14412: -- Attachment: HIVE-14412.8.patch Thanks [~jdere] for your comments! Update v8 patch accordingly. > Add a

[jira] [Commented] (HIVE-14825) Figure out the minimum set of required jars for Hive on Spark after bumping up to Spark 2.0.0

2016-09-22 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15515155#comment-15515155 ] Rui Li commented on HIVE-14825: --- Thanks [~Ferd] for tracking this. I expect the minimum set to be fairly

[jira] [Commented] (HIVE-14029) Update Spark version to 2.0.0

2016-09-22 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513539#comment-15513539 ] Rui Li commented on HIVE-14029: --- I'm using: {noformat} java version "1.8.0_91" Java(TM) SE Runtime

[jira] [Commented] (HIVE-14029) Update Spark version to 2.0.0

2016-09-22 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513473#comment-15513473 ] Rui Li commented on HIVE-14029: --- Seems we have two {{javax.ws.rs.core.UriInfo}} interfaces from two jars:

[jira] [Commented] (HIVE-14029) Update Spark version to 2.0.0

2016-09-22 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512678#comment-15512678 ] Rui Li commented on HIVE-14029: --- When I tried the patch locally, I got a compile error: {noformat}

[jira] [Commented] (HIVE-14029) Update Spark version to 2.0.0

2016-09-22 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512499#comment-15512499 ] Rui Li commented on HIVE-14029: --- Hi [~Ferd], My thought was we should be able to tell the user what are

[jira] [Updated] (HIVE-14412) Add a timezone-aware timestamp

2016-09-21 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-14412: -- Attachment: HIVE-14412.7.patch Thank you very much [~xuefuz] for the review. Update patch to address your

[jira] [Comment Edited] (HIVE-14719) ASTNode rootNode is not maintained properly when changing child/parent relation

2016-09-21 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511742#comment-15511742 ] Rui Li edited comment on HIVE-14719 at 9/22/16 1:17 AM: Guess [~ashutoshc] is busy

[jira] [Commented] (HIVE-14719) ASTNode rootNode is not maintained properly when changing child/parent relation

2016-09-21 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511742#comment-15511742 ] Rui Li commented on HIVE-14719: --- Guess [~ashutoshc] is busy with other tasks. [~ jcamachorodriguez], [~

[jira] [Commented] (HIVE-14412) Add a timezone-aware timestamp

2016-09-21 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511722#comment-15511722 ] Rui Li commented on HIVE-14412: --- Latest failures not related and I've uploaded v6 to RB. [~xuefuz] please

[jira] [Commented] (HIVE-14797) reducer number estimating may lead to data skew

2016-09-21 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511708#comment-15511708 ] Rui Li commented on HIVE-14797: --- I see some failures "did not produce a TEST-*.xml file". Are they related?

[jira] [Commented] (HIVE-14029) Update Spark version to 2.0.0

2016-09-21 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511682#comment-15511682 ] Rui Li commented on HIVE-14029: --- [~Ferd], the classpath is just for HS2/CLI, so I don't think we need all

[jira] [Updated] (HIVE-14412) Add a timezone-aware timestamp

2016-09-21 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-14412: -- Attachment: HIVE-14412.6.patch V6 patch makes TIME as reserved keyword, and ZONE and TIMESTAMPTZ as

[jira] [Commented] (HIVE-14029) Update Spark version to 2.0.0

2016-09-21 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509966#comment-15509966 ] Rui Li commented on HIVE-14029: --- OK. AFAIK, HoS only needs spark-core. So we can try adding spark-core and

[jira] [Commented] (HIVE-14029) Update Spark version to 2.0.0

2016-09-21 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15509231#comment-15509231 ] Rui Li commented on HIVE-14029: --- +1 to the latest patch. [~Ferd], is there anything needs to be updated in

[jira] [Updated] (HIVE-14714) Avoid misleading "java.io.IOException: Stream closed" when shutting down HoS

2016-09-21 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-14714: -- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Committed to

[jira] [Updated] (HIVE-14714) Avoid misleading "java.io.IOException: Stream closed" when shutting down HoS

2016-09-21 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-14714: -- Summary: Avoid misleading "java.io.IOException: Stream closed" when shutting down HoS (was: Finishing Hive on

[jira] [Commented] (HIVE-14797) reducer number estimating may lead to data skew

2016-09-20 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508805#comment-15508805 ] Rui Li commented on HIVE-14797: --- [~roncenzhao] your solution seems also OK and simpler. Would like to know

[jira] [Comment Edited] (HIVE-14797) reducer number estimating may lead to data skew

2016-09-20 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15508789#comment-15508789 ] Rui Li edited comment on HIVE-14797 at 9/21/16 5:16 AM: Hmm random prime won't

<    5   6   7   8   9   10   11   12   13   14   >