[jira] [Created] (HIVE-28121) Use direct SQL for transactional altering table parameter

2024-03-15 Thread Rui Li (Jira)
Rui Li created HIVE-28121: - Summary: Use direct SQL for transactional altering table parameter Key: HIVE-28121 URL: https://issues.apache.org/jira/browse/HIVE-28121 Project: Hive Issue Type: Bug

[jira] [Commented] (HIVE-26882) Allow transactional check of Table parameter before altering the Table

2024-03-14 Thread Rui Li (Jira)
[ https://issues.apache.org/jira/browse/HIVE-26882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17827089#comment-17827089 ] Rui Li commented on HIVE-26882: --- [~pvary] thanks for the pointer. I think it might work. Will run our e2e

[jira] [Commented] (HIVE-26882) Allow transactional check of Table parameter before altering the Table

2024-03-13 Thread Rui Li (Jira)
[ https://issues.apache.org/jira/browse/HIVE-26882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17826001#comment-17826001 ] Rui Li commented on HIVE-26882: --- Hi [~pvary], I tried writing direct SQL with JDO: {code:Java} String dml =

[jira] [Commented] (HIVE-26882) Allow transactional check of Table parameter before altering the Table

2024-03-12 Thread Rui Li (Jira)
[ https://issues.apache.org/jira/browse/HIVE-26882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17825557#comment-17825557 ] Rui Li commented on HIVE-26882: --- bq. I'm trying to suggest to use the direct SQL to update the metadata

[jira] [Commented] (HIVE-26882) Allow transactional check of Table parameter before altering the Table

2024-03-11 Thread Rui Li (Jira)
[ https://issues.apache.org/jira/browse/HIVE-26882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17825297#comment-17825297 ] Rui Li commented on HIVE-26882: --- bq. What do you see as an issue with that? The issue is each alter table

[jira] [Commented] (HIVE-26882) Allow transactional check of Table parameter before altering the Table

2024-03-08 Thread Rui Li (Jira)
[ https://issues.apache.org/jira/browse/HIVE-26882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17824800#comment-17824800 ] Rui Li commented on HIVE-26882: --- I found two options for MariaDB: # Use SERIALIZABLE level. # Use

[jira] [Commented] (HIVE-26882) Allow transactional check of Table parameter before altering the Table

2024-03-08 Thread Rui Li (Jira)
[ https://issues.apache.org/jira/browse/HIVE-26882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17824669#comment-17824669 ] Rui Li commented on HIVE-26882: --- In terms of performance, we tested both SERIALIZABLE, and REPEATABLE_READ

[jira] [Commented] (HIVE-26882) Allow transactional check of Table parameter before altering the Table

2024-03-08 Thread Rui Li (Jira)
[ https://issues.apache.org/jira/browse/HIVE-26882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17824659#comment-17824659 ] Rui Li commented on HIVE-26882: --- Hi [~pvary], I tested using docker image {{{}mysql:5.6.17{}}}. Tried both

[jira] [Commented] (HIVE-26882) Allow transactional check of Table parameter before altering the Table

2024-03-07 Thread Rui Li (Jira)
[ https://issues.apache.org/jira/browse/HIVE-26882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17824618#comment-17824618 ] Rui Li commented on HIVE-26882: --- I tried running these SQLs in each transaction, but none of them made the

[jira] [Commented] (HIVE-26882) Allow transactional check of Table parameter before altering the Table

2024-03-07 Thread Rui Li (Jira)
[ https://issues.apache.org/jira/browse/HIVE-26882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17824352#comment-17824352 ] Rui Li commented on HIVE-26882: --- I run our test again, with MariaDB at SERIALIZABLE level and the results

[jira] [Comment Edited] (HIVE-26882) Allow transactional check of Table parameter before altering the Table

2024-03-07 Thread Rui Li (Jira)
[ https://issues.apache.org/jira/browse/HIVE-26882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17824303#comment-17824303 ] Rui Li edited comment on HIVE-26882 at 3/7/24 8:34 AM: --- [~pvary] Thanks for your

[jira] [Commented] (HIVE-26882) Allow transactional check of Table parameter before altering the Table

2024-03-07 Thread Rui Li (Jira)
[ https://issues.apache.org/jira/browse/HIVE-26882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17824303#comment-17824303 ] Rui Li commented on HIVE-26882: --- [~pvary] Thanks for your comment. I ran some test for the scenarios

[jira] [Comment Edited] (HIVE-26882) Allow transactional check of Table parameter before altering the Table

2024-03-05 Thread Rui Li (Jira)
[ https://issues.apache.org/jira/browse/HIVE-26882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17823671#comment-17823671 ] Rui Li edited comment on HIVE-26882 at 3/5/24 3:50 PM: --- I tested again with MariaDB

[jira] [Commented] (HIVE-26882) Allow transactional check of Table parameter before altering the Table

2024-03-05 Thread Rui Li (Jira)
[ https://issues.apache.org/jira/browse/HIVE-26882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17823678#comment-17823678 ] Rui Li commented on HIVE-26882: --- And if this feature has only been verified with Postgres, perhaps we

[jira] [Comment Edited] (HIVE-26882) Allow transactional check of Table parameter before altering the Table

2024-03-05 Thread Rui Li (Jira)
[ https://issues.apache.org/jira/browse/HIVE-26882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17823671#comment-17823671 ] Rui Li edited comment on HIVE-26882 at 3/5/24 3:22 PM: --- I tested again with MariaDB

[jira] [Commented] (HIVE-26882) Allow transactional check of Table parameter before altering the Table

2024-03-05 Thread Rui Li (Jira)
[ https://issues.apache.org/jira/browse/HIVE-26882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17823671#comment-17823671 ] Rui Li commented on HIVE-26882: --- I tested again with MariaDB and there're fewer commit conflicts in HMS log

[jira] [Commented] (HIVE-26882) Allow transactional check of Table parameter before altering the Table

2024-03-01 Thread Rui Li (Jira)
[ https://issues.apache.org/jira/browse/HIVE-26882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17822776#comment-17822776 ] Rui Li commented on HIVE-26882: --- Hey [~pvary], thanks for your insights! I'll check whether the number of

[jira] [Commented] (HIVE-26882) Allow transactional check of Table parameter before altering the Table

2024-03-01 Thread Rui Li (Jira)
[ https://issues.apache.org/jira/browse/HIVE-26882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17822618#comment-17822618 ] Rui Li commented on HIVE-26882: --- Hi [~pvary], we're trying to improve performance of concurrent write to

[jira] [Assigned] (HIVE-7292) Hive on Spark

2021-10-19 Thread Rui Li (Jira)
[ https://issues.apache.org/jira/browse/HIVE-7292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li reassigned HIVE-7292: Assignee: Xuefu Zhang (was: Hui An) > Hive on Spark > - > > Key: HIVE-7292 >

[jira] [Commented] (HIVE-17133) NoSuchMethodError in Hadoop FileStatus.compareTo

2019-12-26 Thread Rui Li (Jira)
[ https://issues.apache.org/jira/browse/HIVE-17133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17003635#comment-17003635 ] Rui Li commented on HIVE-17133: --- While HADOOP-14683 has been fixed, it seems we still have to fix on our

[jira] [Updated] (HIVE-17133) NoSuchMethodError in Hadoop FileStatus.compareTo

2019-12-26 Thread Rui Li (Jira)
[ https://issues.apache.org/jira/browse/HIVE-17133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-17133: -- Status: Patch Available (was: Open) > NoSuchMethodError in Hadoop FileStatus.compareTo >

[jira] [Assigned] (HIVE-17133) NoSuchMethodError in Hadoop FileStatus.compareTo

2019-12-26 Thread Rui Li (Jira)
[ https://issues.apache.org/jira/browse/HIVE-17133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li reassigned HIVE-17133: - Assignee: Rui Li > NoSuchMethodError in Hadoop FileStatus.compareTo >

[jira] [Updated] (HIVE-17133) NoSuchMethodError in Hadoop FileStatus.compareTo

2019-12-26 Thread Rui Li (Jira)
[ https://issues.apache.org/jira/browse/HIVE-17133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-17133: -- Attachment: HIVE-17133.1.patch > NoSuchMethodError in Hadoop FileStatus.compareTo >

[jira] [Commented] (HIVE-22158) HMS Translation layer - Disallow non-ACID MANAGED tables.

2019-11-12 Thread Rui Li (Jira)
[ https://issues.apache.org/jira/browse/HIVE-22158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973065#comment-16973065 ] Rui Li commented on HIVE-22158: --- Thanks [~ngangam]. Setting {{metastore.warehouse.external.dir}} solves my

[jira] [Commented] (HIVE-22158) HMS Translation layer - Disallow non-ACID MANAGED tables.

2019-11-11 Thread Rui Li (Jira)
[ https://issues.apache.org/jira/browse/HIVE-22158?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971481#comment-16971481 ] Rui Li commented on HIVE-22158: --- Hi [~ngangam], I recently hit some issue with this change. When I create

[jira] [Commented] (HIVE-22053) Function name is not normalized when creating function

2019-07-31 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-22053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16897084#comment-16897084 ] Rui Li commented on HIVE-22053: --- Thanks [~pvary] :) > Function name is not normalized when creating

[jira] [Updated] (HIVE-22053) Function name is not normalized when creating function

2019-07-31 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-22053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-22053: -- Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Guess PR is

[jira] [Commented] (HIVE-22053) Function name is not normalized when creating function

2019-07-27 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-22053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16894617#comment-16894617 ] Rui Li commented on HIVE-22053: --- Thanks [~xuefuz] for the review. [~pvary] it'd be good if you can also

[jira] [Commented] (HIVE-22053) Function name is not normalized when creating function

2019-07-26 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-22053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16893703#comment-16893703 ] Rui Li commented on HIVE-22053: --- cc [~xuefuz] > Function name is not normalized when creating function >

[jira] [Updated] (HIVE-22053) Function name is not normalized when creating function

2019-07-26 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-22053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-22053: -- Status: Patch Available (was: Open) > Function name is not normalized when creating function >

[jira] [Updated] (HIVE-22053) Function name is not normalized when creating function

2019-07-26 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-22053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-22053: -- Attachment: HIVE-22053.1.patch > Function name is not normalized when creating function >

[jira] [Assigned] (HIVE-22053) Function name is not normalized when creating function

2019-07-26 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-22053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li reassigned HIVE-22053: - > Function name is not normalized when creating function >

[jira] [Updated] (HIVE-21111) ConditionalTask cannot be cast to MapRedTask

2019-03-24 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-2?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-2: -- Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) >

[jira] [Commented] (HIVE-21111) ConditionalTask cannot be cast to MapRedTask

2019-03-24 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-2?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16800361#comment-16800361 ] Rui Li commented on HIVE-2: --- [~qunyan] glad to know it helps. Closing this one. > ConditionalTask

[jira] [Commented] (HIVE-21111) ConditionalTask cannot be cast to MapRedTask

2019-03-22 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-2?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798876#comment-16798876 ] Rui Li commented on HIVE-2: --- I'm not sure whether it's valid to have a conditional task in another

[jira] [Commented] (HIVE-21111) ConditionalTask cannot be cast to MapRedTask

2019-03-14 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-2?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16793257#comment-16793257 ] Rui Li commented on HIVE-2: --- [~qunyan], could you provide a case that can reproduce this issue? >

[jira] [Commented] (HIVE-17020) Aggressive RS dedup can incorrectly remove OP tree branch

2018-12-18 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-17020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16724030#comment-16724030 ] Rui Li commented on HIVE-17020: --- Hi [~vgarg], is the failure just due to output diff? > Aggressive RS

[jira] [Commented] (HIVE-17020) Aggressive RS dedup can incorrectly remove OP tree branch

2018-12-11 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-17020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16718383#comment-16718383 ] Rui Li commented on HIVE-17020: --- +1 > Aggressive RS dedup can incorrectly remove OP tree branch >

[jira] [Commented] (HIVE-17020) Aggressive RS dedup can incorrectly remove OP tree branch

2018-12-11 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-17020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16716911#comment-16716911 ] Rui Li commented on HIVE-17020: --- {code} quotedid_smb.q,\ + reducesink_dedup.q\ resourceplan.q,\

[jira] [Updated] (HIVE-14557) Nullpointer When both SkewJoin and Mapjoin Enabled

2018-11-16 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-14557: -- Resolution: Fixed Fix Version/s: 4.0.0 Status: Resolved (was: Patch Available) Pushed to

[jira] [Updated] (HIVE-14557) Nullpointer When both SkewJoin and Mapjoin Enabled

2018-11-14 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-14557: -- Attachment: HIVE-14557.3.patch > Nullpointer When both SkewJoin and Mapjoin Enabled >

[jira] [Commented] (HIVE-14557) Nullpointer When both SkewJoin and Mapjoin Enabled

2018-11-14 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16687395#comment-16687395 ] Rui Li commented on HIVE-14557: --- Sure, attaching the same patch > Nullpointer When both SkewJoin and

[jira] [Commented] (HIVE-14557) Nullpointer When both SkewJoin and Mapjoin Enabled

2018-11-14 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16686476#comment-16686476 ] Rui Li commented on HIVE-14557: --- The metastore tests passed on my laptop. [~ychena], any idea what might be

[jira] [Commented] (HIVE-14557) Nullpointer When both SkewJoin and Mapjoin Enabled

2018-11-09 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681401#comment-16681401 ] Rui Li commented on HIVE-14557: --- Sorry about the late response. Attach an updated patch. > Nullpointer

[jira] [Updated] (HIVE-14557) Nullpointer When both SkewJoin and Mapjoin Enabled

2018-11-09 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-14557: -- Attachment: HIVE-14557.3.patch > Nullpointer When both SkewJoin and Mapjoin Enabled >

[jira] [Commented] (HIVE-14557) Nullpointer When both SkewJoin and Mapjoin Enabled

2018-08-06 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16569935#comment-16569935 ] Rui Li commented on HIVE-14557: --- [~aihuaxu], could you please verify whether the patch can solve your

[jira] [Commented] (HIVE-14557) Nullpointer When both SkewJoin and Mapjoin Enabled

2018-08-06 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16569817#comment-16569817 ] Rui Li commented on HIVE-14557: --- Upload a patch based on Nemon's solution. > Nullpointer When both

[jira] [Updated] (HIVE-14557) Nullpointer When both SkewJoin and Mapjoin Enabled

2018-08-06 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-14557: -- Attachment: HIVE-14557.2.patch > Nullpointer When both SkewJoin and Mapjoin Enabled >

[jira] [Assigned] (HIVE-14557) Nullpointer When both SkewJoin and Mapjoin Enabled

2018-08-06 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-14557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li reassigned HIVE-14557: - Assignee: Rui Li > Nullpointer When both SkewJoin and Mapjoin Enabled >

[jira] [Commented] (HIVE-20032) Don't serialize hashCode for repartitionAndSortWithinPartitions

2018-07-26 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-20032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16559162#comment-16559162 ] Rui Li commented on HIVE-20032: --- +1 > Don't serialize hashCode for repartitionAndSortWithinPartitions >

[jira] [Commented] (HIVE-20032) Don't serialize hashCode for repartitionAndSortWithinPartitions

2018-07-26 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-20032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558085#comment-16558085 ] Rui Li commented on HIVE-20032: --- [~stakiar], thanks for update. I can run my queries with latest patch. For

[jira] [Commented] (HIVE-20032) Don't serialize hashCode for repartitionAndSortWithinPartitions

2018-07-25 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-20032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555466#comment-16555466 ] Rui Li commented on HIVE-20032: --- bq. I originally thought that --jars would add the specified jars to the

[jira] [Commented] (HIVE-20032) Don't serialize hashCode for repartitionAndSortWithinPartitions

2018-07-25 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-20032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1612#comment-1612 ] Rui Li commented on HIVE-20032: --- I run a simple query in yarn-cluster mode with patch v8 and hit an issue:

[jira] [Commented] (HIVE-20032) Don't serialize hashCode for repartitionAndSortWithinPartitions

2018-07-24 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-20032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16554183#comment-16554183 ] Rui Li commented on HIVE-20032: --- Hi [~stakiar], I left some comments on RB. Meanwhile, could you explain

[jira] [Commented] (HIVE-20056) SparkPartitionPruner shouldn't be triggered by Spark tasks

2018-07-22 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-20056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16552297#comment-16552297 ] Rui Li commented on HIVE-20056: --- +1 > SparkPartitionPruner shouldn't be triggered by Spark tasks >

[jira] [Commented] (HIVE-20056) SparkPartitionPruner shouldn't be triggered by Spark tasks

2018-07-19 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-20056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16550173#comment-16550173 ] Rui Li commented on HIVE-20056: --- Thanks [~stakiar], nice catch. Patch looks good to me. Nit: {code} +if

[jira] [Commented] (HIVE-20032) Don't serialize hashCode when groupByShuffle and RDD cacheing is disabled

2018-07-18 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-20032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1654#comment-1654 ] Rui Li commented on HIVE-20032: --- Hi [~stakiar], kryo was relocated not just because Spark uses a different

[jira] [Commented] (HIVE-20032) Don't serialize hashCode when groupByShuffle and RDD cacheing is disabled

2018-07-11 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-20032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16541076#comment-16541076 ] Rui Li commented on HIVE-20032: --- Hi [~stakiar], thanks for working on this. There're other cases where RDD

[jira] [Commented] (HIVE-20007) Hive should carry out timestamp computations in UTC

2018-06-28 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-20007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16527010#comment-16527010 ] Rui Li commented on HIVE-20007: --- I see. Thanks [~jcamachorodriguez]. > Hive should carry out timestamp

[jira] [Commented] (HIVE-20007) Hive should carry out timestamp computations in UTC

2018-06-28 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-20007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16526249#comment-16526249 ] Rui Li commented on HIVE-20007: --- [~ashutoshc] suggested in HIVE-14412 that we should replace

[jira] [Commented] (HIVE-19671) Distribute by rand() can lead to data inconsistency

2018-06-24 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16521790#comment-16521790 ] Rui Li commented on HIVE-19671: --- We can check all RS and look for non-deterministic UDF in partition keys

[jira] [Commented] (HIVE-19671) Distribute by rand() can lead to data inconsistency

2018-06-21 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16519357#comment-16519357 ] Rui Li commented on HIVE-19671: --- [~xuefuz], I agree it's not trivial to solve this on Hive side. Maybe we

[jira] [Commented] (HIVE-19671) Distribute by rand() can lead to data inconsistency

2018-06-20 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16517846#comment-16517846 ] Rui Li commented on HIVE-19671: --- [~xuefuz], thanks for your input. I think rand(seed) may not work if the

[jira] [Updated] (HIVE-19895) The unique ID in SparkPartitionPruningSinkOperator is no longer needed

2018-06-14 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-19895: -- Description: With HIVE-19237 in place, seems we don't need to maintain a unique ID in > The unique ID in

[jira] [Updated] (HIVE-19895) The unique ID in SparkPartitionPruningSinkOperator is no longer needed

2018-06-14 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-19895: -- Description: With HIVE-19237 in place, seems we don't need to maintain a unique ID in

[jira] [Commented] (HIVE-18533) Add option to use InProcessLauncher to submit spark jobs

2018-06-06 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16502983#comment-16502983 ] Rui Li commented on HIVE-18533: --- [~stakiar], please take a look at the Yetus report. I think some warnings

[jira] [Commented] (HIVE-16391) Publish proper Hive 1.2 jars (without including all dependencies in uber jar)

2018-06-05 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501662#comment-16501662 ] Rui Li commented on HIVE-16391: --- [~jerryshao], assigning this to you so you should have permission to

[jira] [Assigned] (HIVE-16391) Publish proper Hive 1.2 jars (without including all dependencies in uber jar)

2018-06-05 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-16391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li reassigned HIVE-16391: - Assignee: Saisai Shao > Publish proper Hive 1.2 jars (without including all dependencies in uber jar) >

[jira] [Commented] (HIVE-19602) Refactor inplace progress code in Hive-on-spark progress monitor to use ProgressMonitor instance

2018-06-02 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16498996#comment-16498996 ] Rui Li commented on HIVE-19602: --- Thanks for pinging me. +1 pending test > Refactor inplace progress code

[jira] [Commented] (HIVE-19237) Only use an operatorId once in a plan

2018-05-30 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16495191#comment-16495191 ] Rui Li commented on HIVE-19237: --- Thanks [~kgyrtkirk], I think it works. For this particular issue, seems

[jira] [Commented] (HIVE-19237) Only use an operatorId once in a plan

2018-05-30 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494782#comment-16494782 ] Rui Li commented on HIVE-19237: --- [~kgyrtkirk], I agree storing origin ID is not ideal and I'd be happy to

[jira] [Commented] (HIVE-18533) Add option to use InProcessLauncher to submit spark jobs

2018-05-29 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494623#comment-16494623 ] Rui Li commented on HIVE-18533: --- [~stakiar] sorry I overlooked your previous comments. +1 pending tests bq.

[jira] [Commented] (HIVE-19237) Only use an operatorId once in a plan

2018-05-29 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16494616#comment-16494616 ] Rui Li commented on HIVE-19237: --- Hey [~kgyrtkirk], how about we do it this way: # The old id should only be

[jira] [Commented] (HIVE-19237) Only use an operatorId once in a plan

2018-05-29 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493479#comment-16493479 ] Rui Li commented on HIVE-19237: --- Hi [~kgyrtkirk], my concern is there can be multiple logically-same

[jira] [Commented] (HIVE-19237) Only use an operatorId once in a plan

2018-05-29 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493443#comment-16493443 ] Rui Li commented on HIVE-19237: --- The spark tests failed because some code relies on the operatorId to

[jira] [Commented] (HIVE-19671) Distribute by rand() can lead to data inconsistency

2018-05-28 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16493065#comment-16493065 ] Rui Li commented on HIVE-19671: --- Verified the issue only happens when there're task retries. I can think of

[jira] [Assigned] (HIVE-19671) Distribute by rand() can lead to data inconsistency

2018-05-28 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li reassigned HIVE-19671: - Assignee: Rui Li > Distribute by rand() can lead to data inconsistency >

[jira] [Commented] (HIVE-19671) Distribute by rand() can lead to data inconsistency

2018-05-23 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16487148#comment-16487148 ] Rui Li commented on HIVE-19671: --- I haven't verified it but my guess is the issue happens with task failover.

[jira] [Updated] (HIVE-19671) Distribute by rand() can lead to data inconsistency

2018-05-23 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-19671: -- Description: Noticed the following queries can give different results: {code} select count(*) from tbl; select

[jira] [Commented] (HIVE-18533) Add option to use InProcessLauncher to submit spark jobs

2018-05-14 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474138#comment-16474138 ] Rui Li commented on HIVE-18533: --- Seems the new qtest has unstable output. Please look into it. Some more

[jira] [Commented] (HIVE-18533) Add option to use InProcessLauncher to submit spark jobs

2018-05-12 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16473112#comment-16473112 ] Rui Li commented on HIVE-18533: --- Thanks for the update [~stakiar]. There's some issue with

[jira] [Commented] (HIVE-18533) Add option to use InProcessLauncher to submit spark jobs

2018-05-07 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466816#comment-16466816 ] Rui Li commented on HIVE-18533: --- Hi [~stakiar], my concern for SparkLauncherFuture is mainly about the

[jira] [Commented] (HIVE-19439) MapWork shouldn't be reused when Spark task fails during initialization

2018-05-07 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466775#comment-16466775 ] Rui Li commented on HIVE-19439: --- BTW, the hash table is loaded when we init the dummy operators

[jira] [Commented] (HIVE-19439) MapWork shouldn't be reused when Spark task fails during initialization

2018-05-07 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466772#comment-16466772 ] Rui Li commented on HIVE-19439: --- Hi [~vihangk1], the task is retried by Spark, and it calls

[jira] [Commented] (HIVE-19388) ClassCastException during VectorMapJoinCommonOperator initialization

2018-05-07 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465911#comment-16465911 ] Rui Li commented on HIVE-19388: --- [~vihangk1], thanks for fixing this. The change looks good. +1 As for your

[jira] [Commented] (HIVE-18533) Add option to use InProcessLauncher to submit spark jobs

2018-05-07 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465624#comment-16465624 ] Rui Li commented on HIVE-18533: --- Hi [~stakiar], for SparkLauncherSparkClient, how about we use a thread to

[jira] [Commented] (HIVE-18533) Add option to use InProcessLauncher to submit spark jobs

2018-05-03 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462612#comment-16462612 ] Rui Li commented on HIVE-18533: --- Hi [~stakiar], could you explain why we now need to monitor the driver with

[jira] [Commented] (HIVE-18533) Add option to use InProcessLauncher to submit spark jobs

2018-05-02 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461318#comment-16461318 ] Rui Li commented on HIVE-18533: --- [~stakiar], thanks for working on this and sorry for the delay. I had a

[jira] [Commented] (HIVE-19316) StatsTask fails due to ClassCastException

2018-04-28 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457537#comment-16457537 ] Rui Li commented on HIVE-19316: --- My understanding is we can't send LongColumnStatsDataInspector over thrift.

[jira] [Commented] (HIVE-19316) StatsTask fails due to ClassCastException

2018-04-26 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16454296#comment-16454296 ] Rui Li commented on HIVE-19316: --- The issue only happens with remote HMS. We're expecting

[jira] [Commented] (HIVE-19266) Use UDFs in Hive-On-Spark complains Unable to find class Exception regarding kryo

2018-04-26 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-19266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453553#comment-16453553 ] Rui Li commented on HIVE-19266: --- Hey [~jason4z], thanks for reporting the this. The issue should have been

[jira] [Updated] (HIVE-17193) HoS: don't combine map works that are targets of different DPPs

2018-04-25 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-17193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-17193: -- Resolution: Fixed Fix Version/s: 3.1.0 Status: Resolved (was: Patch Available) Pushed to

[jira] [Commented] (HIVE-18862) qfiles: prepare .q files for using datasets

2018-04-24 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16451595#comment-16451595 ] Rui Li commented on HIVE-18862: --- It's great if we can have this in branch-3, which makes it easier to cherry

[jira] [Updated] (HIVE-17193) HoS: don't combine map works that are targets of different DPPs

2018-04-24 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-17193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-17193: -- Attachment: HIVE-17193.5.patch > HoS: don't combine map works that are targets of different DPPs >

[jira] [Commented] (HIVE-17193) HoS: don't combine map works that are targets of different DPPs

2018-04-21 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-17193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16447064#comment-16447064 ] Rui Li commented on HIVE-17193: --- The failures are not related. [~stakiar], could you take a look at the

[jira] [Updated] (HIVE-17193) HoS: don't combine map works that are targets of different DPPs

2018-04-17 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-17193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-17193: -- Attachment: HIVE-17193.4.patch > HoS: don't combine map works that are targets of different DPPs >

[jira] [Commented] (HIVE-18831) Differentiate errors that are thrown by Spark tasks

2018-04-10 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16432179#comment-16432179 ] Rui Li commented on HIVE-18831: --- +1 pending test > Differentiate errors that are thrown by Spark tasks >

[jira] [Commented] (HIVE-18831) Differentiate errors that are thrown by Spark tasks

2018-04-08 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429750#comment-16429750 ] Rui Li commented on HIVE-18831: --- Thanks [~stakiar] for the update. The patch looks good to me overall,

[jira] [Updated] (HIVE-17193) HoS: don't combine map works that are targets of different DPPs

2018-04-04 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-17193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-17193: -- Attachment: HIVE-17193.3.patch > HoS: don't combine map works that are targets of different DPPs >

[jira] [Updated] (HIVE-18955) HoS: Unable to create Channel from class NioServerSocketChannel

2018-04-03 Thread Rui Li (JIRA)
[ https://issues.apache.org/jira/browse/HIVE-18955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li updated HIVE-18955: -- Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Pushed to

  1   2   3   4   5   6   7   8   9   10   >