[jira] [Resolved] (HIVE-25446) Wrong exception thrown if capacity<=0
[ https://issues.apache.org/jira/browse/HIVE-25446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan resolved HIVE-25446. - Resolution: Fixed PR merged to master. Thanks [~ashish-kumar-sharma] for the patch and [~gupta.nikhil0007]/[~adeshrao] for the review! > Wrong exception thrown if capacity<=0 > - > > Key: HIVE-25446 > URL: https://issues.apache.org/jira/browse/HIVE-25446 > Project: Hive > Issue Type: Sub-task > Components: Vectorization >Affects Versions: 3.1.2 >Reporter: Matt McCline >Assignee: Ashish Sharma >Priority: Trivial > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Encountered this in a very large query: > Caused by: java.lang.AssertionError: Capacity must be a power of two > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTable.validateCapacity(VectorMapJoinFastHashTable.java:60) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTable.(VectorMapJoinFastHashTable.java:77) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastBytesHashTable.(VectorMapJoinFastBytesHashTable.java:132) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastBytesHashMap.(VectorMapJoinFastBytesHashMap.java:166) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastStringHashMap.(VectorMapJoinFastStringHashMap.java:43) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastTableContainer.createHashTable(VectorMapJoinFastTableContainer.java:137) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastTableContainer.(VectorMapJoinFastTableContainer.java:86) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTableLoader.load(VectorMapJoinFastHashTableLoader.java:122) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTableInternal(MapJoinOperator.java:344) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:413) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.lambda$initializeOp$0(MapJoinOperator.java:215) > at > org.apache.hadoop.hive.ql.exec.tez.ObjectCache.retrieve(ObjectCache.java:96) > at > org.apache.hadoop.hive.ql.exec.tez.ObjectCache$1.call(ObjectCache.java:113) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-25446) Wrong exception thrown if capacity<=0
[ https://issues.apache.org/jira/browse/HIVE-25446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-25446: Affects Version/s: 3.1.2 (was: 3.1.1) > Wrong exception thrown if capacity<=0 > - > > Key: HIVE-25446 > URL: https://issues.apache.org/jira/browse/HIVE-25446 > Project: Hive > Issue Type: Sub-task > Components: Vectorization >Affects Versions: 3.1.2 >Reporter: Matt McCline >Assignee: Ashish Sharma >Priority: Trivial > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Encountered this in a very large query: > Caused by: java.lang.AssertionError: Capacity must be a power of two > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTable.validateCapacity(VectorMapJoinFastHashTable.java:60) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTable.(VectorMapJoinFastHashTable.java:77) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastBytesHashTable.(VectorMapJoinFastBytesHashTable.java:132) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastBytesHashMap.(VectorMapJoinFastBytesHashMap.java:166) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastStringHashMap.(VectorMapJoinFastStringHashMap.java:43) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastTableContainer.createHashTable(VectorMapJoinFastTableContainer.java:137) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastTableContainer.(VectorMapJoinFastTableContainer.java:86) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTableLoader.load(VectorMapJoinFastHashTableLoader.java:122) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTableInternal(MapJoinOperator.java:344) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:413) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.lambda$initializeOp$0(MapJoinOperator.java:215) > at > org.apache.hadoop.hive.ql.exec.tez.ObjectCache.retrieve(ObjectCache.java:96) > at > org.apache.hadoop.hive.ql.exec.tez.ObjectCache$1.call(ObjectCache.java:113) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-25446) Wrong exception thrown if capacity<=0
[ https://issues.apache.org/jira/browse/HIVE-25446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-25446: Component/s: Vectorization > Wrong exception thrown if capacity<=0 > - > > Key: HIVE-25446 > URL: https://issues.apache.org/jira/browse/HIVE-25446 > Project: Hive > Issue Type: Sub-task > Components: Vectorization >Affects Versions: 3.1.1 >Reporter: Matt McCline >Assignee: Ashish Sharma >Priority: Trivial > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Encountered this in a very large query: > Caused by: java.lang.AssertionError: Capacity must be a power of two > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTable.validateCapacity(VectorMapJoinFastHashTable.java:60) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTable.(VectorMapJoinFastHashTable.java:77) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastBytesHashTable.(VectorMapJoinFastBytesHashTable.java:132) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastBytesHashMap.(VectorMapJoinFastBytesHashMap.java:166) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastStringHashMap.(VectorMapJoinFastStringHashMap.java:43) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastTableContainer.createHashTable(VectorMapJoinFastTableContainer.java:137) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastTableContainer.(VectorMapJoinFastTableContainer.java:86) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTableLoader.load(VectorMapJoinFastHashTableLoader.java:122) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTableInternal(MapJoinOperator.java:344) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:413) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.lambda$initializeOp$0(MapJoinOperator.java:215) > at > org.apache.hadoop.hive.ql.exec.tez.ObjectCache.retrieve(ObjectCache.java:96) > at > org.apache.hadoop.hive.ql.exec.tez.ObjectCache$1.call(ObjectCache.java:113) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25446) Wrong exception thrown if capacity<=0
[ https://issues.apache.org/jira/browse/HIVE-25446?focusedWorklogId=746341=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-746341 ] ASF GitHub Bot logged work on HIVE-25446: - Author: ASF GitHub Bot Created on: 23/Mar/22 04:26 Start Date: 23/Mar/22 04:26 Worklog Time Spent: 10m Work Description: sankarh merged pull request #3092: URL: https://github.com/apache/hive/pull/3092 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 746341) Time Spent: 0.5h (was: 20m) > Wrong exception thrown if capacity<=0 > - > > Key: HIVE-25446 > URL: https://issues.apache.org/jira/browse/HIVE-25446 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.1.1 >Reporter: Matt McCline >Assignee: Ashish Sharma >Priority: Trivial > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Encountered this in a very large query: > Caused by: java.lang.AssertionError: Capacity must be a power of two > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTable.validateCapacity(VectorMapJoinFastHashTable.java:60) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTable.(VectorMapJoinFastHashTable.java:77) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastBytesHashTable.(VectorMapJoinFastBytesHashTable.java:132) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastBytesHashMap.(VectorMapJoinFastBytesHashMap.java:166) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastStringHashMap.(VectorMapJoinFastStringHashMap.java:43) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastTableContainer.createHashTable(VectorMapJoinFastTableContainer.java:137) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastTableContainer.(VectorMapJoinFastTableContainer.java:86) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTableLoader.load(VectorMapJoinFastHashTableLoader.java:122) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTableInternal(MapJoinOperator.java:344) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:413) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.lambda$initializeOp$0(MapJoinOperator.java:215) > at > org.apache.hadoop.hive.ql.exec.tez.ObjectCache.retrieve(ObjectCache.java:96) > at > org.apache.hadoop.hive.ql.exec.tez.ObjectCache$1.call(ObjectCache.java:113) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-25446) Wrong exception thrown if capacity<=0
[ https://issues.apache.org/jira/browse/HIVE-25446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-25446: Summary: Wrong exception thrown if capacity<=0 (was: Wrong execption thrown if capacity<=0) > Wrong exception thrown if capacity<=0 > - > > Key: HIVE-25446 > URL: https://issues.apache.org/jira/browse/HIVE-25446 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.1.1 >Reporter: Matt McCline >Assignee: Ashish Sharma >Priority: Trivial > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Encountered this in a very large query: > Caused by: java.lang.AssertionError: Capacity must be a power of two > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTable.validateCapacity(VectorMapJoinFastHashTable.java:60) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTable.(VectorMapJoinFastHashTable.java:77) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastBytesHashTable.(VectorMapJoinFastBytesHashTable.java:132) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastBytesHashMap.(VectorMapJoinFastBytesHashMap.java:166) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastStringHashMap.(VectorMapJoinFastStringHashMap.java:43) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastTableContainer.createHashTable(VectorMapJoinFastTableContainer.java:137) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastTableContainer.(VectorMapJoinFastTableContainer.java:86) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTableLoader.load(VectorMapJoinFastHashTableLoader.java:122) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTableInternal(MapJoinOperator.java:344) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:413) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.lambda$initializeOp$0(MapJoinOperator.java:215) > at > org.apache.hadoop.hive.ql.exec.tez.ObjectCache.retrieve(ObjectCache.java:96) > at > org.apache.hadoop.hive.ql.exec.tez.ObjectCache$1.call(ObjectCache.java:113) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-26058) Choose meaningful names for the Metastore pool threads
[
https://issues.apache.org/jira/browse/HIVE-26058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihua Deng updated HIVE-26058:
---
Fix Version/s: 4.0.0-alpha-2
(was: 4.0.0)
> Choose meaningful names for the Metastore pool threads
> --
>
> Key: HIVE-26058
> URL: https://issues.apache.org/jira/browse/HIVE-26058
> Project: Hive
> Issue Type: Improvement
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-2
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Due to TThreadPoolServer#createDefaultExecutorService setting the thread name
> by
> {code:java}
> thread.setName("TThreadPoolServer WorkerProcess-%d"); {code}
> The logger output the thread name like:
> {noformat}
> [TThreadPoolServer WorkerProcess-%d] utils.FileUtils: Renaming
> pfile:/{noformat}
> , which makes it hard to identify and debug a thread.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (HIVE-26058) Choose meaningful names for the Metastore pool threads
[
https://issues.apache.org/jira/browse/HIVE-26058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihua Deng resolved HIVE-26058.
Fix Version/s: 4.0.0
Assignee: Zhihua Deng
Resolution: Fixed
> Choose meaningful names for the Metastore pool threads
> --
>
> Key: HIVE-26058
> URL: https://issues.apache.org/jira/browse/HIVE-26058
> Project: Hive
> Issue Type: Improvement
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Due to TThreadPoolServer#createDefaultExecutorService setting the thread name
> by
> {code:java}
> thread.setName("TThreadPoolServer WorkerProcess-%d"); {code}
> The logger output the thread name like:
> {noformat}
> [TThreadPoolServer WorkerProcess-%d] utils.FileUtils: Renaming
> pfile:/{noformat}
> , which makes it hard to identify and debug a thread.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HIVE-26058) Choose meaningful names for the Metastore pool threads
[
https://issues.apache.org/jira/browse/HIVE-26058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17511006#comment-17511006
]
Zhihua Deng commented on HIVE-26058:
Merged to master. Thank you for the feedback and review, [~zabetak]!
> Choose meaningful names for the Metastore pool threads
> --
>
> Key: HIVE-26058
> URL: https://issues.apache.org/jira/browse/HIVE-26058
> Project: Hive
> Issue Type: Improvement
>Reporter: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Due to TThreadPoolServer#createDefaultExecutorService setting the thread name
> by
> {code:java}
> thread.setName("TThreadPoolServer WorkerProcess-%d"); {code}
> The logger output the thread name like:
> {noformat}
> [TThreadPoolServer WorkerProcess-%d] utils.FileUtils: Renaming
> pfile:/{noformat}
> , which makes it hard to identify and debug a thread.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-26058) Choose meaningful names for the Metastore pool threads
[
https://issues.apache.org/jira/browse/HIVE-26058?focusedWorklogId=746336=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-746336
]
ASF GitHub Bot logged work on HIVE-26058:
-
Author: ASF GitHub Bot
Created on: 23/Mar/22 04:03
Start Date: 23/Mar/22 04:03
Worklog Time Spent: 10m
Work Description: dengzhhu653 merged pull request #3125:
URL: https://github.com/apache/hive/pull/3125
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 746336)
Time Spent: 20m (was: 10m)
> Choose meaningful names for the Metastore pool threads
> --
>
> Key: HIVE-26058
> URL: https://issues.apache.org/jira/browse/HIVE-26058
> Project: Hive
> Issue Type: Improvement
>Reporter: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Due to TThreadPoolServer#createDefaultExecutorService setting the thread name
> by
> {code:java}
> thread.setName("TThreadPoolServer WorkerProcess-%d"); {code}
> The logger output the thread name like:
> {noformat}
> [TThreadPoolServer WorkerProcess-%d] utils.FileUtils: Renaming
> pfile:/{noformat}
> , which makes it hard to identify and debug a thread.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (HIVE-26043) Use constraint info when creating RexNodes
[
https://issues.apache.org/jira/browse/HIVE-26043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa resolved HIVE-26043.
---
Resolution: Fixed
Pushed to master. Thanks [~zabetak] for review.
> Use constraint info when creating RexNodes
> --
>
> Key: HIVE-26043
> URL: https://issues.apache.org/jira/browse/HIVE-26043
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Prior HIVE-23100 Not null constraints affected newly created RexNode type
> nullability.
> Nullability enables the subquery rewrite algorithm to generate more optimal
> plan.
> [https://github.com/apache/hive/blob/1213ad3f0ae0e21e7519dc28b8b6d1401cdd1441/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSubQueryRemoveRule.java#L324]
> Example:
> {code:java}
> explain cbo
> select ws_sales_price
> from web_sales, customer, item
> where ws_bill_customer_sk = c_customer_sk
> and ws_item_sk = i_item_sk
> and ( c_customer_sk = 1
> or
> i_item_id in (select i_item_id
> from item
> where i_item_sk in (2, 3)
> )
> );
> {code}
> Without not null constraints
> {code:java}
> HiveProject(ws_sales_price=[$2])
> HiveFilter(condition=[OR(AND(<>($6, 0), IS NOT NULL($8)), =($3, 1))])
> HiveProject(ws_item_sk=[$0], ws_bill_customer_sk=[$1],
> ws_sales_price=[$2], c_customer_sk=[$8], i_item_sk=[$3], i_item_id=[$4],
> c=[$5], i_item_id0=[$6], literalTrue=[$7])
> HiveJoin(condition=[=($1, $8)], joinType=[inner], algorithm=[none],
> cost=[not available])
> HiveJoin(condition=[=($0, $3)], joinType=[inner], algorithm=[none],
> cost=[not available])
> HiveProject(ws_item_sk=[$2], ws_bill_customer_sk=[$3],
> ws_sales_price=[$20])
> HiveFilter(condition=[IS NOT NULL($3)])
> HiveTableScan(table=[[default, web_sales]],
> table:alias=[web_sales])
> HiveJoin(condition=[=($1, $3)], joinType=[left], algorithm=[none],
> cost=[not available])
> HiveJoin(condition=[true], joinType=[inner], algorithm=[none],
> cost=[not available])
> HiveProject(i_item_sk=[$0], i_item_id=[$1])
> HiveTableScan(table=[[default, item]], table:alias=[item])
> HiveProject(c=[$0])
> HiveAggregate(group=[{}], c=[COUNT()])
> HiveFilter(condition=[IN($0, 2:BIGINT, 3:BIGINT)])
> HiveTableScan(table=[[default, item]], table:alias=[item])
> HiveProject(i_item_id=[$0], literalTrue=[true])
> HiveAggregate(group=[{1}])
> HiveFilter(condition=[IN($0, 2:BIGINT, 3:BIGINT)])
> HiveTableScan(table=[[default, item]], table:alias=[item])
> HiveProject(c_customer_sk=[$0])
> HiveTableScan(table=[[default, customer]], table:alias=[customer])
> {code}
> With not null constraints
> {code:java}
> HiveProject(ws_sales_price=[$2])
> HiveFilter(condition=[OR(IS NOT NULL($7), =($3, 1))])
> HiveProject(ws_item_sk=[$0], ws_bill_customer_sk=[$1],
> ws_sales_price=[$2], c_customer_sk=[$7], i_item_sk=[$3], i_item_id=[$4],
> i_item_id0=[$5], literalTrue=[$6])
> HiveJoin(condition=[=($1, $7)], joinType=[inner], algorithm=[none],
> cost=[not available])
> HiveJoin(condition=[=($0, $3)], joinType=[inner], algorithm=[none],
> cost=[not available])
> HiveProject(ws_item_sk=[$2], ws_bill_customer_sk=[$3],
> ws_sales_price=[$20])
> HiveFilter(condition=[IS NOT NULL($3)])
> HiveTableScan(table=[[default, web_sales]],
> table:alias=[web_sales])
> HiveJoin(condition=[=($1, $2)], joinType=[left], algorithm=[none],
> cost=[not available])
> HiveProject(i_item_sk=[$0], i_item_id=[$1])
> HiveTableScan(table=[[default, item]], table:alias=[item])
> HiveProject(i_item_id=[$0], literalTrue=[true])
> HiveAggregate(group=[{1}])
> HiveFilter(condition=[IN($0, 2:BIGINT, 3:BIGINT)])
> HiveTableScan(table=[[default, item]], table:alias=[item])
> HiveProject(c_customer_sk=[$0])
> HiveTableScan(table=[[default, customer]], table:alias=[customer])
> {code}
> In the first plan when not null constraints was ignored there is an extra
> {{item}} table join without join condition:
> {code:java}
> HiveJoin(condition=[true], joinType=[inner], algorithm=[none],
> cost=[not available])
> HiveProject(i_item_sk=[$0], i_item_id=[$1])
>
[jira] [Work logged] (HIVE-26043) Use constraint info when creating RexNodes
[
https://issues.apache.org/jira/browse/HIVE-26043?focusedWorklogId=746315=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-746315
]
ASF GitHub Bot logged work on HIVE-26043:
-
Author: ASF GitHub Bot
Created on: 23/Mar/22 03:02
Start Date: 23/Mar/22 03:02
Worklog Time Spent: 10m
Work Description: kasakrisz closed pull request #3120:
URL: https://github.com/apache/hive/pull/3120
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 746315)
Time Spent: 1h 40m (was: 1.5h)
> Use constraint info when creating RexNodes
> --
>
> Key: HIVE-26043
> URL: https://issues.apache.org/jira/browse/HIVE-26043
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Prior HIVE-23100 Not null constraints affected newly created RexNode type
> nullability.
> Nullability enables the subquery rewrite algorithm to generate more optimal
> plan.
> [https://github.com/apache/hive/blob/1213ad3f0ae0e21e7519dc28b8b6d1401cdd1441/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSubQueryRemoveRule.java#L324]
> Example:
> {code:java}
> explain cbo
> select ws_sales_price
> from web_sales, customer, item
> where ws_bill_customer_sk = c_customer_sk
> and ws_item_sk = i_item_sk
> and ( c_customer_sk = 1
> or
> i_item_id in (select i_item_id
> from item
> where i_item_sk in (2, 3)
> )
> );
> {code}
> Without not null constraints
> {code:java}
> HiveProject(ws_sales_price=[$2])
> HiveFilter(condition=[OR(AND(<>($6, 0), IS NOT NULL($8)), =($3, 1))])
> HiveProject(ws_item_sk=[$0], ws_bill_customer_sk=[$1],
> ws_sales_price=[$2], c_customer_sk=[$8], i_item_sk=[$3], i_item_id=[$4],
> c=[$5], i_item_id0=[$6], literalTrue=[$7])
> HiveJoin(condition=[=($1, $8)], joinType=[inner], algorithm=[none],
> cost=[not available])
> HiveJoin(condition=[=($0, $3)], joinType=[inner], algorithm=[none],
> cost=[not available])
> HiveProject(ws_item_sk=[$2], ws_bill_customer_sk=[$3],
> ws_sales_price=[$20])
> HiveFilter(condition=[IS NOT NULL($3)])
> HiveTableScan(table=[[default, web_sales]],
> table:alias=[web_sales])
> HiveJoin(condition=[=($1, $3)], joinType=[left], algorithm=[none],
> cost=[not available])
> HiveJoin(condition=[true], joinType=[inner], algorithm=[none],
> cost=[not available])
> HiveProject(i_item_sk=[$0], i_item_id=[$1])
> HiveTableScan(table=[[default, item]], table:alias=[item])
> HiveProject(c=[$0])
> HiveAggregate(group=[{}], c=[COUNT()])
> HiveFilter(condition=[IN($0, 2:BIGINT, 3:BIGINT)])
> HiveTableScan(table=[[default, item]], table:alias=[item])
> HiveProject(i_item_id=[$0], literalTrue=[true])
> HiveAggregate(group=[{1}])
> HiveFilter(condition=[IN($0, 2:BIGINT, 3:BIGINT)])
> HiveTableScan(table=[[default, item]], table:alias=[item])
> HiveProject(c_customer_sk=[$0])
> HiveTableScan(table=[[default, customer]], table:alias=[customer])
> {code}
> With not null constraints
> {code:java}
> HiveProject(ws_sales_price=[$2])
> HiveFilter(condition=[OR(IS NOT NULL($7), =($3, 1))])
> HiveProject(ws_item_sk=[$0], ws_bill_customer_sk=[$1],
> ws_sales_price=[$2], c_customer_sk=[$7], i_item_sk=[$3], i_item_id=[$4],
> i_item_id0=[$5], literalTrue=[$6])
> HiveJoin(condition=[=($1, $7)], joinType=[inner], algorithm=[none],
> cost=[not available])
> HiveJoin(condition=[=($0, $3)], joinType=[inner], algorithm=[none],
> cost=[not available])
> HiveProject(ws_item_sk=[$2], ws_bill_customer_sk=[$3],
> ws_sales_price=[$20])
> HiveFilter(condition=[IS NOT NULL($3)])
> HiveTableScan(table=[[default, web_sales]],
> table:alias=[web_sales])
> HiveJoin(condition=[=($1, $2)], joinType=[left], algorithm=[none],
> cost=[not available])
> HiveProject(i_item_sk=[$0], i_item_id=[$1])
> HiveTableScan(table=[[default, item]], table:alias=[item])
> HiveProject(i_item_id=[$0],
[jira] [Work logged] (HIVE-26043) Use constraint info when creating RexNodes
[
https://issues.apache.org/jira/browse/HIVE-26043?focusedWorklogId=746314=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-746314
]
ASF GitHub Bot logged work on HIVE-26043:
-
Author: ASF GitHub Bot
Created on: 23/Mar/22 03:01
Start Date: 23/Mar/22 03:01
Worklog Time Spent: 10m
Work Description: kasakrisz merged pull request #3121:
URL: https://github.com/apache/hive/pull/3121
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 746314)
Time Spent: 1.5h (was: 1h 20m)
> Use constraint info when creating RexNodes
> --
>
> Key: HIVE-26043
> URL: https://issues.apache.org/jira/browse/HIVE-26043
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Prior HIVE-23100 Not null constraints affected newly created RexNode type
> nullability.
> Nullability enables the subquery rewrite algorithm to generate more optimal
> plan.
> [https://github.com/apache/hive/blob/1213ad3f0ae0e21e7519dc28b8b6d1401cdd1441/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSubQueryRemoveRule.java#L324]
> Example:
> {code:java}
> explain cbo
> select ws_sales_price
> from web_sales, customer, item
> where ws_bill_customer_sk = c_customer_sk
> and ws_item_sk = i_item_sk
> and ( c_customer_sk = 1
> or
> i_item_id in (select i_item_id
> from item
> where i_item_sk in (2, 3)
> )
> );
> {code}
> Without not null constraints
> {code:java}
> HiveProject(ws_sales_price=[$2])
> HiveFilter(condition=[OR(AND(<>($6, 0), IS NOT NULL($8)), =($3, 1))])
> HiveProject(ws_item_sk=[$0], ws_bill_customer_sk=[$1],
> ws_sales_price=[$2], c_customer_sk=[$8], i_item_sk=[$3], i_item_id=[$4],
> c=[$5], i_item_id0=[$6], literalTrue=[$7])
> HiveJoin(condition=[=($1, $8)], joinType=[inner], algorithm=[none],
> cost=[not available])
> HiveJoin(condition=[=($0, $3)], joinType=[inner], algorithm=[none],
> cost=[not available])
> HiveProject(ws_item_sk=[$2], ws_bill_customer_sk=[$3],
> ws_sales_price=[$20])
> HiveFilter(condition=[IS NOT NULL($3)])
> HiveTableScan(table=[[default, web_sales]],
> table:alias=[web_sales])
> HiveJoin(condition=[=($1, $3)], joinType=[left], algorithm=[none],
> cost=[not available])
> HiveJoin(condition=[true], joinType=[inner], algorithm=[none],
> cost=[not available])
> HiveProject(i_item_sk=[$0], i_item_id=[$1])
> HiveTableScan(table=[[default, item]], table:alias=[item])
> HiveProject(c=[$0])
> HiveAggregate(group=[{}], c=[COUNT()])
> HiveFilter(condition=[IN($0, 2:BIGINT, 3:BIGINT)])
> HiveTableScan(table=[[default, item]], table:alias=[item])
> HiveProject(i_item_id=[$0], literalTrue=[true])
> HiveAggregate(group=[{1}])
> HiveFilter(condition=[IN($0, 2:BIGINT, 3:BIGINT)])
> HiveTableScan(table=[[default, item]], table:alias=[item])
> HiveProject(c_customer_sk=[$0])
> HiveTableScan(table=[[default, customer]], table:alias=[customer])
> {code}
> With not null constraints
> {code:java}
> HiveProject(ws_sales_price=[$2])
> HiveFilter(condition=[OR(IS NOT NULL($7), =($3, 1))])
> HiveProject(ws_item_sk=[$0], ws_bill_customer_sk=[$1],
> ws_sales_price=[$2], c_customer_sk=[$7], i_item_sk=[$3], i_item_id=[$4],
> i_item_id0=[$5], literalTrue=[$6])
> HiveJoin(condition=[=($1, $7)], joinType=[inner], algorithm=[none],
> cost=[not available])
> HiveJoin(condition=[=($0, $3)], joinType=[inner], algorithm=[none],
> cost=[not available])
> HiveProject(ws_item_sk=[$2], ws_bill_customer_sk=[$3],
> ws_sales_price=[$20])
> HiveFilter(condition=[IS NOT NULL($3)])
> HiveTableScan(table=[[default, web_sales]],
> table:alias=[web_sales])
> HiveJoin(condition=[=($1, $2)], joinType=[left], algorithm=[none],
> cost=[not available])
> HiveProject(i_item_sk=[$0], i_item_id=[$1])
> HiveTableScan(table=[[default, item]], table:alias=[item])
> HiveProject(i_item_id=[$0],
[jira] [Work logged] (HIVE-25575) Add support for JWT authentication in HTTP mode
[
https://issues.apache.org/jira/browse/HIVE-25575?focusedWorklogId=746208=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-746208
]
ASF GitHub Bot logged work on HIVE-25575:
-
Author: ASF GitHub Bot
Created on: 22/Mar/22 22:11
Start Date: 22/Mar/22 22:11
Worklog Time Spent: 10m
Work Description: sourabh912 commented on a change in pull request #3006:
URL: https://github.com/apache/hive/pull/3006#discussion_r832664780
##
File path: service/src/java/org/apache/hive/service/auth/jwt/JWTValidator.java
##
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hive.service.auth.jwt;
+
+import com.google.common.base.Preconditions;
+import com.nimbusds.jose.JOSEException;
+import com.nimbusds.jose.JWSHeader;
+import com.nimbusds.jose.JWSObject;
+import com.nimbusds.jose.JWSVerifier;
+import com.nimbusds.jose.crypto.factories.DefaultJWSVerifierFactory;
+import com.nimbusds.jose.jwk.AsymmetricJWK;
+import com.nimbusds.jose.jwk.JWK;
+import com.nimbusds.jwt.JWTClaimsSet;
+import com.nimbusds.jwt.SignedJWT;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.security.sasl.AuthenticationException;
+import java.io.IOException;
+import java.security.Key;
+import java.text.ParseException;
+import java.util.Date;
+import java.util.List;
+
+/**
+ * This class is used to validate JWT. JWKS is fetched during instantiation
and kept in the memory.
+ * We disallow JWT signature verification with symmetric key, because that
means anyone can get the same key
+ * and use it to sign a JWT.
+ */
+public class JWTValidator {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(JWTValidator.class.getName());
+ private static final DefaultJWSVerifierFactory JWS_VERIFIER_FACTORY = new
DefaultJWSVerifierFactory();
+
+ private final URLBasedJWKSProvider jwksProvider;
+
+ public JWTValidator(HiveConf conf) throws IOException, ParseException {
+this.jwksProvider = new URLBasedJWKSProvider(conf);
+ }
+
+ public String validateJWTAndExtractUser(String signedJwt) throws
ParseException, AuthenticationException {
+Preconditions.checkNotNull(jwksProvider);
+Preconditions.checkNotNull(signedJwt, "No token found");
+final SignedJWT parsedJwt = SignedJWT.parse(signedJwt);
+List matchedJWKS = jwksProvider.getJWKs(parsedJwt.getHeader());
+if (matchedJWKS.isEmpty()) {
+ throw new AuthenticationException("Failed to find matched JWKs with the
JWT header: " + parsedJwt.getHeader());
+}
+
+// verify signature
+Exception lastException = null;
+for (JWK matchedJWK : matchedJWKS) {
+ try {
+JWSVerifier verifier = getVerifier(parsedJwt.getHeader(), matchedJWK);
+if (parsedJwt.verify(verifier)) {
+ break;
Review comment:
nit: would be good to add a debug log with with matchedJWK's and
signedJWT's basic info.
##
File path: service/src/java/org/apache/hive/service/auth/jwt/JWTValidator.java
##
@@ -0,0 +1,109 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hive.service.auth.jwt;
+
+import com.google.common.base.Preconditions;
+import com.nimbusds.jose.JOSEException;
+import com.nimbusds.jose.JWSHeader;
+import com.nimbusds.jose.JWSObject;
+import
[jira] [Work logged] (HIVE-25826) Support table defaults at each database level
[
https://issues.apache.org/jira/browse/HIVE-25826?focusedWorklogId=746189=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-746189
]
ASF GitHub Bot logged work on HIVE-25826:
-
Author: ASF GitHub Bot
Created on: 22/Mar/22 21:54
Start Date: 22/Mar/22 21:54
Worklog Time Spent: 10m
Work Description: saihemanth-cloudera commented on a change in pull
request #2924:
URL: https://github.com/apache/hive/pull/2924#discussion_r832659200
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
##
@@ -13625,7 +13656,8 @@ ASTNode analyzeCreateTable(
}
tblProps = validateAndAddDefaultProperties(
tblProps, isExt, storageFormat, dbDotTab, sortCols,
isMaterialization, isTemporary,
- isTransactional, isManaged);
+ isTransactional, isManaged, new String[] {qualifiedTabName.getDb(),
qualifiedTabName.getTable()}, isDefaultTableTypeChanged);
Review comment:
This flag is used as one of the determining factors in the
validateAndAddDefaultProperties() to decide whether to create an acid table or
managed table.
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
##
@@ -13615,7 +13631,22 @@ ASTNode analyzeCreateTable(
// Handle different types of CREATE TABLE command
// Note: each branch must call addDbAndTabToOutputs after finalizing table
properties.
-
+Database database = getDatabase(qualifiedTabName.getDb());
+boolean isDefaultTableTypeChanged = false;
+if(database.getParameters() != null) {
+ String defaultTableType =
database.getParameters().getOrDefault(DEFAULT_TABLE_TYPE, "none");
Review comment:
Ack
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
##
@@ -13245,9 +13247,23 @@ private void updateDefaultTblProps(Map
source, Map tblProp, boolean
isTransactional, boolean isExt, boolean isTableTypeChanged) {
Review comment:
ack
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestDatabaseTableDefault.java
##
@@ -0,0 +1,298 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql;
+
+import static org.junit.Assert.assertEquals;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.common.FileUtils;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf.ConfVars;
+import org.apache.hadoop.hive.metastore.HiveMetaStoreClient;
+import org.apache.hadoop.hive.metastore.utils.TestTxnDbUtil;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
+
+//import org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
+
+/**
+ * This class resides in itests to facilitate running query using Tez engine,
since the jars are
+ * fully loaded here, which is not the case if it stays in ql.
+ */
+public class TestDatabaseTableDefault {
+static final private Logger LOG =
LoggerFactory.getLogger(TestDatabaseTableDefault.class);
+
+private HiveConf hiveConf;
+private IDriver d;
+private static final String database_with_default_table_type =
"database_with_default_table_type";
+private static final String default_db = "default_db";
+private static final String table_type_managed = "MANAGED_TABLE";
+private static final String table_type_external = "EXTERNAL_TABLE";
+File ext_wh = null;
+File wh = null;
+protected static HiveMetaStoreClient client;
+protected static Configuration conf;
+private static enum TableNames {
+TRANSACTIONALTBL1("transactional_table_1"),
+
[jira] [Updated] (HIVE-26015) CREATE HBase table fails when SERDEPROPERTIES contain special characters
[ https://issues.apache.org/jira/browse/HIVE-26015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stamatis Zampetakis updated HIVE-26015: --- Fix Version/s: 4.0.0-alpha-2 > CREATE HBase table fails when SERDEPROPERTIES contain special characters > > > Key: HIVE-26015 > URL: https://issues.apache.org/jira/browse/HIVE-26015 > Project: Hive > Issue Type: New Feature >Reporter: Steve Carlin >Assignee: Steve Carlin >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-2 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > A Create table statement is failing for HBase going through Ranger. > The stack trace shows a problem with the getURIForAuth method. > The table is creating someting like this: > CREATE EXTERNAL TABLE `mytesttbl`( `field1` string COMMENT 'from > deserializer',`field2` string COMMENT 'from deserializer',`field3` string > COMMENT 'from deserializer',`field4` string COMMENT 'from > deserializer',`field5` string COMMENT 'from deserializer',`field6` int > COMMENT 'from deserializer', `field7` string COMMENT 'from deserializer', > `field8` int COMMENT 'from deserializer') ROW FORMAT SERDE > 'org.apache.hadoop.hive.hbase.HBaseSerDe' STORED BY > 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ( > 'hbase.columns.mapping'=':key,field1,field2,field3,field4,field5#b,field6,cf:field7#b','serialization.format'='1') > TBLPROPERTIES ( 'hbase.table.name'='mytesttbl'); > Essentially, the SERDEPROPERTIES contain hash tabs which is causing a problem > when creating a URI -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HIVE-26015) CREATE HBase table fails when SERDEPROPERTIES contain special characters
[ https://issues.apache.org/jira/browse/HIVE-26015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stamatis Zampetakis resolved HIVE-26015. Resolution: Fixed Fixed in https://github.com/apache/hive/commit/2bdec6f63d217cb42720fd7fe5cee804a2e5803c, Thanks for the PR [~scarlin]! > CREATE HBase table fails when SERDEPROPERTIES contain special characters > > > Key: HIVE-26015 > URL: https://issues.apache.org/jira/browse/HIVE-26015 > Project: Hive > Issue Type: New Feature >Reporter: Steve Carlin >Assignee: Steve Carlin >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-2 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > A Create table statement is failing for HBase going through Ranger. > The stack trace shows a problem with the getURIForAuth method. > The table is creating someting like this: > CREATE EXTERNAL TABLE `mytesttbl`( `field1` string COMMENT 'from > deserializer',`field2` string COMMENT 'from deserializer',`field3` string > COMMENT 'from deserializer',`field4` string COMMENT 'from > deserializer',`field5` string COMMENT 'from deserializer',`field6` int > COMMENT 'from deserializer', `field7` string COMMENT 'from deserializer', > `field8` int COMMENT 'from deserializer') ROW FORMAT SERDE > 'org.apache.hadoop.hive.hbase.HBaseSerDe' STORED BY > 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ( > 'hbase.columns.mapping'=':key,field1,field2,field3,field4,field5#b,field6,cf:field7#b','serialization.format'='1') > TBLPROPERTIES ( 'hbase.table.name'='mytesttbl'); > Essentially, the SERDEPROPERTIES contain hash tabs which is causing a problem > when creating a URI -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-26015) HBase table with Ranger authentication fails; needs URLEncoding
[ https://issues.apache.org/jira/browse/HIVE-26015?focusedWorklogId=746172=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-746172 ] ASF GitHub Bot logged work on HIVE-26015: - Author: ASF GitHub Bot Created on: 22/Mar/22 21:38 Start Date: 22/Mar/22 21:38 Worklog Time Spent: 10m Work Description: zabetak closed pull request #3084: URL: https://github.com/apache/hive/pull/3084 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 746172) Time Spent: 1h 40m (was: 1.5h) > HBase table with Ranger authentication fails; needs URLEncoding > --- > > Key: HIVE-26015 > URL: https://issues.apache.org/jira/browse/HIVE-26015 > Project: Hive > Issue Type: New Feature >Reporter: Steve Carlin >Assignee: Steve Carlin >Priority: Major > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > > A Create table statement is failing for HBase going through Ranger. > The stack trace shows a problem with the getURIForAuth method. > The table is creating someting like this: > CREATE EXTERNAL TABLE `mytesttbl`( `field1` string COMMENT 'from > deserializer',`field2` string COMMENT 'from deserializer',`field3` string > COMMENT 'from deserializer',`field4` string COMMENT 'from > deserializer',`field5` string COMMENT 'from deserializer',`field6` int > COMMENT 'from deserializer', `field7` string COMMENT 'from deserializer', > `field8` int COMMENT 'from deserializer') ROW FORMAT SERDE > 'org.apache.hadoop.hive.hbase.HBaseSerDe' STORED BY > 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ( > 'hbase.columns.mapping'=':key,field1,field2,field3,field4,field5#b,field6,cf:field7#b','serialization.format'='1') > TBLPROPERTIES ( 'hbase.table.name'='mytesttbl'); > Essentially, the SERDEPROPERTIES contain hash tabs which is causing a problem > when creating a URI -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-26015) CREATE HBase table fails when SERDEPROPERTIES contain special characters
[ https://issues.apache.org/jira/browse/HIVE-26015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stamatis Zampetakis updated HIVE-26015: --- Summary: CREATE HBase table fails when SERDEPROPERTIES contain special characters (was: HBase table with Ranger authentication fails; needs URLEncoding) > CREATE HBase table fails when SERDEPROPERTIES contain special characters > > > Key: HIVE-26015 > URL: https://issues.apache.org/jira/browse/HIVE-26015 > Project: Hive > Issue Type: New Feature >Reporter: Steve Carlin >Assignee: Steve Carlin >Priority: Major > Labels: pull-request-available > Time Spent: 1h 40m > Remaining Estimate: 0h > > A Create table statement is failing for HBase going through Ranger. > The stack trace shows a problem with the getURIForAuth method. > The table is creating someting like this: > CREATE EXTERNAL TABLE `mytesttbl`( `field1` string COMMENT 'from > deserializer',`field2` string COMMENT 'from deserializer',`field3` string > COMMENT 'from deserializer',`field4` string COMMENT 'from > deserializer',`field5` string COMMENT 'from deserializer',`field6` int > COMMENT 'from deserializer', `field7` string COMMENT 'from deserializer', > `field8` int COMMENT 'from deserializer') ROW FORMAT SERDE > 'org.apache.hadoop.hive.hbase.HBaseSerDe' STORED BY > 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES ( > 'hbase.columns.mapping'=':key,field1,field2,field3,field4,field5#b,field6,cf:field7#b','serialization.format'='1') > TBLPROPERTIES ( 'hbase.table.name'='mytesttbl'); > Essentially, the SERDEPROPERTIES contain hash tabs which is causing a problem > when creating a URI -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-21456) Hive Metastore HTTP Thrift
[
https://issues.apache.org/jira/browse/HIVE-21456?focusedWorklogId=746051=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-746051
]
ASF GitHub Bot logged work on HIVE-21456:
-
Author: ASF GitHub Bot
Created on: 22/Mar/22 19:11
Start Date: 22/Mar/22 19:11
Worklog Time Spent: 10m
Work Description: sourabh912 commented on a change in pull request #3105:
URL: https://github.com/apache/hive/pull/3105#discussion_r832503645
##
File path: itests/hive-unit/src/main/java/org/hadoop/hive/jdbc/SSLTestUtils.java
##
@@ -67,6 +67,12 @@ public static void setMetastoreSslConf(HiveConf conf) {
KEY_STORE_TRUST_STORE_PASSWORD);
}
+ public static void setMetastoreHttpsConf(HiveConf conf) {
+setMetastoreSslConf(conf);
+MetastoreConf.setVar(conf, MetastoreConf.ConfVars.TRANSPORT_MODE, "http");
Review comment:
Yes, it should be THRIFT_TRANSPORT_MODE.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 746051)
Time Spent: 1h 50m (was: 1h 40m)
> Hive Metastore HTTP Thrift
> --
>
> Key: HIVE-21456
> URL: https://issues.apache.org/jira/browse/HIVE-21456
> Project: Hive
> Issue Type: New Feature
> Components: Metastore, Standalone Metastore
>Reporter: Amit Khanna
>Assignee: Sourabh Goyal
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21456.2.patch, HIVE-21456.3.patch,
> HIVE-21456.4.patch, HIVE-21456.patch
>
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Hive Metastore currently doesn't have support for HTTP transport because of
> which it is not possible to access it via Knox. Adding support for Thrift
> over HTTP transport will allow the clients to access via Knox
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-21456) Hive Metastore HTTP Thrift
[
https://issues.apache.org/jira/browse/HIVE-21456?focusedWorklogId=746034=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-746034
]
ASF GitHub Bot logged work on HIVE-21456:
-
Author: ASF GitHub Bot
Created on: 22/Mar/22 18:47
Start Date: 22/Mar/22 18:47
Worklog Time Spent: 10m
Work Description: sourabh912 commented on a change in pull request #3105:
URL: https://github.com/apache/hive/pull/3105#discussion_r832508524
##
File path:
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
##
@@ -1356,6 +1356,13 @@ public static ConfVars getMetaConf(String name) {
"Comma-separated list of tasks that will be started in separate
threads. These will be" +
" started only when the metastore is running as a separate
service. They must " +
"implement " + METASTORE_TASK_THREAD_CLASS),
+THRIFT_TRANSPORT_MODE("metastore.server.thrift.transport.mode",
+"hive.metastore.server.thrift.transport.mode", "binary",
+"Transport mode for thrift server in Metastore. Can be binary or
http"),
+THRIFT_HTTP_PATH("metastore.server.thrift.http.path",
Review comment:
I think a better config name would be
`metastore.server.thrift.transport.http.path`. Thoughts?
##
File path:
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/SecurityUtils.java
##
@@ -271,6 +286,31 @@ public static TTransport getSSLSocket(String host, int
port, int loginTimeout,
return getSSLSocketWithHttps(tSSLSocket);
}
+ public static THttpClient getHttpSSLSocket(String httpsUrl, String
trustStorePath,
+ String trustStorePasswd, String trustStoreAlgorithm, String
trustStoreType)
Review comment:
The server side code for this is in HiveMetaStore.java at
https://github.com/apache/hive/pull/3105/files#diff-00e70b6958060aa36762b21bf16676f83af01c1e09b56816aecc6abe7c8ac866R683
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 746034)
Time Spent: 1h 40m (was: 1.5h)
> Hive Metastore HTTP Thrift
> --
>
> Key: HIVE-21456
> URL: https://issues.apache.org/jira/browse/HIVE-21456
> Project: Hive
> Issue Type: New Feature
> Components: Metastore, Standalone Metastore
>Reporter: Amit Khanna
>Assignee: Sourabh Goyal
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21456.2.patch, HIVE-21456.3.patch,
> HIVE-21456.4.patch, HIVE-21456.patch
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Hive Metastore currently doesn't have support for HTTP transport because of
> which it is not possible to access it via Knox. Adding support for Thrift
> over HTTP transport will allow the clients to access via Knox
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-21456) Hive Metastore HTTP Thrift
[
https://issues.apache.org/jira/browse/HIVE-21456?focusedWorklogId=746024=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-746024
]
ASF GitHub Bot logged work on HIVE-21456:
-
Author: ASF GitHub Bot
Created on: 22/Mar/22 18:38
Start Date: 22/Mar/22 18:38
Worklog Time Spent: 10m
Work Description: sourabh912 commented on a change in pull request #3105:
URL: https://github.com/apache/hive/pull/3105#discussion_r832506838
##
File path:
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
##
@@ -591,6 +592,31 @@ public void renamePartition(String catName, String dbname,
String tableName, Lis
client.rename_partition_req(req);
}
+ /**
+ * The config parameter can be like "path", "/path", "/path/", "path/*",
"/path1/path2/*" and so on.
+ * httpPath should end up as "/*", "/path/*" or "/path1/../pathN/*"
+ * @param httpPath
+ * @return
+ */
+ private String getHttpPath(String httpPath) {
Review comment:
Sure. I will remove it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 746024)
Time Spent: 1.5h (was: 1h 20m)
> Hive Metastore HTTP Thrift
> --
>
> Key: HIVE-21456
> URL: https://issues.apache.org/jira/browse/HIVE-21456
> Project: Hive
> Issue Type: New Feature
> Components: Metastore, Standalone Metastore
>Reporter: Amit Khanna
>Assignee: Sourabh Goyal
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21456.2.patch, HIVE-21456.3.patch,
> HIVE-21456.4.patch, HIVE-21456.patch
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Hive Metastore currently doesn't have support for HTTP transport because of
> which it is not possible to access it via Knox. Adding support for Thrift
> over HTTP transport will allow the clients to access via Knox
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-21456) Hive Metastore HTTP Thrift
[
https://issues.apache.org/jira/browse/HIVE-21456?focusedWorklogId=746022=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-746022
]
ASF GitHub Bot logged work on HIVE-21456:
-
Author: ASF GitHub Bot
Created on: 22/Mar/22 18:36
Start Date: 22/Mar/22 18:36
Worklog Time Spent: 10m
Work Description: sourabh912 commented on a change in pull request #3105:
URL: https://github.com/apache/hive/pull/3105#discussion_r832503645
##
File path: itests/hive-unit/src/main/java/org/hadoop/hive/jdbc/SSLTestUtils.java
##
@@ -67,6 +67,12 @@ public static void setMetastoreSslConf(HiveConf conf) {
KEY_STORE_TRUST_STORE_PASSWORD);
}
+ public static void setMetastoreHttpsConf(HiveConf conf) {
+setMetastoreSslConf(conf);
+MetastoreConf.setVar(conf, MetastoreConf.ConfVars.TRANSPORT_MODE, "http");
Review comment:
Yes, THRIFT_TRANSPORT_MODE is better variable name.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 746022)
Time Spent: 1h 20m (was: 1h 10m)
> Hive Metastore HTTP Thrift
> --
>
> Key: HIVE-21456
> URL: https://issues.apache.org/jira/browse/HIVE-21456
> Project: Hive
> Issue Type: New Feature
> Components: Metastore, Standalone Metastore
>Reporter: Amit Khanna
>Assignee: Sourabh Goyal
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-21456.2.patch, HIVE-21456.3.patch,
> HIVE-21456.4.patch, HIVE-21456.patch
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Hive Metastore currently doesn't have support for HTTP transport because of
> which it is not possible to access it via Knox. Adding support for Thrift
> over HTTP transport will allow the clients to access via Knox
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-26060) Invalidate acid table directory cache on drop table
[ https://issues.apache.org/jira/browse/HIVE-26060?focusedWorklogId=745964=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745964 ] ASF GitHub Bot logged work on HIVE-26060: - Author: ASF GitHub Bot Created on: 22/Mar/22 17:36 Start Date: 22/Mar/22 17:36 Worklog Time Spent: 10m Work Description: asinkovits opened a new pull request #3128: URL: https://github.com/apache/hive/pull/3128 ### What changes were proposed in this pull request? Invalidate acid directory cache on table drop ### Why are the changes needed? Cache can give back invalid directory location on table drop if we recreate the same table. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Manual tests as of now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 745964) Remaining Estimate: 0h Time Spent: 10m > Invalidate acid table directory cache on drop table > --- > > Key: HIVE-26060 > URL: https://issues.apache.org/jira/browse/HIVE-26060 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0-alpha-1 >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Minor > Labels: ACID > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-26060) Invalidate acid table directory cache on drop table
[ https://issues.apache.org/jira/browse/HIVE-26060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-26060: -- Labels: ACID pull-request-available (was: ACID) > Invalidate acid table directory cache on drop table > --- > > Key: HIVE-26060 > URL: https://issues.apache.org/jira/browse/HIVE-26060 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0-alpha-1 >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Minor > Labels: ACID, pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25977) Enhance Compaction Cleaner to skip when there is nothing to do #2
[
https://issues.apache.org/jira/browse/HIVE-25977?focusedWorklogId=745933=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745933
]
ASF GitHub Bot logged work on HIVE-25977:
-
Author: ASF GitHub Bot
Created on: 22/Mar/22 16:51
Start Date: 22/Mar/22 16:51
Worklog Time Spent: 10m
Work Description: klcopp commented on a change in pull request #3102:
URL: https://github.com/apache/hive/pull/3102#discussion_r832393501
##
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java
##
@@ -445,10 +445,17 @@ private boolean removeFiles(String location,
ValidWriteIdList writeIdList, Compa
obsoleteDirs = dir.getAbortedDirectories();
}
-if (obsoleteDirs.isEmpty()
-&& !hasDataBelowWatermark(dir, fs, path, ci.highestWriteId,
writeIdList.getHighWatermark())) {
- LOG.info(idWatermark(ci) + " nothing to remove below watermark " +
ci.highestWriteId + ", ");
- return true;
+if (obsoleteDirs.isEmpty()) {
+ conf.set(ValidTxnList.VALID_TXNS_KEY, new ValidReadTxnList().toString());
+
+ AcidDirectory fullDir = AcidUtils.getAcidState(fs, path, conf,
+new ValidReaderWriteIdList(ci.getFullTableName(), new long[0], new
BitSet(), ci.highestWriteId, Long.MAX_VALUE),
+Ref.from(false), false, hdfsDirSnapshots);
+
+ if (fullDir.getObsolete().isEmpty()) {
+LOG.info(idWatermark(ci) + " nothing to remove below watermark " +
ci.highestWriteId + ", ");
+return true;
+ }
Review comment:
It might be a bit more readable/simpler to add an else clause (handle
the case of !obsoleteDirs.isEmpty()) here, what do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 745933)
Time Spent: 2h 10m (was: 2h)
> Enhance Compaction Cleaner to skip when there is nothing to do #2
> -
>
> Key: HIVE-25977
> URL: https://issues.apache.org/jira/browse/HIVE-25977
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> initially this was just an addendum to the original patch ; but got delayed
> and altered - so it should have its own ticket
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25977) Enhance Compaction Cleaner to skip when there is nothing to do #2
[
https://issues.apache.org/jira/browse/HIVE-25977?focusedWorklogId=745932=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745932
]
ASF GitHub Bot logged work on HIVE-25977:
-
Author: ASF GitHub Bot
Created on: 22/Mar/22 16:49
Start Date: 22/Mar/22 16:49
Worklog Time Spent: 10m
Work Description: klcopp commented on a change in pull request #3102:
URL: https://github.com/apache/hive/pull/3102#discussion_r832388105
##
File path:
ql/src/test/org/apache/hadoop/hive/ql/txn/compactor/CompactorTest.java
##
@@ -401,7 +401,7 @@ private void addFile(Table t, Partition p, long minTxn,
long maxTxn, int numReco
switch (type) {
case BASE: filename = AcidUtils.BASE_PREFIX + maxTxn + (visibilityId > 0
? AcidUtils.VISIBILITY_PREFIX + visibilityId : ""); break;
case LENGTH_FILE: // Fall through to delta
- case DELTA: filename = makeDeltaDirName(minTxn, maxTxn); break;
+ case DELTA: filename =
AcidUtils.addVisibilitySuffix(makeDeltaDirName(minTxn, maxTxn),visibilityId);
break;
Review comment:
This still isn't 100% true-to-life, since only compacted deltas have a
visibility suffix, FWIW. Maybe could add a COMPACTED_DELTA type?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 745932)
Time Spent: 2h (was: 1h 50m)
> Enhance Compaction Cleaner to skip when there is nothing to do #2
> -
>
> Key: HIVE-25977
> URL: https://issues.apache.org/jira/browse/HIVE-25977
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> initially this was just an addendum to the original patch ; but got delayed
> and altered - so it should have its own ticket
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work started] (HIVE-26060) Invalidate acid table directory cache on drop table
[ https://issues.apache.org/jira/browse/HIVE-26060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-26060 started by Antal Sinkovits. -- > Invalidate acid table directory cache on drop table > --- > > Key: HIVE-26060 > URL: https://issues.apache.org/jira/browse/HIVE-26060 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0-alpha-1 >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Minor > Labels: ACID > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HIVE-26060) Invalidate acid table directory cache on drop table
[ https://issues.apache.org/jira/browse/HIVE-26060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Antal Sinkovits reassigned HIVE-26060: -- > Invalidate acid table directory cache on drop table > --- > > Key: HIVE-26060 > URL: https://issues.apache.org/jira/browse/HIVE-26060 > Project: Hive > Issue Type: Bug >Affects Versions: 4.0.0-alpha-1 >Reporter: Antal Sinkovits >Assignee: Antal Sinkovits >Priority: Minor > Labels: ACID > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-26041) Fix wrong type supplied for getLatestCommittedCompaction
[
https://issues.apache.org/jira/browse/HIVE-26041?focusedWorklogId=745892=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745892
]
ASF GitHub Bot logged work on HIVE-26041:
-
Author: ASF GitHub Bot
Created on: 22/Mar/22 15:52
Start Date: 22/Mar/22 15:52
Worklog Time Spent: 10m
Work Description: nrg4878 commented on pull request #3113:
URL: https://github.com/apache/hive/pull/3113#issuecomment-1075345739
Fix has been merged. Please close the jira and the PR. Thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 745892)
Time Spent: 0.5h (was: 20m)
> Fix wrong type supplied for getLatestCommittedCompaction
>
>
> Key: HIVE-26041
> URL: https://issues.apache.org/jira/browse/HIVE-26041
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Reporter: Yu-Wen Lai
>Assignee: Yu-Wen Lai
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> In HIVE-25753, we filter compactions by CC_ID, but I used string type as the
> parameter for the prepared statement. That cause a type error on some
> databases (at least failed on PostgreSQL).
> To correctly handle the filter, we should use
> {code:java}
> preparedStatement.setLong(...){code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-26052) Non blocking DROP MATERIALIZED VIEW implementation
[
https://issues.apache.org/jira/browse/HIVE-26052?focusedWorklogId=745814=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745814
]
ASF GitHub Bot logged work on HIVE-26052:
-
Author: ASF GitHub Bot
Created on: 22/Mar/22 13:09
Start Date: 22/Mar/22 13:09
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on a change in pull request #3111:
URL: https://github.com/apache/hive/pull/3111#discussion_r832155994
##
File path: ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands.java
##
@@ -1768,4 +1768,97 @@ public void testDropTableWithoutSuffix() throws
Exception {
Assert.assertEquals("Unexpected number of compactions in history", 0,
resp.getCompactsSize());
}
}
+
+ @Test
+ public void testDropMaterializedViewWithSuffix() throws Exception {
+String tableName = "tab_acid";
+String mviewName = "mv_" + tableName;
+runStatementOnDriver("drop materialized view if exists " + mviewName);
+runStatementOnDriver("drop table if exists " + tableName);
+HiveConf.setBoolVar(hiveConf,
HiveConf.ConfVars.HIVE_ACID_CREATE_TABLE_USE_SUFFIX, true);
+
+runStatementOnDriver("create table " + tableName + "(a int, b int) stored
as orc TBLPROPERTIES ('transactional'='true')");
+runStatementOnDriver("insert into " + tableName + " values(1,2),(3,4)");
+runStatementOnDriver("create materialized view " + mviewName + " stored as
orc TBLPROPERTIES ('transactional'='true') " +
+ "as select a from tab_acid where b > 1");
+runStatementOnDriver("drop materialized view " + mviewName);
+
+int count = TestTxnDbUtil.countQueryAgent(hiveConf,
+ "select count(*) from TXN_TO_WRITE_ID where T2W_TABLE = '" + mviewName +
"'");
+Assert.assertEquals(1, count);
+
+FileSystem fs = FileSystem.get(hiveConf);
+FileStatus[] stat = fs.listStatus(new Path(getWarehouseDir()),
+ t -> t.getName().matches(mviewName + SOFT_DELETE_TABLE_PATTERN));
+if (1 != stat.length) {
+ Assert.fail("Materialized view data was removed from FS");
+}
+MetastoreTaskThread houseKeeperService = new AcidHouseKeeperService();
+houseKeeperService.setConf(hiveConf);
+
+houseKeeperService.run();
+count = TestTxnDbUtil.countQueryAgent(hiveConf,
+ "select count(*) from TXN_TO_WRITE_ID where T2W_TABLE = '" + mviewName +
"'");
+Assert.assertEquals(0, count);
+
+try {
+ runStatementOnDriver("select * from " + mviewName);
+} catch (Exception ex) {
+ Assert.assertTrue(ex.getMessage().contains(
+ErrorMsg.INVALID_TABLE.getMsg(StringUtils.wrap(mviewName, "'";
+}
+// Check status of compaction job
+TxnStore txnHandler = TxnUtils.getTxnStore(hiveConf);
+ShowCompactResponse resp = txnHandler.showCompact(new
ShowCompactRequest());
+
+Assert.assertEquals("Unexpected number of compactions in history", 1,
resp.getCompactsSize());
+Assert.assertEquals("Unexpected 0 compaction state",
+ TxnStore.CLEANING_RESPONSE, resp.getCompacts().get(0).getState());
+
+runCleaner(hiveConf);
+
+FileStatus[] status = fs.listStatus(new Path(getWarehouseDir()),
+ t -> t.getName().matches(mviewName + SOFT_DELETE_TABLE_PATTERN));
+Assert.assertEquals(0, status.length);
+ }
+
+ @Test
+ public void testDropMaterializedViewWithoutSuffix() throws Exception {
+String tableName = "tab_acid";
+String mviewName = "mv_" + tableName;
+runStatementOnDriver("drop materialized view if exists " + mviewName);
+
+for (boolean enabled : Arrays.asList( true)) {
Review comment:
yes, you're right, looks like I removed 1 of the combinations during the
troubleshooting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 745814)
Time Spent: 50m (was: 40m)
> Non blocking DROP MATERIALIZED VIEW implementation
> --
>
> Key: HIVE-26052
> URL: https://issues.apache.org/jira/browse/HIVE-26052
> Project: Hive
> Issue Type: Task
>Reporter: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-26043) Use constraint info when creating RexNodes
[
https://issues.apache.org/jira/browse/HIVE-26043?focusedWorklogId=745794=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745794
]
ASF GitHub Bot logged work on HIVE-26043:
-
Author: ASF GitHub Bot
Created on: 22/Mar/22 12:41
Start Date: 22/Mar/22 12:41
Worklog Time Spent: 10m
Work Description: kasakrisz commented on pull request #3121:
URL: https://github.com/apache/hive/pull/3121#issuecomment-1075127798
Partition columns can have not null constraint defined so I extended this
patch to handle them.
Virtual columns can have null values. Ex.:
```
SELECT ROW__ID FROM non_transactional;
```
will produce `NULLs`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 745794)
Time Spent: 1h 20m (was: 1h 10m)
> Use constraint info when creating RexNodes
> --
>
> Key: HIVE-26043
> URL: https://issues.apache.org/jira/browse/HIVE-26043
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Prior HIVE-23100 Not null constraints affected newly created RexNode type
> nullability.
> Nullability enables the subquery rewrite algorithm to generate more optimal
> plan.
> [https://github.com/apache/hive/blob/1213ad3f0ae0e21e7519dc28b8b6d1401cdd1441/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSubQueryRemoveRule.java#L324]
> Example:
> {code:java}
> explain cbo
> select ws_sales_price
> from web_sales, customer, item
> where ws_bill_customer_sk = c_customer_sk
> and ws_item_sk = i_item_sk
> and ( c_customer_sk = 1
> or
> i_item_id in (select i_item_id
> from item
> where i_item_sk in (2, 3)
> )
> );
> {code}
> Without not null constraints
> {code:java}
> HiveProject(ws_sales_price=[$2])
> HiveFilter(condition=[OR(AND(<>($6, 0), IS NOT NULL($8)), =($3, 1))])
> HiveProject(ws_item_sk=[$0], ws_bill_customer_sk=[$1],
> ws_sales_price=[$2], c_customer_sk=[$8], i_item_sk=[$3], i_item_id=[$4],
> c=[$5], i_item_id0=[$6], literalTrue=[$7])
> HiveJoin(condition=[=($1, $8)], joinType=[inner], algorithm=[none],
> cost=[not available])
> HiveJoin(condition=[=($0, $3)], joinType=[inner], algorithm=[none],
> cost=[not available])
> HiveProject(ws_item_sk=[$2], ws_bill_customer_sk=[$3],
> ws_sales_price=[$20])
> HiveFilter(condition=[IS NOT NULL($3)])
> HiveTableScan(table=[[default, web_sales]],
> table:alias=[web_sales])
> HiveJoin(condition=[=($1, $3)], joinType=[left], algorithm=[none],
> cost=[not available])
> HiveJoin(condition=[true], joinType=[inner], algorithm=[none],
> cost=[not available])
> HiveProject(i_item_sk=[$0], i_item_id=[$1])
> HiveTableScan(table=[[default, item]], table:alias=[item])
> HiveProject(c=[$0])
> HiveAggregate(group=[{}], c=[COUNT()])
> HiveFilter(condition=[IN($0, 2:BIGINT, 3:BIGINT)])
> HiveTableScan(table=[[default, item]], table:alias=[item])
> HiveProject(i_item_id=[$0], literalTrue=[true])
> HiveAggregate(group=[{1}])
> HiveFilter(condition=[IN($0, 2:BIGINT, 3:BIGINT)])
> HiveTableScan(table=[[default, item]], table:alias=[item])
> HiveProject(c_customer_sk=[$0])
> HiveTableScan(table=[[default, customer]], table:alias=[customer])
> {code}
> With not null constraints
> {code:java}
> HiveProject(ws_sales_price=[$2])
> HiveFilter(condition=[OR(IS NOT NULL($7), =($3, 1))])
> HiveProject(ws_item_sk=[$0], ws_bill_customer_sk=[$1],
> ws_sales_price=[$2], c_customer_sk=[$7], i_item_sk=[$3], i_item_id=[$4],
> i_item_id0=[$5], literalTrue=[$6])
> HiveJoin(condition=[=($1, $7)], joinType=[inner], algorithm=[none],
> cost=[not available])
> HiveJoin(condition=[=($0, $3)], joinType=[inner], algorithm=[none],
> cost=[not available])
> HiveProject(ws_item_sk=[$2], ws_bill_customer_sk=[$3],
> ws_sales_price=[$20])
> HiveFilter(condition=[IS NOT NULL($3)])
> HiveTableScan(table=[[default, web_sales]],
> table:alias=[web_sales])
>
[jira] [Updated] (HIVE-26059) Eventually clean compactions in "refused" state from compaction history
[ https://issues.apache.org/jira/browse/HIVE-26059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-26059: -- Labels: pull-request-available (was: ) > Eventually clean compactions in "refused" state from compaction history > --- > > Key: HIVE-26059 > URL: https://issues.apache.org/jira/browse/HIVE-26059 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: László Végh >Assignee: László Végh >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Compactions in states succeeded, failed, and did not initiate have a > retention threshold (example: > metastore.compactor.history.retention.succeeded) and are purged from > COMPLETED_COMPACTIONS if the number of compactions in this state per > partition/unpartitioned table passes the threshold. This keeps the size of > COMPLETED_COMPACTIONS in check. > We should also purge refused compactions from COMPLETED_COMPACTIONS. > See: > CompactionTxnHandler#purgeCompactionHistory > ! Also: REFUSED_RESPONSE should be added to > org.apache.hadoop.hive.metastore.txn.TxnStore#COMPACTION_STATES so that > metrics will be collected about it. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-26059) Eventually clean compactions in "refused" state from compaction history
[ https://issues.apache.org/jira/browse/HIVE-26059?focusedWorklogId=745792=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745792 ] ASF GitHub Bot logged work on HIVE-26059: - Author: ASF GitHub Bot Created on: 22/Mar/22 12:40 Start Date: 22/Mar/22 12:40 Worklog Time Spent: 10m Work Description: veghlaci05 opened a new pull request #3127: URL: https://github.com/apache/hive/pull/3127 ### What changes were proposed in this pull request? Purge refused compactions from COMPLETED_COMPACTIONS. ### Why are the changes needed? Compactions in states succeeded, failed, and did not initiate have a retention threshold (example: metastore.compactor.history.retention.succeeded) and are purged from COMPLETED_COMPACTIONS if the number of compactions in this state per partition/unpartitioned table passes the threshold. This behaviour should also be applied to compactions in REFUSED state. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Manually, and through unit tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 745792) Remaining Estimate: 0h Time Spent: 10m > Eventually clean compactions in "refused" state from compaction history > --- > > Key: HIVE-26059 > URL: https://issues.apache.org/jira/browse/HIVE-26059 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: László Végh >Assignee: László Végh >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Compactions in states succeeded, failed, and did not initiate have a > retention threshold (example: > metastore.compactor.history.retention.succeeded) and are purged from > COMPLETED_COMPACTIONS if the number of compactions in this state per > partition/unpartitioned table passes the threshold. This keeps the size of > COMPLETED_COMPACTIONS in check. > We should also purge refused compactions from COMPLETED_COMPACTIONS. > See: > CompactionTxnHandler#purgeCompactionHistory > ! Also: REFUSED_RESPONSE should be added to > org.apache.hadoop.hive.metastore.txn.TxnStore#COMPACTION_STATES so that > metrics will be collected about it. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-26043) Use constraint info when creating RexNodes
[
https://issues.apache.org/jira/browse/HIVE-26043?focusedWorklogId=745790=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745790
]
ASF GitHub Bot logged work on HIVE-26043:
-
Author: ASF GitHub Bot
Created on: 22/Mar/22 12:38
Start Date: 22/Mar/22 12:38
Worklog Time Spent: 10m
Work Description: kasakrisz commented on a change in pull request #3121:
URL: https://github.com/apache/hive/pull/3121#discussion_r832123342
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
##
@@ -3119,65 +3123,24 @@ private RelNode genTableLogicalPlan(String tableAlias,
QB qb) throws SemanticExc
return tableRel;
}
-private RelDataType inferNotNullableColumns(Table tabMetaData, RelDataType
rowType)
-throws HiveException {
- final NotNullConstraint nnc = tabMetaData.getNotNullConstraint();
- final PrimaryKeyInfo pkc = tabMetaData.getPrimaryKeyInfo();
- if ((nnc == null || nnc.getNotNullConstraints().isEmpty()) &&
- (pkc == null || pkc.getColNames().isEmpty())) {
-return rowType;
+private boolean isNullable(String colName, NotNullConstraint
notNullConstraints, PrimaryKeyInfo primaryKeyInfo) {
+ if (notNullConstraints == null && primaryKeyInfo == null) {
+return true;
}
- // Build the bitset with not null columns
- ImmutableBitSet.Builder builder = ImmutableBitSet.builder();
- if (nnc != null) {
-for (String nnCol : nnc.getNotNullConstraints().values()) {
- int nnPos = -1;
- for (int i = 0; i < rowType.getFieldNames().size(); i++) {
-if (rowType.getFieldNames().get(i).equals(nnCol)) {
- nnPos = i;
- break;
-}
- }
- if (nnPos == -1) {
-LOG.error("Column for not null constraint definition " + nnCol + "
not found");
-return rowType;
- }
- builder.set(nnPos);
-}
- }
- if (pkc != null) {
-for (String pkCol : pkc.getColNames().values()) {
- int pkPos = -1;
- for (int i = 0; i < rowType.getFieldNames().size(); i++) {
-if (rowType.getFieldNames().get(i).equals(pkCol)) {
- pkPos = i;
- break;
-}
- }
- if (pkPos == -1) {
-LOG.error("Column for not null constraint definition " + pkCol + "
not found");
-return rowType;
- }
- builder.set(pkPos);
-}
+ if (notNullConstraints != null &&
+ !notNullConstraints.getNotNullConstraints().isEmpty() &&
Review comment:
Removed
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
##
@@ -3119,65 +3123,24 @@ private RelNode genTableLogicalPlan(String tableAlias,
QB qb) throws SemanticExc
return tableRel;
}
-private RelDataType inferNotNullableColumns(Table tabMetaData, RelDataType
rowType)
-throws HiveException {
- final NotNullConstraint nnc = tabMetaData.getNotNullConstraint();
- final PrimaryKeyInfo pkc = tabMetaData.getPrimaryKeyInfo();
- if ((nnc == null || nnc.getNotNullConstraints().isEmpty()) &&
- (pkc == null || pkc.getColNames().isEmpty())) {
-return rowType;
+private boolean isNullable(String colName, NotNullConstraint
notNullConstraints, PrimaryKeyInfo primaryKeyInfo) {
+ if (notNullConstraints == null && primaryKeyInfo == null) {
+return true;
}
- // Build the bitset with not null columns
- ImmutableBitSet.Builder builder = ImmutableBitSet.builder();
- if (nnc != null) {
-for (String nnCol : nnc.getNotNullConstraints().values()) {
- int nnPos = -1;
- for (int i = 0; i < rowType.getFieldNames().size(); i++) {
-if (rowType.getFieldNames().get(i).equals(nnCol)) {
- nnPos = i;
- break;
-}
- }
- if (nnPos == -1) {
-LOG.error("Column for not null constraint definition " + nnCol + "
not found");
-return rowType;
- }
- builder.set(nnPos);
-}
- }
- if (pkc != null) {
-for (String pkCol : pkc.getColNames().values()) {
- int pkPos = -1;
- for (int i = 0; i < rowType.getFieldNames().size(); i++) {
-if (rowType.getFieldNames().get(i).equals(pkCol)) {
- pkPos = i;
- break;
-}
- }
- if (pkPos == -1) {
-LOG.error("Column for not null constraint definition " + pkCol + "
not found");
-return rowType;
- }
- builder.set(pkPos);
-}
+ if (notNullConstraints != null &&
+ !notNullConstraints.getNotNullConstraints().isEmpty() &&
+
[jira] [Work logged] (HIVE-26043) Use constraint info when creating RexNodes
[
https://issues.apache.org/jira/browse/HIVE-26043?focusedWorklogId=745791=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745791
]
ASF GitHub Bot logged work on HIVE-26043:
-
Author: ASF GitHub Bot
Created on: 22/Mar/22 12:38
Start Date: 22/Mar/22 12:38
Worklog Time Spent: 10m
Work Description: kasakrisz commented on a change in pull request #3121:
URL: https://github.com/apache/hive/pull/3121#discussion_r832123658
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
##
@@ -3119,65 +3123,24 @@ private RelNode genTableLogicalPlan(String tableAlias,
QB qb) throws SemanticExc
return tableRel;
}
-private RelDataType inferNotNullableColumns(Table tabMetaData, RelDataType
rowType)
-throws HiveException {
- final NotNullConstraint nnc = tabMetaData.getNotNullConstraint();
- final PrimaryKeyInfo pkc = tabMetaData.getPrimaryKeyInfo();
- if ((nnc == null || nnc.getNotNullConstraints().isEmpty()) &&
- (pkc == null || pkc.getColNames().isEmpty())) {
-return rowType;
+private boolean isNullable(String colName, NotNullConstraint
notNullConstraints, PrimaryKeyInfo primaryKeyInfo) {
+ if (notNullConstraints == null && primaryKeyInfo == null) {
+return true;
}
Review comment:
Removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 745791)
Time Spent: 1h 10m (was: 1h)
> Use constraint info when creating RexNodes
> --
>
> Key: HIVE-26043
> URL: https://issues.apache.org/jira/browse/HIVE-26043
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Prior HIVE-23100 Not null constraints affected newly created RexNode type
> nullability.
> Nullability enables the subquery rewrite algorithm to generate more optimal
> plan.
> [https://github.com/apache/hive/blob/1213ad3f0ae0e21e7519dc28b8b6d1401cdd1441/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSubQueryRemoveRule.java#L324]
> Example:
> {code:java}
> explain cbo
> select ws_sales_price
> from web_sales, customer, item
> where ws_bill_customer_sk = c_customer_sk
> and ws_item_sk = i_item_sk
> and ( c_customer_sk = 1
> or
> i_item_id in (select i_item_id
> from item
> where i_item_sk in (2, 3)
> )
> );
> {code}
> Without not null constraints
> {code:java}
> HiveProject(ws_sales_price=[$2])
> HiveFilter(condition=[OR(AND(<>($6, 0), IS NOT NULL($8)), =($3, 1))])
> HiveProject(ws_item_sk=[$0], ws_bill_customer_sk=[$1],
> ws_sales_price=[$2], c_customer_sk=[$8], i_item_sk=[$3], i_item_id=[$4],
> c=[$5], i_item_id0=[$6], literalTrue=[$7])
> HiveJoin(condition=[=($1, $8)], joinType=[inner], algorithm=[none],
> cost=[not available])
> HiveJoin(condition=[=($0, $3)], joinType=[inner], algorithm=[none],
> cost=[not available])
> HiveProject(ws_item_sk=[$2], ws_bill_customer_sk=[$3],
> ws_sales_price=[$20])
> HiveFilter(condition=[IS NOT NULL($3)])
> HiveTableScan(table=[[default, web_sales]],
> table:alias=[web_sales])
> HiveJoin(condition=[=($1, $3)], joinType=[left], algorithm=[none],
> cost=[not available])
> HiveJoin(condition=[true], joinType=[inner], algorithm=[none],
> cost=[not available])
> HiveProject(i_item_sk=[$0], i_item_id=[$1])
> HiveTableScan(table=[[default, item]], table:alias=[item])
> HiveProject(c=[$0])
> HiveAggregate(group=[{}], c=[COUNT()])
> HiveFilter(condition=[IN($0, 2:BIGINT, 3:BIGINT)])
> HiveTableScan(table=[[default, item]], table:alias=[item])
> HiveProject(i_item_id=[$0], literalTrue=[true])
> HiveAggregate(group=[{1}])
> HiveFilter(condition=[IN($0, 2:BIGINT, 3:BIGINT)])
> HiveTableScan(table=[[default, item]], table:alias=[item])
> HiveProject(c_customer_sk=[$0])
> HiveTableScan(table=[[default, customer]], table:alias=[customer])
> {code}
> With not null constraints
> {code:java}
> HiveProject(ws_sales_price=[$2])
>
[jira] [Work started] (HIVE-26059) Eventually clean compactions in "refused" state from compaction history
[ https://issues.apache.org/jira/browse/HIVE-26059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-26059 started by László Végh. -- > Eventually clean compactions in "refused" state from compaction history > --- > > Key: HIVE-26059 > URL: https://issues.apache.org/jira/browse/HIVE-26059 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: László Végh >Assignee: László Végh >Priority: Major > > Compactions in states succeeded, failed, and did not initiate have a > retention threshold (example: > metastore.compactor.history.retention.succeeded) and are purged from > COMPLETED_COMPACTIONS if the number of compactions in this state per > partition/unpartitioned table passes the threshold. This keeps the size of > COMPLETED_COMPACTIONS in check. > We should also purge refused compactions from COMPLETED_COMPACTIONS. > See: > CompactionTxnHandler#purgeCompactionHistory > ! Also: REFUSED_RESPONSE should be added to > org.apache.hadoop.hive.metastore.txn.TxnStore#COMPACTION_STATES so that > metrics will be collected about it. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HIVE-26059) Eventually clean compactions in "refused" state from compaction history
[ https://issues.apache.org/jira/browse/HIVE-26059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Végh reassigned HIVE-26059: -- Assignee: László Végh > Eventually clean compactions in "refused" state from compaction history > --- > > Key: HIVE-26059 > URL: https://issues.apache.org/jira/browse/HIVE-26059 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: László Végh >Assignee: László Végh >Priority: Major > > Compactions in states succeeded, failed, and did not initiate have a > retention threshold (example: > metastore.compactor.history.retention.succeeded) and are purged from > COMPLETED_COMPACTIONS if the number of compactions in this state per > partition/unpartitioned table passes the threshold. This keeps the size of > COMPLETED_COMPACTIONS in check. > We should also purge refused compactions from COMPLETED_COMPACTIONS. > See: > CompactionTxnHandler#purgeCompactionHistory > ! Also: REFUSED_RESPONSE should be added to > org.apache.hadoop.hive.metastore.txn.TxnStore#COMPACTION_STATES so that > metrics will be collected about it. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-26043) Use constraint info when creating RexNodes
[
https://issues.apache.org/jira/browse/HIVE-26043?focusedWorklogId=745737=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745737
]
ASF GitHub Bot logged work on HIVE-26043:
-
Author: ASF GitHub Bot
Created on: 22/Mar/22 11:38
Start Date: 22/Mar/22 11:38
Worklog Time Spent: 10m
Work Description: zabetak commented on a change in pull request #3121:
URL: https://github.com/apache/hive/pull/3121#discussion_r832062572
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
##
@@ -3119,65 +3123,24 @@ private RelNode genTableLogicalPlan(String tableAlias,
QB qb) throws SemanticExc
return tableRel;
}
-private RelDataType inferNotNullableColumns(Table tabMetaData, RelDataType
rowType)
-throws HiveException {
- final NotNullConstraint nnc = tabMetaData.getNotNullConstraint();
- final PrimaryKeyInfo pkc = tabMetaData.getPrimaryKeyInfo();
- if ((nnc == null || nnc.getNotNullConstraints().isEmpty()) &&
- (pkc == null || pkc.getColNames().isEmpty())) {
-return rowType;
+private boolean isNullable(String colName, NotNullConstraint
notNullConstraints, PrimaryKeyInfo primaryKeyInfo) {
+ if (notNullConstraints == null && primaryKeyInfo == null) {
+return true;
}
- // Build the bitset with not null columns
- ImmutableBitSet.Builder builder = ImmutableBitSet.builder();
- if (nnc != null) {
-for (String nnCol : nnc.getNotNullConstraints().values()) {
- int nnPos = -1;
- for (int i = 0; i < rowType.getFieldNames().size(); i++) {
-if (rowType.getFieldNames().get(i).equals(nnCol)) {
- nnPos = i;
- break;
-}
- }
- if (nnPos == -1) {
-LOG.error("Column for not null constraint definition " + nnCol + "
not found");
-return rowType;
- }
- builder.set(nnPos);
-}
- }
- if (pkc != null) {
-for (String pkCol : pkc.getColNames().values()) {
- int pkPos = -1;
- for (int i = 0; i < rowType.getFieldNames().size(); i++) {
-if (rowType.getFieldNames().get(i).equals(pkCol)) {
- pkPos = i;
- break;
-}
- }
- if (pkPos == -1) {
-LOG.error("Column for not null constraint definition " + pkCol + "
not found");
-return rowType;
- }
- builder.set(pkPos);
-}
+ if (notNullConstraints != null &&
+ !notNullConstraints.getNotNullConstraints().isEmpty() &&
+
notNullConstraints.getNotNullConstraints().containsValue(colName)) {
+return false;
}
- ImmutableBitSet bitSet = builder.build();
-
- RexBuilder rexBuilder = cluster.getRexBuilder();
- RelDataTypeFactory dtFactory = rexBuilder.getTypeFactory();
- List fieldTypes = new LinkedList();
- List fieldNames = new LinkedList();
- for (RelDataTypeField rdtf : rowType.getFieldList()) {
-if (bitSet.indexOf(rdtf.getIndex()) != -1) {
- fieldTypes.add(dtFactory.createTypeWithNullability(rdtf.getType(),
false));
-} else {
- fieldTypes.add(rdtf.getType());
-}
-fieldNames.add(rdtf.getName());
+ if (primaryKeyInfo != null &&
+ !primaryKeyInfo.getColNames().isEmpty() &&
Review comment:
Do we need the `isEmpty` check?
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java
##
@@ -3119,65 +3123,24 @@ private RelNode genTableLogicalPlan(String tableAlias,
QB qb) throws SemanticExc
return tableRel;
}
-private RelDataType inferNotNullableColumns(Table tabMetaData, RelDataType
rowType)
-throws HiveException {
- final NotNullConstraint nnc = tabMetaData.getNotNullConstraint();
- final PrimaryKeyInfo pkc = tabMetaData.getPrimaryKeyInfo();
- if ((nnc == null || nnc.getNotNullConstraints().isEmpty()) &&
- (pkc == null || pkc.getColNames().isEmpty())) {
-return rowType;
+private boolean isNullable(String colName, NotNullConstraint
notNullConstraints, PrimaryKeyInfo primaryKeyInfo) {
+ if (notNullConstraints == null && primaryKeyInfo == null) {
+return true;
}
- // Build the bitset with not null columns
- ImmutableBitSet.Builder builder = ImmutableBitSet.builder();
- if (nnc != null) {
-for (String nnCol : nnc.getNotNullConstraints().values()) {
- int nnPos = -1;
- for (int i = 0; i < rowType.getFieldNames().size(); i++) {
-if (rowType.getFieldNames().get(i).equals(nnCol)) {
- nnPos = i;
- break;
-}
- }
- if (nnPos == -1) {
-LOG.error("Column for not null constraint definition " +
[jira] [Resolved] (HIVE-26044) Remove hardcoded version references from the tests
[ https://issues.apache.org/jira/browse/HIVE-26044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary resolved HIVE-26044. --- Fix Version/s: 4.0.0 Resolution: Fixed Thanks for the review [~Marton Bod] and [~zabetak]! > Remove hardcoded version references from the tests > -- > > Key: HIVE-26044 > URL: https://issues.apache.org/jira/browse/HIVE-26044 > Project: Hive > Issue Type: Test >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 2h > Remaining Estimate: 0h > > There are several places where there are hardcoded version references in the > tests. > We should remove them to so it is easier to change versions. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-26044) Remove hardcoded version references from the tests
[ https://issues.apache.org/jira/browse/HIVE-26044?focusedWorklogId=745732=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745732 ] ASF GitHub Bot logged work on HIVE-26044: - Author: ASF GitHub Bot Created on: 22/Mar/22 11:21 Start Date: 22/Mar/22 11:21 Worklog Time Spent: 10m Work Description: pvary merged pull request #3115: URL: https://github.com/apache/hive/pull/3115 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 745732) Time Spent: 2h (was: 1h 50m) > Remove hardcoded version references from the tests > -- > > Key: HIVE-26044 > URL: https://issues.apache.org/jira/browse/HIVE-26044 > Project: Hive > Issue Type: Test >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Major > Labels: pull-request-available > Time Spent: 2h > Remaining Estimate: 0h > > There are several places where there are hardcoded version references in the > tests. > We should remove them to so it is easier to change versions. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-26015) HBase table with Ranger authentication fails; needs URLEncoding
[
https://issues.apache.org/jira/browse/HIVE-26015?focusedWorklogId=745708=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745708
]
ASF GitHub Bot logged work on HIVE-26015:
-
Author: ASF GitHub Bot
Created on: 22/Mar/22 10:50
Start Date: 22/Mar/22 10:50
Worklog Time Spent: 10m
Work Description: zabetak commented on a change in pull request #3084:
URL: https://github.com/apache/hive/pull/3084#discussion_r832014955
##
File path:
hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseStorageHandler.java
##
@@ -46,6 +53,46 @@ public void testHbaseConfigIsAddedToJobConf() {
jobConfToConfigure.get("hbase.some.fake.option.from.xml.file") !=
null);
}
+ @Test
+ public void testGetUriForAuth() {
+try {
+ HBaseStorageHandler hbaseStorageHandler = new HBaseStorageHandler();
+ hbaseStorageHandler.setConf(new JobConf(new HiveConf()));
+ Table table = createMockTable(new HashMap<>());
+ URI uri = hbaseStorageHandler.getURIForAuth(table);
+ // If there is no tablename provided, the default "null" is still
+ // written out. At the time this test was written, this was the current
+ // behavior, so I left this test as/is. Need to research if a null
+ // table can be provided here.
+ Assert.assertEquals("hbase://localhost:2181/null", uri.toString());
+
+ Map serdeParams = new HashMap<>();
+ serdeParams.put("hbase.zookeeper.quorum", "testhost");
+ serdeParams.put("hbase.zookeeper.property.clientPort", "8765");
+ table = createMockTable(serdeParams);
+ uri = hbaseStorageHandler.getURIForAuth(table);
+ Assert.assertEquals("hbase://testhost:8765/null", uri.toString());
+
+ serdeParams.put("hbase.table.name", "mytbl");
+ table = createMockTable(serdeParams);
+ uri = hbaseStorageHandler.getURIForAuth(table);
+ Assert.assertEquals("hbase://testhost:8765/mytbl", uri.toString());
+
+ serdeParams.put("hbase.columns.mapping", "mycolumns");
+ table = createMockTable(serdeParams);
+ uri = hbaseStorageHandler.getURIForAuth(table);
+ Assert.assertEquals("hbase://testhost:8765/mytbl/mycolumns",
uri.toString());
+
+ serdeParams.put("hbase.table.name", "my#tbl");
+ serdeParams.put("hbase.columns.mapping", "myco#lumns");
+ table = createMockTable(serdeParams);
+ uri = hbaseStorageHandler.getURIForAuth(table);
+ Assert.assertEquals("hbase://testhost:8765/my%23tbl/myco%23lumns",
uri.toString());
+} catch (URISyntaxException e) {
+ throw new RuntimeException(e);
+}
Review comment:
The catch and rethrow pattern is rarely necessary in unit tests.
Moreover, it has few disadvantages such as obscuring the original exception &
error message, increasing indentation etc. The try-catch block can be removed
and the `URISyntaxException` can go into the declaration of the method if
necessary.
##
File path:
hbase-handler/src/test/org/apache/hadoop/hive/hbase/TestHBaseStorageHandler.java
##
@@ -56,4 +103,16 @@ private TableDesc getHBaseTableDesc() {
Mockito.when(tableDesc.getProperties()).thenReturn(properties);
return tableDesc;
}
+
+ private Table createMockTable(Map serdeParams) {
+Table table = new Table();
+StorageDescriptor sd = new StorageDescriptor();
+SerDeInfo sdi = new SerDeInfo();
+Map params = new HashMap<>();
+sdi.setParameters(serdeParams);
+sd.setSerdeInfo(sdi);
+table.setSd(sd);
+table.setParameters(params);
Review comment:
nit: Change to
`table.setParameters(Collections.emptyMap())` or
`table.setParameters(new HashMap<>())` or
remove entirely if it doesn't call failures
##
File path:
hbase-handler/src/java/org/apache/hadoop/hive/hbase/HBaseStorageHandler.java
##
@@ -293,14 +294,23 @@ public void configureTableJobProperties(
public URI getURIForAuth(Table table) throws URISyntaxException {
Map tableProperties =
HiveCustomStorageHandlerUtils.getTableProperties(table);
hbaseConf = getConf();
-String hbase_host = tableProperties.containsKey(HBASE_HOST_NAME)?
tableProperties.get(HBASE_HOST_NAME) : hbaseConf.get(HBASE_HOST_NAME);
-String hbase_port = tableProperties.containsKey(HBASE_CLIENT_PORT)?
tableProperties.get(HBASE_CLIENT_PORT) : hbaseConf.get(HBASE_CLIENT_PORT);
-String table_name =
tableProperties.getOrDefault(HBaseSerDe.HBASE_TABLE_NAME, null);
-String column_family =
tableProperties.getOrDefault(HBaseSerDe.HBASE_COLUMNS_MAPPING, null);
-if (column_family != null)
- return new
URI(HBASE_PREFIX+"//"+hbase_host+":"+hbase_port+"/"+table_name+"/"+column_family);
-else
- return new
URI(HBASE_PREFIX+"//"+hbase_host+":"+hbase_port+"/"+table_name);
+String hbase_host = tableProperties.getOrDefault(HBASE_HOST_NAME,
+hbaseConf.get(HBASE_HOST_NAME));
[jira] [Updated] (HIVE-26026) Use the new "REFUSED" compaction state where it makes sense
[ https://issues.apache.org/jira/browse/HIVE-26026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-26026: -- Labels: pull-request-available (was: ) > Use the new "REFUSED" compaction state where it makes sense > --- > > Key: HIVE-26026 > URL: https://issues.apache.org/jira/browse/HIVE-26026 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: László Végh >Assignee: László Végh >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > The > org.apache.hadoop.hive.ql.txn.compactor.Worker#findNextCompactionAndExecute > method does several checks (The table/partition exists, is not sorted, there > are enough files to compact, etc.) before it actually executes the compaction > request. If the compaction request fails on any of these checks, it is put to > "SUCCEEDED" state which is often misleading for users. SHOW COMPACTIONS will > show these requests as succeeded without an error, while the table is not > compacted at all. > For these cases, the state should be "REFUSED" instead of "SUCCEEDED" among > with the appropriate error message. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-26026) Use the new "REFUSED" compaction state where it makes sense
[ https://issues.apache.org/jira/browse/HIVE-26026?focusedWorklogId=745680=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745680 ] ASF GitHub Bot logged work on HIVE-26026: - Author: ASF GitHub Bot Created on: 22/Mar/22 09:34 Start Date: 22/Mar/22 09:34 Worklog Time Spent: 10m Work Description: veghlaci05 opened a new pull request #3126: URL: https://github.com/apache/hive/pull/3126 ### What changes were proposed in this pull request? When the pre-compaction checks fail in Worker, the compaction state should be set to "REFUSED" instead of "SUCCEEDED" among with the appropriate error message. ### Why are the changes needed? The org.apache.hadoop.hive.ql.txn.compactor.Worker#findNextCompactionAndExecute method does several checks (The table/partition exists, is not sorted, there are enough files to compact, etc.) before it actually executes the compaction request. If the compaction request fails on any of these checks, it is put to "SUCCEEDED" state which is often misleading for users. SHOW COMPACTIONS will show these requests as succeeded without an error, while the table is not compacted at all. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Manually and through unit tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 745680) Remaining Estimate: 0h Time Spent: 10m > Use the new "REFUSED" compaction state where it makes sense > --- > > Key: HIVE-26026 > URL: https://issues.apache.org/jira/browse/HIVE-26026 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: László Végh >Assignee: László Végh >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > The > org.apache.hadoop.hive.ql.txn.compactor.Worker#findNextCompactionAndExecute > method does several checks (The table/partition exists, is not sorted, there > are enough files to compact, etc.) before it actually executes the compaction > request. If the compaction request fails on any of these checks, it is put to > "SUCCEEDED" state which is often misleading for users. SHOW COMPACTIONS will > show these requests as succeeded without an error, while the table is not > compacted at all. > For these cases, the state should be "REFUSED" instead of "SUCCEEDED" among > with the appropriate error message. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-26044) Remove hardcoded version references from the tests
[
https://issues.apache.org/jira/browse/HIVE-26044?focusedWorklogId=745679=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745679
]
ASF GitHub Bot logged work on HIVE-26044:
-
Author: ASF GitHub Bot
Created on: 22/Mar/22 09:33
Start Date: 22/Mar/22 09:33
Worklog Time Spent: 10m
Work Description: zabetak commented on a change in pull request #3115:
URL: https://github.com/apache/hive/pull/3115#discussion_r831953781
##
File path:
itests/hive-unit/src/test/java/org/apache/hive/service/server/InformationSchemaWithPrivilegeTestBase.java
##
@@ -287,7 +291,7 @@ public void test() throws Exception {
List args = new ArrayList(baseArgs);
args.add("-f");
-
args.add("../../metastore/scripts/upgrade/hive/hive-schema-4.0.0-alpha-1.hive.sql");
+args.add("../../metastore/scripts/upgrade/hive/hive-schema-" +
hiveSchemaVer + ".hive.sql");
Review comment:
Thanks both for the clarifications!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 745679)
Time Spent: 1h 50m (was: 1h 40m)
> Remove hardcoded version references from the tests
> --
>
> Key: HIVE-26044
> URL: https://issues.apache.org/jira/browse/HIVE-26044
> Project: Hive
> Issue Type: Test
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> There are several places where there are hardcoded version references in the
> tests.
> We should remove them to so it is easier to change versions.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HIVE-26033) Repl Load fails with Wrong FS error.
[
https://issues.apache.org/jira/browse/HIVE-26033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-26033:
--
Fix Version/s: 4.0.0-alpha-1
> Repl Load fails with Wrong FS error.
>
>
> Key: HIVE-26033
> URL: https://issues.apache.org/jira/browse/HIVE-26033
> Project: Hive
> Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0, 4.0.0-alpha-1
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> For External table replication with staging on source, the replication load
> fails with wrong FS error while cleaning up snapshots.
> {noformat}
> Exception : Wrong FS:
> hdfs://cluster1:8020/user/hive/replDir/policy_1646973828/_file_list_external_current,
> expected: hdfs://cluster2:8020
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25867) Partition filter condition should pushed down to metastore query if it is equivalence Predicate
[ https://issues.apache.org/jira/browse/HIVE-25867?focusedWorklogId=745642=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745642 ] ASF GitHub Bot logged work on HIVE-25867: - Author: ASF GitHub Bot Created on: 22/Mar/22 08:15 Start Date: 22/Mar/22 08:15 Worklog Time Spent: 10m Work Description: pvary commented on pull request #2947: URL: https://github.com/apache/hive/pull/2947#issuecomment-1074862624 Thanks for the explanation @ming95! This was a misunderstanding on my side. What happens if your query is like this - would the filtering work in this case? ``` select * from src_play_day WHERE dt='20211125' AND u_gtype='activity_workshop' limit 10; ``` Notice that I used `'`-s to convert 20211125 to string. Another question is, how does this work with the other supported backend databases. The issue with directSql is that we have to handle the differences between the SQL engines ourselves. Also what happens when we have different types, like timestamp/binary/struct? Thanks for investigating this issue, and coming up with possible solutions! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 745642) Time Spent: 1h 50m (was: 1h 40m) > Partition filter condition should pushed down to metastore query if it is > equivalence Predicate > --- > > Key: HIVE-25867 > URL: https://issues.apache.org/jira/browse/HIVE-25867 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: shezm >Assignee: shezm >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > The colnum type of the partition is different from the column type of the hql > query, the metastore will not push down the query to the RDBMS, but will > instead get all PARTITIONS.PART_NAME of the hive table then filter it > according to the hql Expression. > https://github.com/apache/hive/blob/5b112aa6dcc4e374c0a7c2b24042f24ae6815da1/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java#L1316 > If the hive table has too many partitions and there are multiple hql queries > at the same time,RDBMS will increasing CPU IO_WAIT and affect performance. > If the partition filter condition in hql is an equivalent predicate, the > metastore should be pushed down to RDBMS, which can optimize the query > performance of hive large tables. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-26053) Non blocking DROP VIEW
[ https://issues.apache.org/jira/browse/HIVE-26053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-26053: -- Labels: pull-request-available (was: ) > Non blocking DROP VIEW > -- > > Key: HIVE-26053 > URL: https://issues.apache.org/jira/browse/HIVE-26053 > Project: Hive > Issue Type: Task >Reporter: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > DROP VIEW shouldn't acquire any locks -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-26053) Non blocking DROP VIEW
[
https://issues.apache.org/jira/browse/HIVE-26053?focusedWorklogId=745638=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745638
]
ASF GitHub Bot logged work on HIVE-26053:
-
Author: ASF GitHub Bot
Created on: 22/Mar/22 08:05
Start Date: 22/Mar/22 08:05
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #3116:
URL: https://github.com/apache/hive/pull/3116#discussion_r831870673
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/ddl/view/drop/DropViewAnalyzer.java
##
@@ -50,7 +50,7 @@ public void analyzeInternal(ASTNode root) throws
SemanticException {
Table view = getTable(viewName, throwException);
if (view != null) {
inputs.add(new ReadEntity(view));
- outputs.add(new WriteEntity(view, WriteEntity.WriteType.DDL_EXCLUSIVE));
+ outputs.add(new WriteEntity(view, WriteEntity.WriteType.DDL_NO_LOCK));
// see AcidUtils.needsLock(output)
Review comment:
What about materialized view-s where the view could be used for
optimising the query and it could be used as a source instead of the original
source table?
Or that is a different story?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 745638)
Remaining Estimate: 0h
Time Spent: 10m
> Non blocking DROP VIEW
> --
>
> Key: HIVE-26053
> URL: https://issues.apache.org/jira/browse/HIVE-26053
> Project: Hive
> Issue Type: Task
>Reporter: Denys Kuzmenko
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> DROP VIEW shouldn't acquire any locks
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HIVE-26058) Choose meaningful names for the Metastore pool threads
[
https://issues.apache.org/jira/browse/HIVE-26058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-26058:
--
Labels: pull-request-available (was: )
> Choose meaningful names for the Metastore pool threads
> --
>
> Key: HIVE-26058
> URL: https://issues.apache.org/jira/browse/HIVE-26058
> Project: Hive
> Issue Type: Improvement
>Reporter: Zhihua Deng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Due to TThreadPoolServer#createDefaultExecutorService setting the thread name
> by
> {code:java}
> thread.setName("TThreadPoolServer WorkerProcess-%d"); {code}
> The logger output the thread name like:
> {noformat}
> [TThreadPoolServer WorkerProcess-%d] utils.FileUtils: Renaming
> pfile:/{noformat}
> , which makes it hard to identify and debug a thread.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-26058) Choose meaningful names for the Metastore pool threads
[
https://issues.apache.org/jira/browse/HIVE-26058?focusedWorklogId=745613=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745613
]
ASF GitHub Bot logged work on HIVE-26058:
-
Author: ASF GitHub Bot
Created on: 22/Mar/22 07:00
Start Date: 22/Mar/22 07:00
Worklog Time Spent: 10m
Work Description: dengzhhu653 opened a new pull request #3125:
URL: https://github.com/apache/hive/pull/3125
### What changes were proposed in this pull request?
### Why are the changes needed?
### Does this PR introduce _any_ user-facing change?
### How was this patch tested?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 745613)
Remaining Estimate: 0h
Time Spent: 10m
> Choose meaningful names for the Metastore pool threads
> --
>
> Key: HIVE-26058
> URL: https://issues.apache.org/jira/browse/HIVE-26058
> Project: Hive
> Issue Type: Improvement
>Reporter: Zhihua Deng
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Due to TThreadPoolServer#createDefaultExecutorService setting the thread name
> by
> {code:java}
> thread.setName("TThreadPoolServer WorkerProcess-%d"); {code}
> The logger output the thread name like:
> {noformat}
> [TThreadPoolServer WorkerProcess-%d] utils.FileUtils: Renaming
> pfile:/{noformat}
> , which makes it hard to identify and debug a thread.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HIVE-26058) Choose meaningful names for the Metastore pool threads
[
https://issues.apache.org/jira/browse/HIVE-26058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihua Deng updated HIVE-26058:
---
Description:
Due to TThreadPoolServer#createDefaultExecutorService setting the thread name
by
{code:java}
thread.setName("TThreadPoolServer WorkerProcess-%d"); {code}
The logger output the thread name like:
{noformat}
[TThreadPoolServer WorkerProcess-%d] utils.FileUtils: Renaming
pfile:/{noformat}
, which makes it hard to identify and debug a thread.
was:
Due to TThreadPoolServer#createDefaultExecutorService setting the thread name
by
{code:java}
thread.setName("TThreadPoolServer WorkerProcess-%d"); {code}
The logger output the thread name like:
{noformat}
[TThreadPoolServer WorkerProcess-%d] utils.FileUtils: Renaming
pfile:/{noformat}
, which makes it hard to identify and debug a thread.
> Choose meaningful names for the Metastore pool threads
> --
>
> Key: HIVE-26058
> URL: https://issues.apache.org/jira/browse/HIVE-26058
> Project: Hive
> Issue Type: Improvement
>Reporter: Zhihua Deng
>Priority: Major
>
> Due to TThreadPoolServer#createDefaultExecutorService setting the thread name
> by
> {code:java}
> thread.setName("TThreadPoolServer WorkerProcess-%d"); {code}
> The logger output the thread name like:
> {noformat}
> [TThreadPoolServer WorkerProcess-%d] utils.FileUtils: Renaming
> pfile:/{noformat}
> , which makes it hard to identify and debug a thread.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-26043) Use constraint info when creating RexNodes
[
https://issues.apache.org/jira/browse/HIVE-26043?focusedWorklogId=745599=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745599
]
ASF GitHub Bot logged work on HIVE-26043:
-
Author: ASF GitHub Bot
Created on: 22/Mar/22 06:01
Start Date: 22/Mar/22 06:01
Worklog Time Spent: 10m
Work Description: kasakrisz commented on pull request #3120:
URL: https://github.com/apache/hive/pull/3120#issuecomment-1074771467
@zabetak
Including the nullability flag to `ColumnInfo` can be nicer solution: #3121
Thanks for the tip.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 745599)
Time Spent: 40m (was: 0.5h)
> Use constraint info when creating RexNodes
> --
>
> Key: HIVE-26043
> URL: https://issues.apache.org/jira/browse/HIVE-26043
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Prior HIVE-23100 Not null constraints affected newly created RexNode type
> nullability.
> Nullability enables the subquery rewrite algorithm to generate more optimal
> plan.
> [https://github.com/apache/hive/blob/1213ad3f0ae0e21e7519dc28b8b6d1401cdd1441/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSubQueryRemoveRule.java#L324]
> Example:
> {code:java}
> explain cbo
> select ws_sales_price
> from web_sales, customer, item
> where ws_bill_customer_sk = c_customer_sk
> and ws_item_sk = i_item_sk
> and ( c_customer_sk = 1
> or
> i_item_id in (select i_item_id
> from item
> where i_item_sk in (2, 3)
> )
> );
> {code}
> Without not null constraints
> {code:java}
> HiveProject(ws_sales_price=[$2])
> HiveFilter(condition=[OR(AND(<>($6, 0), IS NOT NULL($8)), =($3, 1))])
> HiveProject(ws_item_sk=[$0], ws_bill_customer_sk=[$1],
> ws_sales_price=[$2], c_customer_sk=[$8], i_item_sk=[$3], i_item_id=[$4],
> c=[$5], i_item_id0=[$6], literalTrue=[$7])
> HiveJoin(condition=[=($1, $8)], joinType=[inner], algorithm=[none],
> cost=[not available])
> HiveJoin(condition=[=($0, $3)], joinType=[inner], algorithm=[none],
> cost=[not available])
> HiveProject(ws_item_sk=[$2], ws_bill_customer_sk=[$3],
> ws_sales_price=[$20])
> HiveFilter(condition=[IS NOT NULL($3)])
> HiveTableScan(table=[[default, web_sales]],
> table:alias=[web_sales])
> HiveJoin(condition=[=($1, $3)], joinType=[left], algorithm=[none],
> cost=[not available])
> HiveJoin(condition=[true], joinType=[inner], algorithm=[none],
> cost=[not available])
> HiveProject(i_item_sk=[$0], i_item_id=[$1])
> HiveTableScan(table=[[default, item]], table:alias=[item])
> HiveProject(c=[$0])
> HiveAggregate(group=[{}], c=[COUNT()])
> HiveFilter(condition=[IN($0, 2:BIGINT, 3:BIGINT)])
> HiveTableScan(table=[[default, item]], table:alias=[item])
> HiveProject(i_item_id=[$0], literalTrue=[true])
> HiveAggregate(group=[{1}])
> HiveFilter(condition=[IN($0, 2:BIGINT, 3:BIGINT)])
> HiveTableScan(table=[[default, item]], table:alias=[item])
> HiveProject(c_customer_sk=[$0])
> HiveTableScan(table=[[default, customer]], table:alias=[customer])
> {code}
> With not null constraints
> {code:java}
> HiveProject(ws_sales_price=[$2])
> HiveFilter(condition=[OR(IS NOT NULL($7), =($3, 1))])
> HiveProject(ws_item_sk=[$0], ws_bill_customer_sk=[$1],
> ws_sales_price=[$2], c_customer_sk=[$7], i_item_sk=[$3], i_item_id=[$4],
> i_item_id0=[$5], literalTrue=[$6])
> HiveJoin(condition=[=($1, $7)], joinType=[inner], algorithm=[none],
> cost=[not available])
> HiveJoin(condition=[=($0, $3)], joinType=[inner], algorithm=[none],
> cost=[not available])
> HiveProject(ws_item_sk=[$2], ws_bill_customer_sk=[$3],
> ws_sales_price=[$20])
> HiveFilter(condition=[IS NOT NULL($3)])
> HiveTableScan(table=[[default, web_sales]],
> table:alias=[web_sales])
> HiveJoin(condition=[=($1, $2)], joinType=[left], algorithm=[none],
> cost=[not available])
> HiveProject(i_item_sk=[$0],
