[jira] [Assigned] (HIVE-25790) Make managed table copies handle updates (FileUtils)
[ https://issues.apache.org/jira/browse/HIVE-25790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haymant Mangla reassigned HIVE-25790: - > Make managed table copies handle updates (FileUtils) > > > Key: HIVE-25790 > URL: https://issues.apache.org/jira/browse/HIVE-25790 > Project: Hive > Issue Type: Improvement >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-25782) Create Table As Select fails for managed ACID tables
[
https://issues.apache.org/jira/browse/HIVE-25782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-25782:
--
Labels: pull-request-available (was: )
> Create Table As Select fails for managed ACID tables
>
>
> Key: HIVE-25782
> URL: https://issues.apache.org/jira/browse/HIVE-25782
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Reporter: Csaba Juhász
>Assignee: Sai Hemanth Gantasala
>Priority: Major
> Labels: pull-request-available
> Attachments: ctas_acid_managed.q
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Create Table As Select fails for managed ACID tables:
> *MetaException(message:Processor has no capabilities, cannot create an ACID
> table.)*
> HMSHandler.translate_table_dryrun invokes
> MetastoreDefaultTransformer.transformCreateTable with null
> processorCapabilities and processorId.
> https://github.com/apache/hive/blob/c7fdd459305f4bf6913dc4bed7e8df8c7bf9e458/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java#L2251
> {code:java}
> Dec 06 05:32:47 Starting translation for CreateTable for processor null with
> null on table vectortab10korc
> Dec 06 05:32:47 MetaException(message:Processor has no capabilities, cannot
> create an ACID table.)
> at
> org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer.transformCreateTable(MetastoreDefaultTransformer.java:663)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.translate_table_dryrun(HiveMetaStore.java:2159)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy29.translate_table_dryrun(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$translate_table_dryrun.getResult(ThriftHiveMetastore.java:16981)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$translate_table_dryrun.getResult(ThriftHiveMetastore.java:16965)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:643)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:638)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:638)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> {code}
> Reproduction ([^ctas_acid_managed.q]):
> {code:java}
> set hive.support.concurrency=true;
> set hive.exec.dynamic.partition.mode=nonstrict;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> set
> metastore.metadata.transformer.class=org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer;
> create table test stored as orc tblproperties ('transactional'='true') as
> select from_unixtime(unix_timestamp("0002-01-01 09:57:21", "-MM-dd
> HH:mm:ss")); {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25782) Create Table As Select fails for managed ACID tables
[
https://issues.apache.org/jira/browse/HIVE-25782?focusedWorklogId=692994=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-692994
]
ASF GitHub Bot logged work on HIVE-25782:
-
Author: ASF GitHub Bot
Created on: 09/Dec/21 06:06
Start Date: 09/Dec/21 06:06
Worklog Time Spent: 10m
Work Description: saihemanth-cloudera opened a new pull request #2858:
URL: https://github.com/apache/hive/pull/2858
… query
### What changes were proposed in this pull request?
Added client capabilities in the dry run for CTAS query.
### Why are the changes needed?
The changes are required so that managed tables can be created with client
capabilities.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Local machine, Remote cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 692994)
Remaining Estimate: 0h
Time Spent: 10m
> Create Table As Select fails for managed ACID tables
>
>
> Key: HIVE-25782
> URL: https://issues.apache.org/jira/browse/HIVE-25782
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Reporter: Csaba Juhász
>Assignee: Sai Hemanth Gantasala
>Priority: Major
> Labels: pull-request-available
> Attachments: ctas_acid_managed.q
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Create Table As Select fails for managed ACID tables:
> *MetaException(message:Processor has no capabilities, cannot create an ACID
> table.)*
> HMSHandler.translate_table_dryrun invokes
> MetastoreDefaultTransformer.transformCreateTable with null
> processorCapabilities and processorId.
> https://github.com/apache/hive/blob/c7fdd459305f4bf6913dc4bed7e8df8c7bf9e458/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java#L2251
> {code:java}
> Dec 06 05:32:47 Starting translation for CreateTable for processor null with
> null on table vectortab10korc
> Dec 06 05:32:47 MetaException(message:Processor has no capabilities, cannot
> create an ACID table.)
> at
> org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer.transformCreateTable(MetastoreDefaultTransformer.java:663)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.translate_table_dryrun(HiveMetaStore.java:2159)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy29.translate_table_dryrun(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$translate_table_dryrun.getResult(ThriftHiveMetastore.java:16981)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$translate_table_dryrun.getResult(ThriftHiveMetastore.java:16965)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:643)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:638)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:638)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
[jira] [Work logged] (HIVE-25751) Ignore exceptions related to interruption when the limit is reached
[
https://issues.apache.org/jira/browse/HIVE-25751?focusedWorklogId=692958=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-692958
]
ASF GitHub Bot logged work on HIVE-25751:
-
Author: ASF GitHub Bot
Created on: 09/Dec/21 03:23
Start Date: 09/Dec/21 03:23
Worklog Time Spent: 10m
Work Description: rbalamohan commented on a change in pull request #2828:
URL: https://github.com/apache/hive/pull/2828#discussion_r765405135
##
File path:
llap-server/src/java/org/apache/hadoop/hive/llap/io/api/impl/LlapInputFormat.java
##
@@ -138,10 +142,28 @@
rr.start();
return result;
} catch (Exception ex) {
- throw new IOException(ex);
+ Throwable rootCause = JavaUtils.findRootCause(ex);
Review comment:
Thanks @abstractdog . LGTM. +1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 692958)
Time Spent: 1h 20m (was: 1h 10m)
> Ignore exceptions related to interruption when the limit is reached
> ---
>
> Key: HIVE-25751
> URL: https://issues.apache.org/jira/browse/HIVE-25751
> Project: Hive
> Issue Type: Bug
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Labels: pull-request-available
> Attachments: syslog, syslog_with_fix
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> The problem is well described on TEZ-4356, but it will be handled from hive
> instead.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25786) Auto-close browser window/tab after successful auth with SSO(SAML) authentication
[ https://issues.apache.org/jira/browse/HIVE-25786?focusedWorklogId=692938=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-692938 ] ASF GitHub Bot logged work on HIVE-25786: - Author: ASF GitHub Bot Created on: 09/Dec/21 01:49 Start Date: 09/Dec/21 01:49 Worklog Time Spent: 10m Work Description: nrg4878 commented on pull request #2851: URL: https://github.com/apache/hive/pull/2851#issuecomment-989425005 Change looks good to me. +1 pending clean test run. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 692938) Time Spent: 20m (was: 10m) > Auto-close browser window/tab after successful auth with SSO(SAML) > authentication > - > > Key: HIVE-25786 > URL: https://issues.apache.org/jira/browse/HIVE-25786 > Project: Hive > Issue Type: Bug > Components: Authentication, HiveServer2 >Reporter: Sai Hemanth Gantasala >Assignee: Sai Hemanth Gantasala >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > When a client(beeline/DBeaver) tries to connect to HS2. Each connection opens > a new tab which is a bit of an annoying user experience. This task is to > Auto-close the window on a successful auth after a timed interval (5 sec). On > an unsuccessful auth, the browser window would not be auto-closed. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HIVE-25782) Create Table As Select fails for managed ACID tables
[
https://issues.apache.org/jira/browse/HIVE-25782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sai Hemanth Gantasala reassigned HIVE-25782:
Assignee: Sai Hemanth Gantasala
> Create Table As Select fails for managed ACID tables
>
>
> Key: HIVE-25782
> URL: https://issues.apache.org/jira/browse/HIVE-25782
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Reporter: Csaba Juhász
>Assignee: Sai Hemanth Gantasala
>Priority: Major
> Attachments: ctas_acid_managed.q
>
>
> Create Table As Select fails for managed ACID tables:
> *MetaException(message:Processor has no capabilities, cannot create an ACID
> table.)*
> HMSHandler.translate_table_dryrun invokes
> MetastoreDefaultTransformer.transformCreateTable with null
> processorCapabilities and processorId.
> https://github.com/apache/hive/blob/c7fdd459305f4bf6913dc4bed7e8df8c7bf9e458/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HMSHandler.java#L2251
> {code:java}
> Dec 06 05:32:47 Starting translation for CreateTable for processor null with
> null on table vectortab10korc
> Dec 06 05:32:47 MetaException(message:Processor has no capabilities, cannot
> create an ACID table.)
> at
> org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer.transformCreateTable(MetastoreDefaultTransformer.java:663)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.translate_table_dryrun(HiveMetaStore.java:2159)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> at com.sun.proxy.$Proxy29.translate_table_dryrun(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$translate_table_dryrun.getResult(ThriftHiveMetastore.java:16981)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$translate_table_dryrun.getResult(ThriftHiveMetastore.java:16965)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:643)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor$1.run(HadoopThriftAuthBridge.java:638)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1898)
> at
> org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:638)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:834)
> {code}
> Reproduction ([^ctas_acid_managed.q]):
> {code:java}
> set hive.support.concurrency=true;
> set hive.exec.dynamic.partition.mode=nonstrict;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> set
> metastore.metadata.transformer.class=org.apache.hadoop.hive.metastore.MetastoreDefaultTransformer;
> create table test stored as orc tblproperties ('transactional'='true') as
> select from_unixtime(unix_timestamp("0002-01-01 09:57:21", "-MM-dd
> HH:mm:ss")); {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-24975) Bug in ValidWriteIdList comparison in TxnIdUtils
[
https://issues.apache.org/jira/browse/HIVE-24975?focusedWorklogId=692708=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-692708
]
ASF GitHub Bot logged work on HIVE-24975:
-
Author: ASF GitHub Bot
Created on: 08/Dec/21 16:23
Start Date: 08/Dec/21 16:23
Worklog Time Spent: 10m
Work Description: sourabh912 commented on a change in pull request #2641:
URL: https://github.com/apache/hive/pull/2641#discussion_r765021220
##
File path: storage-api/src/test/org/apache/hive/common/util/TestTxnIdUtils.java
##
@@ -190,5 +191,15 @@ public void testCompareWriteIds() throws Exception {
new ValidReaderWriteIdList("default.table2", new long[] {8,10,11}, new
BitSet(), 11)),
-1);
+ValidWriteIdList a =
Review comment:
Sure @kgyrtkirk . I will keep your suggestion in mind for future
commits. Thanks for the review and approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 692708)
Time Spent: 1h (was: 50m)
> Bug in ValidWriteIdList comparison in TxnIdUtils
>
>
> Key: HIVE-24975
> URL: https://issues.apache.org/jira/browse/HIVE-24975
> Project: Hive
> Issue Type: Bug
>Reporter: Sourabh Goyal
>Assignee: Sourabh Goyal
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> TxnIdUtils's
> [compare|https://github.com/apache/hive/blob/master/storage-api/src/java/org/apache/hive/common/util/TxnIdUtils.java#L38]
> method returns incorrect result for the following validWriteIdLists
> ValidWriteIdList a = new ValidReaderWriteIdList("default.test:1:1:1:");
> ValidWriteIdList b = new
> ValidReaderWriteIdList("default.test:1:9223372036854775807::");
> TxnIdUtils.compare(a, b) returns +1 whereas the expected response is -1 since
> b is more recent.
> cc - [~kishendas] [~vihangk1]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-24975) Bug in ValidWriteIdList comparison in TxnIdUtils
[
https://issues.apache.org/jira/browse/HIVE-24975?focusedWorklogId=692696=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-692696
]
ASF GitHub Bot logged work on HIVE-24975:
-
Author: ASF GitHub Bot
Created on: 08/Dec/21 16:13
Start Date: 08/Dec/21 16:13
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on a change in pull request #2641:
URL: https://github.com/apache/hive/pull/2641#discussion_r765012114
##
File path: storage-api/src/test/org/apache/hive/common/util/TestTxnIdUtils.java
##
@@ -190,5 +191,15 @@ public void testCompareWriteIds() throws Exception {
new ValidReaderWriteIdList("default.table2", new long[] {8,10,11}, new
BitSet(), 11)),
-1);
+ValidWriteIdList a =
Review comment:
putting a lot of cases into the same method is not really development
friendly - next time please put new testcases into new test methods
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 692696)
Time Spent: 50m (was: 40m)
> Bug in ValidWriteIdList comparison in TxnIdUtils
>
>
> Key: HIVE-24975
> URL: https://issues.apache.org/jira/browse/HIVE-24975
> Project: Hive
> Issue Type: Bug
>Reporter: Sourabh Goyal
>Assignee: Sourabh Goyal
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> TxnIdUtils's
> [compare|https://github.com/apache/hive/blob/master/storage-api/src/java/org/apache/hive/common/util/TxnIdUtils.java#L38]
> method returns incorrect result for the following validWriteIdLists
> ValidWriteIdList a = new ValidReaderWriteIdList("default.test:1:1:1:");
> ValidWriteIdList b = new
> ValidReaderWriteIdList("default.test:1:9223372036854775807::");
> TxnIdUtils.compare(a, b) returns +1 whereas the expected response is -1 since
> b is more recent.
> cc - [~kishendas] [~vihangk1]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (HIVE-25735) Improve statestimator in UDFWhen/UDFCase
[ https://issues.apache.org/jira/browse/HIVE-25735?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich resolved HIVE-25735. - Fix Version/s: 4.0.0 Resolution: Fixed merged into master. Thank you [~kkasa] for reviewing the changes! > Improve statestimator in UDFWhen/UDFCase > > > Key: HIVE-25735 > URL: https://issues.apache.org/jira/browse/HIVE-25735 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25735) Improve statestimator in UDFWhen/UDFCase
[ https://issues.apache.org/jira/browse/HIVE-25735?focusedWorklogId=692687=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-692687 ] ASF GitHub Bot logged work on HIVE-25735: - Author: ASF GitHub Bot Created on: 08/Dec/21 15:44 Start Date: 08/Dec/21 15:44 Worklog Time Spent: 10m Work Description: kgyrtkirk merged pull request #2814: URL: https://github.com/apache/hive/pull/2814 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 692687) Time Spent: 20m (was: 10m) > Improve statestimator in UDFWhen/UDFCase > > > Key: HIVE-25735 > URL: https://issues.apache.org/jira/browse/HIVE-25735 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-21172) DEFAULT keyword handling in MERGE UPDATE clause issues
[
https://issues.apache.org/jira/browse/HIVE-21172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-21172:
--
Labels: pull-request-available (was: )
> DEFAULT keyword handling in MERGE UPDATE clause issues
> --
>
> Key: HIVE-21172
> URL: https://issues.apache.org/jira/browse/HIVE-21172
> Project: Hive
> Issue Type: Sub-task
> Components: SQL, Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> once HIVE-21159 lands, enable {{HiveConf.MERGE_SPLIT_UPDATE}} and run these
> tests.
> TestMiniLlapLocalCliDriver.testCliDriver[sqlmerge_stats]
> mvn test -Dtest=TestMiniLlapLocalCliDriver
> -Dqfile=insert_into_default_keyword.q
> Merge is rewritten as a multi-insert. When Update clause has DEFAULT, it's
> not properly replaced with a value in the muli-insert - it's treated as a
> literal
> {noformat}
> INSERT INTO `default`.`acidTable`-- update clause(insert part)
> SELECT `t`.`key`, `DEFAULT`, `t`.`value`
>WHERE `t`.`key` = `s`.`key` AND `s`.`key` > 3 AND NOT(`s`.`key` < 3)
> {noformat}
> See {{LOG.info("Going to reparse <" + originalQuery + "> as \n<" +
> rewrittenQueryStr.toString() + ">");}} in hive.log
> {{MergeSemanticAnalyzer.replaceDefaultKeywordForMerge()}} is only called in
> {{handleInsert}} but not {{handleUpdate()}}. Why does issue only show up with
> {{MERGE_SPLIT_UPDATE}}?
> Once this is fixed, HiveConf.MERGE_SPLIT_UPDATE should be true by default
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-21172) DEFAULT keyword handling in MERGE UPDATE clause issues
[
https://issues.apache.org/jira/browse/HIVE-21172?focusedWorklogId=692674=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-692674
]
ASF GitHub Bot logged work on HIVE-21172:
-
Author: ASF GitHub Bot
Created on: 08/Dec/21 15:33
Start Date: 08/Dec/21 15:33
Worklog Time Spent: 10m
Work Description: kasakrisz opened a new pull request #2857:
URL: https://github.com/apache/hive/pull/2857
### What changes were proposed in this pull request?
When generating the insert part of an update clause of a merge statement
replace the default keywords to the corresponding default values defined in
default constraints.
### Why are the changes needed?
To support splitting update clauses of merge statements into two insert
branches: one for inserting new values and one for inserting into delete
deltas. The first insert doesn't have to be sorted which can give a performance
boost. See [HIVE-21158](https://issues.apache.org/jira/browse/HIVE-21158) and
[HIVE-21159](https://issues.apache.org/jira/browse/HIVE-21159) for more details.
When 'default' keyword is used the optimization can not be triggered because
the compilation of the rewritten statement failed.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
```
mvn test -Dtest.output.overwrite -DskipSparkTests
-Dtest=TestMiniLlapLocalCliDriver -Dqfile=sqlmerge_stats.q -pl itests/qtest
-Pitests
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 692674)
Remaining Estimate: 0h
Time Spent: 10m
> DEFAULT keyword handling in MERGE UPDATE clause issues
> --
>
> Key: HIVE-21172
> URL: https://issues.apache.org/jira/browse/HIVE-21172
> Project: Hive
> Issue Type: Sub-task
> Components: SQL, Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> once HIVE-21159 lands, enable {{HiveConf.MERGE_SPLIT_UPDATE}} and run these
> tests.
> TestMiniLlapLocalCliDriver.testCliDriver[sqlmerge_stats]
> mvn test -Dtest=TestMiniLlapLocalCliDriver
> -Dqfile=insert_into_default_keyword.q
> Merge is rewritten as a multi-insert. When Update clause has DEFAULT, it's
> not properly replaced with a value in the muli-insert - it's treated as a
> literal
> {noformat}
> INSERT INTO `default`.`acidTable`-- update clause(insert part)
> SELECT `t`.`key`, `DEFAULT`, `t`.`value`
>WHERE `t`.`key` = `s`.`key` AND `s`.`key` > 3 AND NOT(`s`.`key` < 3)
> {noformat}
> See {{LOG.info("Going to reparse <" + originalQuery + "> as \n<" +
> rewrittenQueryStr.toString() + ">");}} in hive.log
> {{MergeSemanticAnalyzer.replaceDefaultKeywordForMerge()}} is only called in
> {{handleInsert}} but not {{handleUpdate()}}. Why does issue only show up with
> {{MERGE_SPLIT_UPDATE}}?
> Once this is fixed, HiveConf.MERGE_SPLIT_UPDATE should be true by default
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Assigned] (HIVE-21172) DEFAULT keyword handling in MERGE UPDATE clause issues
[
https://issues.apache.org/jira/browse/HIVE-21172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa reassigned HIVE-21172:
-
Assignee: Krisztian Kasa
> DEFAULT keyword handling in MERGE UPDATE clause issues
> --
>
> Key: HIVE-21172
> URL: https://issues.apache.org/jira/browse/HIVE-21172
> Project: Hive
> Issue Type: Sub-task
> Components: SQL, Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Assignee: Krisztian Kasa
>Priority: Major
>
> once HIVE-21159 lands, enable {{HiveConf.MERGE_SPLIT_UPDATE}} and run these
> tests.
> TestMiniLlapLocalCliDriver.testCliDriver[sqlmerge_stats]
> mvn test -Dtest=TestMiniLlapLocalCliDriver
> -Dqfile=insert_into_default_keyword.q
> Merge is rewritten as a multi-insert. When Update clause has DEFAULT, it's
> not properly replaced with a value in the muli-insert - it's treated as a
> literal
> {noformat}
> INSERT INTO `default`.`acidTable`-- update clause(insert part)
> SELECT `t`.`key`, `DEFAULT`, `t`.`value`
>WHERE `t`.`key` = `s`.`key` AND `s`.`key` > 3 AND NOT(`s`.`key` < 3)
> {noformat}
> See {{LOG.info("Going to reparse <" + originalQuery + "> as \n<" +
> rewrittenQueryStr.toString() + ">");}} in hive.log
> {{MergeSemanticAnalyzer.replaceDefaultKeywordForMerge()}} is only called in
> {{handleInsert}} but not {{handleUpdate()}}. Why does issue only show up with
> {{MERGE_SPLIT_UPDATE}}?
> Once this is fixed, HiveConf.MERGE_SPLIT_UPDATE should be true by default
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (HIVE-25788) Iceberg CTAS should honor location clause and have correct table properties
[ https://issues.apache.org/jira/browse/HIVE-25788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Bod resolved HIVE-25788. --- Resolution: Fixed > Iceberg CTAS should honor location clause and have correct table properties > --- > > Key: HIVE-25788 > URL: https://issues.apache.org/jira/browse/HIVE-25788 > Project: Hive > Issue Type: Bug >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > Currently Iceberg CTAS does not take the LOCATION clause into consideration. > Also, these tables end up with some unintended table properties coming from > the SerDe, such as partition.columns or partition.columns.comments, etc. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25788) Iceberg CTAS should honor location clause and have correct table properties
[ https://issues.apache.org/jira/browse/HIVE-25788?focusedWorklogId=692617=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-692617 ] ASF GitHub Bot logged work on HIVE-25788: - Author: ASF GitHub Bot Created on: 08/Dec/21 14:32 Start Date: 08/Dec/21 14:32 Worklog Time Spent: 10m Work Description: marton-bod merged pull request #2847: URL: https://github.com/apache/hive/pull/2847 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 692617) Time Spent: 1h 20m (was: 1h 10m) > Iceberg CTAS should honor location clause and have correct table properties > --- > > Key: HIVE-25788 > URL: https://issues.apache.org/jira/browse/HIVE-25788 > Project: Hive > Issue Type: Bug >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > Currently Iceberg CTAS does not take the LOCATION clause into consideration. > Also, these tables end up with some unintended table properties coming from > the SerDe, such as partition.columns or partition.columns.comments, etc. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HIVE-25788) Iceberg CTAS should honor location clause and have correct table properties
[ https://issues.apache.org/jira/browse/HIVE-25788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17455277#comment-17455277 ] Marton Bod commented on HIVE-25788: --- Pushed to master. For the reviews, I'd like to thank [~pvary] and [~szita] > Iceberg CTAS should honor location clause and have correct table properties > --- > > Key: HIVE-25788 > URL: https://issues.apache.org/jira/browse/HIVE-25788 > Project: Hive > Issue Type: Bug >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > Currently Iceberg CTAS does not take the LOCATION clause into consideration. > Also, these tables end up with some unintended table properties coming from > the SerDe, such as partition.columns or partition.columns.comments, etc. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-21160) Rewrite Update statement as Multi-insert and do Update split early
[ https://issues.apache.org/jira/browse/HIVE-21160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-21160: -- Labels: pull-request-available (was: ) > Rewrite Update statement as Multi-insert and do Update split early > -- > > Key: HIVE-21160 > URL: https://issues.apache.org/jira/browse/HIVE-21160 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Krisztian Kasa >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-21160) Rewrite Update statement as Multi-insert and do Update split early
[ https://issues.apache.org/jira/browse/HIVE-21160?focusedWorklogId=692539=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-692539 ] ASF GitHub Bot logged work on HIVE-21160: - Author: ASF GitHub Bot Created on: 08/Dec/21 13:07 Start Date: 08/Dec/21 13:07 Worklog Time Spent: 10m Work Description: kasakrisz opened a new pull request #2855: URL: https://github.com/apache/hive/pull/2855 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 692539) Remaining Estimate: 0h Time Spent: 10m > Rewrite Update statement as Multi-insert and do Update split early > -- > > Key: HIVE-21160 > URL: https://issues.apache.org/jira/browse/HIVE-21160 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Krisztian Kasa >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HIVE-21160) Rewrite Update statement as Multi-insert and do Update split early
[ https://issues.apache.org/jira/browse/HIVE-21160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Krisztian Kasa reassigned HIVE-21160: - Assignee: Krisztian Kasa > Rewrite Update statement as Multi-insert and do Update split early > -- > > Key: HIVE-21160 > URL: https://issues.apache.org/jira/browse/HIVE-21160 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Krisztian Kasa >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HIVE-21158) Perform update split early
[
https://issues.apache.org/jira/browse/HIVE-21158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa reassigned HIVE-21158:
-
Assignee: Krisztian Kasa
> Perform update split early
> --
>
> Key: HIVE-21158
> URL: https://issues.apache.org/jira/browse/HIVE-21158
> Project: Hive
> Issue Type: Improvement
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Krisztian Kasa
>Priority: Major
>
> Currently Acid 2.0 does U=D+I in the OrcRecordUpdater. This means that all
> Updates (wide rows) are shuffled AND sorted.
> We could modify the the multi-insert statement which results from Merge
> statement so that instead of having one of the legs represent Update, we
> create 2 legs - 1 representing Delete of original row and 1 representing
> Insert of the new version.
> Delete events are very small so sorting them is cheap. The Insert are
> written to disk in a sorted way by virtue of how ROW__IDs are generated.
> Exactly the same idea applies to regular Update statement.
> Note that the U=D+I in OrcRecordUpdater needs to be kept to keep [Streaming
> Mutate API
> |https://cwiki.apache.org/confluence/display/Hive/HCatalog+Streaming+Mutation+API]
> working on 2.0.
> *This requires that TxnHandler flags 2 Deletes as a conflict - it doesn't
> currently*
> Incidentally, 2.0 + early split allows updating all columns including
> bucketing and partition columns
> What is lock acquisition based on? Need to make sure that conflict detection
> (write set tracking) still works
> So we want to transform
> {noformat}
> update T set B = 7 where A=1
> {noformat}
> into
> {noformat}
> from T
> insert into T select ROW__ID where a = 1 SORT BY ROW__ID
> insert into T select a, 7 where a = 1
> {noformat}
> even better to
> {noformat}
> from T where a = 1
> insert into T select ROW__ID SORT BY ROW__ID
> insert into T select a, 7
> {noformat}
> but this won't parse currently.
> This is very similar to how MERGE stmt is handled.
> Need some though on on how WriteSet tracking works. If we don't allow
> updating partition column, then even with dynamic partitions
> TxnHandler.addDynamicPartitions() should see 1 entry (in Update type) for
> each partition since both the insert and delete land in the same partition.
> If part cols can be updated, then then we may insert a Delete event into P1
> and corresponding Insert event into P2 so addDynamicPartitions() should see
> both parts. I guess both need to be recored in Write_Set but with different
> types. The delete as 'delete' and insert as insert so that it can conflict
> with some IOW on the 'new' partition.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Assigned] (HIVE-25789) Replication metrics and logs show wrong repl id when no of events replicated is 0
[ https://issues.apache.org/jira/browse/HIVE-25789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arko Sharma reassigned HIVE-25789: -- > Replication metrics and logs show wrong repl id when no of events replicated > is 0 > - > > Key: HIVE-25789 > URL: https://issues.apache.org/jira/browse/HIVE-25789 > Project: Hive > Issue Type: Bug >Reporter: Arko Sharma >Assignee: Arko Sharma >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-25789) Replication metrics and logs show wrong repl id when no of events replicated is 0
[ https://issues.apache.org/jira/browse/HIVE-25789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arko Sharma updated HIVE-25789: --- Description: When no of events replicated in incremental cycle is 0, logs and metrics show wrong value of lastReplId. REPL STATUS command still gives the right value. Logs show a value of 'null'. > Replication metrics and logs show wrong repl id when no of events replicated > is 0 > - > > Key: HIVE-25789 > URL: https://issues.apache.org/jira/browse/HIVE-25789 > Project: Hive > Issue Type: Bug >Reporter: Arko Sharma >Assignee: Arko Sharma >Priority: Major > > When no of events replicated in incremental cycle is 0, logs and metrics show > wrong value of lastReplId. REPL STATUS command still gives the right value. > Logs show a value of 'null'. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25335) Unreasonable setting reduce number, when join big size table(but small row count) and small size table
[ https://issues.apache.org/jira/browse/HIVE-25335?focusedWorklogId=692426=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-692426 ] ASF GitHub Bot logged work on HIVE-25335: - Author: ASF GitHub Bot Created on: 08/Dec/21 10:45 Start Date: 08/Dec/21 10:45 Worklog Time Spent: 10m Work Description: zabetak commented on pull request #2490: URL: https://github.com/apache/hive/pull/2490#issuecomment-988700164 Hey @zhengchenyu the change looks reasonable to me. Are you planning to push this forward (update/check) the tests? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 692426) Time Spent: 1.5h (was: 1h 20m) > Unreasonable setting reduce number, when join big size table(but small row > count) and small size table > -- > > Key: HIVE-25335 > URL: https://issues.apache.org/jira/browse/HIVE-25335 > Project: Hive > Issue Type: Improvement >Reporter: zhengchenyu >Assignee: zhengchenyu >Priority: Major > Labels: pull-request-available > Attachments: HIVE-25335.001.patch > > Time Spent: 1.5h > Remaining Estimate: 0h > > I found an application which is slow in our cluster, because the proccess > bytes of one reduce is very huge, but only two reduce. > when I debug, I found the reason. Because in this sql, one big size table > (about 30G) with few row count(about 3.5M), another small size table (about > 100M) have more row count (about 3.6M). So JoinStatsRule.process only use > 100M to estimate reducer's number. But we need to process 30G byte in fact. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25788) Iceberg CTAS should honor location clause and have correct table properties
[ https://issues.apache.org/jira/browse/HIVE-25788?focusedWorklogId=692407=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-692407 ] ASF GitHub Bot logged work on HIVE-25788: - Author: ASF GitHub Bot Created on: 08/Dec/21 10:13 Start Date: 08/Dec/21 10:13 Worklog Time Spent: 10m Work Description: marton-bod commented on a change in pull request #2847: URL: https://github.com/apache/hive/pull/2847#discussion_r764725069 ## File path: iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergSerDe.java ## @@ -166,22 +167,33 @@ private void createTableForCTAS(Configuration configuration, Properties serDePro serDeProperties.put(InputFormatConfig.PARTITION_SPEC, PartitionSpecParser.toJson(spec)); } +// clean up the properties for table creation (so that internal serde props don't become table props) +Properties createProps = getCTASTableCreationProperties(serDeProperties); Review comment: I mainly extracted into a function for readability this time, not reuse. But if you think it's better to inline, I can do it as I don't feel strongly about it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 692407) Time Spent: 1h 10m (was: 1h) > Iceberg CTAS should honor location clause and have correct table properties > --- > > Key: HIVE-25788 > URL: https://issues.apache.org/jira/browse/HIVE-25788 > Project: Hive > Issue Type: Bug >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > Currently Iceberg CTAS does not take the LOCATION clause into consideration. > Also, these tables end up with some unintended table properties coming from > the SerDe, such as partition.columns or partition.columns.comments, etc. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25788) Iceberg CTAS should honor location clause and have correct table properties
[
https://issues.apache.org/jira/browse/HIVE-25788?focusedWorklogId=692402=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-692402
]
ASF GitHub Bot logged work on HIVE-25788:
-
Author: ASF GitHub Bot
Created on: 08/Dec/21 10:01
Start Date: 08/Dec/21 10:01
Worklog Time Spent: 10m
Work Description: marton-bod commented on a change in pull request #2847:
URL: https://github.com/apache/hive/pull/2847#discussion_r764716043
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
##
@@ -7705,6 +7705,13 @@ protected Operator genFileSinkPlan(String dest, QB qb,
Operator input)
} else {
tableDescriptor = PlanUtils.getTableDesc(tblDesc, cols, colTypes);
}
+
+ // if available, set location in table desc properties
+ if (tblDesc != null && tblDesc.getLocation() != null && tableDescriptor
!= null &&
+
!tableDescriptor.getProperties().containsKey(hive_metastoreConstants.META_TABLE_LOCATION))
{
Review comment:
If the location clause is present in the query, the translator should
not touch that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 692402)
Time Spent: 1h (was: 50m)
> Iceberg CTAS should honor location clause and have correct table properties
> ---
>
> Key: HIVE-25788
> URL: https://issues.apache.org/jira/browse/HIVE-25788
> Project: Hive
> Issue Type: Bug
>Reporter: Marton Bod
>Assignee: Marton Bod
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Currently Iceberg CTAS does not take the LOCATION clause into consideration.
> Also, these tables end up with some unintended table properties coming from
> the SerDe, such as partition.columns or partition.columns.comments, etc.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25788) Iceberg CTAS should honor location clause and have correct table properties
[
https://issues.apache.org/jira/browse/HIVE-25788?focusedWorklogId=692400=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-692400
]
ASF GitHub Bot logged work on HIVE-25788:
-
Author: ASF GitHub Bot
Created on: 08/Dec/21 10:01
Start Date: 08/Dec/21 10:01
Worklog Time Spent: 10m
Work Description: marton-bod commented on a change in pull request #2847:
URL: https://github.com/apache/hive/pull/2847#discussion_r764715611
##
File path:
iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergCTAS.java
##
@@ -88,6 +89,35 @@ public void testCTASPartitionedFromHiveTable() throws
TException, InterruptedExc
Assert.assertEquals("name", table.spec().fields().get(1).name());
}
+ @Test
+ public void testCTASTblPropsAndLocationClause() throws Exception {
+Assume.assumeTrue(HiveIcebergSerDe.CTAS_EXCEPTION_MSG, testTableType ==
TestTables.TestTableType.HIVE_CATALOG);
+
+shell.executeStatement("CREATE TABLE source (id bigint, name string)
PARTITIONED BY (dept string) STORED AS ORC");
+shell.executeStatement(testTables.getInsertQuery(
+HiveIcebergStorageHandlerTestUtils.CUSTOMER_RECORDS,
TableIdentifier.of("default", "source"), false));
+
+String location = temp.newFolder().toURI().toString();
+shell.executeStatement(String.format(
Review comment:
This new test contains the user-added location clause: `LOCATION '%s'`
The earlier CTAS tests contain tests without the location clause.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 692400)
Time Spent: 50m (was: 40m)
> Iceberg CTAS should honor location clause and have correct table properties
> ---
>
> Key: HIVE-25788
> URL: https://issues.apache.org/jira/browse/HIVE-25788
> Project: Hive
> Issue Type: Bug
>Reporter: Marton Bod
>Assignee: Marton Bod
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Currently Iceberg CTAS does not take the LOCATION clause into consideration.
> Also, these tables end up with some unintended table properties coming from
> the SerDe, such as partition.columns or partition.columns.comments, etc.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HIVE-25781) Restore multi-threaded support in Cleaner after HIVE-25115
[ https://issues.apache.org/jira/browse/HIVE-25781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-25781: -- Labels: pull-request-available (was: ) > Restore multi-threaded support in Cleaner after HIVE-25115 > -- > > Key: HIVE-25781 > URL: https://issues.apache.org/jira/browse/HIVE-25781 > Project: Hive > Issue Type: Bug >Reporter: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > In the case of accumulated entries for the same table/partition, they should > be processed one by one in ASC order of write_id's. > However, to support multi-threaded processing in the Cleaner, we need to make > sure that the entries from the same group are not processed concurrently by > multiple threads. > One of the solutions that don't involve synchronization (check if already in > flight, re-try when possible) would be to move the entries from the same > group to the next Cleaner cycle. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25781) Restore multi-threaded support in Cleaner after HIVE-25115
[ https://issues.apache.org/jira/browse/HIVE-25781?focusedWorklogId=692326=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-692326 ] ASF GitHub Bot logged work on HIVE-25781: - Author: ASF GitHub Bot Created on: 08/Dec/21 09:00 Start Date: 08/Dec/21 09:00 Worklog Time Spent: 10m Work Description: deniskuzZ merged pull request #2825: URL: https://github.com/apache/hive/pull/2825 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 692326) Remaining Estimate: 0h Time Spent: 10m > Restore multi-threaded support in Cleaner after HIVE-25115 > -- > > Key: HIVE-25781 > URL: https://issues.apache.org/jira/browse/HIVE-25781 > Project: Hive > Issue Type: Bug >Reporter: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > In the case of accumulated entries for the same table/partition, they should > be processed one by one in ASC order of write_id's. > However, to support multi-threaded processing in the Cleaner, we need to make > sure that the entries from the same group are not processed concurrently by > multiple threads. > One of the solutions that don't involve synchronization (check if already in > flight, re-try when possible) would be to move the entries from the same > group to the next Cleaner cycle. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25788) Iceberg CTAS should honor location clause and have correct table properties
[
https://issues.apache.org/jira/browse/HIVE-25788?focusedWorklogId=692325=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-692325
]
ASF GitHub Bot logged work on HIVE-25788:
-
Author: ASF GitHub Bot
Created on: 08/Dec/21 08:59
Start Date: 08/Dec/21 08:59
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #2847:
URL: https://github.com/apache/hive/pull/2847#discussion_r764665772
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
##
@@ -7705,6 +7705,13 @@ protected Operator genFileSinkPlan(String dest, QB qb,
Operator input)
} else {
tableDescriptor = PlanUtils.getTableDesc(tblDesc, cols, colTypes);
}
+
+ // if available, set location in table desc properties
+ if (tblDesc != null && tblDesc.getLocation() != null && tableDescriptor
!= null &&
+
!tableDescriptor.getProperties().containsKey(hive_metastoreConstants.META_TABLE_LOCATION))
{
Review comment:
how does this work with the HMSTranslators? (These are the stuff which
makes sure that external/managed table is put into the correct location)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 692325)
Time Spent: 40m (was: 0.5h)
> Iceberg CTAS should honor location clause and have correct table properties
> ---
>
> Key: HIVE-25788
> URL: https://issues.apache.org/jira/browse/HIVE-25788
> Project: Hive
> Issue Type: Bug
>Reporter: Marton Bod
>Assignee: Marton Bod
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Currently Iceberg CTAS does not take the LOCATION clause into consideration.
> Also, these tables end up with some unintended table properties coming from
> the SerDe, such as partition.columns or partition.columns.comments, etc.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25788) Iceberg CTAS should honor location clause and have correct table properties
[
https://issues.apache.org/jira/browse/HIVE-25788?focusedWorklogId=692323=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-692323
]
ASF GitHub Bot logged work on HIVE-25788:
-
Author: ASF GitHub Bot
Created on: 08/Dec/21 08:57
Start Date: 08/Dec/21 08:57
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #2847:
URL: https://github.com/apache/hive/pull/2847#discussion_r764664451
##
File path:
iceberg/iceberg-handler/src/test/java/org/apache/iceberg/mr/hive/TestHiveIcebergCTAS.java
##
@@ -88,6 +89,35 @@ public void testCTASPartitionedFromHiveTable() throws
TException, InterruptedExc
Assert.assertEquals("name", table.spec().fields().get(1).name());
}
+ @Test
+ public void testCTASTblPropsAndLocationClause() throws Exception {
+Assume.assumeTrue(HiveIcebergSerDe.CTAS_EXCEPTION_MSG, testTableType ==
TestTables.TestTableType.HIVE_CATALOG);
+
+shell.executeStatement("CREATE TABLE source (id bigint, name string)
PARTITIONED BY (dept string) STORED AS ORC");
+shell.executeStatement(testTables.getInsertQuery(
+HiveIcebergStorageHandlerTestUtils.CUSTOMER_RECORDS,
TableIdentifier.of("default", "source"), false));
+
+String location = temp.newFolder().toURI().toString();
+shell.executeStatement(String.format(
Review comment:
Can we add a use-case where the location is provided by the user?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 692323)
Time Spent: 0.5h (was: 20m)
> Iceberg CTAS should honor location clause and have correct table properties
> ---
>
> Key: HIVE-25788
> URL: https://issues.apache.org/jira/browse/HIVE-25788
> Project: Hive
> Issue Type: Bug
>Reporter: Marton Bod
>Assignee: Marton Bod
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Currently Iceberg CTAS does not take the LOCATION clause into consideration.
> Also, these tables end up with some unintended table properties coming from
> the SerDe, such as partition.columns or partition.columns.comments, etc.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25788) Iceberg CTAS should honor location clause and have correct table properties
[ https://issues.apache.org/jira/browse/HIVE-25788?focusedWorklogId=692321=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-692321 ] ASF GitHub Bot logged work on HIVE-25788: - Author: ASF GitHub Bot Created on: 08/Dec/21 08:56 Start Date: 08/Dec/21 08:56 Worklog Time Spent: 10m Work Description: pvary commented on a change in pull request #2847: URL: https://github.com/apache/hive/pull/2847#discussion_r764663048 ## File path: iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergSerDe.java ## @@ -166,22 +167,33 @@ private void createTableForCTAS(Configuration configuration, Properties serDePro serDeProperties.put(InputFormatConfig.PARTITION_SPEC, PartitionSpecParser.toJson(spec)); } +// clean up the properties for table creation (so that internal serde props don't become table props) +Properties createProps = getCTASTableCreationProperties(serDeProperties); Review comment: If not reused, why not call it inline? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 692321) Time Spent: 20m (was: 10m) > Iceberg CTAS should honor location clause and have correct table properties > --- > > Key: HIVE-25788 > URL: https://issues.apache.org/jira/browse/HIVE-25788 > Project: Hive > Issue Type: Bug >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > Currently Iceberg CTAS does not take the LOCATION clause into consideration. > Also, these tables end up with some unintended table properties coming from > the SerDe, such as partition.columns or partition.columns.comments, etc. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25788) Iceberg CTAS should honor location clause and have correct table properties
[
https://issues.apache.org/jira/browse/HIVE-25788?focusedWorklogId=692319=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-692319
]
ASF GitHub Bot logged work on HIVE-25788:
-
Author: ASF GitHub Bot
Created on: 08/Dec/21 08:55
Start Date: 08/Dec/21 08:55
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #2847:
URL: https://github.com/apache/hive/pull/2847#discussion_r764662834
##
File path:
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergSerDe.java
##
@@ -166,22 +167,33 @@ private void createTableForCTAS(Configuration
configuration, Properties serDePro
serDeProperties.put(InputFormatConfig.PARTITION_SPEC,
PartitionSpecParser.toJson(spec));
}
+// clean up the properties for table creation (so that internal serde
props don't become table props)
+Properties createProps = getCTASTableCreationProperties(serDeProperties);
+
// create CTAS table
LOG.info("Creating table {} for CTAS with schema: {}, and spec: {}",
serDeProperties.get(Catalogs.NAME), tableSchema,
serDeProperties.get(InputFormatConfig.PARTITION_SPEC));
-Catalogs.createTable(configuration, serDeProperties);
+Catalogs.createTable(configuration, createProps);
// set this in the query state so that we can rollback the table in the
lifecycle hook in case of failures
SessionStateUtil.addResource(configuration,
InputFormatConfig.CTAS_TABLE_NAME,
serDeProperties.getProperty(Catalogs.NAME));
}
- private void assertNotVectorizedTez(Configuration configuration) {
-if ("tez".equals(configuration.get("hive.execution.engine")) &&
-"true".equals(configuration.get("hive.vectorized.execution.enabled")))
{
- throw new UnsupportedOperationException("Vectorized execution on Tez is
currently not supported when using " +
- "Iceberg tables. Please set hive.vectorized.execution.enabled=false
and rerun the query.");
-}
+ private Properties getCTASTableCreationProperties(Properties
serDeProperties) {
Review comment:
maybe `filterCTASTableCreationProperties`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 692319)
Remaining Estimate: 0h
Time Spent: 10m
> Iceberg CTAS should honor location clause and have correct table properties
> ---
>
> Key: HIVE-25788
> URL: https://issues.apache.org/jira/browse/HIVE-25788
> Project: Hive
> Issue Type: Bug
>Reporter: Marton Bod
>Assignee: Marton Bod
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently Iceberg CTAS does not take the LOCATION clause into consideration.
> Also, these tables end up with some unintended table properties coming from
> the SerDe, such as partition.columns or partition.columns.comments, etc.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HIVE-25788) Iceberg CTAS should honor location clause and have correct table properties
[ https://issues.apache.org/jira/browse/HIVE-25788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-25788: -- Labels: pull-request-available (was: ) > Iceberg CTAS should honor location clause and have correct table properties > --- > > Key: HIVE-25788 > URL: https://issues.apache.org/jira/browse/HIVE-25788 > Project: Hive > Issue Type: Bug >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Currently Iceberg CTAS does not take the LOCATION clause into consideration. > Also, these tables end up with some unintended table properties coming from > the SerDe, such as partition.columns or partition.columns.comments, etc. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HIVE-25788) Iceberg CTAS should honor location clause and have correct table properties
[ https://issues.apache.org/jira/browse/HIVE-25788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marton Bod reassigned HIVE-25788: - > Iceberg CTAS should honor location clause and have correct table properties > --- > > Key: HIVE-25788 > URL: https://issues.apache.org/jira/browse/HIVE-25788 > Project: Hive > Issue Type: Bug >Reporter: Marton Bod >Assignee: Marton Bod >Priority: Major > > Currently Iceberg CTAS does not take the LOCATION clause into consideration. > Also, these tables end up with some unintended table properties coming from > the SerDe, such as partition.columns or partition.columns.comments, etc. -- This message was sent by Atlassian Jira (v8.20.1#820001)
