[jira] [Resolved] (HIVE-25763) HIVE IS BAD
[ https://issues.apache.org/jira/browse/HIVE-25763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pravin Pawar resolved HIVE-25763. - Release Note: OK Resolution: Fixed > HIVE IS BAD > --- > > Key: HIVE-25763 > URL: https://issues.apache.org/jira/browse/HIVE-25763 > Project: Hive > Issue Type: Bug >Reporter: Pravin Pawar >Assignee: Pravin Pawar >Priority: Critical > > BAD -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HIVE-25763) HIVE IS BAD
[ https://issues.apache.org/jira/browse/HIVE-25763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17452180#comment-17452180 ] Pravin Pawar commented on HIVE-25763: - OK > HIVE IS BAD > --- > > Key: HIVE-25763 > URL: https://issues.apache.org/jira/browse/HIVE-25763 > Project: Hive > Issue Type: Bug >Reporter: Pravin Pawar >Assignee: Pravin Pawar >Priority: Critical > > BAD -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work started] (HIVE-25763) HIVE IS BAD
[ https://issues.apache.org/jira/browse/HIVE-25763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-25763 started by Pravin Pawar. --- > HIVE IS BAD > --- > > Key: HIVE-25763 > URL: https://issues.apache.org/jira/browse/HIVE-25763 > Project: Hive > Issue Type: Bug >Reporter: Pravin Pawar >Assignee: Pravin Pawar >Priority: Critical > > BAD -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HIVE-25763) HIVE IS BAD
[ https://issues.apache.org/jira/browse/HIVE-25763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pravin Pawar reassigned HIVE-25763: --- Assignee: Pravin Pawar > HIVE IS BAD > --- > > Key: HIVE-25763 > URL: https://issues.apache.org/jira/browse/HIVE-25763 > Project: Hive > Issue Type: Bug >Reporter: Pravin Pawar >Assignee: Pravin Pawar >Priority: Critical > > BAD -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25729) ThriftUnionObjectInspector should be notified when fully inited
[
https://issues.apache.org/jira/browse/HIVE-25729?focusedWorklogId=689042=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-689042
]
ASF GitHub Bot logged work on HIVE-25729:
-
Author: ASF GitHub Bot
Created on: 02/Dec/21 04:38
Start Date: 02/Dec/21 04:38
Worklog Time Spent: 10m
Work Description: dengzhhu653 commented on pull request #2808:
URL: https://github.com/apache/hive/pull/2808#issuecomment-984286215
> Sorry @dengzhhu653 , was on vacation. Can you add some details on why this
change is necessary? thanks.
Thank you for the feedback.
[HIVE-11586](https://issues.apache.org/jira/browse/HIVE-11586) has solved the
problem that a uninitialized instance of ReflectionStructObjectInspector may be
returned in case of multi threads trying to get it, npe would be thrown if we
refer to some fields of the inspector. In order to make true that the
ReflectionStructObjectInspector is fully inited, [others may wait for up to 3
seconds](https://github.com/apache/hive/blob/master/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java#L106-L121).
The
[ThriftStructObjectInspector](https://github.com/apache/hive/blob/7b3ecf617a6d46f48a3b6f77e0339fd4ad95a420/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ReflectionStructObjectInspector.java#L165-L178)
will notify the monitor on finishing initialization, but
[ThriftUnionObjectInspector](https://github.com/apache/hive/blob/master/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ThriftUnionObjectInspector.java#L95-L104)
misses to do this in his own overriden `init` method.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 689042)
Time Spent: 40m (was: 0.5h)
> ThriftUnionObjectInspector should be notified when fully inited
> ---
>
> Key: HIVE-25729
> URL: https://issues.apache.org/jira/browse/HIVE-25729
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> For thread safe purpose, a ReflectionStructObjectInspector instance would
> wait for 3 seconds to ensure the returning ObjectInspector is fully inited,
> {code:java}
> synchronized (soi) {
> while (!soi.isFullyInited(checkedTypes)) {
> //
> soi.wait(3000);
> }
> } {code}
> It seems that we are missing to notify ThriftUnionObjectInspector when it has
> been inited.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25729) ThriftUnionObjectInspector should be notified when fully inited
[
https://issues.apache.org/jira/browse/HIVE-25729?focusedWorklogId=689033=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-689033
]
ASF GitHub Bot logged work on HIVE-25729:
-
Author: ASF GitHub Bot
Created on: 02/Dec/21 03:48
Start Date: 02/Dec/21 03:48
Worklog Time Spent: 10m
Work Description: sunchao commented on pull request #2808:
URL: https://github.com/apache/hive/pull/2808#issuecomment-984267079
Sorry @dengzhhu653 , was on vacation. Can you add some details on why this
change is necessary? thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 689033)
Time Spent: 0.5h (was: 20m)
> ThriftUnionObjectInspector should be notified when fully inited
> ---
>
> Key: HIVE-25729
> URL: https://issues.apache.org/jira/browse/HIVE-25729
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Reporter: Zhihua Deng
>Assignee: Zhihua Deng
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> For thread safe purpose, a ReflectionStructObjectInspector instance would
> wait for 3 seconds to ensure the returning ObjectInspector is fully inited,
> {code:java}
> synchronized (soi) {
> while (!soi.isFullyInited(checkedTypes)) {
> //
> soi.wait(3000);
> }
> } {code}
> It seems that we are missing to notify ThriftUnionObjectInspector when it has
> been inited.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Assigned] (HIVE-25762) ok
[ https://issues.apache.org/jira/browse/HIVE-25762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pravin Pawar reassigned HIVE-25762: --- Assignee: Pravin Pawar ok > ok > -- > > Key: HIVE-25762 > URL: https://issues.apache.org/jira/browse/HIVE-25762 > Project: Hive > Issue Type: Bug >Reporter: Pravin Pawar >Assignee: Pravin Pawar >Priority: Major > > ok -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HIVE-25761) unable see hive jobs
[ https://issues.apache.org/jira/browse/HIVE-25761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pravin Pawar resolved HIVE-25761. - Release Note: resolved Resolution: Fixed > unable see hive jobs > > > Key: HIVE-25761 > URL: https://issues.apache.org/jira/browse/HIVE-25761 > Project: Hive > Issue Type: Bug >Reporter: Pravin Pawar >Assignee: Pravin Pawar >Priority: Critical > > no able to login -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HIVE-25761) unable see hive jobs
[ https://issues.apache.org/jira/browse/HIVE-25761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17452141#comment-17452141 ] Pravin Pawar commented on HIVE-25761: - rca issue reolved > unable see hive jobs > > > Key: HIVE-25761 > URL: https://issues.apache.org/jira/browse/HIVE-25761 > Project: Hive > Issue Type: Bug >Reporter: Pravin Pawar >Assignee: Pravin Pawar >Priority: Critical > > no able to login -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work started] (HIVE-25761) unable see hive jobs
[ https://issues.apache.org/jira/browse/HIVE-25761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-25761 started by Pravin Pawar. --- > unable see hive jobs > > > Key: HIVE-25761 > URL: https://issues.apache.org/jira/browse/HIVE-25761 > Project: Hive > Issue Type: Bug >Reporter: Pravin Pawar >Assignee: Pravin Pawar >Priority: Critical > > no able to login -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HIVE-25761) unable see hive jobs
[ https://issues.apache.org/jira/browse/HIVE-25761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pravin Pawar reassigned HIVE-25761: --- Assignee: Pravin Pawar > unable see hive jobs > > > Key: HIVE-25761 > URL: https://issues.apache.org/jira/browse/HIVE-25761 > Project: Hive > Issue Type: Bug >Reporter: Pravin Pawar >Assignee: Pravin Pawar >Priority: Critical > > no able to login -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work started] (HIVE-25760) impala is not running
[ https://issues.apache.org/jira/browse/HIVE-25760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-25760 started by Pravin Pawar. --- > impala is not running > - > > Key: HIVE-25760 > URL: https://issues.apache.org/jira/browse/HIVE-25760 > Project: Hive > Issue Type: Bug >Reporter: Pravin Pawar >Assignee: Pravin Pawar >Priority: Critical > > impala is not working properly -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HIVE-25760) impala is not running
[ https://issues.apache.org/jira/browse/HIVE-25760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pravin Pawar resolved HIVE-25760. - Release Note: issue resolved Resolution: Fixed > impala is not running > - > > Key: HIVE-25760 > URL: https://issues.apache.org/jira/browse/HIVE-25760 > Project: Hive > Issue Type: Bug >Reporter: Pravin Pawar >Assignee: Pravin Pawar >Priority: Critical > > impala is not working properly -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HIVE-25760) impala is not running
[ https://issues.apache.org/jira/browse/HIVE-25760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17452136#comment-17452136 ] Pravin Pawar commented on HIVE-25760: - issue resolved > impala is not running > - > > Key: HIVE-25760 > URL: https://issues.apache.org/jira/browse/HIVE-25760 > Project: Hive > Issue Type: Bug >Reporter: Pravin Pawar >Assignee: Pravin Pawar >Priority: Critical > > impala is not working properly -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HIVE-25760) impala is not running
[ https://issues.apache.org/jira/browse/HIVE-25760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pravin Pawar reassigned HIVE-25760: --- Assignee: Pravin Pawar > impala is not running > - > > Key: HIVE-25760 > URL: https://issues.apache.org/jira/browse/HIVE-25760 > Project: Hive > Issue Type: Bug >Reporter: Pravin Pawar >Assignee: Pravin Pawar >Priority: Critical > > impala is not working properly -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HIVE-25759) unable to raise hive jobs
[ https://issues.apache.org/jira/browse/HIVE-25759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pravin Pawar resolved HIVE-25759. - Release Note: issue resolved Resolution: Fixed > unable to raise hive jobs > - > > Key: HIVE-25759 > URL: https://issues.apache.org/jira/browse/HIVE-25759 > Project: Hive > Issue Type: Bug > Components: Avro >Reporter: Pravin Pawar >Assignee: Pravin Pawar >Priority: Minor > > unable to run job -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HIVE-25759) unable to raise hive jobs
[ https://issues.apache.org/jira/browse/HIVE-25759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17452125#comment-17452125 ] Pravin Pawar commented on HIVE-25759: - RCA - issue resolved > unable to raise hive jobs > - > > Key: HIVE-25759 > URL: https://issues.apache.org/jira/browse/HIVE-25759 > Project: Hive > Issue Type: Bug > Components: Avro >Reporter: Pravin Pawar >Assignee: Pravin Pawar >Priority: Minor > > unable to run job -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work started] (HIVE-25759) unable to raise hive jobs
[ https://issues.apache.org/jira/browse/HIVE-25759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-25759 started by Pravin Pawar. --- > unable to raise hive jobs > - > > Key: HIVE-25759 > URL: https://issues.apache.org/jira/browse/HIVE-25759 > Project: Hive > Issue Type: Bug > Components: Avro >Reporter: Pravin Pawar >Assignee: Pravin Pawar >Priority: Minor > > unable to run job -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HIVE-25759) unable to raise hive jobs
[ https://issues.apache.org/jira/browse/HIVE-25759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pravin Pawar reassigned HIVE-25759: --- > unable to raise hive jobs > - > > Key: HIVE-25759 > URL: https://issues.apache.org/jira/browse/HIVE-25759 > Project: Hive > Issue Type: Bug > Components: Avro >Reporter: Pravin Pawar >Assignee: Pravin Pawar >Priority: Minor > > unable to run job -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25652) Add constraints in result of “SHOW CREATE TABLE ”
[
https://issues.apache.org/jira/browse/HIVE-25652?focusedWorklogId=688998=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-688998
]
ASF GitHub Bot logged work on HIVE-25652:
-
Author: ASF GitHub Bot
Created on: 02/Dec/21 01:47
Start Date: 02/Dec/21 01:47
Worklog Time Spent: 10m
Work Description: soumyakanti3578 commented on a change in pull request
#2777:
URL: https://github.com/apache/hive/pull/2777#discussion_r760702646
##
File path:
ql/src/test/results/clientpositive/llap/show_create_table_db_table.q.out
##
@@ -120,8 +120,8 @@ POSTHOOK: query: SHOW CREATE TABLE tmp_feng.tmp_showcrt4
POSTHOOK: type: SHOW_CREATETABLE
POSTHOOK: Input: tmp_feng@tmp_showcrt4
CREATE TABLE `tmp_feng`.`tmp_showcrt4`(
- `s1` struct<`p1`:string>,
- `s2` struct<`p2`:array,
array>)
+ `s1` struct<``p1``:string>,
+ `s2` struct<``p2``:array,
array>)
Review comment:
Hey @kasakrisz , what do you think about these changes? These occurred
due to the change that you requested. I feel these are fine, but please let me
know if you think otherwise.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 688998)
Time Spent: 4h 50m (was: 4h 40m)
> Add constraints in result of “SHOW CREATE TABLE ”
> -
>
> Key: HIVE-25652
> URL: https://issues.apache.org/jira/browse/HIVE-25652
> Project: Hive
> Issue Type: Improvement
>Reporter: Soumyakanti Das
>Assignee: Soumyakanti Das
>Priority: Major
> Labels: pull-request-available
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
> Currently show create table doesn’t pull any constraint info like not null,
> defaults, primary key.
> Example:
> Create table
>
> {code:java}
> CREATE TABLE TEST(
> col1 varchar(100) NOT NULL COMMENT "comment for column 1",
> col2 timestamp DEFAULT CURRENT_TIMESTAMP() COMMENT "comment for column 2",
> col3 decimal,
> col4 varchar(512) NOT NULL,
> col5 varchar(100),
> primary key(col1, col2) disable novalidate)
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat';
> {code}
> Currently {{SHOW CREATE TABLE TEST}} doesn't show the column constraints.
> {code:java}
> CREATE TABLE `test`(
> `col1` varchar(100) COMMENT 'comment for column 1',
> `col2` timestamp COMMENT 'comment for column 2',
> `col3` decimal(10,0),
> `col4` varchar(512),
> `col5` varchar(100))
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.ql.io.orc.OrcSerde'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcInputFormat'
> OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat'
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25708) Implement creation of table_diff
[
https://issues.apache.org/jira/browse/HIVE-25708?focusedWorklogId=688977=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-688977
]
ASF GitHub Bot logged work on HIVE-25708:
-
Author: ASF GitHub Bot
Created on: 01/Dec/21 23:45
Start Date: 01/Dec/21 23:45
Worklog Time Spent: 10m
Work Description: pkumarsinha commented on a change in pull request #2795:
URL: https://github.com/apache/hive/pull/2795#discussion_r760635567
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/repl/OptimisedBootstrapUtils.java
##
@@ -0,0 +1,290 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.exec.repl;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FileStatus;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.RemoteIterator;
+import org.apache.hadoop.hive.common.FileUtils;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.api.Database;
+import org.apache.hadoop.hive.metastore.api.NotificationEvent;
+import
org.apache.hadoop.hive.metastore.messaging.event.filters.DatabaseAndTableFilter;
+import org.apache.hadoop.hive.metastore.utils.MetaStoreUtils;
+import org.apache.hadoop.hive.ql.exec.repl.util.ReplUtils;
+import org.apache.hadoop.hive.ql.metadata.Hive;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.metadata.Partition;
+import org.apache.hadoop.hive.ql.metadata.Table;
+import org.apache.hadoop.hive.ql.parse.SemanticException;
+import org.apache.hadoop.hive.ql.parse.repl.DumpType;
+import org.apache.hadoop.hive.ql.parse.repl.dump.Utils;
+import org.apache.hadoop.hive.ql.parse.repl.load.DumpMetaData;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.nio.charset.Charset;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import static
org.apache.hadoop.hive.ql.parse.ReplicationSpec.getLastReplicatedStateFromParameters;
+
+public class OptimisedBootstrapUtils {
Review comment:
Can you pls add a javadoc comment for the class?
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/parse/TestReplicationOptimisedBootstrap.java
##
@@ -0,0 +1,409 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.parse;
+
+import org.apache.hadoop.fs.ContentSummary;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hdfs.DistributedFileSystem;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import
org.apache.hadoop.hive.metastore.messaging.json.gzip.GzipJSONMessageEncoder;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.security.UserGroupInformation;
+
+import org.junit.After;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+
+import static org.apache.hadoop.hdfs.protocol.HdfsConstants.QUOTA_RESET;
+import static
[jira] [Updated] (HIVE-25758) OOM due to recursive application of CBO rules
[
https://issues.apache.org/jira/browse/HIVE-25758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alessandro Solimando updated HIVE-25758:
Summary: OOM due to recursive application of CBO rules (was: OOM due to
recursive application CBO rules)
> OOM due to recursive application of CBO rules
> -
>
> Key: HIVE-25758
> URL: https://issues.apache.org/jira/browse/HIVE-25758
> Project: Hive
> Issue Type: Bug
> Components: CBO, Query Planning
>Affects Versions: 4.0.0
>Reporter: Alessandro Solimando
>Priority: Major
>
>
> Reproducing query is as follows:
> {code:java}
> create table test1 (act_nbr string);
> create table test2 (month int);
> create table test3 (mth int, con_usd double);
> EXPLAIN
>SELECT c.month,
> d.con_usd
>FROM
> (SELECT
> cast(regexp_replace(substr(add_months(from_unixtime(unix_timestamp(),
> '-MM-dd'), -1), 1, 7), '-', '') AS int) AS month
> FROM test1
> UNION ALL
> SELECT month
> FROM test2
> WHERE month = 202110) c
>JOIN test3 d ON c.month = d.mth; {code}
>
> Different plans are generated during the first CBO steps, last being:
> {noformat}
> 2021-12-01T08:28:08,598 DEBUG [a18191bb-3a2b-4193-9abf-4e37dd1996bb main]
> parse.CalcitePlanner: Plan after decorre
> lation:
> HiveProject(month=[$0], con_usd=[$2])
> HiveJoin(condition=[=($0, $1)], joinType=[inner], algorithm=[none],
> cost=[not available])
> HiveProject(month=[$0])
> HiveUnion(all=[true])
>
> HiveProject(month=[CAST(regexp_replace(substr(add_months(FROM_UNIXTIME(UNIX_TIMESTAMP,
> _UTF-16LE'-MM-d
> d':VARCHAR(2147483647) CHARACTER SET "UTF-16LE"), -1), 1, 7),
> _UTF-16LE'-':VARCHAR(2147483647) CHARACTER SET "UTF-
> 16LE", _UTF-16LE'':VARCHAR(2147483647) CHARACTER SET "UTF-16LE")):INTEGER])
> HiveTableScan(table=[[default, test1]], table:alias=[test1])
> HiveProject(month=[$0])
> HiveFilter(condition=[=($0, CAST(202110):INTEGER)])
> HiveTableScan(table=[[default, test2]], table:alias=[test2])
> HiveTableScan(table=[[default, test3]], table:alias=[d]){noformat}
>
> Then, the HEP planner will keep expanding the filter expression with
> redundant expressions, such as the following, where the identical CAST
> expression is present multiple times:
>
> {noformat}
> rel#118:HiveFilter.HIVE.[].any(input=HepRelVertex#39,condition=IN(CAST(regexp_replace(substr(add_months(FROM_UNIXTIME(UNIX_TIMESTAMP,
> _UTF-16LE'-MM-dd':VARCHAR(2147483647) CHARACTER SET "UTF-16LE"), -1), 1,
> 7), _UTF-16LE'-':VARCHAR(2147483647) CHARACTER SET "UTF-16LE",
> _UTF-16LE'':VARCHAR(2147483647) CHARACTER SET "UTF-16LE")):INTEGER,
> CAST(regexp_replace(substr(add_months(FROM_UNIXTIME(UNIX_TIMESTAMP,
> _UTF-16LE'-MM-dd':VARCHAR(2147483647) CHARACTER SET "UTF-16LE"), -1), 1,
> 7), _UTF-16LE'-':VARCHAR(2147483647) CHARACTER SET "UTF-16LE",
> _UTF-16LE'':VARCHAR(2147483647) CHARACTER SET "UTF-16LE")):INTEGER,
> 202110)){noformat}
>
> The problem seems to come from a bad interaction of at least
> _HiveFilterProjectTransposeRule_ and
> {_}HiveJoinPushTransitivePredicatesRule{_}, possibly more.
> Most probably then UNION part can be removed and the reproducer be simplified
> even further.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25757) Use cached database type to choose metastore backend queries
[ https://issues.apache.org/jira/browse/HIVE-25757?focusedWorklogId=688786=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-688786 ] ASF GitHub Bot logged work on HIVE-25757: - Author: ASF GitHub Bot Created on: 01/Dec/21 16:59 Start Date: 01/Dec/21 16:59 Worklog Time Spent: 10m Work Description: yongzhi opened a new pull request #2833: URL: https://github.com/apache/hive/pull/2833 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 688786) Remaining Estimate: 0h Time Spent: 10m > Use cached database type to choose metastore backend queries > > > Key: HIVE-25757 > URL: https://issues.apache.org/jira/browse/HIVE-25757 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 4.0.0 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > In HIVE-21075, we use DatabaseProduct.determineDatabaseProduct which can be > expensive. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-25757) Use cached database type to choose metastore backend queries
[ https://issues.apache.org/jira/browse/HIVE-25757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-25757: -- Labels: pull-request-available (was: ) > Use cached database type to choose metastore backend queries > > > Key: HIVE-25757 > URL: https://issues.apache.org/jira/browse/HIVE-25757 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 4.0.0 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > In HIVE-21075, we use DatabaseProduct.determineDatabaseProduct which can be > expensive. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HIVE-25609) Preserve XAttrs in normal file copy case.
[ https://issues.apache.org/jira/browse/HIVE-25609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pravin Sinha resolved HIVE-25609. - Resolution: Fixed Committed to master. Thanks for the patch [~haymant] !! > Preserve XAttrs in normal file copy case. > - > > Key: HIVE-25609 > URL: https://issues.apache.org/jira/browse/HIVE-25609 > Project: Hive > Issue Type: Improvement >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25738) NullIf doesn't support complex types
[
https://issues.apache.org/jira/browse/HIVE-25738?focusedWorklogId=688772=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-688772
]
ASF GitHub Bot logged work on HIVE-25738:
-
Author: ASF GitHub Bot
Created on: 01/Dec/21 16:26
Start Date: 01/Dec/21 16:26
Worklog Time Spent: 10m
Work Description: kgyrtkirk merged pull request #2816:
URL: https://github.com/apache/hive/pull/2816
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 688772)
Time Spent: 2.5h (was: 2h 20m)
> NullIf doesn't support complex types
>
>
> Key: HIVE-25738
> URL: https://issues.apache.org/jira/browse/HIVE-25738
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> {code}
> SELECT NULLIF(array(1,2,3),array(1,2,3))
> {code}
> results in:
> {code}
> java.lang.ClassCastException:
> org.apache.hadoop.hive.serde2.objectinspector.StandardListObjectInspector
> cannot be cast to
> org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFNullif.evaluate(GenericUDFNullif.java:96)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDF.initializeAndFoldConstants(GenericUDF.java:177)
> at
> org.apache.hadoop.hive.ql.parse.type.HiveFunctionHelper.getReturnType(HiveFunctionHelper.java:135)
> at
> org.apache.hadoop.hive.ql.parse.type.RexNodeExprFactory.createFuncCallExpr(RexNodeExprFactory.java:647)
> [...]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (HIVE-25738) NullIf doesn't support complex types
[
https://issues.apache.org/jira/browse/HIVE-25738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich resolved HIVE-25738.
-
Fix Version/s: 4.0.0
Assignee: Zoltan Haindrich
Resolution: Fixed
merged into master. Thank you [~zabetak] and [~dengzh] for reviewing the
changes!
> NullIf doesn't support complex types
>
>
> Key: HIVE-25738
> URL: https://issues.apache.org/jira/browse/HIVE-25738
> Project: Hive
> Issue Type: Improvement
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> {code}
> SELECT NULLIF(array(1,2,3),array(1,2,3))
> {code}
> results in:
> {code}
> java.lang.ClassCastException:
> org.apache.hadoop.hive.serde2.objectinspector.StandardListObjectInspector
> cannot be cast to
> org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDFNullif.evaluate(GenericUDFNullif.java:96)
> at
> org.apache.hadoop.hive.ql.udf.generic.GenericUDF.initializeAndFoldConstants(GenericUDF.java:177)
> at
> org.apache.hadoop.hive.ql.parse.type.HiveFunctionHelper.getReturnType(HiveFunctionHelper.java:135)
> at
> org.apache.hadoop.hive.ql.parse.type.RexNodeExprFactory.createFuncCallExpr(RexNodeExprFactory.java:647)
> [...]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Assigned] (HIVE-25757) Use cached database type to choose metastore backend queries
[ https://issues.apache.org/jira/browse/HIVE-25757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongzhi Chen reassigned HIVE-25757: --- > Use cached database type to choose metastore backend queries > > > Key: HIVE-25757 > URL: https://issues.apache.org/jira/browse/HIVE-25757 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 4.0.0 >Reporter: Yongzhi Chen >Assignee: Yongzhi Chen >Priority: Major > > In HIVE-21075, we use DatabaseProduct.determineDatabaseProduct which can be > expensive. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HIVE-25756) Fix replication metrics backward compatibility issue.
[ https://issues.apache.org/jira/browse/HIVE-25756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haymant Mangla reassigned HIVE-25756: - > Fix replication metrics backward compatibility issue. > - > > Key: HIVE-25756 > URL: https://issues.apache.org/jira/browse/HIVE-25756 > Project: Hive > Issue Type: Bug >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-24445) Non blocking DROP TABLE implementation
[
https://issues.apache.org/jira/browse/HIVE-24445?focusedWorklogId=688714=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-688714
]
ASF GitHub Bot logged work on HIVE-24445:
-
Author: ASF GitHub Bot
Created on: 01/Dec/21 14:52
Start Date: 01/Dec/21 14:52
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on a change in pull request #2772:
URL: https://github.com/apache/hive/pull/2772#discussion_r760259374
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
##
@@ -336,9 +336,9 @@ public void markCompacted(CompactionInfo info) throws
MetaException {
* By filtering on minOpenTxnWaterMark, we will only cleanup after
every transaction is committed, that could see
* the uncompacted deltas. This way the cleaner can clean up
everything that was made obsolete by this compaction.
*/
-String s = "SELECT \"CQ_ID\", \"CQ_DATABASE\", \"CQ_TABLE\",
\"CQ_PARTITION\", "
-+ "\"CQ_TYPE\", \"CQ_RUN_AS\", \"CQ_HIGHEST_WRITE_ID\" FROM
\"COMPACTION_QUEUE\" WHERE \"CQ_STATE\" = '"
-+ READY_FOR_CLEANING + "'";
+String s = "SELECT \"CQ_ID\", \"CQ_DATABASE\", \"CQ_TABLE\",
\"CQ_PARTITION\", \"CQ_TYPE\", "
++ "\"CQ_RUN_AS\", \"CQ_HIGHEST_WRITE_ID\", \"CQ_TBLPROPERTIES\"
FROM \"COMPACTION_QUEUE\" WHERE \"CQ_STATE\" = '"
Review comment:
it's an existing field, so should be in sysdb tables. This part only
affects the Cleaner.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 688714)
Time Spent: 11.5h (was: 11h 20m)
> Non blocking DROP TABLE implementation
> --
>
> Key: HIVE-24445
> URL: https://issues.apache.org/jira/browse/HIVE-24445
> Project: Hive
> Issue Type: New Feature
> Components: Hive
>Reporter: Zoltan Chovan
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 11.5h
> Remaining Estimate: 0h
>
> Implement a way to execute drop table operations in a way that doesn't have
> to wait for currently running read operations to be finished.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-24445) Non blocking DROP TABLE implementation
[
https://issues.apache.org/jira/browse/HIVE-24445?focusedWorklogId=688712=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-688712
]
ASF GitHub Bot logged work on HIVE-24445:
-
Author: ASF GitHub Bot
Created on: 01/Dec/21 14:50
Start Date: 01/Dec/21 14:50
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on a change in pull request #2772:
URL: https://github.com/apache/hive/pull/2772#discussion_r760257691
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
##
@@ -1451,11 +1453,11 @@ public void commitTxn(CommitTxnRequest rqst) throws
NoSuchTxnException, TxnAbort
OperationType.UPDATE + "," + OperationType.DELETE + ")";
long tempCommitId = generateTemporaryId();
-if (txnType.get() == TxnType.COMPACTION) {
+if (txnType == TxnType.SOFT_DELETE || txnType == TxnType.COMPACTION) {
acquireTxnLock(stmt, false);
commitId = getHighWaterMark(stmt);
-} else if (txnType.get() != TxnType.READ_ONLY && !isReplayedReplTxn) {
+} else if (txnType != TxnType.READ_ONLY && !isReplayedReplTxn) {
Review comment:
I need to check if we send the metadata delete event for repl, however,
for the data removal, we are doing this in Cleaner.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 688712)
Time Spent: 11h 20m (was: 11h 10m)
> Non blocking DROP TABLE implementation
> --
>
> Key: HIVE-24445
> URL: https://issues.apache.org/jira/browse/HIVE-24445
> Project: Hive
> Issue Type: New Feature
> Components: Hive
>Reporter: Zoltan Chovan
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 11h 20m
> Remaining Estimate: 0h
>
> Implement a way to execute drop table operations in a way that doesn't have
> to wait for currently running read operations to be finished.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-24445) Non blocking DROP TABLE implementation
[
https://issues.apache.org/jira/browse/HIVE-24445?focusedWorklogId=688710=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-688710
]
ASF GitHub Bot logged work on HIVE-24445:
-
Author: ASF GitHub Bot
Created on: 01/Dec/21 14:48
Start Date: 01/Dec/21 14:48
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on a change in pull request #2772:
URL: https://github.com/apache/hive/pull/2772#discussion_r760255174
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/AcidEventListener.java
##
@@ -60,9 +65,30 @@ public void onDropDatabase (DropDatabaseEvent dbEvent)
throws MetaException {
@Override
public void onDropTable(DropTableEvent tableEvent) throws MetaException {
-if (TxnUtils.isTransactionalTable(tableEvent.getTable())) {
+Table table = tableEvent.getTable();
+
+if (TxnUtils.isTransactionalTable(table)) {
txnHandler = getTxnHandler();
- txnHandler.cleanupRecords(HiveObjectType.TABLE, null,
tableEvent.getTable(), null);
+ txnHandler.cleanupRecords(HiveObjectType.TABLE, null, table, null,
!tableEvent.getDeleteData());
+
+ if (!tableEvent.getDeleteData()) {
+long currentTxn =
Optional.ofNullable(tableEvent.getEnvironmentContext())
+ .map(EnvironmentContext::getProperties)
+ .map(prop -> prop.get("txnId"))
+ .map(Long::parseLong)
+ .orElse(0L);
+
+if (currentTxn > 0) {
+ CompactionRequest rqst = new CompactionRequest(table.getDbName(),
table.getTableName(), CompactionType.MAJOR);
+ rqst.putToProperties("location", table.getSd().getLocation());
Review comment:
I didn't want to change the compaction queue schema + it kinda fits the
definition of CQ_TBLPROPERTIES column
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 688710)
Time Spent: 11h 10m (was: 11h)
> Non blocking DROP TABLE implementation
> --
>
> Key: HIVE-24445
> URL: https://issues.apache.org/jira/browse/HIVE-24445
> Project: Hive
> Issue Type: New Feature
> Components: Hive
>Reporter: Zoltan Chovan
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 11h 10m
> Remaining Estimate: 0h
>
> Implement a way to execute drop table operations in a way that doesn't have
> to wait for currently running read operations to be finished.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-24445) Non blocking DROP TABLE implementation
[
https://issues.apache.org/jira/browse/HIVE-24445?focusedWorklogId=688707=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-688707
]
ASF GitHub Bot logged work on HIVE-24445:
-
Author: ASF GitHub Bot
Created on: 01/Dec/21 14:46
Start Date: 01/Dec/21 14:46
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on a change in pull request #2772:
URL: https://github.com/apache/hive/pull/2772#discussion_r760253006
##
File path:
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
##
@@ -1772,6 +1773,22 @@ public void dropTable(String dbname, String name,
boolean deleteData,
dropTable(getDefaultCatalog(conf), dbname, name, deleteData,
ignoreUnknownTab, ifPurge);
}
+ @Override
+ public void dropTable(Table tbl, boolean deleteData, boolean
ignoreUnknownTbl, boolean ifPurge) throws TException {
+EnvironmentContext context = null;
+if (ifPurge) {
Review comment:
good catch, it's not handled yet, as we don't support purge in Cleaner,
however, we could pass `purge` attribute via the queue tblproperties.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 688707)
Time Spent: 11h (was: 10h 50m)
> Non blocking DROP TABLE implementation
> --
>
> Key: HIVE-24445
> URL: https://issues.apache.org/jira/browse/HIVE-24445
> Project: Hive
> Issue Type: New Feature
> Components: Hive
>Reporter: Zoltan Chovan
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 11h
> Remaining Estimate: 0h
>
> Implement a way to execute drop table operations in a way that doesn't have
> to wait for currently running read operations to be finished.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-24445) Non blocking DROP TABLE implementation
[
https://issues.apache.org/jira/browse/HIVE-24445?focusedWorklogId=688703=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-688703
]
ASF GitHub Bot logged work on HIVE-24445:
-
Author: ASF GitHub Bot
Created on: 01/Dec/21 14:43
Start Date: 01/Dec/21 14:43
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on a change in pull request #2772:
URL: https://github.com/apache/hive/pull/2772#discussion_r760251029
##
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Initiator.java
##
@@ -249,7 +249,7 @@ protected String resolveUserToRunAs(Map
cache, Table t, Partitio
String user = cache.get(fullTableName);
if (user == null) {
- user = findUserToRunAs(sd.getLocation(), t);
+ user = TxnUtils.findUserToRunAs(sd.getLocation(), t, conf);
Review comment:
nope, refactor needed to set runAs user for `safe-delete` in
AcidEventListener.dropTable()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 688703)
Time Spent: 10h 50m (was: 10h 40m)
> Non blocking DROP TABLE implementation
> --
>
> Key: HIVE-24445
> URL: https://issues.apache.org/jira/browse/HIVE-24445
> Project: Hive
> Issue Type: New Feature
> Components: Hive
>Reporter: Zoltan Chovan
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10h 50m
> Remaining Estimate: 0h
>
> Implement a way to execute drop table operations in a way that doesn't have
> to wait for currently running read operations to be finished.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-24445) Non blocking DROP TABLE implementation
[
https://issues.apache.org/jira/browse/HIVE-24445?focusedWorklogId=688701=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-688701
]
ASF GitHub Bot logged work on HIVE-24445:
-
Author: ASF GitHub Bot
Created on: 01/Dec/21 14:42
Start Date: 01/Dec/21 14:42
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on a change in pull request #2772:
URL: https://github.com/apache/hive/pull/2772#discussion_r760249779
##
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java
##
@@ -318,8 +335,8 @@ private static String idWatermark(CompactionInfo ci) {
private boolean removeFiles(String location, ValidWriteIdList writeIdList,
CompactionInfo ci)
throws IOException, NoSuchObjectException, MetaException {
Path locPath = new Path(location);
-AcidDirectory dir = AcidUtils.getAcidState(locPath.getFileSystem(conf),
locPath, conf, writeIdList, Ref.from(
-false), false);
+FileSystem fs = locPath.getFileSystem(conf);
Review comment:
no, created a separate cleanup method to handle `safe-delete` +
extracted the common part
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 688701)
Time Spent: 10h 40m (was: 10.5h)
> Non blocking DROP TABLE implementation
> --
>
> Key: HIVE-24445
> URL: https://issues.apache.org/jira/browse/HIVE-24445
> Project: Hive
> Issue Type: New Feature
> Components: Hive
>Reporter: Zoltan Chovan
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10h 40m
> Remaining Estimate: 0h
>
> Implement a way to execute drop table operations in a way that doesn't have
> to wait for currently running read operations to be finished.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-24445) Non blocking DROP TABLE implementation
[
https://issues.apache.org/jira/browse/HIVE-24445?focusedWorklogId=688699=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-688699
]
ASF GitHub Bot logged work on HIVE-24445:
-
Author: ASF GitHub Bot
Created on: 01/Dec/21 14:40
Start Date: 01/Dec/21 14:40
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on a change in pull request #2772:
URL: https://github.com/apache/hive/pull/2772#discussion_r76024
##
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java
##
@@ -304,7 +321,7 @@ private ValidReaderWriteIdList
getValidCleanerWriteIdList(CompactionInfo ci, Tab
}
private static boolean isDynPartAbort(Table t, CompactionInfo ci) {
-return t.getPartitionKeys() != null && t.getPartitionKeys().size() > 0
+return Optional.ofNullable(t).map(Table::getPartitionKeys).filter(pk ->
pk.size() > 0).isPresent()
Review comment:
`safe-delete` use case - table object would be null here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 688699)
Time Spent: 10.5h (was: 10h 20m)
> Non blocking DROP TABLE implementation
> --
>
> Key: HIVE-24445
> URL: https://issues.apache.org/jira/browse/HIVE-24445
> Project: Hive
> Issue Type: New Feature
> Components: Hive
>Reporter: Zoltan Chovan
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10.5h
> Remaining Estimate: 0h
>
> Implement a way to execute drop table operations in a way that doesn't have
> to wait for currently running read operations to be finished.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-24445) Non blocking DROP TABLE implementation
[
https://issues.apache.org/jira/browse/HIVE-24445?focusedWorklogId=688698=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-688698
]
ASF GitHub Bot logged work on HIVE-24445:
-
Author: ASF GitHub Bot
Created on: 01/Dec/21 14:39
Start Date: 01/Dec/21 14:39
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on a change in pull request #2772:
URL: https://github.com/apache/hive/pull/2772#discussion_r760246130
##
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java
##
@@ -279,10 +292,14 @@ private void clean(CompactionInfo ci, long minOpenTxnGLB,
boolean metricsEnabled
}
}
}
+
+ private interface CleanerTask {
+T execute() throws Exception;
+ }
- private ValidReaderWriteIdList getValidCleanerWriteIdList(CompactionInfo ci,
Table t, ValidTxnList validTxnList)
+ private ValidReaderWriteIdList getValidCleanerWriteIdList(CompactionInfo ci,
ValidTxnList validTxnList)
Review comment:
it's needed for `safe_delete` use-case as we won't be able to resolve
the table object - metadata would be already removed, however, we could get
full tableName from the ci object.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 688698)
Time Spent: 10h 20m (was: 10h 10m)
> Non blocking DROP TABLE implementation
> --
>
> Key: HIVE-24445
> URL: https://issues.apache.org/jira/browse/HIVE-24445
> Project: Hive
> Issue Type: New Feature
> Components: Hive
>Reporter: Zoltan Chovan
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10h 20m
> Remaining Estimate: 0h
>
> Implement a way to execute drop table operations in a way that doesn't have
> to wait for currently running read operations to be finished.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-24445) Non blocking DROP TABLE implementation
[
https://issues.apache.org/jira/browse/HIVE-24445?focusedWorklogId=688697=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-688697
]
ASF GitHub Bot logged work on HIVE-24445:
-
Author: ASF GitHub Bot
Created on: 01/Dec/21 14:38
Start Date: 01/Dec/21 14:38
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on a change in pull request #2772:
URL: https://github.com/apache/hive/pull/2772#discussion_r760246130
##
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java
##
@@ -279,10 +292,14 @@ private void clean(CompactionInfo ci, long minOpenTxnGLB,
boolean metricsEnabled
}
}
}
+
+ private interface CleanerTask {
+T execute() throws Exception;
+ }
- private ValidReaderWriteIdList getValidCleanerWriteIdList(CompactionInfo ci,
Table t, ValidTxnList validTxnList)
+ private ValidReaderWriteIdList getValidCleanerWriteIdList(CompactionInfo ci,
ValidTxnList validTxnList)
Review comment:
it's needed for `safe_delete` use case as we won't be able to resolve
table object - metadata would be already removed, we should get full tableName
from the ci object.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 688697)
Time Spent: 10h 10m (was: 10h)
> Non blocking DROP TABLE implementation
> --
>
> Key: HIVE-24445
> URL: https://issues.apache.org/jira/browse/HIVE-24445
> Project: Hive
> Issue Type: New Feature
> Components: Hive
>Reporter: Zoltan Chovan
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10h 10m
> Remaining Estimate: 0h
>
> Implement a way to execute drop table operations in a way that doesn't have
> to wait for currently running read operations to be finished.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-24445) Non blocking DROP TABLE implementation
[
https://issues.apache.org/jira/browse/HIVE-24445?focusedWorklogId=688693=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-688693
]
ASF GitHub Bot logged work on HIVE-24445:
-
Author: ASF GitHub Bot
Created on: 01/Dec/21 14:34
Start Date: 01/Dec/21 14:34
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on a change in pull request #2772:
URL: https://github.com/apache/hive/pull/2772#discussion_r760241861
##
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java
##
@@ -279,10 +292,14 @@ private void clean(CompactionInfo ci, long minOpenTxnGLB,
boolean metricsEnabled
}
}
}
+
+ private interface CleanerTask {
Review comment:
not really, good catch!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 688693)
Time Spent: 10h (was: 9h 50m)
> Non blocking DROP TABLE implementation
> --
>
> Key: HIVE-24445
> URL: https://issues.apache.org/jira/browse/HIVE-24445
> Project: Hive
> Issue Type: New Feature
> Components: Hive
>Reporter: Zoltan Chovan
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10h
> Remaining Estimate: 0h
>
> Implement a way to execute drop table operations in a way that doesn't have
> to wait for currently running read operations to be finished.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-24445) Non blocking DROP TABLE implementation
[
https://issues.apache.org/jira/browse/HIVE-24445?focusedWorklogId=688691=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-688691
]
ASF GitHub Bot logged work on HIVE-24445:
-
Author: ASF GitHub Bot
Created on: 01/Dec/21 14:31
Start Date: 01/Dec/21 14:31
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on a change in pull request #2772:
URL: https://github.com/apache/hive/pull/2772#discussion_r760238917
##
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java
##
@@ -172,22 +174,29 @@ private void clean(CompactionInfo ci, long minOpenTxnGLB,
boolean metricsEnabled
if (metricsEnabled) {
perfLogger.perfLogBegin(CLASS_NAME, cleanerMetric);
}
- Table t = resolveTable(ci);
- if (t == null) {
-// The table was dropped before we got around to cleaning it.
-LOG.info("Unable to find table " + ci.getFullTableName() + ", assuming
it was dropped." +
+ String location =
Optional.ofNullable(ci.properties).map(StringableMap::new)
Review comment:
we only support this feature for acid tables with a non-customized
location:
if (!TableType.VIRTUAL_VIEW.toString().equals(tbl.getTableType())) {
if (tbl.getSd().getLocation() == null ||
tbl.getSd().getLocation().isEmpty()) {
tblPath = wh.getDefaultTablePath(db,
getPathWithSuffix(tbl.getTableName(), tbl), isExternal(tbl));
}
String getPathWithSuffix(String path, Table tbl) {
return path + (tbl.isSetTxnId() ?
VISIBILITY_PREFIX + String.format(DELTA_DIGITS, tbl.getTxnId()) :
"");
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 688691)
Time Spent: 9h 50m (was: 9h 40m)
> Non blocking DROP TABLE implementation
> --
>
> Key: HIVE-24445
> URL: https://issues.apache.org/jira/browse/HIVE-24445
> Project: Hive
> Issue Type: New Feature
> Components: Hive
>Reporter: Zoltan Chovan
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 9h 50m
> Remaining Estimate: 0h
>
> Implement a way to execute drop table operations in a way that doesn't have
> to wait for currently running read operations to be finished.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-24445) Non blocking DROP TABLE implementation
[
https://issues.apache.org/jira/browse/HIVE-24445?focusedWorklogId=688688=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-688688
]
ASF GitHub Bot logged work on HIVE-24445:
-
Author: ASF GitHub Bot
Created on: 01/Dec/21 14:26
Start Date: 01/Dec/21 14:26
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on a change in pull request #2772:
URL: https://github.com/apache/hive/pull/2772#discussion_r760233519
##
File path: ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
##
@@ -1334,6 +1344,20 @@ public void dropTable(String tableName, boolean ifPurge)
throws HiveException {
dropTable(names[0], names[1], true, true, ifPurge);
}
+ public void dropTable(Table table, boolean ifPurge) throws HiveException {
+boolean tableWithSuffix = (HiveConf.getBoolVar(conf,
ConfVars.HIVE_ACID_CREATE_TABLE_USE_SUFFIX)
+|| HiveConf.getBoolVar(conf,
ConfVars.HIVE_ACID_LOCKLESS_READS_ENABLED))
+ && AcidUtils.isTransactionalTable(table)
+ && table.getSd().getLocation().matches(PATH_WITH_VERSION_ID);
+
+if (tableWithSuffix) {
+ long txnId = Optional.ofNullable(SessionState.get())
+.map(ss -> ss.getTxnMgr().getCurrentTxnId()).orElse(0L);
+ table.getTTable().setTxnId(txnId);
+}
Review comment:
we could have old tables that were created without this feature being
enabled, and we still should support regular drops for them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 688688)
Time Spent: 9h 40m (was: 9.5h)
> Non blocking DROP TABLE implementation
> --
>
> Key: HIVE-24445
> URL: https://issues.apache.org/jira/browse/HIVE-24445
> Project: Hive
> Issue Type: New Feature
> Components: Hive
>Reporter: Zoltan Chovan
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 9h 40m
> Remaining Estimate: 0h
>
> Implement a way to execute drop table operations in a way that doesn't have
> to wait for currently running read operations to be finished.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-24445) Non blocking DROP TABLE implementation
[
https://issues.apache.org/jira/browse/HIVE-24445?focusedWorklogId=688684=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-688684
]
ASF GitHub Bot logged work on HIVE-24445:
-
Author: ASF GitHub Bot
Created on: 01/Dec/21 14:21
Start Date: 01/Dec/21 14:21
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on a change in pull request #2772:
URL: https://github.com/apache/hive/pull/2772#discussion_r760229198
##
File path: ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
##
@@ -1273,14 +1276,21 @@ public void createTable(Table tbl, boolean ifNotExists,
principalPrivs.setRolePrivileges(grants.getRoleGrants());
tTbl.setPrivileges(principalPrivs);
}
+boolean createTableUseSuffix = (HiveConf.getBoolVar(conf,
ConfVars.HIVE_ACID_CREATE_TABLE_USE_SUFFIX)
+|| HiveConf.getBoolVar(conf,
ConfVars.HIVE_ACID_LOCKLESS_READS_ENABLED))
+ && AcidUtils.isTransactionalTable(tbl);
+
+if (createTableUseSuffix) {
+ tTbl.setTxnId(ss.getTxnMgr().getCurrentTxnId());
Review comment:
I am relying on txnId a flag that shows if the non-blocking drop is
supported. if not set, we won't create a versioned table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 688684)
Time Spent: 9.5h (was: 9h 20m)
> Non blocking DROP TABLE implementation
> --
>
> Key: HIVE-24445
> URL: https://issues.apache.org/jira/browse/HIVE-24445
> Project: Hive
> Issue Type: New Feature
> Components: Hive
>Reporter: Zoltan Chovan
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 9.5h
> Remaining Estimate: 0h
>
> Implement a way to execute drop table operations in a way that doesn't have
> to wait for currently running read operations to be finished.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-24445) Non blocking DROP TABLE implementation
[
https://issues.apache.org/jira/browse/HIVE-24445?focusedWorklogId=688673=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-688673
]
ASF GitHub Bot logged work on HIVE-24445:
-
Author: ASF GitHub Bot
Created on: 01/Dec/21 14:09
Start Date: 01/Dec/21 14:09
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on a change in pull request #2772:
URL: https://github.com/apache/hive/pull/2772#discussion_r760218118
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/drop/DropTableAnalyzer.java
##
@@ -51,7 +55,14 @@ public void analyzeInternal(ASTNode root) throws
SemanticException {
Table table = getTable(tableName, throwException);
if (table != null) {
inputs.add(new ReadEntity(table));
- outputs.add(new WriteEntity(table, WriteEntity.WriteType.DDL_EXCLUSIVE));
+
+ boolean tableWithSuffix = (HiveConf.getBoolVar(conf,
ConfVars.HIVE_ACID_CREATE_TABLE_USE_SUFFIX)
+ || HiveConf.getBoolVar(conf,
ConfVars.HIVE_ACID_LOCKLESS_READS_ENABLED))
+&& AcidUtils.isTransactionalTable(table)
+&& table.getSd().getLocation().matches(PATH_WITH_VERSION_ID);
+
+ outputs.add(new WriteEntity(table,
Review comment:
it's already logged by DbTxnManager when calling
AcidUtils.makeLockComponents
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 688673)
Time Spent: 9h 20m (was: 9h 10m)
> Non blocking DROP TABLE implementation
> --
>
> Key: HIVE-24445
> URL: https://issues.apache.org/jira/browse/HIVE-24445
> Project: Hive
> Issue Type: New Feature
> Components: Hive
>Reporter: Zoltan Chovan
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 9h 20m
> Remaining Estimate: 0h
>
> Implement a way to execute drop table operations in a way that doesn't have
> to wait for currently running read operations to be finished.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-24445) Non blocking DROP TABLE implementation
[
https://issues.apache.org/jira/browse/HIVE-24445?focusedWorklogId=688670=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-688670
]
ASF GitHub Bot logged work on HIVE-24445:
-
Author: ASF GitHub Bot
Created on: 01/Dec/21 14:06
Start Date: 01/Dec/21 14:06
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on a change in pull request #2772:
URL: https://github.com/apache/hive/pull/2772#discussion_r760214887
##
File path: ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java
##
@@ -3109,10 +3109,16 @@ public static TxnType getTxnType(Configuration conf,
ASTNode tree) {
if (tree.getFirstChildWithType(HiveParser.TOK_ALTERTABLE_COMPACT) != null){
return TxnType.COMPACTION;
}
+// check if soft delete
+if (tree.getToken().getType() == HiveParser.TOK_DROPTABLE
+ && (HiveConf.getBoolVar(conf, ConfVars.HIVE_ACID_CREATE_TABLE_USE_SUFFIX)
+|| HiveConf.getBoolVar(conf,
ConfVars.HIVE_ACID_LOCKLESS_READS_ENABLED))){
+ return TxnType.SOFT_DELETE;
Review comment:
I don't think so as we still should be able to drop tables created
without that option (not versioned)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 688670)
Time Spent: 9h 10m (was: 9h)
> Non blocking DROP TABLE implementation
> --
>
> Key: HIVE-24445
> URL: https://issues.apache.org/jira/browse/HIVE-24445
> Project: Hive
> Issue Type: New Feature
> Components: Hive
>Reporter: Zoltan Chovan
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 9h 10m
> Remaining Estimate: 0h
>
> Implement a way to execute drop table operations in a way that doesn't have
> to wait for currently running read operations to be finished.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-24445) Non blocking DROP TABLE implementation
[
https://issues.apache.org/jira/browse/HIVE-24445?focusedWorklogId=688669=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-688669
]
ASF GitHub Bot logged work on HIVE-24445:
-
Author: ASF GitHub Bot
Created on: 01/Dec/21 14:04
Start Date: 01/Dec/21 14:04
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on a change in pull request #2772:
URL: https://github.com/apache/hive/pull/2772#discussion_r760213666
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/drop/DropTableOperation.java
##
@@ -109,7 +109,7 @@ public int execute() throws HiveException {
}
// TODO: API w/catalog name
-context.getDb().dropTable(desc.getTableName(), desc.isPurge());
+context.getDb().dropTable(table, desc.isPurge());
Review comment:
I need to pass extra info like txnId through the tbl object
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 688669)
Time Spent: 9h (was: 8h 50m)
> Non blocking DROP TABLE implementation
> --
>
> Key: HIVE-24445
> URL: https://issues.apache.org/jira/browse/HIVE-24445
> Project: Hive
> Issue Type: New Feature
> Components: Hive
>Reporter: Zoltan Chovan
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 9h
> Remaining Estimate: 0h
>
> Implement a way to execute drop table operations in a way that doesn't have
> to wait for currently running read operations to be finished.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-21075) Metastore: Drop partition performance downgrade with Postgres DB
[ https://issues.apache.org/jira/browse/HIVE-21075?focusedWorklogId=688661=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-688661 ] ASF GitHub Bot logged work on HIVE-21075: - Author: ASF GitHub Bot Created on: 01/Dec/21 13:51 Start Date: 01/Dec/21 13:51 Worklog Time Spent: 10m Work Description: yongzhi merged pull request #2826: URL: https://github.com/apache/hive/pull/2826 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 688661) Time Spent: 7h 50m (was: 7h 40m) > Metastore: Drop partition performance downgrade with Postgres DB > > > Key: HIVE-21075 > URL: https://issues.apache.org/jira/browse/HIVE-21075 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Yongzhi Chen >Assignee: Oleksiy Sayankin >Priority: Major > Labels: pull-request-available > Attachments: HIVE-21075.2.patch > > Time Spent: 7h 50m > Remaining Estimate: 0h > > In order to workaround oracle not supporting limit statement caused > performance issue, HIVE-9447 makes all the backend DB run select count(1) > from SDS where SDS.CD_ID=? to check if the specific CD_ID is referenced in > SDS table before drop a partition. This select count(1) statement does not > scale well in Postgres, and there is no index for CD_ID column in SDS table. > For a SDS table with with 1.5 million rows, select count(1) has average 700ms > without index, while in 10-20ms with index. But the statement before > HIVE-9447( SELECT * FROM "SDS" "A0" WHERE "A0"."CD_ID" = $1 limit 1) uses > less than 10ms . -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-24445) Non blocking DROP TABLE implementation
[
https://issues.apache.org/jira/browse/HIVE-24445?focusedWorklogId=688643=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-688643
]
ASF GitHub Bot logged work on HIVE-24445:
-
Author: ASF GitHub Bot
Created on: 01/Dec/21 13:13
Start Date: 01/Dec/21 13:13
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on a change in pull request #2772:
URL: https://github.com/apache/hive/pull/2772#discussion_r760170333
##
File path:
standalone-metastore/metastore-common/src/gen/thrift/gen-cpp/hive_metastore_types.cpp
##
@@ -615,16 +615,18 @@ int _kTxnTypeValues[] = {
TxnType::REPL_CREATED,
TxnType::READ_ONLY,
TxnType::COMPACTION,
- TxnType::MATER_VIEW_REBUILD
+ TxnType::MATER_VIEW_REBUILD,
+ TxnType::SOFT_DELETE
Review comment:
however, old HMS won't have support for lockless drop as well, so we
should be good
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 688643)
Time Spent: 8h 50m (was: 8h 40m)
> Non blocking DROP TABLE implementation
> --
>
> Key: HIVE-24445
> URL: https://issues.apache.org/jira/browse/HIVE-24445
> Project: Hive
> Issue Type: New Feature
> Components: Hive
>Reporter: Zoltan Chovan
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 8h 50m
> Remaining Estimate: 0h
>
> Implement a way to execute drop table operations in a way that doesn't have
> to wait for currently running read operations to be finished.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-24445) Non blocking DROP TABLE implementation
[
https://issues.apache.org/jira/browse/HIVE-24445?focusedWorklogId=688642=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-688642
]
ASF GitHub Bot logged work on HIVE-24445:
-
Author: ASF GitHub Bot
Created on: 01/Dec/21 13:13
Start Date: 01/Dec/21 13:13
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on a change in pull request #2772:
URL: https://github.com/apache/hive/pull/2772#discussion_r760170333
##
File path:
standalone-metastore/metastore-common/src/gen/thrift/gen-cpp/hive_metastore_types.cpp
##
@@ -615,16 +615,18 @@ int _kTxnTypeValues[] = {
TxnType::REPL_CREATED,
TxnType::READ_ONLY,
TxnType::COMPACTION,
- TxnType::MATER_VIEW_REBUILD
+ TxnType::MATER_VIEW_REBUILD,
+ TxnType::SOFT_DELETE
Review comment:
however, old HMS won't have support for lockless read, so we should be
good
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 688642)
Time Spent: 8h 40m (was: 8.5h)
> Non blocking DROP TABLE implementation
> --
>
> Key: HIVE-24445
> URL: https://issues.apache.org/jira/browse/HIVE-24445
> Project: Hive
> Issue Type: New Feature
> Components: Hive
>Reporter: Zoltan Chovan
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 8h 40m
> Remaining Estimate: 0h
>
> Implement a way to execute drop table operations in a way that doesn't have
> to wait for currently running read operations to be finished.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-24445) Non blocking DROP TABLE implementation
[
https://issues.apache.org/jira/browse/HIVE-24445?focusedWorklogId=688640=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-688640
]
ASF GitHub Bot logged work on HIVE-24445:
-
Author: ASF GitHub Bot
Created on: 01/Dec/21 13:11
Start Date: 01/Dec/21 13:11
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on a change in pull request #2772:
URL: https://github.com/apache/hive/pull/2772#discussion_r760168937
##
File path:
standalone-metastore/metastore-common/src/gen/thrift/gen-cpp/hive_metastore_types.cpp
##
@@ -615,16 +615,18 @@ int _kTxnTypeValues[] = {
TxnType::REPL_CREATED,
TxnType::READ_ONLY,
TxnType::COMPACTION,
- TxnType::MATER_VIEW_REBUILD
+ TxnType::MATER_VIEW_REBUILD,
+ TxnType::SOFT_DELETE
Review comment:
Old versions of HMS would treat new values as null (default - null), in
this case, we won't ever enter the if branch for SOFT_DELETE, meaning, we won't
populate the highest write id required by cleaner and might clean up some
deltas prematurely, see HIVE-24291.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 688640)
Time Spent: 8.5h (was: 8h 20m)
> Non blocking DROP TABLE implementation
> --
>
> Key: HIVE-24445
> URL: https://issues.apache.org/jira/browse/HIVE-24445
> Project: Hive
> Issue Type: New Feature
> Components: Hive
>Reporter: Zoltan Chovan
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 8.5h
> Remaining Estimate: 0h
>
> Implement a way to execute drop table operations in a way that doesn't have
> to wait for currently running read operations to be finished.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-24445) Non blocking DROP TABLE implementation
[ https://issues.apache.org/jira/browse/HIVE-24445?focusedWorklogId=688633=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-688633 ] ASF GitHub Bot logged work on HIVE-24445: - Author: ASF GitHub Bot Created on: 01/Dec/21 12:55 Start Date: 01/Dec/21 12:55 Worklog Time Spent: 10m Work Description: deniskuzZ commented on a change in pull request #2772: URL: https://github.com/apache/hive/pull/2772#discussion_r760156827 ## File path: itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestAcidOnTez.java ## @@ -18,6 +18,7 @@ package org.apache.hadoop.hive.ql; +import static org.apache.hadoop.hive.ql.TxnCommandsBaseForTests.*; Review comment: import is required after the tests refactor, previous visibility was package-private. ps: moved tests refactor under https://github.com/apache/hive/pull/2831 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 688633) Time Spent: 8h 10m (was: 8h) > Non blocking DROP TABLE implementation > -- > > Key: HIVE-24445 > URL: https://issues.apache.org/jira/browse/HIVE-24445 > Project: Hive > Issue Type: New Feature > Components: Hive >Reporter: Zoltan Chovan >Assignee: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Time Spent: 8h 10m > Remaining Estimate: 0h > > Implement a way to execute drop table operations in a way that doesn't have > to wait for currently running read operations to be finished. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-24445) Non blocking DROP TABLE implementation
[
https://issues.apache.org/jira/browse/HIVE-24445?focusedWorklogId=688634=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-688634
]
ASF GitHub Bot logged work on HIVE-24445:
-
Author: ASF GitHub Bot
Created on: 01/Dec/21 12:55
Start Date: 01/Dec/21 12:55
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on a change in pull request #2772:
URL: https://github.com/apache/hive/pull/2772#discussion_r760157455
##
File path: ql/src/test/org/apache/hadoop/hive/ql/TxnCommandsBaseForTests.java
##
@@ -202,8 +208,29 @@ public int compare(int[] row1, int[] row2) {
return 0;
}
}
- protected String makeValuesClause(int[][] rows) {
-return TestTxnCommands2.makeValuesClause(rows);
+ public static String makeValuesClause(int[][] rows) {
Review comment:
moved test refactor under: https://github.com/apache/hive/pull/2831
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 688634)
Time Spent: 8h 20m (was: 8h 10m)
> Non blocking DROP TABLE implementation
> --
>
> Key: HIVE-24445
> URL: https://issues.apache.org/jira/browse/HIVE-24445
> Project: Hive
> Issue Type: New Feature
> Components: Hive
>Reporter: Zoltan Chovan
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 8h 20m
> Remaining Estimate: 0h
>
> Implement a way to execute drop table operations in a way that doesn't have
> to wait for currently running read operations to be finished.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HIVE-25754) Fix column projection for union all queries with multiple aliases
[
https://issues.apache.org/jira/browse/HIVE-25754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marton Bod updated HIVE-25754:
--
Description:
Given two tables:
{code:java}
create table source1 (dt string, d1 int, d2 int) stored as orc;
create table source2 (dt string, d1 int, d2 int) stored as orc;
insert into source1 values ('20211107', 1, 2);
insert into source2 values ('20211108', 11, 22);{code}
If you run this query with UNION ALL, the {{key}} column will be missing from
the output:
{code:java}
select explode(map('D219', D219
,'D220', D220)) as (key, value) from (
{{select '20211107' as date_key
,1 as D219
,2 as D220
) t}}
union all
select explode(map('D221', D221
,'D222', D222)) as (key, value)
from (}}
{{select '20211107' as date_key
,1 as D221
,2 as D222
) t{code}
Result:
{code:java}
1
2
11
22{code}
Correct result should be:
{code:java}
D219 1
D220 2
D221 11
D222 22{code}
was:
Given two tables:
{code:java}
create table source1 (dt string, d1 int, d2 int) stored as orc;
create table source2 (dt string, d1 int, d2 int) stored as orc;
insert into source1 values ('20211107', 1, 2);
insert into source2 values ('20211108', 11, 22);{code}
If you run this query with UNION ALL, the {{key}} column will be missing from
the output:
{code:java}
select explode(map('D219', D219
,'D220', D220)) as (key, value) from (
{{select '20211107' as date_key
,1 as D219
,2 as D220
) t}}
union all
select explode(map('D221', D221
,'D222', D222)) as (key, value)
from (}}
{{select '20211107' as date_key
,1 as D221
,2 as D222
) t{code}
Result:
{code:java}
1
2
11
22{code}
Correct result should be:
{code:java}
D219 1
D220 2
D221 11
D222 22{code}
> Fix column projection for union all queries with multiple aliases
> -
>
> Key: HIVE-25754
> URL: https://issues.apache.org/jira/browse/HIVE-25754
> Project: Hive
> Issue Type: Bug
>Reporter: Marton Bod
>Assignee: Marton Bod
>Priority: Major
>
> Given two tables:
> {code:java}
> create table source1 (dt string, d1 int, d2 int) stored as orc;
> create table source2 (dt string, d1 int, d2 int) stored as orc;
> insert into source1 values ('20211107', 1, 2);
> insert into source2 values ('20211108', 11, 22);{code}
> If you run this query with UNION ALL, the {{key}} column will be missing from
> the output:
> {code:java}
> select explode(map('D219', D219
> ,'D220', D220)) as (key, value) from (
> {{select '20211107' as date_key
> ,1 as D219
> ,2 as D220
> ) t}}
> union all
> select explode(map('D221', D221
> ,'D222', D222)) as (key, value)
> from (}}
> {{select '20211107' as date_key
> ,1 as D221
> ,2 as D222
> ) t{code}
> Result:
> {code:java}
> 1
> 2
> 11
> 22{code}
> Correct result should be:
> {code:java}
> D219 1
> D220 2
> D221 11
> D222 22{code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25755) TestTxnCommands classes refactor
[
https://issues.apache.org/jira/browse/HIVE-25755?focusedWorklogId=688625=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-688625
]
ASF GitHub Bot logged work on HIVE-25755:
-
Author: ASF GitHub Bot
Created on: 01/Dec/21 12:37
Start Date: 01/Dec/21 12:37
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #2831:
URL: https://github.com/apache/hive/pull/2831#discussion_r760143965
##
File path: ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java
##
@@ -767,7 +709,7 @@ public void testNonAcidToAcidConversion3() throws Exception
{
// 4. Update the existing row, and insert another row to newly-converted
ACID table
runStatementOnDriver("update " + Table.NONACIDORCTBL + " set b=3 where
a=1");
runStatementOnDriver("insert into " + Table.NONACIDORCTBL + "(a,b)
values(3,4)");
-status = fs.listStatus(new Path(TEST_WAREHOUSE_DIR + "/" +
+status = fs.listStatus(new Path(getWarehouseDir() + "/" +
Review comment:
Question: Would it worth to create a method for this?
`listStatus(String tableName)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 688625)
Time Spent: 1h (was: 50m)
> TestTxnCommands classes refactor
>
>
> Key: HIVE-25755
> URL: https://issues.apache.org/jira/browse/HIVE-25755
> Project: Hive
> Issue Type: Test
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25755) TestTxnCommands classes refactor
[
https://issues.apache.org/jira/browse/HIVE-25755?focusedWorklogId=688624=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-688624
]
ASF GitHub Bot logged work on HIVE-25755:
-
Author: ASF GitHub Bot
Created on: 01/Dec/21 12:36
Start Date: 01/Dec/21 12:36
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #2831:
URL: https://github.com/apache/hive/pull/2831#discussion_r760143118
##
File path: ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java
##
@@ -1172,7 +1114,7 @@ private int getFileSystemCacheSize() throws Exception {
this.total = total;
}
}
- private static CompactionsByState countCompacts(TxnStore txnHandler) throws
MetaException {
+ private CompactionsByState countCompacts(TxnStore txnHandler) throws
MetaException {
Review comment:
nit: new line before
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 688624)
Time Spent: 50m (was: 40m)
> TestTxnCommands classes refactor
>
>
> Key: HIVE-25755
> URL: https://issues.apache.org/jira/browse/HIVE-25755
> Project: Hive
> Issue Type: Test
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25755) TestTxnCommands classes refactor
[
https://issues.apache.org/jira/browse/HIVE-25755?focusedWorklogId=688623=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-688623
]
ASF GitHub Bot logged work on HIVE-25755:
-
Author: ASF GitHub Bot
Created on: 01/Dec/21 12:35
Start Date: 01/Dec/21 12:35
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #2831:
URL: https://github.com/apache/hive/pull/2831#discussion_r760142963
##
File path: ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java
##
@@ -1010,7 +952,7 @@ public void testInsertOverwriteWithSelfJoin() throws
Exception {
Assert.assertEquals("Insert overwrite partition failed",
stringifyValues(updatedData), rs2);
//insert overwrite not supported for ACID tables
}
- private static void checkCompactionState(CompactionsByState expected,
CompactionsByState actual) {
+ private void checkCompactionState(CompactionsByState expected,
CompactionsByState actual) {
Review comment:
nit: new line before
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 688623)
Time Spent: 40m (was: 0.5h)
> TestTxnCommands classes refactor
>
>
> Key: HIVE-25755
> URL: https://issues.apache.org/jira/browse/HIVE-25755
> Project: Hive
> Issue Type: Test
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25755) TestTxnCommands classes refactor
[
https://issues.apache.org/jira/browse/HIVE-25755?focusedWorklogId=688622=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-688622
]
ASF GitHub Bot logged work on HIVE-25755:
-
Author: ASF GitHub Bot
Created on: 01/Dec/21 12:34
Start Date: 01/Dec/21 12:34
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #2831:
URL: https://github.com/apache/hive/pull/2831#discussion_r760141833
##
File path: ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands.java
##
@@ -1397,7 +1396,7 @@ public void testNonAcidToAcidConversion01() throws
Exception {
BucketCodec.V1.encode(new
AcidOutputFormat.Options(hiveConf).bucket(1)));
//run Compaction
-runStatementOnDriver("alter table "+ TestTxnCommands2.Table.NONACIDORCTBL
+" compact 'major'");
+runStatementOnDriver("alter table "+ Table.NONACIDORCTBL +" compact
'major'");
Review comment:
nit: Maybe OCD but put spaces around the `+` signs please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 688622)
Time Spent: 0.5h (was: 20m)
> TestTxnCommands classes refactor
>
>
> Key: HIVE-25755
> URL: https://issues.apache.org/jira/browse/HIVE-25755
> Project: Hive
> Issue Type: Test
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25755) TestTxnCommands classes refactor
[
https://issues.apache.org/jira/browse/HIVE-25755?focusedWorklogId=688621=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-688621
]
ASF GitHub Bot logged work on HIVE-25755:
-
Author: ASF GitHub Bot
Created on: 01/Dec/21 12:33
Start Date: 01/Dec/21 12:33
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #2831:
URL: https://github.com/apache/hive/pull/2831#discussion_r760141497
##
File path: ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands.java
##
@@ -1352,7 +1351,7 @@ public void run() {
}
}, 5000);
long start = System.currentTimeMillis();
-runStatementOnDriver("alter table "+ TestTxnCommands2.Table.ACIDTBL +"
compact 'major' AND WAIT");
+runStatementOnDriver("alter table "+ Table.ACIDTBL +" compact 'major' AND
WAIT");
Review comment:
nit: Maybe OCD but put spaces around the `+` signs please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 688621)
Time Spent: 20m (was: 10m)
> TestTxnCommands classes refactor
>
>
> Key: HIVE-25755
> URL: https://issues.apache.org/jira/browse/HIVE-25755
> Project: Hive
> Issue Type: Test
>Reporter: Denys Kuzmenko
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25755) TestTxnCommands classes refactor
[ https://issues.apache.org/jira/browse/HIVE-25755?focusedWorklogId=688618=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-688618 ] ASF GitHub Bot logged work on HIVE-25755: - Author: ASF GitHub Bot Created on: 01/Dec/21 12:32 Start Date: 01/Dec/21 12:32 Worklog Time Spent: 10m Work Description: deniskuzZ opened a new pull request #2831: URL: https://github.com/apache/hive/pull/2831 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 688618) Remaining Estimate: 0h Time Spent: 10m > TestTxnCommands classes refactor > > > Key: HIVE-25755 > URL: https://issues.apache.org/jira/browse/HIVE-25755 > Project: Hive > Issue Type: Test >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-25755) TestTxnCommands classes refactor
[ https://issues.apache.org/jira/browse/HIVE-25755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-25755: -- Labels: pull-request-available (was: ) > TestTxnCommands classes refactor > > > Key: HIVE-25755 > URL: https://issues.apache.org/jira/browse/HIVE-25755 > Project: Hive > Issue Type: Test >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Assigned] (HIVE-25755) TestTxnCommands classes refactor
[ https://issues.apache.org/jira/browse/HIVE-25755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko reassigned HIVE-25755: - Assignee: Denys Kuzmenko > TestTxnCommands classes refactor > > > Key: HIVE-25755 > URL: https://issues.apache.org/jira/browse/HIVE-25755 > Project: Hive > Issue Type: Test >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HIVE-25754) Fix column projection for union all queries with multiple aliases
[
https://issues.apache.org/jira/browse/HIVE-25754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17451733#comment-17451733
]
Marton Bod commented on HIVE-25754:
---
PR: [https://github.com/apache/hive/pull/2822]
> Fix column projection for union all queries with multiple aliases
> -
>
> Key: HIVE-25754
> URL: https://issues.apache.org/jira/browse/HIVE-25754
> Project: Hive
> Issue Type: Bug
>Reporter: Marton Bod
>Assignee: Marton Bod
>Priority: Major
>
> Given two tables:
>
> {code:java}
> create table source1 (dt string, d1 int, d2 int) stored as orc;
> create table source2 (dt string, d1 int, d2 int) stored as orc;
> insert into source1 values ('20211107', 1, 2);
> insert into source2 values ('20211108', 11, 22);{code}
> If you run this query with UNION ALL, the {{key}} column will be missing from
> the output:
>
>
> {code:java}
> select explode(map('D219', D219
> ,'D220', D220)) as (key, value) from (
> {{select '20211107' as date_key
> ,1 as D219
> ,2 as D220
> ) t}}
> union all
> select explode(map('D221', D221
> ,'D222', D222)) as (key, value)
> from (}}
> {{select '20211107' as date_key
> ,1 as D221
> ,2 as D222
> ) t{code}
> Result:
>
> {code:java}
> 1
> 2
> 11
> 22{code}
>
> Correct result should be:
> {code:java}
> D219 1
> D220 2
> D221 11
> D222 22{code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HIVE-25754) Fix column projection for union all queries with multiple aliases
[
https://issues.apache.org/jira/browse/HIVE-25754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marton Bod updated HIVE-25754:
--
Description:
Given two tables:
{code:java}
create table source1 (dt string, d1 int, d2 int) stored as orc;
create table source2 (dt string, d1 int, d2 int) stored as orc;
insert into source1 values ('20211107', 1, 2);
insert into source2 values ('20211108', 11, 22);{code}
If you run this query with UNION ALL, the {{key}} column will be missing from
the output:
{code:java}
select explode(map('D219', D219
,'D220', D220)) as (key, value) from (
{{select '20211107' as date_key
,1 as D219
,2 as D220
) t}}
union all
select explode(map('D221', D221
,'D222', D222)) as (key, value)
from (}}
{{select '20211107' as date_key
,1 as D221
,2 as D222
) t{code}
Result:
{code:java}
1
2
11
22{code}
Correct result should be:
{code:java}
D219 1
D220 2
D221 11
D222 22{code}
was:
Given two tables:
{{create table source1 (dt string, d1 int, d2 int) stored as orc;
create table source2 (dt string, d1 int, d2 int) stored as orc;
insert into source1 values ('20211107', 1, 2);
insert into source2 values ('20211108', 11, 22);}}
If you run this query with UNION ALL, the {{key}} column will be missing from
the output:
{{select explode(map('D219', D219
,'D220', D220)) as (key, value) from (}}
{{select '20211107' as date_key
,1 as D219
,2 as D220
) t}}
{{union all}}
{{select explode(map('D221', D221
,'D222', D222)) as (key, value)
from (}}
{{select '20211107' as date_key
,1 as D221
,2 as D222
) t}}
Result:
{{1}}
{{2}}
{{11}}
{{22}}
Correct result should be:
{{D219 1}}
{{D220 2}}
{{D221 11}}
{{D222 22}}
> Fix column projection for union all queries with multiple aliases
> -
>
> Key: HIVE-25754
> URL: https://issues.apache.org/jira/browse/HIVE-25754
> Project: Hive
> Issue Type: Bug
>Reporter: Marton Bod
>Assignee: Marton Bod
>Priority: Major
>
> Given two tables:
>
> {code:java}
> create table source1 (dt string, d1 int, d2 int) stored as orc;
> create table source2 (dt string, d1 int, d2 int) stored as orc;
> insert into source1 values ('20211107', 1, 2);
> insert into source2 values ('20211108', 11, 22);{code}
> If you run this query with UNION ALL, the {{key}} column will be missing from
> the output:
>
>
> {code:java}
> select explode(map('D219', D219
> ,'D220', D220)) as (key, value) from (
> {{select '20211107' as date_key
> ,1 as D219
> ,2 as D220
> ) t}}
> union all
> select explode(map('D221', D221
> ,'D222', D222)) as (key, value)
> from (}}
> {{select '20211107' as date_key
> ,1 as D221
> ,2 as D222
> ) t{code}
> Result:
>
> {code:java}
> 1
> 2
> 11
> 22{code}
>
> Correct result should be:
> {code:java}
> D219 1
> D220 2
> D221 11
> D222 22{code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Assigned] (HIVE-25754) Fix column projection for union all queries with multiple aliases
[
https://issues.apache.org/jira/browse/HIVE-25754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marton Bod reassigned HIVE-25754:
-
> Fix column projection for union all queries with multiple aliases
> -
>
> Key: HIVE-25754
> URL: https://issues.apache.org/jira/browse/HIVE-25754
> Project: Hive
> Issue Type: Bug
>Reporter: Marton Bod
>Assignee: Marton Bod
>Priority: Major
>
> Given two tables:
> {{create table source1 (dt string, d1 int, d2 int) stored as orc;
> create table source2 (dt string, d1 int, d2 int) stored as orc;
> insert into source1 values ('20211107', 1, 2);
> insert into source2 values ('20211108', 11, 22);}}
> If you run this query with UNION ALL, the {{key}} column will be missing from
> the output:
> {{select explode(map('D219', D219
> ,'D220', D220)) as (key, value) from (}}
> {{select '20211107' as date_key
> ,1 as D219
> ,2 as D220
> ) t}}
> {{union all}}
> {{select explode(map('D221', D221
> ,'D222', D222)) as (key, value)
> from (}}
> {{select '20211107' as date_key
> ,1 as D221
> ,2 as D222
> ) t}}
> Result:
> {{1}}
> {{2}}
> {{11}}
> {{22}}
>
> Correct result should be:
> {{D2191}}
> {{D2202}}
> {{D22111}}
> {{D22222}}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25740) Handle race condition between compaction txn abort/commit and heartbeater
[
https://issues.apache.org/jira/browse/HIVE-25740?focusedWorklogId=688576=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-688576
]
ASF GitHub Bot logged work on HIVE-25740:
-
Author: ASF GitHub Bot
Created on: 01/Dec/21 10:45
Start Date: 01/Dec/21 10:45
Worklog Time Spent: 10m
Work Description: marton-bod commented on a change in pull request #2817:
URL: https://github.com/apache/hive/pull/2817#discussion_r760063403
##
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java
##
@@ -233,39 +234,36 @@ void gatherStats() {
private final CompactionTxn compactionTxn;
private final String tableName;
private final HiveConf conf;
-private final long txnTimeout;
+private final AtomicBoolean shouldLogError;
public CompactionHeartbeater(CompactionTxn compactionTxn, String
tableName, HiveConf conf) {
this.tableName = Objects.requireNonNull(tableName);
this.compactionTxn = Objects.requireNonNull(compactionTxn);
this.conf = Objects.requireNonNull(conf);
-
- this.txnTimeout = MetastoreConf.getTimeVar(conf,
MetastoreConf.ConfVars.TXN_TIMEOUT, TimeUnit.MILLISECONDS);
+ this.shouldLogError = new AtomicBoolean(true);
setDaemon(true);
setPriority(MIN_PRIORITY);
setName("CompactionHeartbeater-" + compactionTxn.getTxnId());
}
+public void shouldLogError(boolean shouldLogError) {
+ this.shouldLogError.set(shouldLogError);
+}
+
@Override
public void run() {
LOG.debug("Heartbeating compaction transaction id {} for table: {}",
compactionTxn, tableName);
-
IMetaStoreClient msc = null;
try {
// Create a metastore client for each thread since it is not thread
safe
msc = HiveMetaStoreUtils.getHiveMetastoreClient(conf);
Review comment:
That makes sense, and I was debating that internally as well. However,
it would complicate the cleanup of the thread quite a bit. Since the thread
runs infrequently (every 2.5 minutes or so) I think the cost of client creation
should be pretty low, and not worth the extra complexity of storing it in a
field and worry about clean up in the various scenarios. Let me know if that
makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 688576)
Time Spent: 2h (was: 1h 50m)
> Handle race condition between compaction txn abort/commit and heartbeater
> -
>
> Key: HIVE-25740
> URL: https://issues.apache.org/jira/browse/HIVE-25740
> Project: Hive
> Issue Type: Bug
>Reporter: Marton Bod
>Assignee: Marton Bod
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> This issue is the following: once the compaction worker finishes,
> commitTxn/abortTxn is invoked first, and the heartbeater thread is only
> interrupted after that. This can lead to race conditions where the txn has
> already been deleted from the backend DB via commit/abort, but the
> concurrently running heartbeater thread still attempts to send a last
> heartbeat after that, but the txn id won't be found in the DB, leading to
> {{{}NoSuchTxnException{}}}.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25740) Handle race condition between compaction txn abort/commit and heartbeater
[
https://issues.apache.org/jira/browse/HIVE-25740?focusedWorklogId=688575=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-688575
]
ASF GitHub Bot logged work on HIVE-25740:
-
Author: ASF GitHub Bot
Created on: 01/Dec/21 10:43
Start Date: 01/Dec/21 10:43
Worklog Time Spent: 10m
Work Description: marton-bod commented on a change in pull request #2817:
URL: https://github.com/apache/hive/pull/2817#discussion_r760061381
##
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java
##
@@ -751,11 +748,28 @@ void wasSuccessful() {
if (status == TxnStatus.UNKNOWN) {
return;
Review comment:
Good idea. I've refactored the `close()` method accordingly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 688575)
Time Spent: 1h 50m (was: 1h 40m)
> Handle race condition between compaction txn abort/commit and heartbeater
> -
>
> Key: HIVE-25740
> URL: https://issues.apache.org/jira/browse/HIVE-25740
> Project: Hive
> Issue Type: Bug
>Reporter: Marton Bod
>Assignee: Marton Bod
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> This issue is the following: once the compaction worker finishes,
> commitTxn/abortTxn is invoked first, and the heartbeater thread is only
> interrupted after that. This can lead to race conditions where the txn has
> already been deleted from the backend DB via commit/abort, but the
> concurrently running heartbeater thread still attempts to send a last
> heartbeat after that, but the txn id won't be found in the DB, leading to
> {{{}NoSuchTxnException{}}}.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25740) Handle race condition between compaction txn abort/commit and heartbeater
[
https://issues.apache.org/jira/browse/HIVE-25740?focusedWorklogId=688574=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-688574
]
ASF GitHub Bot logged work on HIVE-25740:
-
Author: ASF GitHub Bot
Created on: 01/Dec/21 10:42
Start Date: 01/Dec/21 10:42
Worklog Time Spent: 10m
Work Description: marton-bod commented on a change in pull request #2817:
URL: https://github.com/apache/hive/pull/2817#discussion_r760061124
##
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Worker.java
##
@@ -233,39 +234,36 @@ void gatherStats() {
private final CompactionTxn compactionTxn;
private final String tableName;
private final HiveConf conf;
-private final long txnTimeout;
+private final AtomicBoolean shouldLogError;
public CompactionHeartbeater(CompactionTxn compactionTxn, String
tableName, HiveConf conf) {
this.tableName = Objects.requireNonNull(tableName);
this.compactionTxn = Objects.requireNonNull(compactionTxn);
this.conf = Objects.requireNonNull(conf);
-
- this.txnTimeout = MetastoreConf.getTimeVar(conf,
MetastoreConf.ConfVars.TXN_TIMEOUT, TimeUnit.MILLISECONDS);
+ this.shouldLogError = new AtomicBoolean(true);
setDaemon(true);
setPriority(MIN_PRIORITY);
setName("CompactionHeartbeater-" + compactionTxn.getTxnId());
}
+public void shouldLogError(boolean shouldLogError) {
+ this.shouldLogError.set(shouldLogError);
+}
+
@Override
public void run() {
LOG.debug("Heartbeating compaction transaction id {} for table: {}",
compactionTxn, tableName);
Review comment:
Yep, makes sense. I remove one of the logs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 688574)
Time Spent: 1h 40m (was: 1.5h)
> Handle race condition between compaction txn abort/commit and heartbeater
> -
>
> Key: HIVE-25740
> URL: https://issues.apache.org/jira/browse/HIVE-25740
> Project: Hive
> Issue Type: Bug
>Reporter: Marton Bod
>Assignee: Marton Bod
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> This issue is the following: once the compaction worker finishes,
> commitTxn/abortTxn is invoked first, and the heartbeater thread is only
> interrupted after that. This can lead to race conditions where the txn has
> already been deleted from the backend DB via commit/abort, but the
> concurrently running heartbeater thread still attempts to send a last
> heartbeat after that, but the txn id won't be found in the DB, leading to
> {{{}NoSuchTxnException{}}}.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-24445) Non blocking DROP TABLE implementation
[
https://issues.apache.org/jira/browse/HIVE-24445?focusedWorklogId=688566=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-688566
]
ASF GitHub Bot logged work on HIVE-24445:
-
Author: ASF GitHub Bot
Created on: 01/Dec/21 10:28
Start Date: 01/Dec/21 10:28
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #2772:
URL: https://github.com/apache/hive/pull/2772#discussion_r760050115
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
##
@@ -336,9 +336,9 @@ public void markCompacted(CompactionInfo info) throws
MetaException {
* By filtering on minOpenTxnWaterMark, we will only cleanup after
every transaction is committed, that could see
* the uncompacted deltas. This way the cleaner can clean up
everything that was made obsolete by this compaction.
*/
-String s = "SELECT \"CQ_ID\", \"CQ_DATABASE\", \"CQ_TABLE\",
\"CQ_PARTITION\", "
-+ "\"CQ_TYPE\", \"CQ_RUN_AS\", \"CQ_HIGHEST_WRITE_ID\" FROM
\"COMPACTION_QUEUE\" WHERE \"CQ_STATE\" = '"
-+ READY_FOR_CLEANING + "'";
+String s = "SELECT \"CQ_ID\", \"CQ_DATABASE\", \"CQ_TABLE\",
\"CQ_PARTITION\", \"CQ_TYPE\", "
++ "\"CQ_RUN_AS\", \"CQ_HIGHEST_WRITE_ID\", \"CQ_TBLPROPERTIES\"
FROM \"COMPACTION_QUEUE\" WHERE \"CQ_STATE\" = '"
Review comment:
Is this an already existing field?
Do we show this in the sysdb tables?
What happens with this compaction if an old VW picks this up?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 688566)
Time Spent: 8h (was: 7h 50m)
> Non blocking DROP TABLE implementation
> --
>
> Key: HIVE-24445
> URL: https://issues.apache.org/jira/browse/HIVE-24445
> Project: Hive
> Issue Type: New Feature
> Components: Hive
>Reporter: Zoltan Chovan
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 8h
> Remaining Estimate: 0h
>
> Implement a way to execute drop table operations in a way that doesn't have
> to wait for currently running read operations to be finished.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-24445) Non blocking DROP TABLE implementation
[
https://issues.apache.org/jira/browse/HIVE-24445?focusedWorklogId=688564=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-688564
]
ASF GitHub Bot logged work on HIVE-24445:
-
Author: ASF GitHub Bot
Created on: 01/Dec/21 10:26
Start Date: 01/Dec/21 10:26
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #2772:
URL: https://github.com/apache/hive/pull/2772#discussion_r760047913
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnHandler.java
##
@@ -1451,11 +1453,11 @@ public void commitTxn(CommitTxnRequest rqst) throws
NoSuchTxnException, TxnAbort
OperationType.UPDATE + "," + OperationType.DELETE + ")";
long tempCommitId = generateTemporaryId();
-if (txnType.get() == TxnType.COMPACTION) {
+if (txnType == TxnType.SOFT_DELETE || txnType == TxnType.COMPACTION) {
acquireTxnLock(stmt, false);
commitId = getHighWaterMark(stmt);
-} else if (txnType.get() != TxnType.READ_ONLY && !isReplayedReplTxn) {
+} else if (txnType != TxnType.READ_ONLY && !isReplayedReplTxn) {
Review comment:
What will happen wrt replication of these soft delete-s? Do they know
about the changes we propose?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 688564)
Time Spent: 7h 50m (was: 7h 40m)
> Non blocking DROP TABLE implementation
> --
>
> Key: HIVE-24445
> URL: https://issues.apache.org/jira/browse/HIVE-24445
> Project: Hive
> Issue Type: New Feature
> Components: Hive
>Reporter: Zoltan Chovan
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 7h 50m
> Remaining Estimate: 0h
>
> Implement a way to execute drop table operations in a way that doesn't have
> to wait for currently running read operations to be finished.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-24445) Non blocking DROP TABLE implementation
[
https://issues.apache.org/jira/browse/HIVE-24445?focusedWorklogId=688561=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-688561
]
ASF GitHub Bot logged work on HIVE-24445:
-
Author: ASF GitHub Bot
Created on: 01/Dec/21 10:23
Start Date: 01/Dec/21 10:23
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #2772:
URL: https://github.com/apache/hive/pull/2772#discussion_r760045866
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/AcidEventListener.java
##
@@ -60,9 +65,30 @@ public void onDropDatabase (DropDatabaseEvent dbEvent)
throws MetaException {
@Override
public void onDropTable(DropTableEvent tableEvent) throws MetaException {
-if (TxnUtils.isTransactionalTable(tableEvent.getTable())) {
+Table table = tableEvent.getTable();
+
+if (TxnUtils.isTransactionalTable(table)) {
txnHandler = getTxnHandler();
- txnHandler.cleanupRecords(HiveObjectType.TABLE, null,
tableEvent.getTable(), null);
+ txnHandler.cleanupRecords(HiveObjectType.TABLE, null, table, null,
!tableEvent.getDeleteData());
+
+ if (!tableEvent.getDeleteData()) {
+long currentTxn =
Optional.ofNullable(tableEvent.getEnvironmentContext())
+ .map(EnvironmentContext::getProperties)
+ .map(prop -> prop.get("txnId"))
+ .map(Long::parseLong)
+ .orElse(0L);
+
+if (currentTxn > 0) {
+ CompactionRequest rqst = new CompactionRequest(table.getDbName(),
table.getTableName(), CompactionType.MAJOR);
+ rqst.putToProperties("location", table.getSd().getLocation());
Review comment:
Why properties, and not optional field?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 688561)
Time Spent: 7h 40m (was: 7.5h)
> Non blocking DROP TABLE implementation
> --
>
> Key: HIVE-24445
> URL: https://issues.apache.org/jira/browse/HIVE-24445
> Project: Hive
> Issue Type: New Feature
> Components: Hive
>Reporter: Zoltan Chovan
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 7h 40m
> Remaining Estimate: 0h
>
> Implement a way to execute drop table operations in a way that doesn't have
> to wait for currently running read operations to be finished.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-24445) Non blocking DROP TABLE implementation
[
https://issues.apache.org/jira/browse/HIVE-24445?focusedWorklogId=688560=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-688560
]
ASF GitHub Bot logged work on HIVE-24445:
-
Author: ASF GitHub Bot
Created on: 01/Dec/21 10:21
Start Date: 01/Dec/21 10:21
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #2772:
URL: https://github.com/apache/hive/pull/2772#discussion_r760043808
##
File path:
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
##
@@ -1772,6 +1773,22 @@ public void dropTable(String dbname, String name,
boolean deleteData,
dropTable(getDefaultCatalog(conf), dbname, name, deleteData,
ignoreUnknownTab, ifPurge);
}
+ @Override
+ public void dropTable(Table tbl, boolean deleteData, boolean
ignoreUnknownTbl, boolean ifPurge) throws TException {
+EnvironmentContext context = null;
+if (ifPurge) {
Review comment:
How will we handle purge with soft delete?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 688560)
Time Spent: 7.5h (was: 7h 20m)
> Non blocking DROP TABLE implementation
> --
>
> Key: HIVE-24445
> URL: https://issues.apache.org/jira/browse/HIVE-24445
> Project: Hive
> Issue Type: New Feature
> Components: Hive
>Reporter: Zoltan Chovan
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 7.5h
> Remaining Estimate: 0h
>
> Implement a way to execute drop table operations in a way that doesn't have
> to wait for currently running read operations to be finished.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-24445) Non blocking DROP TABLE implementation
[
https://issues.apache.org/jira/browse/HIVE-24445?focusedWorklogId=688554=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-688554
]
ASF GitHub Bot logged work on HIVE-24445:
-
Author: ASF GitHub Bot
Created on: 01/Dec/21 10:13
Start Date: 01/Dec/21 10:13
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #2772:
URL: https://github.com/apache/hive/pull/2772#discussion_r760037581
##
File path: ql/src/test/org/apache/hadoop/hive/ql/TxnCommandsBaseForTests.java
##
@@ -202,8 +208,29 @@ public int compare(int[] row1, int[] row2) {
return 0;
}
}
- protected String makeValuesClause(int[][] rows) {
-return TestTxnCommands2.makeValuesClause(rows);
+ public static String makeValuesClause(int[][] rows) {
Review comment:
nit: empty line
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 688554)
Time Spent: 7h 10m (was: 7h)
> Non blocking DROP TABLE implementation
> --
>
> Key: HIVE-24445
> URL: https://issues.apache.org/jira/browse/HIVE-24445
> Project: Hive
> Issue Type: New Feature
> Components: Hive
>Reporter: Zoltan Chovan
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 7h 10m
> Remaining Estimate: 0h
>
> Implement a way to execute drop table operations in a way that doesn't have
> to wait for currently running read operations to be finished.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-24445) Non blocking DROP TABLE implementation
[
https://issues.apache.org/jira/browse/HIVE-24445?focusedWorklogId=688558=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-688558
]
ASF GitHub Bot logged work on HIVE-24445:
-
Author: ASF GitHub Bot
Created on: 01/Dec/21 10:20
Start Date: 01/Dec/21 10:20
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #2772:
URL: https://github.com/apache/hive/pull/2772#discussion_r760042920
##
File path:
standalone-metastore/metastore-common/src/gen/thrift/gen-cpp/hive_metastore_types.cpp
##
@@ -615,16 +615,18 @@ int _kTxnTypeValues[] = {
TxnType::REPL_CREATED,
TxnType::READ_ONLY,
TxnType::COMPACTION,
- TxnType::MATER_VIEW_REBUILD
+ TxnType::MATER_VIEW_REBUILD,
+ TxnType::SOFT_DELETE
Review comment:
Will this cause any compatibility issues?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 688558)
Time Spent: 7h 20m (was: 7h 10m)
> Non blocking DROP TABLE implementation
> --
>
> Key: HIVE-24445
> URL: https://issues.apache.org/jira/browse/HIVE-24445
> Project: Hive
> Issue Type: New Feature
> Components: Hive
>Reporter: Zoltan Chovan
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 7h 20m
> Remaining Estimate: 0h
>
> Implement a way to execute drop table operations in a way that doesn't have
> to wait for currently running read operations to be finished.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HIVE-14261) Support set/unset partition parameters
[ https://issues.apache.org/jira/browse/HIVE-14261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17451692#comment-17451692 ] xiepengjie commented on HIVE-14261: --- Yes, you'r right. This pr is just about enriching the syntax of HS2. For other example, we want to have a life cycle management system about the partition, so we need to set different parameters for different partition. > Support set/unset partition parameters > -- > > Key: HIVE-14261 > URL: https://issues.apache.org/jira/browse/HIVE-14261 > Project: Hive > Issue Type: New Feature >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong >Priority: Major > Attachments: HIVE-14261.01.patch > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-24445) Non blocking DROP TABLE implementation
[
https://issues.apache.org/jira/browse/HIVE-24445?focusedWorklogId=688552=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-688552
]
ASF GitHub Bot logged work on HIVE-24445:
-
Author: ASF GitHub Bot
Created on: 01/Dec/21 10:08
Start Date: 01/Dec/21 10:08
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #2772:
URL: https://github.com/apache/hive/pull/2772#discussion_r760033191
##
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java
##
@@ -279,10 +292,14 @@ private void clean(CompactionInfo ci, long minOpenTxnGLB,
boolean metricsEnabled
}
}
}
+
+ private interface CleanerTask {
Review comment:
Why do we need this specific interface instead of `Callable`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 688552)
Time Spent: 7h (was: 6h 50m)
> Non blocking DROP TABLE implementation
> --
>
> Key: HIVE-24445
> URL: https://issues.apache.org/jira/browse/HIVE-24445
> Project: Hive
> Issue Type: New Feature
> Components: Hive
>Reporter: Zoltan Chovan
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 7h
> Remaining Estimate: 0h
>
> Implement a way to execute drop table operations in a way that doesn't have
> to wait for currently running read operations to be finished.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-24445) Non blocking DROP TABLE implementation
[
https://issues.apache.org/jira/browse/HIVE-24445?focusedWorklogId=688549=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-688549
]
ASF GitHub Bot logged work on HIVE-24445:
-
Author: ASF GitHub Bot
Created on: 01/Dec/21 10:05
Start Date: 01/Dec/21 10:05
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #2772:
URL: https://github.com/apache/hive/pull/2772#discussion_r760030923
##
File path: ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/Cleaner.java
##
@@ -172,22 +174,29 @@ private void clean(CompactionInfo ci, long minOpenTxnGLB,
boolean metricsEnabled
if (metricsEnabled) {
perfLogger.perfLogBegin(CLASS_NAME, cleanerMetric);
}
- Table t = resolveTable(ci);
- if (t == null) {
-// The table was dropped before we got around to cleaning it.
-LOG.info("Unable to find table " + ci.getFullTableName() + ", assuming
it was dropped." +
+ String location =
Optional.ofNullable(ci.properties).map(StringableMap::new)
Review comment:
How this behaves with non-standard locations, and transformer generated
locations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 688549)
Time Spent: 6h 50m (was: 6h 40m)
> Non blocking DROP TABLE implementation
> --
>
> Key: HIVE-24445
> URL: https://issues.apache.org/jira/browse/HIVE-24445
> Project: Hive
> Issue Type: New Feature
> Components: Hive
>Reporter: Zoltan Chovan
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 6h 50m
> Remaining Estimate: 0h
>
> Implement a way to execute drop table operations in a way that doesn't have
> to wait for currently running read operations to be finished.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-24445) Non blocking DROP TABLE implementation
[
https://issues.apache.org/jira/browse/HIVE-24445?focusedWorklogId=688546=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-688546
]
ASF GitHub Bot logged work on HIVE-24445:
-
Author: ASF GitHub Bot
Created on: 01/Dec/21 10:01
Start Date: 01/Dec/21 10:01
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #2772:
URL: https://github.com/apache/hive/pull/2772#discussion_r760027429
##
File path: ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
##
@@ -1334,6 +1344,20 @@ public void dropTable(String tableName, boolean ifPurge)
throws HiveException {
dropTable(names[0], names[1], true, true, ifPurge);
}
+ public void dropTable(Table table, boolean ifPurge) throws HiveException {
+boolean tableWithSuffix = (HiveConf.getBoolVar(conf,
ConfVars.HIVE_ACID_CREATE_TABLE_USE_SUFFIX)
+|| HiveConf.getBoolVar(conf,
ConfVars.HIVE_ACID_LOCKLESS_READS_ENABLED))
+ && AcidUtils.isTransactionalTable(table)
+ && table.getSd().getLocation().matches(PATH_WITH_VERSION_ID);
+
+if (tableWithSuffix) {
+ long txnId = Optional.ofNullable(SessionState.get())
+.map(ss -> ss.getTxnMgr().getCurrentTxnId()).orElse(0L);
+ table.getTTable().setTxnId(txnId);
+}
Review comment:
if the location is not PATH_WITH_VERSION_ID, we might have a situation
were we are trying to do a softdelete, but could not do it. Do we need to
handle this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 688546)
Time Spent: 6h 40m (was: 6.5h)
> Non blocking DROP TABLE implementation
> --
>
> Key: HIVE-24445
> URL: https://issues.apache.org/jira/browse/HIVE-24445
> Project: Hive
> Issue Type: New Feature
> Components: Hive
>Reporter: Zoltan Chovan
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 6h 40m
> Remaining Estimate: 0h
>
> Implement a way to execute drop table operations in a way that doesn't have
> to wait for currently running read operations to be finished.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HIVE-14261) Support set/unset partition parameters
[ https://issues.apache.org/jira/browse/HIVE-14261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17451682#comment-17451682 ] Stamatis Zampetakis commented on HIVE-14261: [~xiepengjie] Your mentioned previously that there are a lot of projects who use exclusively HMS thus use the JAVA snippet above. However, the change you propose here is about enriching the syntax of HS2 to be able to change the properties via HiveQL. How will this help other projects who use only HMS? Did I misunderstood something ? > Support set/unset partition parameters > -- > > Key: HIVE-14261 > URL: https://issues.apache.org/jira/browse/HIVE-14261 > Project: Hive > Issue Type: New Feature >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong >Priority: Major > Attachments: HIVE-14261.01.patch > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-24445) Non blocking DROP TABLE implementation
[
https://issues.apache.org/jira/browse/HIVE-24445?focusedWorklogId=688545=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-688545
]
ASF GitHub Bot logged work on HIVE-24445:
-
Author: ASF GitHub Bot
Created on: 01/Dec/21 09:59
Start Date: 01/Dec/21 09:59
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #2772:
URL: https://github.com/apache/hive/pull/2772#discussion_r760026327
##
File path: ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
##
@@ -1273,14 +1276,21 @@ public void createTable(Table tbl, boolean ifNotExists,
principalPrivs.setRolePrivileges(grants.getRoleGrants());
tTbl.setPrivileges(principalPrivs);
}
+boolean createTableUseSuffix = (HiveConf.getBoolVar(conf,
ConfVars.HIVE_ACID_CREATE_TABLE_USE_SUFFIX)
+|| HiveConf.getBoolVar(conf,
ConfVars.HIVE_ACID_LOCKLESS_READS_ENABLED))
+ && AcidUtils.isTransactionalTable(tbl);
+
+if (createTableUseSuffix) {
+ tTbl.setTxnId(ss.getTxnMgr().getCurrentTxnId());
Review comment:
Shall we just push this transactionId for every transactional table?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 688545)
Time Spent: 6.5h (was: 6h 20m)
> Non blocking DROP TABLE implementation
> --
>
> Key: HIVE-24445
> URL: https://issues.apache.org/jira/browse/HIVE-24445
> Project: Hive
> Issue Type: New Feature
> Components: Hive
>Reporter: Zoltan Chovan
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 6.5h
> Remaining Estimate: 0h
>
> Implement a way to execute drop table operations in a way that doesn't have
> to wait for currently running read operations to be finished.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-24445) Non blocking DROP TABLE implementation
[
https://issues.apache.org/jira/browse/HIVE-24445?focusedWorklogId=688544=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-688544
]
ASF GitHub Bot logged work on HIVE-24445:
-
Author: ASF GitHub Bot
Created on: 01/Dec/21 09:58
Start Date: 01/Dec/21 09:58
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #2772:
URL: https://github.com/apache/hive/pull/2772#discussion_r760025563
##
File path: ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java
##
@@ -3109,10 +3109,16 @@ public static TxnType getTxnType(Configuration conf,
ASTNode tree) {
if (tree.getFirstChildWithType(HiveParser.TOK_ALTERTABLE_COMPACT) != null){
return TxnType.COMPACTION;
}
+// check if soft delete
+if (tree.getToken().getType() == HiveParser.TOK_DROPTABLE
+ && (HiveConf.getBoolVar(conf, ConfVars.HIVE_ACID_CREATE_TABLE_USE_SUFFIX)
+|| HiveConf.getBoolVar(conf,
ConfVars.HIVE_ACID_LOCKLESS_READS_ENABLED))){
+ return TxnType.SOFT_DELETE;
Review comment:
Shall we throw an exception if we try to drop a table but the location
is not PATH_WITH_VERSION_ID?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 688544)
Time Spent: 6h 20m (was: 6h 10m)
> Non blocking DROP TABLE implementation
> --
>
> Key: HIVE-24445
> URL: https://issues.apache.org/jira/browse/HIVE-24445
> Project: Hive
> Issue Type: New Feature
> Components: Hive
>Reporter: Zoltan Chovan
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 6h 20m
> Remaining Estimate: 0h
>
> Implement a way to execute drop table operations in a way that doesn't have
> to wait for currently running read operations to be finished.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-24445) Non blocking DROP TABLE implementation
[
https://issues.apache.org/jira/browse/HIVE-24445?focusedWorklogId=688542=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-688542
]
ASF GitHub Bot logged work on HIVE-24445:
-
Author: ASF GitHub Bot
Created on: 01/Dec/21 09:57
Start Date: 01/Dec/21 09:57
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #2772:
URL: https://github.com/apache/hive/pull/2772#discussion_r760024012
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/drop/DropTableOperation.java
##
@@ -109,7 +109,7 @@ public int execute() throws HiveException {
}
// TODO: API w/catalog name
-context.getDb().dropTable(desc.getTableName(), desc.isPurge());
+context.getDb().dropTable(table, desc.isPurge());
Review comment:
nit: Why is this change needed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 688542)
Time Spent: 6h 10m (was: 6h)
> Non blocking DROP TABLE implementation
> --
>
> Key: HIVE-24445
> URL: https://issues.apache.org/jira/browse/HIVE-24445
> Project: Hive
> Issue Type: New Feature
> Components: Hive
>Reporter: Zoltan Chovan
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 6h 10m
> Remaining Estimate: 0h
>
> Implement a way to execute drop table operations in a way that doesn't have
> to wait for currently running read operations to be finished.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-24445) Non blocking DROP TABLE implementation
[
https://issues.apache.org/jira/browse/HIVE-24445?focusedWorklogId=688541=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-688541
]
ASF GitHub Bot logged work on HIVE-24445:
-
Author: ASF GitHub Bot
Created on: 01/Dec/21 09:56
Start Date: 01/Dec/21 09:56
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #2772:
URL: https://github.com/apache/hive/pull/2772#discussion_r760023405
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/ddl/table/drop/DropTableAnalyzer.java
##
@@ -51,7 +55,14 @@ public void analyzeInternal(ASTNode root) throws
SemanticException {
Table table = getTable(tableName, throwException);
if (table != null) {
inputs.add(new ReadEntity(table));
- outputs.add(new WriteEntity(table, WriteEntity.WriteType.DDL_EXCLUSIVE));
+
+ boolean tableWithSuffix = (HiveConf.getBoolVar(conf,
ConfVars.HIVE_ACID_CREATE_TABLE_USE_SUFFIX)
+ || HiveConf.getBoolVar(conf,
ConfVars.HIVE_ACID_LOCKLESS_READS_ENABLED))
+&& AcidUtils.isTransactionalTable(table)
+&& table.getSd().getLocation().matches(PATH_WITH_VERSION_ID);
+
+ outputs.add(new WriteEntity(table,
Review comment:
It would be good to see this in the log. Minimally with debug level
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 688541)
Time Spent: 6h (was: 5h 50m)
> Non blocking DROP TABLE implementation
> --
>
> Key: HIVE-24445
> URL: https://issues.apache.org/jira/browse/HIVE-24445
> Project: Hive
> Issue Type: New Feature
> Components: Hive
>Reporter: Zoltan Chovan
>Assignee: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 6h
> Remaining Estimate: 0h
>
> Implement a way to execute drop table operations in a way that doesn't have
> to wait for currently running read operations to be finished.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-24445) Non blocking DROP TABLE implementation
[ https://issues.apache.org/jira/browse/HIVE-24445?focusedWorklogId=688540=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-688540 ] ASF GitHub Bot logged work on HIVE-24445: - Author: ASF GitHub Bot Created on: 01/Dec/21 09:54 Start Date: 01/Dec/21 09:54 Worklog Time Spent: 10m Work Description: pvary commented on a change in pull request #2772: URL: https://github.com/apache/hive/pull/2772#discussion_r760021737 ## File path: itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestAcidOnTez.java ## @@ -18,6 +18,7 @@ package org.apache.hadoop.hive.ql; +import static org.apache.hadoop.hive.ql.TxnCommandsBaseForTests.*; Review comment: Is there any other change in this file than the import change? If not I would revert this file -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 688540) Time Spent: 5h 50m (was: 5h 40m) > Non blocking DROP TABLE implementation > -- > > Key: HIVE-24445 > URL: https://issues.apache.org/jira/browse/HIVE-24445 > Project: Hive > Issue Type: New Feature > Components: Hive >Reporter: Zoltan Chovan >Assignee: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Time Spent: 5h 50m > Remaining Estimate: 0h > > Implement a way to execute drop table operations in a way that doesn't have > to wait for currently running read operations to be finished. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-24445) Non blocking DROP TABLE implementation
[ https://issues.apache.org/jira/browse/HIVE-24445?focusedWorklogId=688538=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-688538 ] ASF GitHub Bot logged work on HIVE-24445: - Author: ASF GitHub Bot Created on: 01/Dec/21 09:53 Start Date: 01/Dec/21 09:53 Worklog Time Spent: 10m Work Description: pvary commented on a change in pull request #2772: URL: https://github.com/apache/hive/pull/2772#discussion_r760021077 ## File path: itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestAcidOnTez.java ## @@ -18,6 +18,7 @@ package org.apache.hadoop.hive.ql; +import static org.apache.hadoop.hive.ql.TxnCommandsBaseForTests.*; Review comment: nit: do not use wildcard imports -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 688538) Time Spent: 5h 40m (was: 5.5h) > Non blocking DROP TABLE implementation > -- > > Key: HIVE-24445 > URL: https://issues.apache.org/jira/browse/HIVE-24445 > Project: Hive > Issue Type: New Feature > Components: Hive >Reporter: Zoltan Chovan >Assignee: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Time Spent: 5h 40m > Remaining Estimate: 0h > > Implement a way to execute drop table operations in a way that doesn't have > to wait for currently running read operations to be finished. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25609) Preserve XAttrs in normal file copy case.
[ https://issues.apache.org/jira/browse/HIVE-25609?focusedWorklogId=688514=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-688514 ] ASF GitHub Bot logged work on HIVE-25609: - Author: ASF GitHub Bot Created on: 01/Dec/21 09:14 Start Date: 01/Dec/21 09:14 Worklog Time Spent: 10m Work Description: pkumarsinha merged pull request #2793: URL: https://github.com/apache/hive/pull/2793 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 688514) Time Spent: 40m (was: 0.5h) > Preserve XAttrs in normal file copy case. > - > > Key: HIVE-25609 > URL: https://issues.apache.org/jira/browse/HIVE-25609 > Project: Hive > Issue Type: Improvement >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001)
