[jira] [Resolved] (HIVE-26033) Repl Load fails with Wrong FS error.
[
https://issues.apache.org/jira/browse/HIVE-26033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ayush Saxena resolved HIVE-26033.
-
Fix Version/s: 4.0.0
Resolution: Fixed
> Repl Load fails with Wrong FS error.
>
>
> Key: HIVE-26033
> URL: https://issues.apache.org/jira/browse/HIVE-26033
> Project: Hive
> Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> For External table replication with staging on source, the replication load
> fails with wrong FS error while cleaning up snapshots.
> {noformat}
> Exception : Wrong FS:
> hdfs://cluster1:8020/user/hive/replDir/policy_1646973828/_file_list_external_current,
> expected: hdfs://cluster2:8020
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HIVE-26033) Repl Load fails with Wrong FS error.
[
https://issues.apache.org/jira/browse/HIVE-26033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17510231#comment-17510231
]
Ayush Saxena commented on HIVE-26033:
-
Merged PR to master. Thanx [~maheshk114] for the review!!!
> Repl Load fails with Wrong FS error.
>
>
> Key: HIVE-26033
> URL: https://issues.apache.org/jira/browse/HIVE-26033
> Project: Hive
> Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> For External table replication with staging on source, the replication load
> fails with wrong FS error while cleaning up snapshots.
> {noformat}
> Exception : Wrong FS:
> hdfs://cluster1:8020/user/hive/replDir/policy_1646973828/_file_list_external_current,
> expected: hdfs://cluster2:8020
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-26033) Repl Load fails with Wrong FS error.
[
https://issues.apache.org/jira/browse/HIVE-26033?focusedWorklogId=745588=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745588
]
ASF GitHub Bot logged work on HIVE-26033:
-
Author: ASF GitHub Bot
Created on: 22/Mar/22 05:39
Start Date: 22/Mar/22 05:39
Worklog Time Spent: 10m
Work Description: ayushtkn merged pull request #3100:
URL: https://github.com/apache/hive/pull/3100
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 745588)
Time Spent: 20m (was: 10m)
> Repl Load fails with Wrong FS error.
>
>
> Key: HIVE-26033
> URL: https://issues.apache.org/jira/browse/HIVE-26033
> Project: Hive
> Issue Type: Bug
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> For External table replication with staging on source, the replication load
> fails with wrong FS error while cleaning up snapshots.
> {noformat}
> Exception : Wrong FS:
> hdfs://cluster1:8020/user/hive/replDir/policy_1646973828/_file_list_external_current,
> expected: hdfs://cluster2:8020
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-26041) Fix wrong type supplied for getLatestCommittedCompaction
[
https://issues.apache.org/jira/browse/HIVE-26041?focusedWorklogId=745510=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745510
]
ASF GitHub Bot logged work on HIVE-26041:
-
Author: ASF GitHub Bot
Created on: 21/Mar/22 22:57
Start Date: 21/Mar/22 22:57
Worklog Time Spent: 10m
Work Description: nrg4878 commented on pull request #3113:
URL: https://github.com/apache/hive/pull/3113#issuecomment-1074501550
Change looks good to me. +1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 745510)
Time Spent: 20m (was: 10m)
> Fix wrong type supplied for getLatestCommittedCompaction
>
>
> Key: HIVE-26041
> URL: https://issues.apache.org/jira/browse/HIVE-26041
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Reporter: Yu-Wen Lai
>Assignee: Yu-Wen Lai
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> In HIVE-25753, we filter compactions by CC_ID, but I used string type as the
> parameter for the prepared statement. That cause a type error on some
> databases (at least failed on PostgreSQL).
> To correctly handle the filter, we should use
> {code:java}
> preparedStatement.setLong(...){code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Assigned] (HIVE-26055) Fix the HivePrivilegesObjects for Alter table rename command
[ https://issues.apache.org/jira/browse/HIVE-26055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Hemanth Gantasala reassigned HIVE-26055: > Fix the HivePrivilegesObjects for Alter table rename command > > > Key: HIVE-26055 > URL: https://issues.apache.org/jira/browse/HIVE-26055 > Project: Hive > Issue Type: Bug > Components: HiveServer2, Security >Reporter: Sai Hemanth Gantasala >Assignee: Sai Hemanth Gantasala >Priority: Major > > Fix the HivePrivilegeObjects for Alter table rename query in a way that it > includes source table information in the output objects and destination table > information in the input objects. > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25826) Support table defaults at each database level
[
https://issues.apache.org/jira/browse/HIVE-25826?focusedWorklogId=745497=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745497
]
ASF GitHub Bot logged work on HIVE-25826:
-
Author: ASF GitHub Bot
Created on: 21/Mar/22 22:44
Start Date: 21/Mar/22 22:44
Worklog Time Spent: 10m
Work Description: nrg4878 commented on a change in pull request #2924:
URL: https://github.com/apache/hive/pull/2924#discussion_r831597354
##
File path: ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
##
@@ -13615,7 +13631,22 @@ ASTNode analyzeCreateTable(
// Handle different types of CREATE TABLE command
// Note: each branch must call addDbAndTabToOutputs after finalizing table
properties.
-
+Database database = getDatabase(qualifiedTabName.getDb());
+boolean isDefaultTableTypeChanged = false;
+if(database.getParameters() != null) {
+ String defaultTableType =
database.getParameters().getOrDefault(DEFAULT_TABLE_TYPE, "none");
Review comment:
not sure why we are using the default of "none" if not set. We are not
doing anything with the value. why not just allow it to return a null when not
set, and check for null?
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestDatabaseTableDefault.java
##
@@ -0,0 +1,298 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql;
+
+import static org.junit.Assert.assertEquals;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.common.FileUtils;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf.ConfVars;
+import org.apache.hadoop.hive.metastore.HiveMetaStoreClient;
+import org.apache.hadoop.hive.metastore.utils.TestTxnDbUtil;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
Review comment:
nit: rearrange imports in the alphabetical order
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/TestDatabaseTableDefault.java
##
@@ -0,0 +1,298 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql;
+
+import static org.junit.Assert.assertEquals;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Set;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.common.FileUtils;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf.ConfVars;
+import org.apache.hadoop.hive.metastore.HiveMetaStoreClient;
+import org.apache.hadoop.hive.metastore.utils.TestTxnDbUtil;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.junit.After;
+import org.junit.Assert;
+import
[jira] [Commented] (HIVE-25963) Temporary table creation with not null constraint gets converted to external table
[
https://issues.apache.org/jira/browse/HIVE-25963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17510044#comment-17510044
]
Sourabh Goyal commented on HIVE-25963:
--
Merged upstream. Commits:
#
[https://github.com/apache/hive/commit/7a5ec913db6ca294a2753a21f6b14e78006287cc]
#
[https://github.com/apache/hive/commit/dc625fd3c2d1a5bd7ef046ec6d2f549992d37857]
Thank you [~ngangam] [~gsaihemanth] [~hsnusonic] for the review
> Temporary table creation with not null constraint gets converted to external
> table
> ---
>
> Key: HIVE-25963
> URL: https://issues.apache.org/jira/browse/HIVE-25963
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Standalone Metastore
>Reporter: Sourabh Goyal
>Assignee: Sourabh Goyal
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> When creating a temporary table with not null, constraint it gets covered to
> external table. For example:
> create temporary table t2 (a int not null);
> table t2' metadata looks like:
> {code:java}
> +---+++
> | col_name| data_type
>| comment |
> +---+++
> | a | int
>||
> | | NULL
>| NULL |
> | # Detailed Table Information | NULL
>| NULL |
> | Database: | default
>| NULL |
> | OwnerType:| USER
>| NULL |
> | Owner:| sourabh
>| NULL |
> | CreateTime: | Tue Feb 15 15:20:13 PST 2022
>| NULL |
> | LastAccessTime: | UNKNOWN
>| NULL |
> | Retention:| 0
>| NULL |
> | Location: |
> hdfs://localhost:9000/tmp/hive/sourabh/80d374a8-cd7a-4fcf-ae72-51b04ff9c3d8/_tmp_space.db/4574446d-c144-48f9-b4b6-2e9ee0ce5be4
> | NULL |
> | Table Type: | EXTERNAL_TABLE
>| NULL |
> | Table Parameters: | NULL
>| NULL |
> | | COLUMN_STATS_ACCURATE
>| {\"BASIC_STATS\":\"true\",\"COLUMN_STATS\":{\"a\":\"true\"}} |
> | | EXTERNAL
>| TRUE |
> | | TRANSLATED_TO_EXTERNAL
>| TRUE |
> | | bucketing_version
>| 2 |
> | | external.table.purge
>| TRUE |
> | | numFiles
>| 0 |
> | | numRows
>| 0 |
> | | rawDataSize
>| 0 |
> | | totalSize
>| 0 |
> | | transient_lastDdlTime
>| 1644967213
[jira] [Resolved] (HIVE-25963) Temporary table creation with not null constraint gets converted to external table
[
https://issues.apache.org/jira/browse/HIVE-25963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sourabh Goyal resolved HIVE-25963.
--
Resolution: Fixed
> Temporary table creation with not null constraint gets converted to external
> table
> ---
>
> Key: HIVE-25963
> URL: https://issues.apache.org/jira/browse/HIVE-25963
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, Standalone Metastore
>Reporter: Sourabh Goyal
>Assignee: Sourabh Goyal
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> When creating a temporary table with not null, constraint it gets covered to
> external table. For example:
> create temporary table t2 (a int not null);
> table t2' metadata looks like:
> {code:java}
> +---+++
> | col_name| data_type
>| comment |
> +---+++
> | a | int
>||
> | | NULL
>| NULL |
> | # Detailed Table Information | NULL
>| NULL |
> | Database: | default
>| NULL |
> | OwnerType:| USER
>| NULL |
> | Owner:| sourabh
>| NULL |
> | CreateTime: | Tue Feb 15 15:20:13 PST 2022
>| NULL |
> | LastAccessTime: | UNKNOWN
>| NULL |
> | Retention:| 0
>| NULL |
> | Location: |
> hdfs://localhost:9000/tmp/hive/sourabh/80d374a8-cd7a-4fcf-ae72-51b04ff9c3d8/_tmp_space.db/4574446d-c144-48f9-b4b6-2e9ee0ce5be4
> | NULL |
> | Table Type: | EXTERNAL_TABLE
>| NULL |
> | Table Parameters: | NULL
>| NULL |
> | | COLUMN_STATS_ACCURATE
>| {\"BASIC_STATS\":\"true\",\"COLUMN_STATS\":{\"a\":\"true\"}} |
> | | EXTERNAL
>| TRUE |
> | | TRANSLATED_TO_EXTERNAL
>| TRUE |
> | | bucketing_version
>| 2 |
> | | external.table.purge
>| TRUE |
> | | numFiles
>| 0 |
> | | numRows
>| 0 |
> | | rawDataSize
>| 0 |
> | | totalSize
>| 0 |
> | | transient_lastDdlTime
>| 1644967213 |
> | | NULL
>| NULL |
> | # Storage Information | NULL
>| NULL
[jira] [Work logged] (HIVE-26043) Use constraint info when creating RexNodes
[
https://issues.apache.org/jira/browse/HIVE-26043?focusedWorklogId=745220=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745220
]
ASF GitHub Bot logged work on HIVE-26043:
-
Author: ASF GitHub Bot
Created on: 21/Mar/22 16:29
Start Date: 21/Mar/22 16:29
Worklog Time Spent: 10m
Work Description: kasakrisz opened a new pull request #3121:
URL: https://github.com/apache/hive/pull/3121
### What changes were proposed in this pull request?
Add nullability to `ColumnInfo` and use it when creating `RexNodes`
### Why are the changes needed?
When TableScan logical operators are created table constraint information is
pulled from the metastore and applied the RelDataType associated to the TS
operator.
Constraints like not null has an affect the way Hive rewrites subqueries to
joins. Example: if the optimizer is aware that the lhs of an in subquery can
not be null a simpler rewrite is chosen. See jira for example plans.
### Does this PR introduce _any_ user-facing change?
No, but `explain` command output may be different in older Hive versions.
### How was this patch tested?
```
mvn test -Dtest.output.overwrite -DskipSparkTests
-Dtest=TestTezTPCDS30TBPerfCliDriver -Dqfile=query45.q -pl itests/qtest -Pitests
mvn test -Dtest.output.overwrite -DskipSparkTests
-Dtest=TestMiniLlapLocalCliDriver -Dqfile=subquery_in.q -pl itests/qtest
-Pitests
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 745220)
Time Spent: 0.5h (was: 20m)
> Use constraint info when creating RexNodes
> --
>
> Key: HIVE-26043
> URL: https://issues.apache.org/jira/browse/HIVE-26043
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Prior HIVE-23100 Not null constraints affected newly created RexNode type
> nullability.
> Nullability enables the subquery rewrite algorithm to generate more optimal
> plan.
> [https://github.com/apache/hive/blob/1213ad3f0ae0e21e7519dc28b8b6d1401cdd1441/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSubQueryRemoveRule.java#L324]
> Example:
> {code:java}
> explain cbo
> select ws_sales_price
> from web_sales, customer, item
> where ws_bill_customer_sk = c_customer_sk
> and ws_item_sk = i_item_sk
> and ( c_customer_sk = 1
> or
> i_item_id in (select i_item_id
> from item
> where i_item_sk in (2, 3)
> )
> );
> {code}
> Without not null constraints
> {code:java}
> HiveProject(ws_sales_price=[$2])
> HiveFilter(condition=[OR(AND(<>($6, 0), IS NOT NULL($8)), =($3, 1))])
> HiveProject(ws_item_sk=[$0], ws_bill_customer_sk=[$1],
> ws_sales_price=[$2], c_customer_sk=[$8], i_item_sk=[$3], i_item_id=[$4],
> c=[$5], i_item_id0=[$6], literalTrue=[$7])
> HiveJoin(condition=[=($1, $8)], joinType=[inner], algorithm=[none],
> cost=[not available])
> HiveJoin(condition=[=($0, $3)], joinType=[inner], algorithm=[none],
> cost=[not available])
> HiveProject(ws_item_sk=[$2], ws_bill_customer_sk=[$3],
> ws_sales_price=[$20])
> HiveFilter(condition=[IS NOT NULL($3)])
> HiveTableScan(table=[[default, web_sales]],
> table:alias=[web_sales])
> HiveJoin(condition=[=($1, $3)], joinType=[left], algorithm=[none],
> cost=[not available])
> HiveJoin(condition=[true], joinType=[inner], algorithm=[none],
> cost=[not available])
> HiveProject(i_item_sk=[$0], i_item_id=[$1])
> HiveTableScan(table=[[default, item]], table:alias=[item])
> HiveProject(c=[$0])
> HiveAggregate(group=[{}], c=[COUNT()])
> HiveFilter(condition=[IN($0, 2:BIGINT, 3:BIGINT)])
> HiveTableScan(table=[[default, item]], table:alias=[item])
> HiveProject(i_item_id=[$0], literalTrue=[true])
> HiveAggregate(group=[{1}])
> HiveFilter(condition=[IN($0, 2:BIGINT, 3:BIGINT)])
> HiveTableScan(table=[[default, item]], table:alias=[item])
> HiveProject(c_customer_sk=[$0])
> HiveTableScan(table=[[default, customer]], table:alias=[customer])
> {code}
> With not
[jira] [Work logged] (HIVE-25867) Partition filter condition should pushed down to metastore query if it is equivalence Predicate
[
https://issues.apache.org/jira/browse/HIVE-25867?focusedWorklogId=745216=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745216
]
ASF GitHub Bot logged work on HIVE-25867:
-
Author: ASF GitHub Bot
Created on: 21/Mar/22 16:27
Start Date: 21/Mar/22 16:27
Worklog Time Spent: 10m
Work Description: ming95 commented on pull request #2947:
URL: https://github.com/apache/hive/pull/2947#issuecomment-1074123353
> 600k partitions are always a pain. We are working on integrating Iceberg
that would help there. I would guess that the solution is this case should be a
configuration to turn off DB level filtering on query level. Maybe we already
have a config for that.
@pvary
I don't quite understand "turn off DB level filtering"
Let me give an example to detail my issue, the hql is :
```
select * from src_play_day WHERE dt=20211125 AND u_gtype='activity_workshop'
limit 10;
```
the `dt` and `u_gtype` are both partition field , **and column type are both
string.**
Before this change , metastore **not push down partition** to HMS DB, it
will get all partitions and prune partition by expression.**Because in
`dt=20211125`, dt on the left is of type string and 20211125 on the right is of
type Long.** Finally,metastore get the required partition information in the
HMS DB.
In MYSQL ,it will run **two sql** :
```
SELECT `A0`.`PART_NAME` AS NUCORDER0
FROM `PARTITIONS` `A0`
LEFT JOIN `TBLS` `B0` ON `A0`.`TBL_ID` = `B0`.`TBL_ID`
LEFT JOIN `DBS` `C0` ON `B0`.`DB_ID` = `C0`.`DB_ID`
WHERE `C0`.`NAME` = 'default'
AND `B0`.`TBL_NAME` = 'src_play_day'
ORDER BY NUCORDER0;
+
SELECT "PARTITIONS"."PART_ID"
FROM "PARTITIONS"
INNER JOIN "TBLS"
ON "PARTITIONS"."TBL_ID" = "TBLS"."TBL_ID"
AND "TBLS"."TBL_NAME" = 'src_play_day'
INNER JOIN "DBS"
ON "TBLS"."DB_ID" = "DBS"."DB_ID"
AND "DBS"."NAME" = 'default'
WHERE "PARTITIONS"."PART_NAME" IN ('dt=20211125/u_gtype=activity_workshop');
```
Get all partitions will easy increase the load on the HMS DB.
After this change, metastore will push down partition to HMS DB, and not get
all partitions.
In MYSQL ,it will run just one sql :
```
SELECT "PARTITIONS"."PART_ID"
FROM "PARTITIONS"
INNER JOIN "TBLS"
ON "PARTITIONS"."TBL_ID" = "TBLS"."TBL_ID"
AND "TBLS"."TBL_NAME" = 'src_play_day'
INNER JOIN "DBS"
ON "TBLS"."DB_ID" = "DBS"."DB_ID"
AND "DBS"."NAME" = 'default'
INNER JOIN "PARTITION_KEY_VALS" "FILTER0"
ON "FILTER0"."PART_ID" = "PARTITIONS"."PART_ID"
AND "FILTER0"."INTEGER_IDX" = 0
INNER JOIN "PARTITION_KEY_VALS" "FILTER1"
ON "FILTER1"."PART_ID" = "PARTITIONS"."PART_ID"
AND "FILTER1"."INTEGER_IDX" = 1
WHERE "FILTER0"."PART_KEY_VAL" = '20211125'
AND "FILTER1"."PART_KEY_VAL" = 'activity_workshop';
```
So this issue is to allow the partition to be pushed down to the HMS DB in
some cases, so as to avoid getting all the partition of the table.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 745216)
Time Spent: 1h 40m (was: 1.5h)
> Partition filter condition should pushed down to metastore query if it is
> equivalence Predicate
> ---
>
> Key: HIVE-25867
> URL: https://issues.apache.org/jira/browse/HIVE-25867
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Reporter: shezm
>Assignee: shezm
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> The colnum type of the partition is different from the column type of the hql
> query, the metastore will not push down the query to the RDBMS, but will
> instead get all PARTITIONS.PART_NAME of the hive table then filter it
> according to the hql Expression.
> https://github.com/apache/hive/blob/5b112aa6dcc4e374c0a7c2b24042f24ae6815da1/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java#L1316
> If the hive table has too many partitions and there are multiple hql queries
> at the same time,RDBMS will increasing CPU IO_WAIT and affect performance.
> If the partition filter condition in hql is an
[jira] [Resolved] (HIVE-26049) Inconsistent TBL_NAME lengths in HMS schema
[ https://issues.apache.org/jira/browse/HIVE-26049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary resolved HIVE-26049. --- Fix Version/s: 4.0.0-alpha-1 Resolution: Fixed Pushed to master. Thanks for the patch [~kovjanos]! > Inconsistent TBL_NAME lengths in HMS schema > > > Key: HIVE-26049 > URL: https://issues.apache.org/jira/browse/HIVE-26049 > Project: Hive > Issue Type: Bug > Components: Database/Schema >Affects Versions: 4.0.0-alpha-1 >Reporter: Janos Kovacs >Assignee: Janos Kovacs >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-1 > > Time Spent: 40m > Remaining Estimate: 0h > > The transaction specific tables have TBL_NAME length set to 128 while the > TBLS itself allows 256. When a table is created with such a long name hive > fails e.g. to lock the table. > TBL_NAME needs to be consistent across the whole schema. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-26049) Inconsistent TBL_NAME lengths in HMS schema
[ https://issues.apache.org/jira/browse/HIVE-26049?focusedWorklogId=745151=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745151 ] ASF GitHub Bot logged work on HIVE-26049: - Author: ASF GitHub Bot Created on: 21/Mar/22 14:54 Start Date: 21/Mar/22 14:54 Worklog Time Spent: 10m Work Description: pvary merged pull request #3119: URL: https://github.com/apache/hive/pull/3119 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 745151) Time Spent: 40m (was: 0.5h) > Inconsistent TBL_NAME lengths in HMS schema > > > Key: HIVE-26049 > URL: https://issues.apache.org/jira/browse/HIVE-26049 > Project: Hive > Issue Type: Bug > Components: Database/Schema >Affects Versions: 4.0.0-alpha-1 >Reporter: Janos Kovacs >Assignee: Janos Kovacs >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > The transaction specific tables have TBL_NAME length set to 128 while the > TBLS itself allows 256. When a table is created with such a long name hive > fails e.g. to lock the table. > TBL_NAME needs to be consistent across the whole schema. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-26044) Remove hardcoded version references from the tests
[
https://issues.apache.org/jira/browse/HIVE-26044?focusedWorklogId=745140=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745140
]
ASF GitHub Bot logged work on HIVE-26044:
-
Author: ASF GitHub Bot
Created on: 21/Mar/22 14:36
Start Date: 21/Mar/22 14:36
Worklog Time Spent: 10m
Work Description: marton-bod commented on a change in pull request #3115:
URL: https://github.com/apache/hive/pull/3115#discussion_r831178770
##
File path:
itests/hive-unit/src/test/java/org/apache/hive/service/server/InformationSchemaWithPrivilegeTestBase.java
##
@@ -287,7 +291,7 @@ public void test() throws Exception {
List args = new ArrayList(baseArgs);
args.add("-f");
-
args.add("../../metastore/scripts/upgrade/hive/hive-schema-4.0.0-alpha-1.hive.sql");
+args.add("../../metastore/scripts/upgrade/hive/hive-schema-" +
hiveSchemaVer + ".hive.sql");
Review comment:
So HS2 version could be like `3.1.3000.2022.0.7.0`, HMS be like
`3.1.3000.7.2.11.0`, and and HMS DB version like `3.1.3000`? And the above 3
classes encode these 3 versions, respectively
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 745140)
Time Spent: 1h 40m (was: 1.5h)
> Remove hardcoded version references from the tests
> --
>
> Key: HIVE-26044
> URL: https://issues.apache.org/jira/browse/HIVE-26044
> Project: Hive
> Issue Type: Test
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> There are several places where there are hardcoded version references in the
> tests.
> We should remove them to so it is easier to change versions.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-26044) Remove hardcoded version references from the tests
[
https://issues.apache.org/jira/browse/HIVE-26044?focusedWorklogId=745139=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745139
]
ASF GitHub Bot logged work on HIVE-26044:
-
Author: ASF GitHub Bot
Created on: 21/Mar/22 14:35
Start Date: 21/Mar/22 14:35
Worklog Time Spent: 10m
Work Description: marton-bod commented on a change in pull request #3115:
URL: https://github.com/apache/hive/pull/3115#discussion_r831178770
##
File path:
itests/hive-unit/src/test/java/org/apache/hive/service/server/InformationSchemaWithPrivilegeTestBase.java
##
@@ -287,7 +291,7 @@ public void test() throws Exception {
List args = new ArrayList(baseArgs);
args.add("-f");
-
args.add("../../metastore/scripts/upgrade/hive/hive-schema-4.0.0-alpha-1.hive.sql");
+args.add("../../metastore/scripts/upgrade/hive/hive-schema-" +
hiveSchemaVer + ".hive.sql");
Review comment:
So HS2 version could be like `3.1.3000.2022.0.7.0`, HMS be like
`3.1.3000.7.2.11.0`, and and HMS DB version like `3.1.3000`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 745139)
Time Spent: 1.5h (was: 1h 20m)
> Remove hardcoded version references from the tests
> --
>
> Key: HIVE-26044
> URL: https://issues.apache.org/jira/browse/HIVE-26044
> Project: Hive
> Issue Type: Test
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> There are several places where there are hardcoded version references in the
> tests.
> We should remove them to so it is easier to change versions.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25867) Partition filter condition should pushed down to metastore query if it is equivalence Predicate
[ https://issues.apache.org/jira/browse/HIVE-25867?focusedWorklogId=745134=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745134 ] ASF GitHub Bot logged work on HIVE-25867: - Author: ASF GitHub Bot Created on: 21/Mar/22 14:29 Start Date: 21/Mar/22 14:29 Worklog Time Spent: 10m Work Description: pvary commented on pull request #2947: URL: https://github.com/apache/hive/pull/2947#issuecomment-1073969722 600k partitions are always a pain. We are working on integrating Iceberg that would help there. I would guess that the solution is this case should be a configuration to turn off DB level filtering on query level. Maybe we already have a config for that. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 745134) Time Spent: 1.5h (was: 1h 20m) > Partition filter condition should pushed down to metastore query if it is > equivalence Predicate > --- > > Key: HIVE-25867 > URL: https://issues.apache.org/jira/browse/HIVE-25867 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: shezm >Assignee: shezm >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > The colnum type of the partition is different from the column type of the hql > query, the metastore will not push down the query to the RDBMS, but will > instead get all PARTITIONS.PART_NAME of the hive table then filter it > according to the hql Expression. > https://github.com/apache/hive/blob/5b112aa6dcc4e374c0a7c2b24042f24ae6815da1/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java#L1316 > If the hive table has too many partitions and there are multiple hql queries > at the same time,RDBMS will increasing CPU IO_WAIT and affect performance. > If the partition filter condition in hql is an equivalent predicate, the > metastore should be pushed down to RDBMS, which can optimize the query > performance of hive large tables. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-26036) NPE caused by getMTable() in ObjectStore
[
https://issues.apache.org/jira/browse/HIVE-26036?focusedWorklogId=745126=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745126
]
ASF GitHub Bot logged work on HIVE-26036:
-
Author: ASF GitHub Bot
Created on: 21/Mar/22 14:23
Start Date: 21/Mar/22 14:23
Worklog Time Spent: 10m
Work Description: marton-bod commented on a change in pull request #3104:
URL: https://github.com/apache/hive/pull/3104#discussion_r831165554
##
File path:
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java
##
@@ -393,6 +393,14 @@ public void testPartitionOps() throws Exception {
try (AutoCloseable c = deadline()) {
objectStore.addPartition(part2);
}
+List value3 = Arrays.asList("US", "MA");
Review comment:
Can you add test cases for the other changed API methods too? Maybe
create a new method for them, e.g. `testPartitionOpsWhenTableDoesNotExist`?
This test method already seems quite messy and cluttered so maybe it's worth
separating it out
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 745126)
Time Spent: 40m (was: 0.5h)
> NPE caused by getMTable() in ObjectStore
>
>
> Key: HIVE-26036
> URL: https://issues.apache.org/jira/browse/HIVE-26036
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.2, 4.0.0
>Reporter: Yu Weiqiang
>Assignee: Yu Weiqiang
>Priority: Minor
> Labels: pull-request-available
> Attachments: NPE_From_getMTable.png
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> *Issue*
> Some api in ObjectStore invoke getMTable() but not check that if the returned
> value is null, which caused the NPE, like addPartitions(), addPartition(),
> alterPartition() et.
> *Reason*
> Such api described above will check that whether the table exists in
> HMSHandler (first check), but if the table is dropped by other threads after
> the first check, the NPE will happen.
> The simple idea is that we can check the table each time we get from
> getMTable().
> *Stack example*
> !NPE_From_getMTable.png!
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25867) Partition filter condition should pushed down to metastore query if it is equivalence Predicate
[ https://issues.apache.org/jira/browse/HIVE-25867?focusedWorklogId=745102=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745102 ] ASF GitHub Bot logged work on HIVE-25867: - Author: ASF GitHub Bot Created on: 21/Mar/22 14:12 Start Date: 21/Mar/22 14:12 Worklog Time Spent: 10m Work Description: ming95 edited a comment on pull request #2947: URL: https://github.com/apache/hive/pull/2947#issuecomment-1073947248 > 60w? - is this 60? That should not cause to big of an issue for any backend db @pvary Sorry , "w" is a unique shorthand, I overlooked that. 60w=60 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 745102) Time Spent: 1h 20m (was: 1h 10m) > Partition filter condition should pushed down to metastore query if it is > equivalence Predicate > --- > > Key: HIVE-25867 > URL: https://issues.apache.org/jira/browse/HIVE-25867 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: shezm >Assignee: shezm >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > The colnum type of the partition is different from the column type of the hql > query, the metastore will not push down the query to the RDBMS, but will > instead get all PARTITIONS.PART_NAME of the hive table then filter it > according to the hql Expression. > https://github.com/apache/hive/blob/5b112aa6dcc4e374c0a7c2b24042f24ae6815da1/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java#L1316 > If the hive table has too many partitions and there are multiple hql queries > at the same time,RDBMS will increasing CPU IO_WAIT and affect performance. > If the partition filter condition in hql is an equivalent predicate, the > metastore should be pushed down to RDBMS, which can optimize the query > performance of hive large tables. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25867) Partition filter condition should pushed down to metastore query if it is equivalence Predicate
[ https://issues.apache.org/jira/browse/HIVE-25867?focusedWorklogId=745099=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745099 ] ASF GitHub Bot logged work on HIVE-25867: - Author: ASF GitHub Bot Created on: 21/Mar/22 14:11 Start Date: 21/Mar/22 14:11 Worklog Time Spent: 10m Work Description: ming95 commented on pull request #2947: URL: https://github.com/apache/hive/pull/2947#issuecomment-1073947248 > 60w? - is this 60? That should not cause to big of an issue for any backend db Sorry , "w" is a unique shorthand, I overlooked that. 60w=60 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 745099) Time Spent: 1h 10m (was: 1h) > Partition filter condition should pushed down to metastore query if it is > equivalence Predicate > --- > > Key: HIVE-25867 > URL: https://issues.apache.org/jira/browse/HIVE-25867 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: shezm >Assignee: shezm >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > The colnum type of the partition is different from the column type of the hql > query, the metastore will not push down the query to the RDBMS, but will > instead get all PARTITIONS.PART_NAME of the hive table then filter it > according to the hql Expression. > https://github.com/apache/hive/blob/5b112aa6dcc4e374c0a7c2b24042f24ae6815da1/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java#L1316 > If the hive table has too many partitions and there are multiple hql queries > at the same time,RDBMS will increasing CPU IO_WAIT and affect performance. > If the partition filter condition in hql is an equivalent predicate, the > metastore should be pushed down to RDBMS, which can optimize the query > performance of hive large tables. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25867) Partition filter condition should pushed down to metastore query if it is equivalence Predicate
[ https://issues.apache.org/jira/browse/HIVE-25867?focusedWorklogId=745092=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745092 ] ASF GitHub Bot logged work on HIVE-25867: - Author: ASF GitHub Bot Created on: 21/Mar/22 14:04 Start Date: 21/Mar/22 14:04 Worklog Time Spent: 10m Work Description: pvary commented on pull request #2947: URL: https://github.com/apache/hive/pull/2947#issuecomment-1073938660 60w? - is this 60? That should not cause to big of an issue for any backend db -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 745092) Time Spent: 1h (was: 50m) > Partition filter condition should pushed down to metastore query if it is > equivalence Predicate > --- > > Key: HIVE-25867 > URL: https://issues.apache.org/jira/browse/HIVE-25867 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: shezm >Assignee: shezm >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1h > Remaining Estimate: 0h > > The colnum type of the partition is different from the column type of the hql > query, the metastore will not push down the query to the RDBMS, but will > instead get all PARTITIONS.PART_NAME of the hive table then filter it > according to the hql Expression. > https://github.com/apache/hive/blob/5b112aa6dcc4e374c0a7c2b24042f24ae6815da1/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java#L1316 > If the hive table has too many partitions and there are multiple hql queries > at the same time,RDBMS will increasing CPU IO_WAIT and affect performance. > If the partition filter condition in hql is an equivalent predicate, the > metastore should be pushed down to RDBMS, which can optimize the query > performance of hive large tables. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-26044) Remove hardcoded version references from the tests
[
https://issues.apache.org/jira/browse/HIVE-26044?focusedWorklogId=745089=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745089
]
ASF GitHub Bot logged work on HIVE-26044:
-
Author: ASF GitHub Bot
Created on: 21/Mar/22 14:01
Start Date: 21/Mar/22 14:01
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #3115:
URL: https://github.com/apache/hive/pull/3115#discussion_r831141105
##
File path:
itests/hive-unit/src/test/java/org/apache/hive/service/server/InformationSchemaWithPrivilegeTestBase.java
##
@@ -287,7 +291,7 @@ public void test() throws Exception {
List args = new ArrayList(baseArgs);
args.add("-f");
-
args.add("../../metastore/scripts/upgrade/hive/hive-schema-4.0.0-alpha-1.hive.sql");
+args.add("../../metastore/scripts/upgrade/hive/hive-schema-" +
hiveSchemaVer + ".hive.sql");
Review comment:
* `HiveVersionInfo` is the code version of the HS2
* `MetastoreVersionInfo` is the code version of the HMS
* `MetaStoreSchemaInfo` is the db schema version under the HMS - some HMS
changes do not require schema changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 745089)
Time Spent: 1h 20m (was: 1h 10m)
> Remove hardcoded version references from the tests
> --
>
> Key: HIVE-26044
> URL: https://issues.apache.org/jira/browse/HIVE-26044
> Project: Hive
> Issue Type: Test
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> There are several places where there are hardcoded version references in the
> tests.
> We should remove them to so it is easier to change versions.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25867) Partition filter condition should pushed down to metastore query if it is equivalence Predicate
[ https://issues.apache.org/jira/browse/HIVE-25867?focusedWorklogId=745031=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745031 ] ASF GitHub Bot logged work on HIVE-25867: - Author: ASF GitHub Bot Created on: 21/Mar/22 12:57 Start Date: 21/Mar/22 12:57 Worklog Time Spent: 10m Work Description: ming95 commented on pull request #2947: URL: https://github.com/apache/hive/pull/2947#issuecomment-1073862137 > With this change I think the filtering on the HMS DB side is effectively turned off. This would cause queries with smaller number of partitions become slow. Is this issue happening when there are too many partitions in the filter? Could we just turn off this filter if the number of partitions are too high? What is the number of partitions in the query when you had experienced problems? thanks @pvary . in our case , We have a table (two partition fields) with about 60w partitions. When multiple sql jobs are executed at the same time, the HMS DB load will increase. > Could we just turn off this filter if the number of partitions are too high? According to my understanding, there should be no way for us to get the number of partitions before sql execution. Do you mean to cache the partitions number information in HMS to optimize ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 745031) Time Spent: 50m (was: 40m) > Partition filter condition should pushed down to metastore query if it is > equivalence Predicate > --- > > Key: HIVE-25867 > URL: https://issues.apache.org/jira/browse/HIVE-25867 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: shezm >Assignee: shezm >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 50m > Remaining Estimate: 0h > > The colnum type of the partition is different from the column type of the hql > query, the metastore will not push down the query to the RDBMS, but will > instead get all PARTITIONS.PART_NAME of the hive table then filter it > according to the hql Expression. > https://github.com/apache/hive/blob/5b112aa6dcc4e374c0a7c2b24042f24ae6815da1/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java#L1316 > If the hive table has too many partitions and there are multiple hql queries > at the same time,RDBMS will increasing CPU IO_WAIT and affect performance. > If the partition filter condition in hql is an equivalent predicate, the > metastore should be pushed down to RDBMS, which can optimize the query > performance of hive large tables. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-26048) Missing quotation mark in findReadyToClean query
[
https://issues.apache.org/jira/browse/HIVE-26048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antal Sinkovits updated HIVE-26048:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Missing quotation mark in findReadyToClean query
>
>
> Key: HIVE-26048
> URL: https://issues.apache.org/jira/browse/HIVE-26048
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0-alpha-1
>Reporter: Antal Sinkovits
>Assignee: Antal Sinkovits
>Priority: Major
> Labels: ACID, pull-request-available
> Fix For: 4.0.0-alpha-1
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Missing quotation mark causes postgres column error
> {code}
> 2022-03-18T00:53:43,314 ERROR [Thread-651] compactor.Cleaner: Caught an
> exception in the main loop of compactor cleaner, MetaException(message:Unable
> to connect to transaction database org.postgresql.util.PSQLException: ERROR:
> column "cq_retry_retention" does not exist
> Position: 485
> at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2433)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2178)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:306)
> at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441)
> at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365)
> at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:307)
> at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:293)
> at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:270)
> at org.postgresql.jdbc.PgStatement.executeQuery(PgStatement.java:224)
> at
> org.apache.hive.com.zaxxer.hikari.pool.ProxyStatement.executeQuery(ProxyStatement.java:108)
> at
> org.apache.hive.com.zaxxer.hikari.pool.HikariProxyStatement.executeQuery(HikariProxyStatement.java)
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findReadyToClean(CompactionTxnHandler.java:374)
> at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.run(Cleaner.java:146)
> )
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findReadyToClean(CompactionTxnHandler.java:397)
> at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.run(Cleaner.java:146)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HIVE-26048) Missing quotation mark in findReadyToClean query
[
https://issues.apache.org/jira/browse/HIVE-26048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17509814#comment-17509814
]
Antal Sinkovits commented on HIVE-26048:
Pushed to master. Thanks for the. review [~klcopp] and [~dkuzmenko].
> Missing quotation mark in findReadyToClean query
>
>
> Key: HIVE-26048
> URL: https://issues.apache.org/jira/browse/HIVE-26048
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0-alpha-1
>Reporter: Antal Sinkovits
>Assignee: Antal Sinkovits
>Priority: Major
> Labels: ACID, pull-request-available
> Fix For: 4.0.0-alpha-1
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Missing quotation mark causes postgres column error
> {code}
> 2022-03-18T00:53:43,314 ERROR [Thread-651] compactor.Cleaner: Caught an
> exception in the main loop of compactor cleaner, MetaException(message:Unable
> to connect to transaction database org.postgresql.util.PSQLException: ERROR:
> column "cq_retry_retention" does not exist
> Position: 485
> at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2433)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2178)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:306)
> at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441)
> at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365)
> at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:307)
> at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:293)
> at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:270)
> at org.postgresql.jdbc.PgStatement.executeQuery(PgStatement.java:224)
> at
> org.apache.hive.com.zaxxer.hikari.pool.ProxyStatement.executeQuery(ProxyStatement.java:108)
> at
> org.apache.hive.com.zaxxer.hikari.pool.HikariProxyStatement.executeQuery(HikariProxyStatement.java)
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findReadyToClean(CompactionTxnHandler.java:374)
> at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.run(Cleaner.java:146)
> )
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findReadyToClean(CompactionTxnHandler.java:397)
> at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.run(Cleaner.java:146)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HIVE-26048) Missing quotation mark in findReadyToClean query
[
https://issues.apache.org/jira/browse/HIVE-26048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antal Sinkovits updated HIVE-26048:
---
Affects Version/s: 4.0.0-alpha-1
(was: 4.0.0)
> Missing quotation mark in findReadyToClean query
>
>
> Key: HIVE-26048
> URL: https://issues.apache.org/jira/browse/HIVE-26048
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0-alpha-1
>Reporter: Antal Sinkovits
>Assignee: Antal Sinkovits
>Priority: Major
> Labels: ACID, pull-request-available
> Fix For: 4.0.0-alpha-1
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Missing quotation mark causes postgres column error
> {code}
> 2022-03-18T00:53:43,314 ERROR [Thread-651] compactor.Cleaner: Caught an
> exception in the main loop of compactor cleaner, MetaException(message:Unable
> to connect to transaction database org.postgresql.util.PSQLException: ERROR:
> column "cq_retry_retention" does not exist
> Position: 485
> at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2433)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2178)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:306)
> at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441)
> at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365)
> at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:307)
> at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:293)
> at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:270)
> at org.postgresql.jdbc.PgStatement.executeQuery(PgStatement.java:224)
> at
> org.apache.hive.com.zaxxer.hikari.pool.ProxyStatement.executeQuery(ProxyStatement.java:108)
> at
> org.apache.hive.com.zaxxer.hikari.pool.HikariProxyStatement.executeQuery(HikariProxyStatement.java)
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findReadyToClean(CompactionTxnHandler.java:374)
> at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.run(Cleaner.java:146)
> )
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findReadyToClean(CompactionTxnHandler.java:397)
> at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.run(Cleaner.java:146)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HIVE-26048) Missing quotation mark in findReadyToClean query
[
https://issues.apache.org/jira/browse/HIVE-26048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Antal Sinkovits updated HIVE-26048:
---
Fix Version/s: 4.0.0-alpha-1
> Missing quotation mark in findReadyToClean query
>
>
> Key: HIVE-26048
> URL: https://issues.apache.org/jira/browse/HIVE-26048
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Antal Sinkovits
>Assignee: Antal Sinkovits
>Priority: Major
> Labels: ACID, pull-request-available
> Fix For: 4.0.0-alpha-1
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Missing quotation mark causes postgres column error
> {code}
> 2022-03-18T00:53:43,314 ERROR [Thread-651] compactor.Cleaner: Caught an
> exception in the main loop of compactor cleaner, MetaException(message:Unable
> to connect to transaction database org.postgresql.util.PSQLException: ERROR:
> column "cq_retry_retention" does not exist
> Position: 485
> at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2433)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2178)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:306)
> at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441)
> at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365)
> at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:307)
> at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:293)
> at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:270)
> at org.postgresql.jdbc.PgStatement.executeQuery(PgStatement.java:224)
> at
> org.apache.hive.com.zaxxer.hikari.pool.ProxyStatement.executeQuery(ProxyStatement.java:108)
> at
> org.apache.hive.com.zaxxer.hikari.pool.HikariProxyStatement.executeQuery(HikariProxyStatement.java)
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findReadyToClean(CompactionTxnHandler.java:374)
> at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.run(Cleaner.java:146)
> )
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findReadyToClean(CompactionTxnHandler.java:397)
> at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.run(Cleaner.java:146)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-26048) Missing quotation mark in findReadyToClean query
[
https://issues.apache.org/jira/browse/HIVE-26048?focusedWorklogId=745008=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-745008
]
ASF GitHub Bot logged work on HIVE-26048:
-
Author: ASF GitHub Bot
Created on: 21/Mar/22 12:20
Start Date: 21/Mar/22 12:20
Worklog Time Spent: 10m
Work Description: asinkovits merged pull request #3118:
URL: https://github.com/apache/hive/pull/3118
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 745008)
Time Spent: 0.5h (was: 20m)
> Missing quotation mark in findReadyToClean query
>
>
> Key: HIVE-26048
> URL: https://issues.apache.org/jira/browse/HIVE-26048
> Project: Hive
> Issue Type: Bug
>Affects Versions: 4.0.0
>Reporter: Antal Sinkovits
>Assignee: Antal Sinkovits
>Priority: Major
> Labels: ACID, pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Missing quotation mark causes postgres column error
> {code}
> 2022-03-18T00:53:43,314 ERROR [Thread-651] compactor.Cleaner: Caught an
> exception in the main loop of compactor cleaner, MetaException(message:Unable
> to connect to transaction database org.postgresql.util.PSQLException: ERROR:
> column "cq_retry_retention" does not exist
> Position: 485
> at
> org.postgresql.core.v3.QueryExecutorImpl.receiveErrorResponse(QueryExecutorImpl.java:2433)
> at
> org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:2178)
> at
> org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:306)
> at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:441)
> at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:365)
> at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:307)
> at org.postgresql.jdbc.PgStatement.executeCachedSql(PgStatement.java:293)
> at org.postgresql.jdbc.PgStatement.executeWithFlags(PgStatement.java:270)
> at org.postgresql.jdbc.PgStatement.executeQuery(PgStatement.java:224)
> at
> org.apache.hive.com.zaxxer.hikari.pool.ProxyStatement.executeQuery(ProxyStatement.java:108)
> at
> org.apache.hive.com.zaxxer.hikari.pool.HikariProxyStatement.executeQuery(HikariProxyStatement.java)
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findReadyToClean(CompactionTxnHandler.java:374)
> at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.run(Cleaner.java:146)
> )
> at
> org.apache.hadoop.hive.metastore.txn.CompactionTxnHandler.findReadyToClean(CompactionTxnHandler.java:397)
> at org.apache.hadoop.hive.ql.txn.compactor.Cleaner.run(Cleaner.java:146)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-26044) Remove hardcoded version references from the tests
[
https://issues.apache.org/jira/browse/HIVE-26044?focusedWorklogId=744996=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-744996
]
ASF GitHub Bot logged work on HIVE-26044:
-
Author: ASF GitHub Bot
Created on: 21/Mar/22 12:01
Start Date: 21/Mar/22 12:01
Worklog Time Spent: 10m
Work Description: zabetak commented on a change in pull request #3115:
URL: https://github.com/apache/hive/pull/3115#discussion_r831028381
##
File path:
itests/hive-unit/src/test/java/org/apache/hive/service/server/InformationSchemaWithPrivilegeTestBase.java
##
@@ -287,7 +291,7 @@ public void test() throws Exception {
List args = new ArrayList(baseArgs);
args.add("-f");
-
args.add("../../metastore/scripts/upgrade/hive/hive-schema-4.0.0-alpha-1.hive.sql");
+args.add("../../metastore/scripts/upgrade/hive/hive-schema-" +
hiveSchemaVer + ".hive.sql");
Review comment:
So far we have:
- `HiveVersionInfo`
- `MetastoreVersionInfo`
- `MetaStoreSchemaInfo`
I understand there is a concern between HS2 and HMS version being different
but what about `MetastoreVersionInfo`
vs `MetaStoreSchemaInfo`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 744996)
Time Spent: 1h (was: 50m)
> Remove hardcoded version references from the tests
> --
>
> Key: HIVE-26044
> URL: https://issues.apache.org/jira/browse/HIVE-26044
> Project: Hive
> Issue Type: Test
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h
> Remaining Estimate: 0h
>
> There are several places where there are hardcoded version references in the
> tests.
> We should remove them to so it is easier to change versions.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-26044) Remove hardcoded version references from the tests
[
https://issues.apache.org/jira/browse/HIVE-26044?focusedWorklogId=744997=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-744997
]
ASF GitHub Bot logged work on HIVE-26044:
-
Author: ASF GitHub Bot
Created on: 21/Mar/22 12:01
Start Date: 21/Mar/22 12:01
Worklog Time Spent: 10m
Work Description: zabetak commented on a change in pull request #3115:
URL: https://github.com/apache/hive/pull/3115#discussion_r831028381
##
File path:
itests/hive-unit/src/test/java/org/apache/hive/service/server/InformationSchemaWithPrivilegeTestBase.java
##
@@ -287,7 +291,7 @@ public void test() throws Exception {
List args = new ArrayList(baseArgs);
args.add("-f");
-
args.add("../../metastore/scripts/upgrade/hive/hive-schema-4.0.0-alpha-1.hive.sql");
+args.add("../../metastore/scripts/upgrade/hive/hive-schema-" +
hiveSchemaVer + ".hive.sql");
Review comment:
So far we have:
- `HiveVersionInfo`
- `MetastoreVersionInfo`
- `MetaStoreSchemaInfo`
I understand there is a concern between HS2 and HMS version being different
but what about `MetastoreVersionInfo`
vs `MetaStoreSchemaInfo`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 744997)
Time Spent: 1h 10m (was: 1h)
> Remove hardcoded version references from the tests
> --
>
> Key: HIVE-26044
> URL: https://issues.apache.org/jira/browse/HIVE-26044
> Project: Hive
> Issue Type: Test
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> There are several places where there are hardcoded version references in the
> tests.
> We should remove them to so it is easier to change versions.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-26043) Use constraint info when creating RexNodes
[
https://issues.apache.org/jira/browse/HIVE-26043?focusedWorklogId=744985=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-744985
]
ASF GitHub Bot logged work on HIVE-26043:
-
Author: ASF GitHub Bot
Created on: 21/Mar/22 11:43
Start Date: 21/Mar/22 11:43
Worklog Time Spent: 10m
Work Description: zabetak commented on pull request #3120:
URL: https://github.com/apache/hive/pull/3120#issuecomment-1073795144
@kasakrisz Have you explored the option of including the nullability
information in `ColumnInfo`? I think that doing this would allow changes to be
centralized in `TypeConverter` and keep the code more uniform.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 744985)
Time Spent: 20m (was: 10m)
> Use constraint info when creating RexNodes
> --
>
> Key: HIVE-26043
> URL: https://issues.apache.org/jira/browse/HIVE-26043
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Prior HIVE-23100 Not null constraints affected newly created RexNode type
> nullability.
> Nullability enables the subquery rewrite algorithm to generate more optimal
> plan.
> [https://github.com/apache/hive/blob/1213ad3f0ae0e21e7519dc28b8b6d1401cdd1441/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSubQueryRemoveRule.java#L324]
> Example:
> {code:java}
> explain cbo
> select ws_sales_price
> from web_sales, customer, item
> where ws_bill_customer_sk = c_customer_sk
> and ws_item_sk = i_item_sk
> and ( c_customer_sk = 1
> or
> i_item_id in (select i_item_id
> from item
> where i_item_sk in (2, 3)
> )
> );
> {code}
> Without not null constraints
> {code:java}
> HiveProject(ws_sales_price=[$2])
> HiveFilter(condition=[OR(AND(<>($6, 0), IS NOT NULL($8)), =($3, 1))])
> HiveProject(ws_item_sk=[$0], ws_bill_customer_sk=[$1],
> ws_sales_price=[$2], c_customer_sk=[$8], i_item_sk=[$3], i_item_id=[$4],
> c=[$5], i_item_id0=[$6], literalTrue=[$7])
> HiveJoin(condition=[=($1, $8)], joinType=[inner], algorithm=[none],
> cost=[not available])
> HiveJoin(condition=[=($0, $3)], joinType=[inner], algorithm=[none],
> cost=[not available])
> HiveProject(ws_item_sk=[$2], ws_bill_customer_sk=[$3],
> ws_sales_price=[$20])
> HiveFilter(condition=[IS NOT NULL($3)])
> HiveTableScan(table=[[default, web_sales]],
> table:alias=[web_sales])
> HiveJoin(condition=[=($1, $3)], joinType=[left], algorithm=[none],
> cost=[not available])
> HiveJoin(condition=[true], joinType=[inner], algorithm=[none],
> cost=[not available])
> HiveProject(i_item_sk=[$0], i_item_id=[$1])
> HiveTableScan(table=[[default, item]], table:alias=[item])
> HiveProject(c=[$0])
> HiveAggregate(group=[{}], c=[COUNT()])
> HiveFilter(condition=[IN($0, 2:BIGINT, 3:BIGINT)])
> HiveTableScan(table=[[default, item]], table:alias=[item])
> HiveProject(i_item_id=[$0], literalTrue=[true])
> HiveAggregate(group=[{1}])
> HiveFilter(condition=[IN($0, 2:BIGINT, 3:BIGINT)])
> HiveTableScan(table=[[default, item]], table:alias=[item])
> HiveProject(c_customer_sk=[$0])
> HiveTableScan(table=[[default, customer]], table:alias=[customer])
> {code}
> With not null constraints
> {code:java}
> HiveProject(ws_sales_price=[$2])
> HiveFilter(condition=[OR(IS NOT NULL($7), =($3, 1))])
> HiveProject(ws_item_sk=[$0], ws_bill_customer_sk=[$1],
> ws_sales_price=[$2], c_customer_sk=[$7], i_item_sk=[$3], i_item_id=[$4],
> i_item_id0=[$5], literalTrue=[$6])
> HiveJoin(condition=[=($1, $7)], joinType=[inner], algorithm=[none],
> cost=[not available])
> HiveJoin(condition=[=($0, $3)], joinType=[inner], algorithm=[none],
> cost=[not available])
> HiveProject(ws_item_sk=[$2], ws_bill_customer_sk=[$3],
> ws_sales_price=[$20])
> HiveFilter(condition=[IS NOT NULL($3)])
> HiveTableScan(table=[[default, web_sales]],
> table:alias=[web_sales])
> HiveJoin(condition=[=($1, $2)],
[jira] [Assigned] (HIVE-26026) Use the new "REFUSED" compaction state where it makes sense
[ https://issues.apache.org/jira/browse/HIVE-26026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] László Végh reassigned HIVE-26026: -- Assignee: László Végh > Use the new "REFUSED" compaction state where it makes sense > --- > > Key: HIVE-26026 > URL: https://issues.apache.org/jira/browse/HIVE-26026 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: László Végh >Assignee: László Végh >Priority: Major > > The > org.apache.hadoop.hive.ql.txn.compactor.Worker#findNextCompactionAndExecute > method does several checks (The table/partition exists, is not sorted, there > are enough files to compact, etc.) before it actually executes the compaction > request. If the compaction request fails on any of these checks, it is put to > "SUCCEEDED" state which is often misleading for users. SHOW COMPACTIONS will > show these requests as succeeded without an error, while the table is not > compacted at all. > For these cases, the state should be "REFUSED" instead of "SUCCEEDED" among > with the appropriate error message. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work started] (HIVE-26026) Use the new "REFUSED" compaction state where it makes sense
[ https://issues.apache.org/jira/browse/HIVE-26026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-26026 started by László Végh. -- > Use the new "REFUSED" compaction state where it makes sense > --- > > Key: HIVE-26026 > URL: https://issues.apache.org/jira/browse/HIVE-26026 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: László Végh >Assignee: László Végh >Priority: Major > > The > org.apache.hadoop.hive.ql.txn.compactor.Worker#findNextCompactionAndExecute > method does several checks (The table/partition exists, is not sorted, there > are enough files to compact, etc.) before it actually executes the compaction > request. If the compaction request fails on any of these checks, it is put to > "SUCCEEDED" state which is often misleading for users. SHOW COMPACTIONS will > show these requests as succeeded without an error, while the table is not > compacted at all. > For these cases, the state should be "REFUSED" instead of "SUCCEEDED" among > with the appropriate error message. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HIVE-25540) Enable batch update of column stats only for MySql and Postgres
[ https://issues.apache.org/jira/browse/HIVE-25540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17509772#comment-17509772 ] Stamatis Zampetakis commented on HIVE-25540: [~pvary] The changes in HIVE-26040 do seem reasonable for the problem I discovered, many thanks for the quick fix. However, I found this issue just by running one random qtest on a metastore using mssql so I cannot say with confidence that now we have sufficient test coverage for claiming that this feature (HIVE-25181) is production ready in *all* databases; I let [~maheshk114] answer this question. > Enable batch update of column stats only for MySql and Postgres > > > Key: HIVE-25540 > URL: https://issues.apache.org/jira/browse/HIVE-25540 > Project: Hive > Issue Type: Sub-task >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-1 > > Time Spent: 50m > Remaining Estimate: 0h > > The batch updation of partition column stats using direct sql is tested only > for MySql and Postgres. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-26052) Non blocking DROP MATERIALIZED VIEW implementation
[
https://issues.apache.org/jira/browse/HIVE-26052?focusedWorklogId=744951=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-744951
]
ASF GitHub Bot logged work on HIVE-26052:
-
Author: ASF GitHub Bot
Created on: 21/Mar/22 10:39
Start Date: 21/Mar/22 10:39
Worklog Time Spent: 10m
Work Description: kasakrisz commented on a change in pull request #3111:
URL: https://github.com/apache/hive/pull/3111#discussion_r830958725
##
File path: ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands.java
##
@@ -1768,4 +1768,97 @@ public void testDropTableWithoutSuffix() throws
Exception {
Assert.assertEquals("Unexpected number of compactions in history", 0,
resp.getCompactsSize());
}
}
+
+ @Test
+ public void testDropMaterializedViewWithSuffix() throws Exception {
+String tableName = "tab_acid";
+String mviewName = "mv_" + tableName;
+runStatementOnDriver("drop materialized view if exists " + mviewName);
+runStatementOnDriver("drop table if exists " + tableName);
+HiveConf.setBoolVar(hiveConf,
HiveConf.ConfVars.HIVE_ACID_CREATE_TABLE_USE_SUFFIX, true);
+
+runStatementOnDriver("create table " + tableName + "(a int, b int) stored
as orc TBLPROPERTIES ('transactional'='true')");
+runStatementOnDriver("insert into " + tableName + " values(1,2),(3,4)");
+runStatementOnDriver("create materialized view " + mviewName + " stored as
orc TBLPROPERTIES ('transactional'='true') " +
+ "as select a from tab_acid where b > 1");
+runStatementOnDriver("drop materialized view " + mviewName);
+
+int count = TestTxnDbUtil.countQueryAgent(hiveConf,
+ "select count(*) from TXN_TO_WRITE_ID where T2W_TABLE = '" + mviewName +
"'");
+Assert.assertEquals(1, count);
+
+FileSystem fs = FileSystem.get(hiveConf);
+FileStatus[] stat = fs.listStatus(new Path(getWarehouseDir()),
+ t -> t.getName().matches(mviewName + SOFT_DELETE_TABLE_PATTERN));
+if (1 != stat.length) {
+ Assert.fail("Materialized view data was removed from FS");
+}
+MetastoreTaskThread houseKeeperService = new AcidHouseKeeperService();
+houseKeeperService.setConf(hiveConf);
+
+houseKeeperService.run();
+count = TestTxnDbUtil.countQueryAgent(hiveConf,
+ "select count(*) from TXN_TO_WRITE_ID where T2W_TABLE = '" + mviewName +
"'");
+Assert.assertEquals(0, count);
+
+try {
+ runStatementOnDriver("select * from " + mviewName);
+} catch (Exception ex) {
+ Assert.assertTrue(ex.getMessage().contains(
+ErrorMsg.INVALID_TABLE.getMsg(StringUtils.wrap(mviewName, "'";
+}
+// Check status of compaction job
+TxnStore txnHandler = TxnUtils.getTxnStore(hiveConf);
+ShowCompactResponse resp = txnHandler.showCompact(new
ShowCompactRequest());
+
+Assert.assertEquals("Unexpected number of compactions in history", 1,
resp.getCompactsSize());
+Assert.assertEquals("Unexpected 0 compaction state",
+ TxnStore.CLEANING_RESPONSE, resp.getCompacts().get(0).getState());
+
+runCleaner(hiveConf);
+
+FileStatus[] status = fs.listStatus(new Path(getWarehouseDir()),
+ t -> t.getName().matches(mviewName + SOFT_DELETE_TABLE_PATTERN));
+Assert.assertEquals(0, status.length);
+ }
+
+ @Test
+ public void testDropMaterializedViewWithoutSuffix() throws Exception {
+String tableName = "tab_acid";
+String mviewName = "mv_" + tableName;
+runStatementOnDriver("drop materialized view if exists " + mviewName);
+
+for (boolean enabled : Arrays.asList( true)) {
Review comment:
Maybe I'm missing something bit it seems that it checks only one of the
combinations
```
Arrays.asList( true)
```
To check both combinations I would expect
```
Arrays.asList(true, false)
```
Personally I would create 2 separate tests named exactly what the test does
and a method with the common parts instead of the for loop. Like you did later:
```
@Test
public void testDropMaterializedViewNonBlocking() throws Exception {
testDropMaterializedView(false);
}
@Test
public void testDropMaterializedViewBlocking() throws Exception {
testDropMaterializedView(true);
}
private void testDropMaterializedView(boolean blocking) throws Exception {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 744951)
Time Spent: 40m (was: 0.5h)
> Non blocking DROP MATERIALIZED VIEW implementation
> --
>
> Key: HIVE-26052
>
[jira] [Updated] (HIVE-26016) Remove duplicate table exists check in create_table_core api of HMSHandler
[ https://issues.apache.org/jira/browse/HIVE-26016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-26016: -- Fix Version/s: 4.0.0-alpha-1 Status: Patch Available (was: Open) Pushed to master. Thanks for your patch [~wechar] and [~rbalamohan] for the review! > Remove duplicate table exists check in create_table_core api of HMSHandler > -- > > Key: HIVE-26016 > URL: https://issues.apache.org/jira/browse/HIVE-26016 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Yu Weiqiang >Assignee: Yu Weiqiang >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0-alpha-1 > > Time Spent: 50m > Remaining Estimate: 0h > > Currently create_table_core() will check if table exists before opening > transaction, it does not need to check again after opening transaction, can > remove the unnecessary check code. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-26016) Remove duplicate table exists check in create_table_core api of HMSHandler
[ https://issues.apache.org/jira/browse/HIVE-26016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Vary updated HIVE-26016: -- Resolution: Fixed Status: Resolved (was: Patch Available) > Remove duplicate table exists check in create_table_core api of HMSHandler > -- > > Key: HIVE-26016 > URL: https://issues.apache.org/jira/browse/HIVE-26016 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Yu Weiqiang >Assignee: Yu Weiqiang >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0-alpha-1 > > Time Spent: 50m > Remaining Estimate: 0h > > Currently create_table_core() will check if table exists before opening > transaction, it does not need to check again after opening transaction, can > remove the unnecessary check code. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-26016) Remove duplicate table exists check in create_table_core api of HMSHandler
[ https://issues.apache.org/jira/browse/HIVE-26016?focusedWorklogId=744939=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-744939 ] ASF GitHub Bot logged work on HIVE-26016: - Author: ASF GitHub Bot Created on: 21/Mar/22 10:18 Start Date: 21/Mar/22 10:18 Worklog Time Spent: 10m Work Description: pvary merged pull request #3085: URL: https://github.com/apache/hive/pull/3085 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 744939) Time Spent: 50m (was: 40m) > Remove duplicate table exists check in create_table_core api of HMSHandler > -- > > Key: HIVE-26016 > URL: https://issues.apache.org/jira/browse/HIVE-26016 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Yu Weiqiang >Assignee: Yu Weiqiang >Priority: Minor > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > Currently create_table_core() will check if table exists before opening > transaction, it does not need to check again after opening transaction, can > remove the unnecessary check code. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25867) Partition filter condition should pushed down to metastore query if it is equivalence Predicate
[ https://issues.apache.org/jira/browse/HIVE-25867?focusedWorklogId=744938=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-744938 ] ASF GitHub Bot logged work on HIVE-25867: - Author: ASF GitHub Bot Created on: 21/Mar/22 10:17 Start Date: 21/Mar/22 10:17 Worklog Time Spent: 10m Work Description: pvary commented on pull request #2947: URL: https://github.com/apache/hive/pull/2947#issuecomment-1073720618 With this change I think the filtering on the HMS DB side is effectively turned off. This would cause queries with smaller number of partitions become slow. Is this issue happening when there are too many partitions in the filter? Could we just turn off this filter if the number of partitions are too high? What is the number of partitions in the query when you had experienced problems? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 744938) Time Spent: 40m (was: 0.5h) > Partition filter condition should pushed down to metastore query if it is > equivalence Predicate > --- > > Key: HIVE-25867 > URL: https://issues.apache.org/jira/browse/HIVE-25867 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Reporter: shezm >Assignee: shezm >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 40m > Remaining Estimate: 0h > > The colnum type of the partition is different from the column type of the hql > query, the metastore will not push down the query to the RDBMS, but will > instead get all PARTITIONS.PART_NAME of the hive table then filter it > according to the hql Expression. > https://github.com/apache/hive/blob/5b112aa6dcc4e374c0a7c2b24042f24ae6815da1/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java#L1316 > If the hive table has too many partitions and there are multiple hql queries > at the same time,RDBMS will increasing CPU IO_WAIT and affect performance. > If the partition filter condition in hql is an equivalent predicate, the > metastore should be pushed down to RDBMS, which can optimize the query > performance of hive large tables. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-26044) Remove hardcoded version references from the tests
[
https://issues.apache.org/jira/browse/HIVE-26044?focusedWorklogId=744935=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-744935
]
ASF GitHub Bot logged work on HIVE-26044:
-
Author: ASF GitHub Bot
Created on: 21/Mar/22 10:11
Start Date: 21/Mar/22 10:11
Worklog Time Spent: 10m
Work Description: marton-bod commented on a change in pull request #3115:
URL: https://github.com/apache/hive/pull/3115#discussion_r830935136
##
File path:
itests/hive-unit/src/test/java/org/apache/hive/service/server/InformationSchemaWithPrivilegeTestBase.java
##
@@ -287,7 +291,7 @@ public void test() throws Exception {
List args = new ArrayList(baseArgs);
args.add("-f");
-
args.add("../../metastore/scripts/upgrade/hive/hive-schema-4.0.0-alpha-1.hive.sql");
+args.add("../../metastore/scripts/upgrade/hive/hive-schema-" +
hiveSchemaVer + ".hive.sql");
Review comment:
Oh I see. Then I think you did it correctly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 744935)
Time Spent: 50m (was: 40m)
> Remove hardcoded version references from the tests
> --
>
> Key: HIVE-26044
> URL: https://issues.apache.org/jira/browse/HIVE-26044
> Project: Hive
> Issue Type: Test
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> There are several places where there are hardcoded version references in the
> tests.
> We should remove them to so it is easier to change versions.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-26044) Remove hardcoded version references from the tests
[
https://issues.apache.org/jira/browse/HIVE-26044?focusedWorklogId=744932=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-744932
]
ASF GitHub Bot logged work on HIVE-26044:
-
Author: ASF GitHub Bot
Created on: 21/Mar/22 10:08
Start Date: 21/Mar/22 10:08
Worklog Time Spent: 10m
Work Description: pvary commented on a change in pull request #3115:
URL: https://github.com/apache/hive/pull/3115#discussion_r830932705
##
File path:
itests/hive-unit/src/test/java/org/apache/hive/service/server/InformationSchemaWithPrivilegeTestBase.java
##
@@ -287,7 +291,7 @@ public void test() throws Exception {
List args = new ArrayList(baseArgs);
args.add("-f");
-
args.add("../../metastore/scripts/upgrade/hive/hive-schema-4.0.0-alpha-1.hive.sql");
+args.add("../../metastore/scripts/upgrade/hive/hive-schema-" +
hiveSchemaVer + ".hive.sql");
Review comment:
I think that the `HiveVersionInfo.getShortVersion()` and the
`MetaStoreSchemaInfoFactory.get(miniHS2.getServerConf()).getHiveSchemaVersion()`
might and should be different in several deployments. Some random vendor ()
might release different code version (hotfixes, maintenance releases) for the
same HMS DB version.
I think this is why `MetaStoreSchemaInfoFactory` has been created, and I
tried to use the appropriate version in the appropriate place.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 744932)
Time Spent: 40m (was: 0.5h)
> Remove hardcoded version references from the tests
> --
>
> Key: HIVE-26044
> URL: https://issues.apache.org/jira/browse/HIVE-26044
> Project: Hive
> Issue Type: Test
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> There are several places where there are hardcoded version references in the
> tests.
> We should remove them to so it is easier to change versions.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-26052) Non blocking DROP MATERIALIZED VIEW implementation
[
https://issues.apache.org/jira/browse/HIVE-26052?focusedWorklogId=744907=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-744907
]
ASF GitHub Bot logged work on HIVE-26052:
-
Author: ASF GitHub Bot
Created on: 21/Mar/22 09:19
Start Date: 21/Mar/22 09:19
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on a change in pull request #3111:
URL: https://github.com/apache/hive/pull/3111#discussion_r830891092
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/ddl/view/materialized/drop/DropMaterializedViewAnalyzer.java
##
@@ -50,7 +54,14 @@ public void analyzeInternal(ASTNode root) throws
SemanticException {
Table materializedView = getTable(viewName, throwException);
if (materializedView != null) {
inputs.add(new ReadEntity(materializedView));
- outputs.add(new WriteEntity(materializedView,
WriteEntity.WriteType.DDL_EXCLUSIVE));
+
+ boolean tableWithSuffix = (HiveConf.getBoolVar(conf,
ConfVars.HIVE_ACID_CREATE_TABLE_USE_SUFFIX)
+ || HiveConf.getBoolVar(conf,
ConfVars.HIVE_ACID_LOCKLESS_READS_ENABLED))
+&& AcidUtils.isTransactionalTable(materializedView)
+&&
Boolean.parseBoolean(materializedView.getProperty(SOFT_DELETE_TABLE));
+
+ outputs.add(new WriteEntity(materializedView,
+tableWithSuffix ? WriteType.DDL_EXCL_WRITE : WriteType.DDL_EXCLUSIVE));
Review comment:
refactored
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 744907)
Time Spent: 20m (was: 10m)
> Non blocking DROP MATERIALIZED VIEW implementation
> --
>
> Key: HIVE-26052
> URL: https://issues.apache.org/jira/browse/HIVE-26052
> Project: Hive
> Issue Type: Task
>Reporter: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-26052) Non blocking DROP MATERIALIZED VIEW implementation
[
https://issues.apache.org/jira/browse/HIVE-26052?focusedWorklogId=744908=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-744908
]
ASF GitHub Bot logged work on HIVE-26052:
-
Author: ASF GitHub Bot
Created on: 21/Mar/22 09:19
Start Date: 21/Mar/22 09:19
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on a change in pull request #3111:
URL: https://github.com/apache/hive/pull/3111#discussion_r830880753
##
File path: ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands.java
##
@@ -1768,4 +1768,97 @@ public void testDropTableWithoutSuffix() throws
Exception {
Assert.assertEquals("Unexpected number of compactions in history", 0,
resp.getCompactsSize());
}
}
+
+ @Test
+ public void testDropMaterializedViewWithSuffix() throws Exception {
+String tableName = "tab_acid";
+String mviewName = "mv_" + tableName;
+runStatementOnDriver("drop materialized view if exists " + mviewName);
+runStatementOnDriver("drop table if exists " + tableName);
+HiveConf.setBoolVar(hiveConf,
HiveConf.ConfVars.HIVE_ACID_CREATE_TABLE_USE_SUFFIX, true);
+
+runStatementOnDriver("create table " + tableName + "(a int, b int) stored
as orc TBLPROPERTIES ('transactional'='true')");
+runStatementOnDriver("insert into " + tableName + " values(1,2),(3,4)");
+runStatementOnDriver("create materialized view " + mviewName + " stored as
orc TBLPROPERTIES ('transactional'='true') " +
+ "as select a from tab_acid where b > 1");
+runStatementOnDriver("drop materialized view " + mviewName);
+
+int count = TestTxnDbUtil.countQueryAgent(hiveConf,
+ "select count(*) from TXN_TO_WRITE_ID where T2W_TABLE = '" + mviewName +
"'");
+Assert.assertEquals(1, count);
+
+FileSystem fs = FileSystem.get(hiveConf);
+FileStatus[] stat = fs.listStatus(new Path(getWarehouseDir()),
+ t -> t.getName().matches(mviewName + SOFT_DELETE_TABLE_PATTERN));
+if (1 != stat.length) {
+ Assert.fail("Materialized view data was removed from FS");
+}
+MetastoreTaskThread houseKeeperService = new AcidHouseKeeperService();
+houseKeeperService.setConf(hiveConf);
+
+houseKeeperService.run();
+count = TestTxnDbUtil.countQueryAgent(hiveConf,
+ "select count(*) from TXN_TO_WRITE_ID where T2W_TABLE = '" + mviewName +
"'");
+Assert.assertEquals(0, count);
+
+try {
+ runStatementOnDriver("select * from " + mviewName);
+} catch (Exception ex) {
+ Assert.assertTrue(ex.getMessage().contains(
+ErrorMsg.INVALID_TABLE.getMsg(StringUtils.wrap(mviewName, "'";
+}
+// Check status of compaction job
+TxnStore txnHandler = TxnUtils.getTxnStore(hiveConf);
+ShowCompactResponse resp = txnHandler.showCompact(new
ShowCompactRequest());
+
+Assert.assertEquals("Unexpected number of compactions in history", 1,
resp.getCompactsSize());
+Assert.assertEquals("Unexpected 0 compaction state",
+ TxnStore.CLEANING_RESPONSE, resp.getCompacts().get(0).getState());
+
+runCleaner(hiveConf);
+
+FileStatus[] status = fs.listStatus(new Path(getWarehouseDir()),
+ t -> t.getName().matches(mviewName + SOFT_DELETE_TABLE_PATTERN));
+Assert.assertEquals(0, status.length);
+ }
+
+ @Test
+ public void testDropMaterializedViewWithoutSuffix() throws Exception {
+String tableName = "tab_acid";
+String mviewName = "mv_" + tableName;
+runStatementOnDriver("drop materialized view if exists " + mviewName);
+
+for (boolean enabled : Arrays.asList( true)) {
Review comment:
checks 2 combinations when MV was created with flag on/off and drop was
called with opposite values
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 744908)
Time Spent: 0.5h (was: 20m)
> Non blocking DROP MATERIALIZED VIEW implementation
> --
>
> Key: HIVE-26052
> URL: https://issues.apache.org/jira/browse/HIVE-26052
> Project: Hive
> Issue Type: Task
>Reporter: Denys Kuzmenko
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (HIVE-26052) Non blocking DROP MATERIALIZED VIEW implementation
[ https://issues.apache.org/jira/browse/HIVE-26052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-26052: -- Labels: pull-request-available (was: ) > Non blocking DROP MATERIALIZED VIEW implementation > -- > > Key: HIVE-26052 > URL: https://issues.apache.org/jira/browse/HIVE-26052 > Project: Hive > Issue Type: Task >Reporter: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-26052) Non blocking DROP MATERIALIZED VIEW implementation
[
https://issues.apache.org/jira/browse/HIVE-26052?focusedWorklogId=744892=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-744892
]
ASF GitHub Bot logged work on HIVE-26052:
-
Author: ASF GitHub Bot
Created on: 21/Mar/22 09:06
Start Date: 21/Mar/22 09:06
Worklog Time Spent: 10m
Work Description: deniskuzZ commented on a change in pull request #3111:
URL: https://github.com/apache/hive/pull/3111#discussion_r830880753
##
File path: ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands.java
##
@@ -1768,4 +1768,97 @@ public void testDropTableWithoutSuffix() throws
Exception {
Assert.assertEquals("Unexpected number of compactions in history", 0,
resp.getCompactsSize());
}
}
+
+ @Test
+ public void testDropMaterializedViewWithSuffix() throws Exception {
+String tableName = "tab_acid";
+String mviewName = "mv_" + tableName;
+runStatementOnDriver("drop materialized view if exists " + mviewName);
+runStatementOnDriver("drop table if exists " + tableName);
+HiveConf.setBoolVar(hiveConf,
HiveConf.ConfVars.HIVE_ACID_CREATE_TABLE_USE_SUFFIX, true);
+
+runStatementOnDriver("create table " + tableName + "(a int, b int) stored
as orc TBLPROPERTIES ('transactional'='true')");
+runStatementOnDriver("insert into " + tableName + " values(1,2),(3,4)");
+runStatementOnDriver("create materialized view " + mviewName + " stored as
orc TBLPROPERTIES ('transactional'='true') " +
+ "as select a from tab_acid where b > 1");
+runStatementOnDriver("drop materialized view " + mviewName);
+
+int count = TestTxnDbUtil.countQueryAgent(hiveConf,
+ "select count(*) from TXN_TO_WRITE_ID where T2W_TABLE = '" + mviewName +
"'");
+Assert.assertEquals(1, count);
+
+FileSystem fs = FileSystem.get(hiveConf);
+FileStatus[] stat = fs.listStatus(new Path(getWarehouseDir()),
+ t -> t.getName().matches(mviewName + SOFT_DELETE_TABLE_PATTERN));
+if (1 != stat.length) {
+ Assert.fail("Materialized view data was removed from FS");
+}
+MetastoreTaskThread houseKeeperService = new AcidHouseKeeperService();
+houseKeeperService.setConf(hiveConf);
+
+houseKeeperService.run();
+count = TestTxnDbUtil.countQueryAgent(hiveConf,
+ "select count(*) from TXN_TO_WRITE_ID where T2W_TABLE = '" + mviewName +
"'");
+Assert.assertEquals(0, count);
+
+try {
+ runStatementOnDriver("select * from " + mviewName);
+} catch (Exception ex) {
+ Assert.assertTrue(ex.getMessage().contains(
+ErrorMsg.INVALID_TABLE.getMsg(StringUtils.wrap(mviewName, "'";
+}
+// Check status of compaction job
+TxnStore txnHandler = TxnUtils.getTxnStore(hiveConf);
+ShowCompactResponse resp = txnHandler.showCompact(new
ShowCompactRequest());
+
+Assert.assertEquals("Unexpected number of compactions in history", 1,
resp.getCompactsSize());
+Assert.assertEquals("Unexpected 0 compaction state",
+ TxnStore.CLEANING_RESPONSE, resp.getCompacts().get(0).getState());
+
+runCleaner(hiveConf);
+
+FileStatus[] status = fs.listStatus(new Path(getWarehouseDir()),
+ t -> t.getName().matches(mviewName + SOFT_DELETE_TABLE_PATTERN));
+Assert.assertEquals(0, status.length);
+ }
+
+ @Test
+ public void testDropMaterializedViewWithoutSuffix() throws Exception {
+String tableName = "tab_acid";
+String mviewName = "mv_" + tableName;
+runStatementOnDriver("drop materialized view if exists " + mviewName);
+
+for (boolean enabled : Arrays.asList( true)) {
Review comment:
check 2 combinations when MV was created with flag on/off and drop was
called with opposite values
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 744892)
Remaining Estimate: 0h
Time Spent: 10m
> Non blocking DROP MATERIALIZED VIEW implementation
> --
>
> Key: HIVE-26052
> URL: https://issues.apache.org/jira/browse/HIVE-26052
> Project: Hive
> Issue Type: Task
>Reporter: Denys Kuzmenko
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Work logged] (HIVE-25446) Wrong execption thrown if capacity<=0
[ https://issues.apache.org/jira/browse/HIVE-25446?focusedWorklogId=744891=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-744891 ] ASF GitHub Bot logged work on HIVE-25446: - Author: ASF GitHub Bot Created on: 21/Mar/22 09:05 Start Date: 21/Mar/22 09:05 Worklog Time Spent: 10m Work Description: ashish-kumar-sharma commented on pull request #3092: URL: https://github.com/apache/hive/pull/3092#issuecomment-1073656047 @sankarh @adesh-rao @zabetak could you guys please review the PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 744891) Time Spent: 20m (was: 10m) > Wrong execption thrown if capacity<=0 > - > > Key: HIVE-25446 > URL: https://issues.apache.org/jira/browse/HIVE-25446 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.1.1 >Reporter: Matt McCline >Assignee: Ashish Sharma >Priority: Trivial > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Encountered this in a very large query: > Caused by: java.lang.AssertionError: Capacity must be a power of two > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTable.validateCapacity(VectorMapJoinFastHashTable.java:60) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTable.(VectorMapJoinFastHashTable.java:77) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastBytesHashTable.(VectorMapJoinFastBytesHashTable.java:132) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastBytesHashMap.(VectorMapJoinFastBytesHashMap.java:166) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastStringHashMap.(VectorMapJoinFastStringHashMap.java:43) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastTableContainer.createHashTable(VectorMapJoinFastTableContainer.java:137) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastTableContainer.(VectorMapJoinFastTableContainer.java:86) > at > org.apache.hadoop.hive.ql.exec.vector.mapjoin.fast.VectorMapJoinFastHashTableLoader.load(VectorMapJoinFastHashTableLoader.java:122) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTableInternal(MapJoinOperator.java:344) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:413) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.lambda$initializeOp$0(MapJoinOperator.java:215) > at > org.apache.hadoop.hive.ql.exec.tez.ObjectCache.retrieve(ObjectCache.java:96) > at > org.apache.hadoop.hive.ql.exec.tez.ObjectCache$1.call(ObjectCache.java:113) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-26053) Non blocking DROP VIEW
[ https://issues.apache.org/jira/browse/HIVE-26053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-26053: -- Description: DROP VIEW shouldn't acquire any locks > Non blocking DROP VIEW > -- > > Key: HIVE-26053 > URL: https://issues.apache.org/jira/browse/HIVE-26053 > Project: Hive > Issue Type: Task >Reporter: Denys Kuzmenko >Priority: Major > > DROP VIEW shouldn't acquire any locks -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-26051) Non blocking ALTER MATERIALIZED VIEW REWRITE
[ https://issues.apache.org/jira/browse/HIVE-26051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-26051: -- Summary: Non blocking ALTER MATERIALIZED VIEW REWRITE (was: Non blocking ALTER MATERIALIZED VIEW REWRITE implementation) > Non blocking ALTER MATERIALIZED VIEW REWRITE > > > Key: HIVE-26051 > URL: https://issues.apache.org/jira/browse/HIVE-26051 > Project: Hive > Issue Type: Task >Reporter: Denys Kuzmenko >Priority: Major > > AlterMaterializedViewRewriteAnalyzer does not change the view content It just > controls whether the view can be used in automatic query plan rewrite. No > need for an EXCLUSIVE lock here. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-26044) Remove hardcoded version references from the tests
[
https://issues.apache.org/jira/browse/HIVE-26044?focusedWorklogId=744887=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-744887
]
ASF GitHub Bot logged work on HIVE-26044:
-
Author: ASF GitHub Bot
Created on: 21/Mar/22 08:49
Start Date: 21/Mar/22 08:49
Worklog Time Spent: 10m
Work Description: marton-bod commented on a change in pull request #3115:
URL: https://github.com/apache/hive/pull/3115#discussion_r830867915
##
File path:
itests/hive-unit/src/test/java/org/apache/hive/service/server/InformationSchemaWithPrivilegeTestBase.java
##
@@ -287,7 +291,7 @@ public void test() throws Exception {
List args = new ArrayList(baseArgs);
args.add("-f");
-
args.add("../../metastore/scripts/upgrade/hive/hive-schema-4.0.0-alpha-1.hive.sql");
+args.add("../../metastore/scripts/upgrade/hive/hive-schema-" +
hiveSchemaVer + ".hive.sql");
Review comment:
Can we use `HiveVersionInfo.getShortVersion()` in these places too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 744887)
Time Spent: 0.5h (was: 20m)
> Remove hardcoded version references from the tests
> --
>
> Key: HIVE-26044
> URL: https://issues.apache.org/jira/browse/HIVE-26044
> Project: Hive
> Issue Type: Test
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> There are several places where there are hardcoded version references in the
> tests.
> We should remove them to so it is easier to change versions.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (HIVE-26042) Fix flaky streaming tests
[
https://issues.apache.org/jira/browse/HIVE-26042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary resolved HIVE-26042.
---
Fix Version/s: 4.0.0-alpha-1
Resolution: Fixed
Pushed to master.
Thanks for the review [~Marton Bod]
> Fix flaky streaming tests
> -
>
> Key: HIVE-26042
> URL: https://issues.apache.org/jira/browse/HIVE-26042
> Project: Hive
> Issue Type: Test
>Reporter: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0-alpha-1
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The TestStreamingDynamicPartitioning / TestStreaming tests are often failing
> because of problems with creating directories.
> Example:
> {code}
> 2022-03-17T04:03:39,024 ERROR [main] metastore.RetryingHMSHandler:
> MetaException(message:Unable to create database managed directory
> pfile:/home/jenkins/agent/workspace/hive-precommit_PR-3081/streaming/target/warehouse/testing5.db,
> failed to create database testing5)
> at
> org.apache.hadoop.hive.metastore.HMSHandler.create_database_core(HMSHandler.java:1269)
> at
> org.apache.hadoop.hive.metastore.HMSHandler.create_database(HMSHandler.java:1389)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:146)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> at com.sun.proxy.$Proxy38.create_database(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createDatabase(HiveMetaStoreClient.java:1144)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:218)
> at com.sun.proxy.$Proxy47.createDatabase(Unknown Source)
> at org.apache.hadoop.hive.ql.metadata.Hive.createDatabase(Hive.java:608)
> at
> org.apache.hadoop.hive.ql.ddl.database.create.CreateDatabaseOperation.execute(CreateDatabaseOperation.java:68)
> at org.apache.hadoop.hive.ql.ddl.DDLTask.execute(DDLTask.java:84)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:212)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
> at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:361)
> at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:334)
> at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:245)
> at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:106)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:348)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:204)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:153)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:148)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:185)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:228)
> at
> org.apache.hive.streaming.TestStreaming.runDDL(TestStreaming.java:3202)
> at
> org.apache.hive.streaming.TestStreaming.createStoreSales(TestStreaming.java:267)
> at org.apache.hive.streaming.TestStreaming.setup(TestStreaming.java:250)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.RunBefores.invokeMethod(RunBefores.java:33)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
[jira] [Work logged] (HIVE-26042) Fix flaky streaming tests
[
https://issues.apache.org/jira/browse/HIVE-26042?focusedWorklogId=744880=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-744880
]
ASF GitHub Bot logged work on HIVE-26042:
-
Author: ASF GitHub Bot
Created on: 21/Mar/22 08:37
Start Date: 21/Mar/22 08:37
Worklog Time Spent: 10m
Work Description: pvary merged pull request #3114:
URL: https://github.com/apache/hive/pull/3114
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 744880)
Time Spent: 0.5h (was: 20m)
> Fix flaky streaming tests
> -
>
> Key: HIVE-26042
> URL: https://issues.apache.org/jira/browse/HIVE-26042
> Project: Hive
> Issue Type: Test
>Reporter: Peter Vary
>Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The TestStreamingDynamicPartitioning / TestStreaming tests are often failing
> because of problems with creating directories.
> Example:
> {code}
> 2022-03-17T04:03:39,024 ERROR [main] metastore.RetryingHMSHandler:
> MetaException(message:Unable to create database managed directory
> pfile:/home/jenkins/agent/workspace/hive-precommit_PR-3081/streaming/target/warehouse/testing5.db,
> failed to create database testing5)
> at
> org.apache.hadoop.hive.metastore.HMSHandler.create_database_core(HMSHandler.java:1269)
> at
> org.apache.hadoop.hive.metastore.HMSHandler.create_database(HMSHandler.java:1389)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:146)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> at com.sun.proxy.$Proxy38.create_database(Unknown Source)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.createDatabase(HiveMetaStoreClient.java:1144)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:218)
> at com.sun.proxy.$Proxy47.createDatabase(Unknown Source)
> at org.apache.hadoop.hive.ql.metadata.Hive.createDatabase(Hive.java:608)
> at
> org.apache.hadoop.hive.ql.ddl.database.create.CreateDatabaseOperation.execute(CreateDatabaseOperation.java:68)
> at org.apache.hadoop.hive.ql.ddl.DDLTask.execute(DDLTask.java:84)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:212)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105)
> at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:361)
> at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:334)
> at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:245)
> at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:106)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:348)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:204)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:153)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:148)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:185)
> at
> org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:228)
> at
> org.apache.hive.streaming.TestStreaming.runDDL(TestStreaming.java:3202)
> at
> org.apache.hive.streaming.TestStreaming.createStoreSales(TestStreaming.java:267)
> at org.apache.hive.streaming.TestStreaming.setup(TestStreaming.java:250)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
>
[jira] [Updated] (HIVE-26051) Non blocking ALTER MATERIALIZED VIEW REWRITE implementation
[ https://issues.apache.org/jira/browse/HIVE-26051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-26051: -- Summary: Non blocking ALTER MATERIALIZED VIEW REWRITE implementation (was: Non blocking Alter Materialized View Rewrite) > Non blocking ALTER MATERIALIZED VIEW REWRITE implementation > --- > > Key: HIVE-26051 > URL: https://issues.apache.org/jira/browse/HIVE-26051 > Project: Hive > Issue Type: Task >Reporter: Denys Kuzmenko >Priority: Major > > AlterMaterializedViewRewriteAnalyzer does not change the view content It just > controls whether the view can be used in automatic query plan rewrite. No > need for an EXCLUSIVE lock here. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (HIVE-26052) Non blocking DROP MATERIALIZED VIEW implementation
[ https://issues.apache.org/jira/browse/HIVE-26052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-26052: -- Summary: Non blocking DROP MATERIALIZED VIEW implementation (was: Non-blocking DROP MATERIALIZED VIEW implementation) > Non blocking DROP MATERIALIZED VIEW implementation > -- > > Key: HIVE-26052 > URL: https://issues.apache.org/jira/browse/HIVE-26052 > Project: Hive > Issue Type: Task >Reporter: Denys Kuzmenko >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-26043) Use constraint info when creating RexNodes
[
https://issues.apache.org/jira/browse/HIVE-26043?focusedWorklogId=744870=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-744870
]
ASF GitHub Bot logged work on HIVE-26043:
-
Author: ASF GitHub Bot
Created on: 21/Mar/22 08:12
Start Date: 21/Mar/22 08:12
Worklog Time Spent: 10m
Work Description: kasakrisz opened a new pull request #3120:
URL: https://github.com/apache/hive/pull/3120
### What changes were proposed in this pull request?
Use the data type coming from the Calcite `RelDataType` when creating
`RexNode`s from `ColumnInfo` if possible. This process is part of the
compilation: converting the query represented by QB objects and AST subtrees to
Calcite logical plan.
### Why are the changes needed?
When TableScan logical operators are created table constraint information is
pulled from the metastore and applied the `RelDataType` associated to the TS
operator.
Constraints like `not null` has an affect the way Hive rewrites subqueries
to joins. Example: if the optimizer is aware that the lhs of an in subquery can
not be null a simpler rewrite is chosen. See jira for example plans.
### Does this PR introduce _any_ user-facing change?
No, but `explain` command output may be different in older Hive versions.
### How was this patch tested?
```
mvn test -Dtest.output.overwrite -DskipSparkTests
-Dtest=TestTezTPCDS30TBPerfCliDriver -Dqfile=query45.q -pl itests/qtest -Pitests
mvn test -Dtest.output.overwrite -DskipSparkTests
-Dtest=TestMiniLlapLocalCliDriver -Dqfile=subquery_in.q -pl itests/qtest
-Pitests
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 744870)
Remaining Estimate: 0h
Time Spent: 10m
> Use constraint info when creating RexNodes
> --
>
> Key: HIVE-26043
> URL: https://issues.apache.org/jira/browse/HIVE-26043
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Prior HIVE-23100 Not null constraints affected newly created RexNode type
> nullability.
> Nullability enables the subquery rewrite algorithm to generate more optimal
> plan.
> [https://github.com/apache/hive/blob/1213ad3f0ae0e21e7519dc28b8b6d1401cdd1441/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSubQueryRemoveRule.java#L324]
> Example:
> {code:java}
> explain cbo
> select ws_sales_price
> from web_sales, customer, item
> where ws_bill_customer_sk = c_customer_sk
> and ws_item_sk = i_item_sk
> and ( c_customer_sk = 1
> or
> i_item_id in (select i_item_id
> from item
> where i_item_sk in (2, 3)
> )
> );
> {code}
> Without not null constraints
> {code:java}
> HiveProject(ws_sales_price=[$2])
> HiveFilter(condition=[OR(AND(<>($6, 0), IS NOT NULL($8)), =($3, 1))])
> HiveProject(ws_item_sk=[$0], ws_bill_customer_sk=[$1],
> ws_sales_price=[$2], c_customer_sk=[$8], i_item_sk=[$3], i_item_id=[$4],
> c=[$5], i_item_id0=[$6], literalTrue=[$7])
> HiveJoin(condition=[=($1, $8)], joinType=[inner], algorithm=[none],
> cost=[not available])
> HiveJoin(condition=[=($0, $3)], joinType=[inner], algorithm=[none],
> cost=[not available])
> HiveProject(ws_item_sk=[$2], ws_bill_customer_sk=[$3],
> ws_sales_price=[$20])
> HiveFilter(condition=[IS NOT NULL($3)])
> HiveTableScan(table=[[default, web_sales]],
> table:alias=[web_sales])
> HiveJoin(condition=[=($1, $3)], joinType=[left], algorithm=[none],
> cost=[not available])
> HiveJoin(condition=[true], joinType=[inner], algorithm=[none],
> cost=[not available])
> HiveProject(i_item_sk=[$0], i_item_id=[$1])
> HiveTableScan(table=[[default, item]], table:alias=[item])
> HiveProject(c=[$0])
> HiveAggregate(group=[{}], c=[COUNT()])
> HiveFilter(condition=[IN($0, 2:BIGINT, 3:BIGINT)])
> HiveTableScan(table=[[default, item]], table:alias=[item])
> HiveProject(i_item_id=[$0], literalTrue=[true])
> HiveAggregate(group=[{1}])
> HiveFilter(condition=[IN($0, 2:BIGINT, 3:BIGINT)])
> HiveTableScan(table=[[default,
[jira] [Updated] (HIVE-26043) Use constraint info when creating RexNodes
[
https://issues.apache.org/jira/browse/HIVE-26043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-26043:
--
Labels: pull-request-available (was: )
> Use constraint info when creating RexNodes
> --
>
> Key: HIVE-26043
> URL: https://issues.apache.org/jira/browse/HIVE-26043
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Prior HIVE-23100 Not null constraints affected newly created RexNode type
> nullability.
> Nullability enables the subquery rewrite algorithm to generate more optimal
> plan.
> [https://github.com/apache/hive/blob/1213ad3f0ae0e21e7519dc28b8b6d1401cdd1441/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HiveSubQueryRemoveRule.java#L324]
> Example:
> {code:java}
> explain cbo
> select ws_sales_price
> from web_sales, customer, item
> where ws_bill_customer_sk = c_customer_sk
> and ws_item_sk = i_item_sk
> and ( c_customer_sk = 1
> or
> i_item_id in (select i_item_id
> from item
> where i_item_sk in (2, 3)
> )
> );
> {code}
> Without not null constraints
> {code:java}
> HiveProject(ws_sales_price=[$2])
> HiveFilter(condition=[OR(AND(<>($6, 0), IS NOT NULL($8)), =($3, 1))])
> HiveProject(ws_item_sk=[$0], ws_bill_customer_sk=[$1],
> ws_sales_price=[$2], c_customer_sk=[$8], i_item_sk=[$3], i_item_id=[$4],
> c=[$5], i_item_id0=[$6], literalTrue=[$7])
> HiveJoin(condition=[=($1, $8)], joinType=[inner], algorithm=[none],
> cost=[not available])
> HiveJoin(condition=[=($0, $3)], joinType=[inner], algorithm=[none],
> cost=[not available])
> HiveProject(ws_item_sk=[$2], ws_bill_customer_sk=[$3],
> ws_sales_price=[$20])
> HiveFilter(condition=[IS NOT NULL($3)])
> HiveTableScan(table=[[default, web_sales]],
> table:alias=[web_sales])
> HiveJoin(condition=[=($1, $3)], joinType=[left], algorithm=[none],
> cost=[not available])
> HiveJoin(condition=[true], joinType=[inner], algorithm=[none],
> cost=[not available])
> HiveProject(i_item_sk=[$0], i_item_id=[$1])
> HiveTableScan(table=[[default, item]], table:alias=[item])
> HiveProject(c=[$0])
> HiveAggregate(group=[{}], c=[COUNT()])
> HiveFilter(condition=[IN($0, 2:BIGINT, 3:BIGINT)])
> HiveTableScan(table=[[default, item]], table:alias=[item])
> HiveProject(i_item_id=[$0], literalTrue=[true])
> HiveAggregate(group=[{1}])
> HiveFilter(condition=[IN($0, 2:BIGINT, 3:BIGINT)])
> HiveTableScan(table=[[default, item]], table:alias=[item])
> HiveProject(c_customer_sk=[$0])
> HiveTableScan(table=[[default, customer]], table:alias=[customer])
> {code}
> With not null constraints
> {code:java}
> HiveProject(ws_sales_price=[$2])
> HiveFilter(condition=[OR(IS NOT NULL($7), =($3, 1))])
> HiveProject(ws_item_sk=[$0], ws_bill_customer_sk=[$1],
> ws_sales_price=[$2], c_customer_sk=[$7], i_item_sk=[$3], i_item_id=[$4],
> i_item_id0=[$5], literalTrue=[$6])
> HiveJoin(condition=[=($1, $7)], joinType=[inner], algorithm=[none],
> cost=[not available])
> HiveJoin(condition=[=($0, $3)], joinType=[inner], algorithm=[none],
> cost=[not available])
> HiveProject(ws_item_sk=[$2], ws_bill_customer_sk=[$3],
> ws_sales_price=[$20])
> HiveFilter(condition=[IS NOT NULL($3)])
> HiveTableScan(table=[[default, web_sales]],
> table:alias=[web_sales])
> HiveJoin(condition=[=($1, $2)], joinType=[left], algorithm=[none],
> cost=[not available])
> HiveProject(i_item_sk=[$0], i_item_id=[$1])
> HiveTableScan(table=[[default, item]], table:alias=[item])
> HiveProject(i_item_id=[$0], literalTrue=[true])
> HiveAggregate(group=[{1}])
> HiveFilter(condition=[IN($0, 2:BIGINT, 3:BIGINT)])
> HiveTableScan(table=[[default, item]], table:alias=[item])
> HiveProject(c_customer_sk=[$0])
> HiveTableScan(table=[[default, customer]], table:alias=[customer])
> {code}
> In the first plan when not null constraints was ignored there is an extra
> {{item}} table join without join condition:
> {code:java}
> HiveJoin(condition=[true], joinType=[inner], algorithm=[none],
> cost=[not available])
> HiveProject(i_item_sk=[$0], i_item_id=[$1])
> HiveTableScan(table=[[default, item]],
[jira] [Updated] (HIVE-26051) Non blocking Alter Materialized View Rewrite
[ https://issues.apache.org/jira/browse/HIVE-26051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-26051: -- Description: AlterMaterializedViewRewriteAnalyzer does not change the view content It just controls whether the view can be used in automatic query plan rewrite. No need for an EXCLUSIVE lock here. > Non blocking Alter Materialized View Rewrite > > > Key: HIVE-26051 > URL: https://issues.apache.org/jira/browse/HIVE-26051 > Project: Hive > Issue Type: Task >Reporter: Denys Kuzmenko >Priority: Major > > AlterMaterializedViewRewriteAnalyzer does not change the view content It just > controls whether the view can be used in automatic query plan rewrite. No > need for an EXCLUSIVE lock here. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HIVE-25956) Non blocking RENAME TABLE implementation
[ https://issues.apache.org/jira/browse/HIVE-25956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko resolved HIVE-25956. --- Resolution: Fixed > Non blocking RENAME TABLE implementation > > > Key: HIVE-25956 > URL: https://issues.apache.org/jira/browse/HIVE-25956 > Project: Hive > Issue Type: Task >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > Implement ALTER TABLE RENAME in a way that doesn't have to wait for currently > running read operations to be finished. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HIVE-25956) Non blocking RENAME TABLE implementation
[ https://issues.apache.org/jira/browse/HIVE-25956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17509659#comment-17509659 ] Denys Kuzmenko commented on HIVE-25956: --- Merged to master. [~lpinter], thank you for the review! > Non blocking RENAME TABLE implementation > > > Key: HIVE-25956 > URL: https://issues.apache.org/jira/browse/HIVE-25956 > Project: Hive > Issue Type: Task >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > Implement ALTER TABLE RENAME in a way that doesn't have to wait for currently > running read operations to be finished. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25956) Non blocking RENAME TABLE implementation
[ https://issues.apache.org/jira/browse/HIVE-25956?focusedWorklogId=744862=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-744862 ] ASF GitHub Bot logged work on HIVE-25956: - Author: ASF GitHub Bot Created on: 21/Mar/22 07:50 Start Date: 21/Mar/22 07:50 Worklog Time Spent: 10m Work Description: deniskuzZ merged pull request #3022: URL: https://github.com/apache/hive/pull/3022 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 744862) Time Spent: 0.5h (was: 20m) > Non blocking RENAME TABLE implementation > > > Key: HIVE-25956 > URL: https://issues.apache.org/jira/browse/HIVE-25956 > Project: Hive > Issue Type: Task >Reporter: Denys Kuzmenko >Assignee: Denys Kuzmenko >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > Implement ALTER TABLE RENAME in a way that doesn't have to wait for currently > running read operations to be finished. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HIVE-25814) Add entry in replication_metrics table for skipped replication iterations.
[ https://issues.apache.org/jira/browse/HIVE-25814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayush Saxena resolved HIVE-25814. - Fix Version/s: 4.0.0 Resolution: Fixed > Add entry in replication_metrics table for skipped replication iterations. > -- > > Key: HIVE-25814 > URL: https://issues.apache.org/jira/browse/HIVE-25814 > Project: Hive > Issue Type: Improvement >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HIVE-25814) Add entry in replication_metrics table for skipped replication iterations.
[ https://issues.apache.org/jira/browse/HIVE-25814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17509638#comment-17509638 ] Ayush Saxena commented on HIVE-25814: - Merged PR to master. Thanx [~haymant] for the contribution!!! > Add entry in replication_metrics table for skipped replication iterations. > -- > > Key: HIVE-25814 > URL: https://issues.apache.org/jira/browse/HIVE-25814 > Project: Hive > Issue Type: Improvement >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Work logged] (HIVE-25814) Add entry in replication_metrics table for skipped replication iterations.
[ https://issues.apache.org/jira/browse/HIVE-25814?focusedWorklogId=744853=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-744853 ] ASF GitHub Bot logged work on HIVE-25814: - Author: ASF GitHub Bot Created on: 21/Mar/22 06:53 Start Date: 21/Mar/22 06:53 Worklog Time Spent: 10m Work Description: ayushtkn merged pull request #2907: URL: https://github.com/apache/hive/pull/2907 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: gitbox-unsubscr...@hive.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 744853) Time Spent: 1h 10m (was: 1h) > Add entry in replication_metrics table for skipped replication iterations. > -- > > Key: HIVE-25814 > URL: https://issues.apache.org/jira/browse/HIVE-25814 > Project: Hive > Issue Type: Improvement >Reporter: Haymant Mangla >Assignee: Haymant Mangla >Priority: Major > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.1#820001)
