[jira] [Work logged] (HIVE-23960) Partition with no column statistics leads to unbalanced calls to openTransaction/commitTransaction error during get_partitions_by_names
[
https://issues.apache.org/jira/browse/HIVE-23960?focusedWorklogId=501715=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-501715
]
ASF GitHub Bot logged work on HIVE-23960:
-
Author: ASF GitHub Bot
Created on: 17/Oct/20 00:56
Start Date: 17/Oct/20 00:56
Worklog Time Spent: 10m
Work Description: github-actions[bot] closed pull request #1343:
URL: https://github.com/apache/hive/pull/1343
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 501715)
Time Spent: 1h (was: 50m)
> Partition with no column statistics leads to unbalanced calls to
> openTransaction/commitTransaction error during get_partitions_by_names
> ---
>
> Key: HIVE-23960
> URL: https://issues.apache.org/jira/browse/HIVE-23960
> Project: Hive
> Issue Type: Task
>Reporter: Pravin Sinha
>Assignee: Pravin Sinha
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-23960.01.patch, HIVE-23960.02.patch,
> HIVE-23960.03.patch
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> {color:#172b4d}Creating a partition with data and adding another partition is
> leading to unbalanced calls to open/commit transaction during
> get_partitions_by_names call.{color}
> {color:#172b4d}Issue was discovered during REPL DUMP operation which uses
> this HMS call to get the metadata of partition. This error occurs when there
> is a partition with no column statistics.{color}
> {color:#172b4d}To reproduce:{color}
> {code:java}
> CREATE TABLE student_part_acid(name string, age int, gpa double) PARTITIONED
> BY (ds string) STORED AS orc;
> LOAD DATA INPATH ‘/user/hive/partDir/student_part_acid/ds=20110924’ INTO
> TABLE student_part_acid partition(ds=20110924);
> ALTER TABLE student_part_acid ADD PARTITION (ds=20110925);
> Now if we try to preform REPL DUMP it fails with this the error "Unbalanced
> calls to open/commit transaction" on the HS2 side.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24004) Improve performance for filter hook for superuser path
[ https://issues.apache.org/jira/browse/HIVE-24004?focusedWorklogId=501714=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-501714 ] ASF GitHub Bot logged work on HIVE-24004: - Author: ASF GitHub Bot Created on: 17/Oct/20 00:55 Start Date: 17/Oct/20 00:55 Worklog Time Spent: 10m Work Description: github-actions[bot] closed pull request #1373: URL: https://github.com/apache/hive/pull/1373 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 501714) Time Spent: 1h 10m (was: 1h) > Improve performance for filter hook for superuser path > -- > > Key: HIVE-24004 > URL: https://issues.apache.org/jira/browse/HIVE-24004 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 4.0.0 >Reporter: Sam An >Assignee: Sam An >Priority: Minor > Labels: pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > In HiveMetastoreAuthoriver, the sequence of creating the authorizer can be > optimized so that for super user when we can skip authorization, we can skip > creating authorizer. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-24281) Unable to reopen a closed bug report
[ https://issues.apache.org/jira/browse/HIVE-24281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankur Tagra resolved HIVE-24281. Resolution: Fixed > Unable to reopen a closed bug report > > > Key: HIVE-24281 > URL: https://issues.apache.org/jira/browse/HIVE-24281 > Project: Hive > Issue Type: Bug >Reporter: Ankur Tagra >Assignee: Ankur Tagra >Priority: Trivial > > Unable to reopen a closed bug report -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (HIVE-24281) Unable to reopen a closed bug report
[ https://issues.apache.org/jira/browse/HIVE-24281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankur Tagra reopened HIVE-24281: > Unable to reopen a closed bug report > > > Key: HIVE-24281 > URL: https://issues.apache.org/jira/browse/HIVE-24281 > Project: Hive > Issue Type: Bug >Reporter: Ankur Tagra >Assignee: Ankur Tagra >Priority: Trivial > > Unable to reopen a closed bug report -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-24281) Unable to reopen a closed bug report
[ https://issues.apache.org/jira/browse/HIVE-24281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-24281 started by Ankur Tagra. -- > Unable to reopen a closed bug report > > > Key: HIVE-24281 > URL: https://issues.apache.org/jira/browse/HIVE-24281 > Project: Hive > Issue Type: Bug >Reporter: Ankur Tagra >Assignee: Ankur Tagra >Priority: Trivial > > Unable to reopen a closed bug report -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-24281) Unable to reopen a closed bug report
[ https://issues.apache.org/jira/browse/HIVE-24281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-24281 started by Ankur Tagra. -- > Unable to reopen a closed bug report > > > Key: HIVE-24281 > URL: https://issues.apache.org/jira/browse/HIVE-24281 > Project: Hive > Issue Type: Bug >Reporter: Ankur Tagra >Assignee: Ankur Tagra >Priority: Trivial > > Unable to reopen a closed bug report -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work stopped] (HIVE-24281) Unable to reopen a closed bug report
[ https://issues.apache.org/jira/browse/HIVE-24281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-24281 stopped by Ankur Tagra. -- > Unable to reopen a closed bug report > > > Key: HIVE-24281 > URL: https://issues.apache.org/jira/browse/HIVE-24281 > Project: Hive > Issue Type: Bug >Reporter: Ankur Tagra >Assignee: Ankur Tagra >Priority: Trivial > > Unable to reopen a closed bug report -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (HIVE-24281) Unable to reopen a closed bug report
[ https://issues.apache.org/jira/browse/HIVE-24281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankur Tagra reopened HIVE-24281: > Unable to reopen a closed bug report > > > Key: HIVE-24281 > URL: https://issues.apache.org/jira/browse/HIVE-24281 > Project: Hive > Issue Type: Bug >Reporter: Ankur Tagra >Assignee: Ankur Tagra >Priority: Trivial > > Unable to reopen a closed bug report -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HIVE-24281) Unable to reopen a closed bug report
[ https://issues.apache.org/jira/browse/HIVE-24281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankur Tagra resolved HIVE-24281. Resolution: Fixed > Unable to reopen a closed bug report > > > Key: HIVE-24281 > URL: https://issues.apache.org/jira/browse/HIVE-24281 > Project: Hive > Issue Type: Bug >Reporter: Ankur Tagra >Assignee: Ankur Tagra >Priority: Trivial > > Unable to reopen a closed bug report -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work started] (HIVE-24281) Unable to reopen a closed bug report
[ https://issues.apache.org/jira/browse/HIVE-24281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-24281 started by Ankur Tagra. -- > Unable to reopen a closed bug report > > > Key: HIVE-24281 > URL: https://issues.apache.org/jira/browse/HIVE-24281 > Project: Hive > Issue Type: Bug >Reporter: Ankur Tagra >Assignee: Ankur Tagra >Priority: Trivial > > Unable to reopen a closed bug report -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-24281) Unable to reopen a closed bug report
[ https://issues.apache.org/jira/browse/HIVE-24281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ankur Tagra reassigned HIVE-24281: -- > Unable to reopen a closed bug report > > > Key: HIVE-24281 > URL: https://issues.apache.org/jira/browse/HIVE-24281 > Project: Hive > Issue Type: Bug >Reporter: Ankur Tagra >Assignee: Ankur Tagra >Priority: Trivial > > Unable to reopen a closed bug report -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-23695) [CachedStore] Add check/default constraints in CachedStore
[
https://issues.apache.org/jira/browse/HIVE-23695?focusedWorklogId=501602=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-501602
]
ASF GitHub Bot logged work on HIVE-23695:
-
Author: ASF GitHub Bot
Created on: 16/Oct/20 15:46
Start Date: 16/Oct/20 15:46
Worklog Time Spent: 10m
Work Description: ashish-kumar-sharma commented on a change in pull
request #1527:
URL: https://github.com/apache/hive/pull/1527#discussion_r506560970
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
##
@@ -749,6 +797,52 @@ public void
refreshUniqueConstraints(List constraints) {
}
}
+public void refreshDefaultConstraints(List
constraints) {
+ Map newConstraints = new
ConcurrentHashMap<>();
+ try {
+tableLock.writeLock().lock();
+int size = 0;
+for (SQLDefaultConstraint constraint : constraints) {
+ if
(compareAndSetMemberCacheUpdated(MemberName.DEFAULT_CONSTRAINT_CACHE, true,
false)) {
+LOG.debug("Skipping default constraint cache update for table: " +
getTable().getTableName()
++ "; the default constraint are already refreshed.");
+return;
+ }
+ newConstraints.put(constraint.getDc_name().toLowerCase(),
constraint);
+ size += getObjectSize(SQLUniqueConstraint.class, constraint);
+}
+defaultConstraintCache = newConstraints;
+updateMemberSize(MemberName.DEFAULT_CONSTRAINT_CACHE, size,
SizeMode.Snapshot);
+LOG.debug("Default constraints refresh in cache was successful for
{}.{}.{}", this.getTable().getCatName(),
+this.getTable().getDbName(), this.getTable().getTableName());
+ } finally {
+tableLock.writeLock().unlock();
+ }
+}
+
+public void refreshCheckConstraints(List constraints) {
+ Map newConstraints = new
ConcurrentHashMap<>();
+ try {
+tableLock.writeLock().lock();
+int size = 0;
+for (SQLCheckConstraint constraint : constraints) {
+ if
(compareAndSetMemberCacheUpdated(MemberName.CHECK_CONSTRAINT_CACHE, true,
false)) {
+LOG.debug("Skipping check constraint cache update for table: " +
getTable().getTableName()
++ "; the check constraint are already refreshed.");
+return;
+ }
+ newConstraints.put(constraint.getDc_name().toLowerCase(),
constraint);
+ size += getObjectSize(SQLCheckConstraint.class, constraint);
+}
+checkConstraintCache = newConstraints;
+updateMemberSize(MemberName.CHECK_CONSTRAINT_CACHE, size,
SizeMode.Snapshot);
+LOG.debug("Unique constraints refresh in cache was successful for
{}.{}.{}", this.getTable().getCatName(),
Review comment:
Done
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
##
@@ -1669,33 +1763,47 @@ public boolean populateTableInCache(Table table,
TableCacheObjects cacheObjects)
tblWrapper.setMemberCacheUpdated(MemberName.PARTITION_COL_STATS_CACHE,
false);
tblWrapper.setMemberCacheUpdated(MemberName.AGGR_COL_STATS_CACHE, false);
-if (cacheObjects.getPrimaryKeys() != null) {
- if(!tblWrapper.cachePrimaryKeys(cacheObjects.getPrimaryKeys(), true)) {
+if
(CollectionUtils.isNotEmpty(cacheObjects.getTableConstraints().getPrimaryKeys()))
{
+ if
(!tblWrapper.cachePrimaryKeys(cacheObjects.getTableConstraints().getPrimaryKeys(),
true)) {
+return false;
+ }
+}
+tblWrapper.setMemberCacheUpdated(MemberName.PRIMARY_KEY_CACHE, false);
+
+if
(CollectionUtils.isNotEmpty(cacheObjects.getTableConstraints().getForeignKeys()))
{
Review comment:
Done
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 501602)
Time Spent: 1h 20m (was: 1h 10m)
> [CachedStore] Add check/default constraints in CachedStore
> --
>
> Key: HIVE-23695
> URL: https://issues.apache.org/jira/browse/HIVE-23695
> Project: Hive
> Issue Type: Sub-task
> Components: Standalone Metastore
>Reporter: Adesh Kumar Rao
>Assignee: Ashish Sharma
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> This is blocked by HIVE-23618 (notification
[jira] [Work logged] (HIVE-23695) [CachedStore] Add check/default constraints in CachedStore
[
https://issues.apache.org/jira/browse/HIVE-23695?focusedWorklogId=501603=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-501603
]
ASF GitHub Bot logged work on HIVE-23695:
-
Author: ASF GitHub Bot
Created on: 16/Oct/20 15:46
Start Date: 16/Oct/20 15:46
Worklog Time Spent: 10m
Work Description: ashish-kumar-sharma commented on a change in pull
request #1527:
URL: https://github.com/apache/hive/pull/1527#discussion_r506561396
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/cache/TestCachedStoreUpdateUsingEvents.java
##
@@ -314,160 +308,129 @@ public void testConstraintsForUpdateUsingEvents()
throws Exception {
hmsHandler.create_database(db);
db = rawStore.getDatabase(DEFAULT_CATALOG_NAME, dbName);
-String foreignDbName = "test_table_ops_foreign";
-Database foreignDb = createTestDb(foreignDbName, dbOwner);
-hmsHandler.create_database(foreignDb);
-foreignDb = rawStore.getDatabase(DEFAULT_CATALOG_NAME, foreignDbName);
// Add a table via rawStore
+String parentTableName = "ftbl";
String tblName = "tbl";
String tblOwner = "user1";
FieldSchema col1 = new FieldSchema("col1", "int", "integer column");
FieldSchema col2 = new FieldSchema("col2", "string", "string column");
+FieldSchema col3 = new FieldSchema("col3", "int", "integer column");
List cols = new ArrayList();
cols.add(col1);
cols.add(col2);
+cols.add(col3);
List ptnCols = new ArrayList();
+Table parentTable = createTestTbl(dbName, parentTableName, tblOwner, cols,
ptnCols);
Table tbl = createTestTbl(dbName, tblName, tblOwner, cols, ptnCols);
-String foreignTblName = "ftbl";
-Table foreignTbl = createTestTbl(foreignDbName, foreignTblName, tblOwner,
cols, ptnCols);
-
-SQLPrimaryKey key = new SQLPrimaryKey(dbName, tblName, col1.getName(), 1,
"pk1",
-false, false, false);
-SQLUniqueConstraint uC = new SQLUniqueConstraint(DEFAULT_CATALOG_NAME,
dbName, tblName,
-col1.getName(), 2, "uc1", false, false, false);
-SQLNotNullConstraint nN = new SQLNotNullConstraint(DEFAULT_CATALOG_NAME,
dbName, tblName,
-col1.getName(), "nn1", false, false, false);
-SQLForeignKey foreignKey = new SQLForeignKey(key.getTable_db(),
key.getTable_name(), key.getColumn_name(),
-foreignDbName, foreignTblName, key.getColumn_name(), 2, 1,2,
-"fk1", key.getPk_name(), false, false, false);
-
-hmsHandler.create_table_with_constraints(tbl,
-Arrays.asList(key), null, Arrays.asList(uC), Arrays.asList(nN),
null, null);
-hmsHandler.create_table_with_constraints(foreignTbl, null,
Arrays.asList(foreignKey),
-null, null, null, null);
+
+// Constraints for parent Table
+List parentPkBase =
+Arrays.asList(new SQLPrimaryKey(dbName, parentTableName,
col1.getName(), 1, "parentpk1", false, false, false));
+
+// Constraints for table
+List pkBase =
+Arrays.asList(new SQLPrimaryKey(dbName, tblName, col1.getName(), 1,
"pk1", false, false, false));
+List ucBase = Arrays.asList(
+new SQLUniqueConstraint(DEFAULT_CATALOG_NAME, dbName, tblName,
col1.getName(), 2, "uc1", false, false, false));
+List nnBase = Arrays.asList(
+new SQLNotNullConstraint(DEFAULT_CATALOG_NAME, dbName, tblName,
col1.getName(), "nn1", false, false, false));
+List dcBase = Arrays.asList(
+new SQLDefaultConstraint(DEFAULT_CATALOG_NAME, tbl.getDbName(),
tbl.getTableName(), col2.getName(), "1", "dc1",
+false, false, false));
+List ccBase = Arrays.asList(
+new SQLCheckConstraint(DEFAULT_CATALOG_NAME, tbl.getDbName(),
tbl.getTableName(), col2.getName(), "1", "cc1",
+false, false, false));
+List fkBase = Arrays.asList(
+new SQLForeignKey(parentPkBase.get(0).getTable_db(),
parentPkBase.get(0).getTable_name(),
+parentPkBase.get(0).getColumn_name(), dbName, tblName,
col3.getName(), 2, 1, 2, "fk1",
+parentPkBase.get(0).getPk_name(), false, false, false));
+
+// Create table and parent table
+hmsHandler.create_table_with_constraints(parentTable, parentPkBase, null,
null, null, null, null);
+hmsHandler.create_table_with_constraints(tbl, pkBase, fkBase, ucBase,
nnBase, dcBase, ccBase);
tbl = rawStore.getTable(DEFAULT_CATALOG_NAME, dbName, tblName);
-foreignTbl = rawStore.getTable(DEFAULT_CATALOG_NAME, foreignDbName,
foreignTblName);
+parentTable = rawStore.getTable(DEFAULT_CATALOG_NAME, dbName,
parentTableName);
// Read database, table via CachedStore
-Database dbRead= sharedCache.getDatabaseFromCache(DEFAULT_CATALOG_NAME,
dbName);
+Database dbRead = sharedCache.getDatabaseFromCache(DEFAULT_CATALOG_NAME,
dbName);
Assert.assertEquals(db, dbRead);
+
+// Read table via
[jira] [Work logged] (HIVE-23695) [CachedStore] Add check/default constraints in CachedStore
[
https://issues.apache.org/jira/browse/HIVE-23695?focusedWorklogId=501601=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-501601
]
ASF GitHub Bot logged work on HIVE-23695:
-
Author: ASF GitHub Bot
Created on: 16/Oct/20 15:45
Start Date: 16/Oct/20 15:45
Worklog Time Spent: 10m
Work Description: ashish-kumar-sharma commented on a change in pull
request #1527:
URL: https://github.com/apache/hive/pull/1527#discussion_r506560659
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
##
@@ -749,6 +797,52 @@ public void
refreshUniqueConstraints(List constraints) {
}
}
+public void refreshDefaultConstraints(List
constraints) {
+ Map newConstraints = new
ConcurrentHashMap<>();
+ try {
+tableLock.writeLock().lock();
+int size = 0;
+for (SQLDefaultConstraint constraint : constraints) {
+ if
(compareAndSetMemberCacheUpdated(MemberName.DEFAULT_CONSTRAINT_CACHE, true,
false)) {
+LOG.debug("Skipping default constraint cache update for table: " +
getTable().getTableName()
++ "; the default constraint are already refreshed.");
+return;
+ }
+ newConstraints.put(constraint.getDc_name().toLowerCase(),
constraint);
+ size += getObjectSize(SQLUniqueConstraint.class, constraint);
Review comment:
done
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
##
@@ -909,82 +906,137 @@ private void updateTableColStats(RawStore rawStore,
String catName, String dbNam
}
private void updateTableForeignKeys(RawStore rawStore, String catName,
String dbName, String tblName) {
+ catName = StringUtils.normalizeIdentifier(catName);
+ dbName = StringUtils.normalizeIdentifier(dbName);
+ tblName = StringUtils.normalizeIdentifier(tblName);
LOG.debug("CachedStore: updating cached foreign keys objects for
catalog: {}, database: {}, table: {}", catName,
- dbName, tblName);
+ dbName, tblName);
List fks = null;
try {
Deadline.startTimer("getForeignKeys");
fks = rawStore.getForeignKeys(catName, null, null, dbName, tblName);
Deadline.stopTimer();
} catch (MetaException e) {
-LOG.info("Updating CachedStore: unable to update foreign keys of
catalog: " + catName + ", database: "
-+ dbName + ", table: " + tblName, e);
+LOG.info("Updating CachedStore: unable to update foreign keys of
catalog: " + catName + ", database: " + dbName
++ ", table: " + tblName, e);
}
if (fks != null) {
-
sharedCache.refreshForeignKeysInCache(StringUtils.normalizeIdentifier(catName),
- StringUtils.normalizeIdentifier(dbName),
StringUtils.normalizeIdentifier(tblName), fks);
-LOG.debug("CachedStore: updated cached foreign keys objects for
catalog: {}, database: {}, table: {}",
- catName, dbName, tblName);
+sharedCache.refreshForeignKeysInCache(catName, dbName, tblName, fks);
+LOG.debug("CachedStore: updated cached foreign keys objects for
catalog: {}, database: {}, table: {}", catName,
+dbName, tblName);
}
}
private void updateTableNotNullConstraints(RawStore rawStore, String
catName, String dbName, String tblName) {
+ catName = StringUtils.normalizeIdentifier(catName);
+ dbName = StringUtils.normalizeIdentifier(dbName);
+ tblName = StringUtils.normalizeIdentifier(tblName);
LOG.debug("CachedStore: updating cached not null constraints for
catalog: {}, database: {}, table: {}", catName,
- dbName, tblName);
+ dbName, tblName);
List nns = null;
try {
Deadline.startTimer("getNotNullConstraints");
nns = rawStore.getNotNullConstraints(catName, dbName, tblName);
Deadline.stopTimer();
} catch (MetaException e) {
LOG.info("Updating CachedStore: unable to update not null constraints
of catalog: " + catName + ", database: "
-+ dbName + ", table: " + tblName, e);
++ dbName + ", table: " + tblName, e);
}
if (nns != null) {
-
sharedCache.refreshNotNullConstraintsInCache(StringUtils.normalizeIdentifier(catName),
- StringUtils.normalizeIdentifier(dbName),
StringUtils.normalizeIdentifier(tblName), nns);
-LOG.debug("CachedStore: updated cached not null constraints for
catalog: {}, database: {}, table: {}",
- catName, dbName, tblName);
+sharedCache.refreshNotNullConstraintsInCache(catName, dbName, tblName,
nns);
+LOG.debug("CachedStore: updated cached not null constraints for
catalog: {}, database: {}, table: {}", catName,
+
[jira] [Work logged] (HIVE-23695) [CachedStore] Add check/default constraints in CachedStore
[
https://issues.apache.org/jira/browse/HIVE-23695?focusedWorklogId=501564=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-501564
]
ASF GitHub Bot logged work on HIVE-23695:
-
Author: ASF GitHub Bot
Created on: 16/Oct/20 14:10
Start Date: 16/Oct/20 14:10
Worklog Time Spent: 10m
Work Description: ashish-kumar-sharma commented on a change in pull
request #1527:
URL: https://github.com/apache/hive/pull/1527#discussion_r506468233
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/TableCacheObjects.java
##
@@ -3,59 +3,21 @@
import org.apache.hadoop.hive.metastore.api.AggrStats;
import org.apache.hadoop.hive.metastore.api.ColumnStatistics;
import org.apache.hadoop.hive.metastore.api.Partition;
-import org.apache.hadoop.hive.metastore.api.SQLForeignKey;
-import org.apache.hadoop.hive.metastore.api.SQLNotNullConstraint;
-import org.apache.hadoop.hive.metastore.api.SQLPrimaryKey;
-import org.apache.hadoop.hive.metastore.api.SQLUniqueConstraint;
+import org.apache.hadoop.hive.metastore.api.SQLAllTableConstraints;
import java.util.List;
/*
* Holder class for table objects like partitions, statistics, constraints etc.
*/
public class TableCacheObjects {
- private List primaryKeys;
- private List foreignKeys;
- private List notNullConstraints;
- private List uniqueConstraints;
+ private SQLAllTableConstraints tableConstraints;
Review comment:
Here tableConstraints is just a place holder to keep all constraint at
one place. In cache read and write flow each constraint is treated
independently. Maintained a fail safe flow by falling back to rawstore in case
of dirty read, wrong values and partial info present in cached store.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 501564)
Time Spent: 1h (was: 50m)
> [CachedStore] Add check/default constraints in CachedStore
> --
>
> Key: HIVE-23695
> URL: https://issues.apache.org/jira/browse/HIVE-23695
> Project: Hive
> Issue Type: Sub-task
> Components: Standalone Metastore
>Reporter: Adesh Kumar Rao
>Assignee: Ashish Sharma
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> This is blocked by HIVE-23618 (notification events are not generated for
> default/unique constraints, hence created a separate sub-task from
> HIVE-22015).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-23695) [CachedStore] Add check/default constraints in CachedStore
[
https://issues.apache.org/jira/browse/HIVE-23695?focusedWorklogId=501518=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-501518
]
ASF GitHub Bot logged work on HIVE-23695:
-
Author: ASF GitHub Bot
Created on: 16/Oct/20 11:00
Start Date: 16/Oct/20 11:00
Worklog Time Spent: 10m
Work Description: sankarh commented on a change in pull request #1527:
URL: https://github.com/apache/hive/pull/1527#discussion_r506285109
##
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
##
@@ -749,6 +797,52 @@ public void
refreshUniqueConstraints(List constraints) {
}
}
+public void refreshDefaultConstraints(List
constraints) {
+ Map newConstraints = new
ConcurrentHashMap<>();
+ try {
+tableLock.writeLock().lock();
+int size = 0;
+for (SQLDefaultConstraint constraint : constraints) {
+ if
(compareAndSetMemberCacheUpdated(MemberName.DEFAULT_CONSTRAINT_CACHE, true,
false)) {
+LOG.debug("Skipping default constraint cache update for table: " +
getTable().getTableName()
++ "; the default constraint are already refreshed.");
+return;
+ }
+ newConstraints.put(constraint.getDc_name().toLowerCase(),
constraint);
+ size += getObjectSize(SQLUniqueConstraint.class, constraint);
+}
+defaultConstraintCache = newConstraints;
+updateMemberSize(MemberName.DEFAULT_CONSTRAINT_CACHE, size,
SizeMode.Snapshot);
+LOG.debug("Default constraints refresh in cache was successful for
{}.{}.{}", this.getTable().getCatName(),
+this.getTable().getDbName(), this.getTable().getTableName());
+ } finally {
+tableLock.writeLock().unlock();
+ }
+}
+
+public void refreshCheckConstraints(List constraints) {
+ Map newConstraints = new
ConcurrentHashMap<>();
+ try {
+tableLock.writeLock().lock();
+int size = 0;
+for (SQLCheckConstraint constraint : constraints) {
+ if
(compareAndSetMemberCacheUpdated(MemberName.CHECK_CONSTRAINT_CACHE, true,
false)) {
+LOG.debug("Skipping check constraint cache update for table: " +
getTable().getTableName()
++ "; the check constraint are already refreshed.");
+return;
+ }
+ newConstraints.put(constraint.getDc_name().toLowerCase(),
constraint);
+ size += getObjectSize(SQLCheckConstraint.class, constraint);
+}
+checkConstraintCache = newConstraints;
+updateMemberSize(MemberName.CHECK_CONSTRAINT_CACHE, size,
SizeMode.Snapshot);
+LOG.debug("Unique constraints refresh in cache was successful for
{}.{}.{}", this.getTable().getCatName(),
Review comment:
Copy paste error: "Check constraints .."
##
File path:
itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/cache/TestCachedStoreUpdateUsingEvents.java
##
@@ -314,160 +308,129 @@ public void testConstraintsForUpdateUsingEvents()
throws Exception {
hmsHandler.create_database(db);
db = rawStore.getDatabase(DEFAULT_CATALOG_NAME, dbName);
-String foreignDbName = "test_table_ops_foreign";
-Database foreignDb = createTestDb(foreignDbName, dbOwner);
-hmsHandler.create_database(foreignDb);
-foreignDb = rawStore.getDatabase(DEFAULT_CATALOG_NAME, foreignDbName);
// Add a table via rawStore
+String parentTableName = "ftbl";
String tblName = "tbl";
String tblOwner = "user1";
FieldSchema col1 = new FieldSchema("col1", "int", "integer column");
FieldSchema col2 = new FieldSchema("col2", "string", "string column");
+FieldSchema col3 = new FieldSchema("col3", "int", "integer column");
List cols = new ArrayList();
cols.add(col1);
cols.add(col2);
+cols.add(col3);
List ptnCols = new ArrayList();
+Table parentTable = createTestTbl(dbName, parentTableName, tblOwner, cols,
ptnCols);
Table tbl = createTestTbl(dbName, tblName, tblOwner, cols, ptnCols);
-String foreignTblName = "ftbl";
-Table foreignTbl = createTestTbl(foreignDbName, foreignTblName, tblOwner,
cols, ptnCols);
-
-SQLPrimaryKey key = new SQLPrimaryKey(dbName, tblName, col1.getName(), 1,
"pk1",
-false, false, false);
-SQLUniqueConstraint uC = new SQLUniqueConstraint(DEFAULT_CATALOG_NAME,
dbName, tblName,
-col1.getName(), 2, "uc1", false, false, false);
-SQLNotNullConstraint nN = new SQLNotNullConstraint(DEFAULT_CATALOG_NAME,
dbName, tblName,
-col1.getName(), "nn1", false, false, false);
-SQLForeignKey foreignKey = new SQLForeignKey(key.getTable_db(),
key.getTable_name(), key.getColumn_name(),
-foreignDbName, foreignTblName, key.getColumn_name(), 2, 1,2,
-"fk1",
[jira] [Updated] (HIVE-19103) Nested structure Projection Push Down in Hive with ORC
[ https://issues.apache.org/jira/browse/HIVE-19103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Sharma updated HIVE-19103: - Resolution: Resolved Status: Resolved (was: Patch Available) > Nested structure Projection Push Down in Hive with ORC > -- > > Key: HIVE-19103 > URL: https://issues.apache.org/jira/browse/HIVE-19103 > Project: Hive > Issue Type: Improvement > Components: Hive, ORC >Reporter: Ashish Sharma >Assignee: Ashish Sharma >Priority: Critical > Labels: pull-request-available > Attachments: HIVE-19103.2.patch, HIVE-19103.3.patch, HIVE-19103.patch > > Time Spent: 20m > Remaining Estimate: 0h > > Reading required columns only in nested structure schema > Example - > *Current state* - > Schema - struct,g:string>> > Query - select c.e.f from t where c.e.f > 10; > Current state - read entire c struct from the file and then filter because > "hive.io.file.readcolumn.ids" is referred due to which all the children > column are select to read from the file. > Conf - > _hive.io.file.readcolumn.ids = "2" > hive.io.file.readNestedColumn.paths = "c.e.f"_ > Result - > boolean[ ] include = [true,false,false,true,true,true,true,true] > *Expected state* - > Schema - struct,g:string>> > Query - select c.e.f from t where c.e.f > 10; > Expected state - instead of reading entire c struct from the file just read > only the f column by referring the " hive.io.file.readNestedColumn.paths". > Conf - > _hive.io.file.readcolumn.ids = "2" > hive.io.file.readNestedColumn.paths = "c.e.f"_ > Result - > boolean[ ] include = [true,false,false,true,false,true,true,false] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-24275) Configurations to delay the deletion of obsolete files by the Cleaner
[ https://issues.apache.org/jira/browse/HIVE-24275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-24275: -- Labels: pull-request-available (was: ) > Configurations to delay the deletion of obsolete files by the Cleaner > - > > Key: HIVE-24275 > URL: https://issues.apache.org/jira/browse/HIVE-24275 > Project: Hive > Issue Type: New Feature >Reporter: Kishen Das >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Whenever compaction happens, the cleaner immediately deletes older obsolete > files. In certain cases it would be beneficial to retain these for certain > period. For example : if you are serving the file metadata from cache and > don't want to invalidate the cache during compaction because of performance > reasons. > For this purpose we should introduce a configuration > hive.compactor.delayed.cleanup.enabled, which if enabled will delay the > cleaning up obsolete files. There should be a separate configuration > CLEANER_RETENTION_TIME to specify the duration till which we should retain > these older obsolete files. > It might be beneficial to have one more configuration to decide whether to > retain files involved in an aborted transaction > hive.compactor.aborted.txn.delayed.cleanup.enabled . -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24275) Configurations to delay the deletion of obsolete files by the Cleaner
[ https://issues.apache.org/jira/browse/HIVE-24275?focusedWorklogId=501509=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-501509 ] ASF GitHub Bot logged work on HIVE-24275: - Author: ASF GitHub Bot Created on: 16/Oct/20 10:31 Start Date: 16/Oct/20 10:31 Worklog Time Spent: 10m Work Description: vpnvishv opened a new pull request #1583: URL: https://github.com/apache/hive/pull/1583 ### What changes were proposed in this pull request? Added support to delay cleanup of obsolete files/dirs with user configured value. I have just ported this change from Qubole's hive-3 branch to master with some changes, it still requires thorough review wrt to Master branch. In this change we have reused CQ_META_INFO field from COMPACTION_QUEUE to store compaction end time. As we didn't wanted to change schema in branch-3. This can be changed in master by adding a new field in compaction_queue. ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? added unit test This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 501509) Remaining Estimate: 0h Time Spent: 10m > Configurations to delay the deletion of obsolete files by the Cleaner > - > > Key: HIVE-24275 > URL: https://issues.apache.org/jira/browse/HIVE-24275 > Project: Hive > Issue Type: New Feature >Reporter: Kishen Das >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Whenever compaction happens, the cleaner immediately deletes older obsolete > files. In certain cases it would be beneficial to retain these for certain > period. For example : if you are serving the file metadata from cache and > don't want to invalidate the cache during compaction because of performance > reasons. > For this purpose we should introduce a configuration > hive.compactor.delayed.cleanup.enabled, which if enabled will delay the > cleaning up obsolete files. There should be a separate configuration > CLEANER_RETENTION_TIME to specify the duration till which we should retain > these older obsolete files. > It might be beneficial to have one more configuration to decide whether to > retain files involved in an aborted transaction > hive.compactor.aborted.txn.delayed.cleanup.enabled . -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24217) HMS storage backend for HPL/SQL stored procedures
[ https://issues.apache.org/jira/browse/HIVE-24217?focusedWorklogId=501467=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-501467 ] ASF GitHub Bot logged work on HIVE-24217: - Author: ASF GitHub Bot Created on: 16/Oct/20 08:39 Start Date: 16/Oct/20 08:39 Worklog Time Spent: 10m Work Description: kgyrtkirk commented on a change in pull request #1542: URL: https://github.com/apache/hive/pull/1542#discussion_r506176698 ## File path: standalone-metastore/metastore-server/src/main/resources/package.jdo ## @@ -1549,6 +1549,83 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + Review comment: > What was proposed (but @kgyrtkirk correct me if I'm wrong) as an alternative solution is store the signature together with the body and optionally add a signature string to the table as well (this would be redundant). I think regardless of the representation, backward compatibility will always be a concern. That was a suggestion to provide a way to store a human readable 1 liner for the function - I never thinked that we should parse it; the system should rely on the definition - the human readable field would be just for "show" to be used sysdb related tables/etc. > This is only true for now because HPL/SQL is an AST interpreter but if we ever want to make this performant probably we'll need to introduce a byte code VM at some point in the near future. If at some point in time the "parsing" will prove to be a bottle neck - then we should address it at that point ; rather than try to address a problem which we may not even face - correct me if I'm wrong but I think we will only parse the function 1 time during the HS2 life cycle - which doesn't seem to be a big deal to me. > One thing I don't like about storing the signature in text is that the assumption that invoking the procedure will always require parsing. This is only true for now because HPL/SQL is an AST interpreter but if we ever want to make this performant probably we'll need to introduce a byte code VM at some point in the near future. I think we should clarify/separate 2 things - for stored procedures usually there will be 2 languages at play: * the "host" language - in this case HiveQL - which will accept the function definition * the "foreign" language - which may or may not need parsing; in case of "HPL/SQL" it will but that will not be true for all languages because this patch doesn't contain any HiveQL parser changes - I don't know how that part will work; could you give some example how these functions could be used by the user? ``` create procedure x(a integer) as $$ insert into t values(a); $$ langugage hplsql; call x(12); ``` > For example postgres allows you to invoke procedures implemented in C where runtime information about the signature is not available. I don't know what feature you are refering to ; but afaik to register a function in postgres you should [run a `create function` stmt](https://www.postgresql.org/docs/9.1/sql-createfunction.html) - which will contain all the arguments/etc; and because of that I don't see any issue with other languages - as a create procedure call for any language; must contain the full defintion - which includes argument types/etc - so we will store that for any supported language.. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 501467) Time Spent: 2h 10m (was: 2h) > HMS storage backend for HPL/SQL stored procedures > - > > Key: HIVE-24217 > URL: https://issues.apache.org/jira/browse/HIVE-24217 > Project: Hive > Issue Type: Bug > Components: Hive, hpl/sql, Metastore >Reporter: Attila Magyar >Assignee: Attila Magyar >Priority: Major > Labels: pull-request-available > Attachments: HPL_SQL storedproc
