subject:"\[jira\] \[Work logged\] \(HIVE\-23695\) \[CachedStore\] Add check\/default constraints in CachedStore"

[jira] [Work logged] (HIVE-23695) [CachedStore] Add check/default constraints in CachedStore

2020-10-18 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-23695?focusedWorklogId=502009=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-502009
 ]

ASF GitHub Bot logged work on HIVE-23695:
-

Author: ASF GitHub Bot
Created on: 19/Oct/20 05:43
Start Date: 19/Oct/20 05:43
Worklog Time Spent: 10m 
  Work Description: sankarh merged pull request #1527:
URL: https://github.com/apache/hive/pull/1527


   



This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 502009)
Time Spent: 1h 50m  (was: 1h 40m)

> [CachedStore] Add check/default constraints in CachedStore
> --
>
> Key: HIVE-23695
> URL: https://issues.apache.org/jira/browse/HIVE-23695
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Adesh Kumar Rao
>Assignee: Ashish Sharma
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> This is blocked by HIVE-23618 (notification events are not generated for 
> default/unique constraints, hence created a separate sub-task from 
> HIVE-22015).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (HIVE-23695) [CachedStore] Add check/default constraints in CachedStore

2020-10-18 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-23695?focusedWorklogId=502007=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-502007
 ]

ASF GitHub Bot logged work on HIVE-23695:
-

Author: ASF GitHub Bot
Created on: 19/Oct/20 05:39
Start Date: 19/Oct/20 05:39
Worklog Time Spent: 10m 
  Work Description: sankarh commented on a change in pull request #1527:
URL: https://github.com/apache/hive/pull/1527#discussion_r506278466



##
File path: 
itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/cache/TestCachedStoreUpdateUsingEvents.java
##
@@ -314,160 +308,129 @@ public void testConstraintsForUpdateUsingEvents() 
throws Exception {
 hmsHandler.create_database(db);
 db = rawStore.getDatabase(DEFAULT_CATALOG_NAME, dbName);
 
-String foreignDbName = "test_table_ops_foreign";
-Database foreignDb = createTestDb(foreignDbName, dbOwner);
-hmsHandler.create_database(foreignDb);
-foreignDb = rawStore.getDatabase(DEFAULT_CATALOG_NAME, foreignDbName);
 // Add a table via rawStore
+String parentTableName = "ftbl";
 String tblName = "tbl";
 String tblOwner = "user1";
 FieldSchema col1 = new FieldSchema("col1", "int", "integer column");
 FieldSchema col2 = new FieldSchema("col2", "string", "string column");
+FieldSchema col3 = new FieldSchema("col3", "int", "integer column");
 List cols = new ArrayList();
 cols.add(col1);
 cols.add(col2);
+cols.add(col3);
 List ptnCols = new ArrayList();
+Table parentTable = createTestTbl(dbName, parentTableName, tblOwner, cols, 
ptnCols);
 Table tbl = createTestTbl(dbName, tblName, tblOwner, cols, ptnCols);
-String foreignTblName = "ftbl";
-Table foreignTbl = createTestTbl(foreignDbName, foreignTblName, tblOwner, 
cols, ptnCols);
-
-SQLPrimaryKey key = new SQLPrimaryKey(dbName, tblName, col1.getName(), 1, 
"pk1",
-false, false, false);
-SQLUniqueConstraint uC = new SQLUniqueConstraint(DEFAULT_CATALOG_NAME, 
dbName, tblName,
-col1.getName(), 2, "uc1", false, false, false);
-SQLNotNullConstraint nN = new SQLNotNullConstraint(DEFAULT_CATALOG_NAME, 
dbName, tblName,
-col1.getName(), "nn1", false, false, false);
-SQLForeignKey foreignKey = new SQLForeignKey(key.getTable_db(), 
key.getTable_name(), key.getColumn_name(),
-foreignDbName, foreignTblName, key.getColumn_name(), 2, 1,2,
-"fk1", key.getPk_name(), false, false, false);
-
-hmsHandler.create_table_with_constraints(tbl,
-Arrays.asList(key), null, Arrays.asList(uC), Arrays.asList(nN), 
null, null);
-hmsHandler.create_table_with_constraints(foreignTbl, null, 
Arrays.asList(foreignKey),
-null, null, null, null);
+
+// Constraints for parent Table
+List parentPkBase =
+Arrays.asList(new SQLPrimaryKey(dbName, parentTableName, 
col1.getName(), 1, "parentpk1", false, false, false));
+
+// Constraints for table
+List pkBase =
+Arrays.asList(new SQLPrimaryKey(dbName, tblName, col1.getName(), 1, 
"pk1", false, false, false));
+List ucBase = Arrays.asList(
+new SQLUniqueConstraint(DEFAULT_CATALOG_NAME, dbName, tblName, 
col1.getName(), 2, "uc1", false, false, false));
+List nnBase = Arrays.asList(
+new SQLNotNullConstraint(DEFAULT_CATALOG_NAME, dbName, tblName, 
col1.getName(), "nn1", false, false, false));
+List dcBase = Arrays.asList(
+new SQLDefaultConstraint(DEFAULT_CATALOG_NAME, tbl.getDbName(), 
tbl.getTableName(), col2.getName(), "1", "dc1",
+false, false, false));
+List ccBase = Arrays.asList(
+new SQLCheckConstraint(DEFAULT_CATALOG_NAME, tbl.getDbName(), 
tbl.getTableName(), col2.getName(), "1", "cc1",
+false, false, false));
+List fkBase = Arrays.asList(
+new SQLForeignKey(parentPkBase.get(0).getTable_db(), 
parentPkBase.get(0).getTable_name(),
+parentPkBase.get(0).getColumn_name(), dbName, tblName, 
col3.getName(), 2, 1, 2, "fk1",
+parentPkBase.get(0).getPk_name(), false, false, false));
+
+// Create table and parent table
+hmsHandler.create_table_with_constraints(parentTable, parentPkBase, null, 
null, null, null, null);
+hmsHandler.create_table_with_constraints(tbl, pkBase, fkBase, ucBase, 
nnBase, dcBase, ccBase);
 
 tbl = rawStore.getTable(DEFAULT_CATALOG_NAME, dbName, tblName);
-foreignTbl = rawStore.getTable(DEFAULT_CATALOG_NAME, foreignDbName, 
foreignTblName);
+parentTable = rawStore.getTable(DEFAULT_CATALOG_NAME, dbName, 
parentTableName);
 
 // Read database, table via CachedStore
-Database dbRead= sharedCache.getDatabaseFromCache(DEFAULT_CATALOG_NAME, 
dbName);
+Database dbRead = sharedCache.getDatabaseFromCache(DEFAULT_CATALOG_NAME, 
dbName);
 Assert.assertEquals(db, dbRead);
+
+// Read table via CachedStore

[jira] [Work logged] (HIVE-23695) [CachedStore] Add check/default constraints in CachedStore

2020-10-16 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-23695?focusedWorklogId=501602=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-501602
 ]

ASF GitHub Bot logged work on HIVE-23695:
-

Author: ASF GitHub Bot
Created on: 16/Oct/20 15:46
Start Date: 16/Oct/20 15:46
Worklog Time Spent: 10m 
  Work Description: ashish-kumar-sharma commented on a change in pull 
request #1527:
URL: https://github.com/apache/hive/pull/1527#discussion_r506560970



##
File path: 
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
##
@@ -749,6 +797,52 @@ public void 
refreshUniqueConstraints(List constraints) {
   }
 }
 
+public void refreshDefaultConstraints(List 
constraints) {
+  Map newConstraints = new 
ConcurrentHashMap<>();
+  try {
+tableLock.writeLock().lock();
+int size = 0;
+for (SQLDefaultConstraint constraint : constraints) {
+  if 
(compareAndSetMemberCacheUpdated(MemberName.DEFAULT_CONSTRAINT_CACHE, true, 
false)) {
+LOG.debug("Skipping default constraint cache update for table: " + 
getTable().getTableName()
++ "; the default constraint are already refreshed.");
+return;
+  }
+  newConstraints.put(constraint.getDc_name().toLowerCase(), 
constraint);
+  size += getObjectSize(SQLUniqueConstraint.class, constraint);
+}
+defaultConstraintCache = newConstraints;
+updateMemberSize(MemberName.DEFAULT_CONSTRAINT_CACHE, size, 
SizeMode.Snapshot);
+LOG.debug("Default constraints refresh in cache was successful for 
{}.{}.{}", this.getTable().getCatName(),
+this.getTable().getDbName(), this.getTable().getTableName());
+  } finally {
+tableLock.writeLock().unlock();
+  }
+}
+
+public void refreshCheckConstraints(List constraints) {
+  Map newConstraints = new 
ConcurrentHashMap<>();
+  try {
+tableLock.writeLock().lock();
+int size = 0;
+for (SQLCheckConstraint constraint : constraints) {
+  if 
(compareAndSetMemberCacheUpdated(MemberName.CHECK_CONSTRAINT_CACHE, true, 
false)) {
+LOG.debug("Skipping check constraint cache update for table: " + 
getTable().getTableName()
++ "; the check constraint are already refreshed.");
+return;
+  }
+  newConstraints.put(constraint.getDc_name().toLowerCase(), 
constraint);
+  size += getObjectSize(SQLCheckConstraint.class, constraint);
+}
+checkConstraintCache = newConstraints;
+updateMemberSize(MemberName.CHECK_CONSTRAINT_CACHE, size, 
SizeMode.Snapshot);
+LOG.debug("Unique constraints refresh in cache was successful for 
{}.{}.{}", this.getTable().getCatName(),

Review comment:
   Done

##
File path: 
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
##
@@ -1669,33 +1763,47 @@ public boolean populateTableInCache(Table table, 
TableCacheObjects cacheObjects)
 tblWrapper.setMemberCacheUpdated(MemberName.PARTITION_COL_STATS_CACHE, 
false);
 tblWrapper.setMemberCacheUpdated(MemberName.AGGR_COL_STATS_CACHE, false);
 
-if (cacheObjects.getPrimaryKeys() != null) {
-  if(!tblWrapper.cachePrimaryKeys(cacheObjects.getPrimaryKeys(), true)) {
+if 
(CollectionUtils.isNotEmpty(cacheObjects.getTableConstraints().getPrimaryKeys()))
 {
+  if 
(!tblWrapper.cachePrimaryKeys(cacheObjects.getTableConstraints().getPrimaryKeys(),
 true)) {
+return false;
+  }
+}
+tblWrapper.setMemberCacheUpdated(MemberName.PRIMARY_KEY_CACHE, false);
+
+if 
(CollectionUtils.isNotEmpty(cacheObjects.getTableConstraints().getForeignKeys()))
 {

Review comment:
   Done





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 501602)
Time Spent: 1h 20m  (was: 1h 10m)

> [CachedStore] Add check/default constraints in CachedStore
> --
>
> Key: HIVE-23695
> URL: https://issues.apache.org/jira/browse/HIVE-23695
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Adesh Kumar Rao
>Assignee: Ashish Sharma
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> This is blocked by HIVE-23618 (notification

[jira] [Work logged] (HIVE-23695) [CachedStore] Add check/default constraints in CachedStore

2020-10-16 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-23695?focusedWorklogId=501603=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-501603
 ]

ASF GitHub Bot logged work on HIVE-23695:
-

Author: ASF GitHub Bot
Created on: 16/Oct/20 15:46
Start Date: 16/Oct/20 15:46
Worklog Time Spent: 10m 
  Work Description: ashish-kumar-sharma commented on a change in pull 
request #1527:
URL: https://github.com/apache/hive/pull/1527#discussion_r506561396



##
File path: 
itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/cache/TestCachedStoreUpdateUsingEvents.java
##
@@ -314,160 +308,129 @@ public void testConstraintsForUpdateUsingEvents() 
throws Exception {
 hmsHandler.create_database(db);
 db = rawStore.getDatabase(DEFAULT_CATALOG_NAME, dbName);
 
-String foreignDbName = "test_table_ops_foreign";
-Database foreignDb = createTestDb(foreignDbName, dbOwner);
-hmsHandler.create_database(foreignDb);
-foreignDb = rawStore.getDatabase(DEFAULT_CATALOG_NAME, foreignDbName);
 // Add a table via rawStore
+String parentTableName = "ftbl";
 String tblName = "tbl";
 String tblOwner = "user1";
 FieldSchema col1 = new FieldSchema("col1", "int", "integer column");
 FieldSchema col2 = new FieldSchema("col2", "string", "string column");
+FieldSchema col3 = new FieldSchema("col3", "int", "integer column");
 List cols = new ArrayList();
 cols.add(col1);
 cols.add(col2);
+cols.add(col3);
 List ptnCols = new ArrayList();
+Table parentTable = createTestTbl(dbName, parentTableName, tblOwner, cols, 
ptnCols);
 Table tbl = createTestTbl(dbName, tblName, tblOwner, cols, ptnCols);
-String foreignTblName = "ftbl";
-Table foreignTbl = createTestTbl(foreignDbName, foreignTblName, tblOwner, 
cols, ptnCols);
-
-SQLPrimaryKey key = new SQLPrimaryKey(dbName, tblName, col1.getName(), 1, 
"pk1",
-false, false, false);
-SQLUniqueConstraint uC = new SQLUniqueConstraint(DEFAULT_CATALOG_NAME, 
dbName, tblName,
-col1.getName(), 2, "uc1", false, false, false);
-SQLNotNullConstraint nN = new SQLNotNullConstraint(DEFAULT_CATALOG_NAME, 
dbName, tblName,
-col1.getName(), "nn1", false, false, false);
-SQLForeignKey foreignKey = new SQLForeignKey(key.getTable_db(), 
key.getTable_name(), key.getColumn_name(),
-foreignDbName, foreignTblName, key.getColumn_name(), 2, 1,2,
-"fk1", key.getPk_name(), false, false, false);
-
-hmsHandler.create_table_with_constraints(tbl,
-Arrays.asList(key), null, Arrays.asList(uC), Arrays.asList(nN), 
null, null);
-hmsHandler.create_table_with_constraints(foreignTbl, null, 
Arrays.asList(foreignKey),
-null, null, null, null);
+
+// Constraints for parent Table
+List parentPkBase =
+Arrays.asList(new SQLPrimaryKey(dbName, parentTableName, 
col1.getName(), 1, "parentpk1", false, false, false));
+
+// Constraints for table
+List pkBase =
+Arrays.asList(new SQLPrimaryKey(dbName, tblName, col1.getName(), 1, 
"pk1", false, false, false));
+List ucBase = Arrays.asList(
+new SQLUniqueConstraint(DEFAULT_CATALOG_NAME, dbName, tblName, 
col1.getName(), 2, "uc1", false, false, false));
+List nnBase = Arrays.asList(
+new SQLNotNullConstraint(DEFAULT_CATALOG_NAME, dbName, tblName, 
col1.getName(), "nn1", false, false, false));
+List dcBase = Arrays.asList(
+new SQLDefaultConstraint(DEFAULT_CATALOG_NAME, tbl.getDbName(), 
tbl.getTableName(), col2.getName(), "1", "dc1",
+false, false, false));
+List ccBase = Arrays.asList(
+new SQLCheckConstraint(DEFAULT_CATALOG_NAME, tbl.getDbName(), 
tbl.getTableName(), col2.getName(), "1", "cc1",
+false, false, false));
+List fkBase = Arrays.asList(
+new SQLForeignKey(parentPkBase.get(0).getTable_db(), 
parentPkBase.get(0).getTable_name(),
+parentPkBase.get(0).getColumn_name(), dbName, tblName, 
col3.getName(), 2, 1, 2, "fk1",
+parentPkBase.get(0).getPk_name(), false, false, false));
+
+// Create table and parent table
+hmsHandler.create_table_with_constraints(parentTable, parentPkBase, null, 
null, null, null, null);
+hmsHandler.create_table_with_constraints(tbl, pkBase, fkBase, ucBase, 
nnBase, dcBase, ccBase);
 
 tbl = rawStore.getTable(DEFAULT_CATALOG_NAME, dbName, tblName);
-foreignTbl = rawStore.getTable(DEFAULT_CATALOG_NAME, foreignDbName, 
foreignTblName);
+parentTable = rawStore.getTable(DEFAULT_CATALOG_NAME, dbName, 
parentTableName);
 
 // Read database, table via CachedStore
-Database dbRead= sharedCache.getDatabaseFromCache(DEFAULT_CATALOG_NAME, 
dbName);
+Database dbRead = sharedCache.getDatabaseFromCache(DEFAULT_CATALOG_NAME, 
dbName);
 Assert.assertEquals(db, dbRead);
+
+// Read table via

[jira] [Work logged] (HIVE-23695) [CachedStore] Add check/default constraints in CachedStore

2020-10-16 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-23695?focusedWorklogId=501601=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-501601
 ]

ASF GitHub Bot logged work on HIVE-23695:
-

Author: ASF GitHub Bot
Created on: 16/Oct/20 15:45
Start Date: 16/Oct/20 15:45
Worklog Time Spent: 10m 
  Work Description: ashish-kumar-sharma commented on a change in pull 
request #1527:
URL: https://github.com/apache/hive/pull/1527#discussion_r506560659



##
File path: 
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
##
@@ -749,6 +797,52 @@ public void 
refreshUniqueConstraints(List constraints) {
   }
 }
 
+public void refreshDefaultConstraints(List 
constraints) {
+  Map newConstraints = new 
ConcurrentHashMap<>();
+  try {
+tableLock.writeLock().lock();
+int size = 0;
+for (SQLDefaultConstraint constraint : constraints) {
+  if 
(compareAndSetMemberCacheUpdated(MemberName.DEFAULT_CONSTRAINT_CACHE, true, 
false)) {
+LOG.debug("Skipping default constraint cache update for table: " + 
getTable().getTableName()
++ "; the default constraint are already refreshed.");
+return;
+  }
+  newConstraints.put(constraint.getDc_name().toLowerCase(), 
constraint);
+  size += getObjectSize(SQLUniqueConstraint.class, constraint);

Review comment:
   done

##
File path: 
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/CachedStore.java
##
@@ -909,82 +906,137 @@ private void updateTableColStats(RawStore rawStore, 
String catName, String dbNam
 }
 
 private void updateTableForeignKeys(RawStore rawStore, String catName, 
String dbName, String tblName) {
+  catName = StringUtils.normalizeIdentifier(catName);
+  dbName = StringUtils.normalizeIdentifier(dbName);
+  tblName = StringUtils.normalizeIdentifier(tblName);
   LOG.debug("CachedStore: updating cached foreign keys objects for 
catalog: {}, database: {}, table: {}", catName,
-  dbName, tblName);
+  dbName, tblName);
   List fks = null;
   try {
 Deadline.startTimer("getForeignKeys");
 fks = rawStore.getForeignKeys(catName, null, null, dbName, tblName);
 Deadline.stopTimer();
   } catch (MetaException e) {
-LOG.info("Updating CachedStore: unable to update foreign keys of 
catalog: " + catName + ", database: "
-+ dbName + ", table: " + tblName, e);
+LOG.info("Updating CachedStore: unable to update foreign keys of 
catalog: " + catName + ", database: " + dbName
++ ", table: " + tblName, e);
   }
   if (fks != null) {
-
sharedCache.refreshForeignKeysInCache(StringUtils.normalizeIdentifier(catName),
-  StringUtils.normalizeIdentifier(dbName), 
StringUtils.normalizeIdentifier(tblName), fks);
-LOG.debug("CachedStore: updated cached foreign keys objects for 
catalog: {}, database: {}, table: {}",
-  catName, dbName, tblName);
+sharedCache.refreshForeignKeysInCache(catName, dbName, tblName, fks);
+LOG.debug("CachedStore: updated cached foreign keys objects for 
catalog: {}, database: {}, table: {}", catName,
+dbName, tblName);
   }
 }
 
 private void updateTableNotNullConstraints(RawStore rawStore, String 
catName, String dbName, String tblName) {
+  catName = StringUtils.normalizeIdentifier(catName);
+  dbName = StringUtils.normalizeIdentifier(dbName);
+  tblName = StringUtils.normalizeIdentifier(tblName);
   LOG.debug("CachedStore: updating cached not null constraints for 
catalog: {}, database: {}, table: {}", catName,
-  dbName, tblName);
+  dbName, tblName);
   List nns = null;
   try {
 Deadline.startTimer("getNotNullConstraints");
 nns = rawStore.getNotNullConstraints(catName, dbName, tblName);
 Deadline.stopTimer();
   } catch (MetaException e) {
 LOG.info("Updating CachedStore: unable to update not null constraints 
of catalog: " + catName + ", database: "
-+ dbName + ", table: " + tblName, e);
++ dbName + ", table: " + tblName, e);
   }
   if (nns != null) {
-
sharedCache.refreshNotNullConstraintsInCache(StringUtils.normalizeIdentifier(catName),
-  StringUtils.normalizeIdentifier(dbName), 
StringUtils.normalizeIdentifier(tblName), nns);
-LOG.debug("CachedStore: updated cached not null constraints for 
catalog: {}, database: {}, table: {}",
-  catName, dbName, tblName);
+sharedCache.refreshNotNullConstraintsInCache(catName, dbName, tblName, 
nns);
+LOG.debug("CachedStore: updated cached not null constraints for 
catalog: {}, database: {}, table: {}", catName,
+

[jira] [Work logged] (HIVE-23695) [CachedStore] Add check/default constraints in CachedStore

2020-10-16 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-23695?focusedWorklogId=501564=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-501564
 ]

ASF GitHub Bot logged work on HIVE-23695:
-

Author: ASF GitHub Bot
Created on: 16/Oct/20 14:10
Start Date: 16/Oct/20 14:10
Worklog Time Spent: 10m 
  Work Description: ashish-kumar-sharma commented on a change in pull 
request #1527:
URL: https://github.com/apache/hive/pull/1527#discussion_r506468233



##
File path: 
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/TableCacheObjects.java
##
@@ -3,59 +3,21 @@
 import org.apache.hadoop.hive.metastore.api.AggrStats;
 import org.apache.hadoop.hive.metastore.api.ColumnStatistics;
 import org.apache.hadoop.hive.metastore.api.Partition;
-import org.apache.hadoop.hive.metastore.api.SQLForeignKey;
-import org.apache.hadoop.hive.metastore.api.SQLNotNullConstraint;
-import org.apache.hadoop.hive.metastore.api.SQLPrimaryKey;
-import org.apache.hadoop.hive.metastore.api.SQLUniqueConstraint;
+import org.apache.hadoop.hive.metastore.api.SQLAllTableConstraints;
 
 import java.util.List;
 
 /*
  * Holder class for table objects like partitions, statistics, constraints etc.
  */
 public class TableCacheObjects {
-  private List primaryKeys;
-  private List foreignKeys;
-  private List notNullConstraints;
-  private List uniqueConstraints;
+  private SQLAllTableConstraints tableConstraints;

Review comment:
   Here tableConstraints is just a place holder to keep all constraint at 
one place. In cache read and write flow each constraint is treated 
independently. Maintained a fail safe flow by falling back to rawstore in case 
of dirty read, wrong values and partial info present in cached store.





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 501564)
Time Spent: 1h  (was: 50m)

> [CachedStore] Add check/default constraints in CachedStore
> --
>
> Key: HIVE-23695
> URL: https://issues.apache.org/jira/browse/HIVE-23695
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Adesh Kumar Rao
>Assignee: Ashish Sharma
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> This is blocked by HIVE-23618 (notification events are not generated for 
> default/unique constraints, hence created a separate sub-task from 
> HIVE-22015).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (HIVE-23695) [CachedStore] Add check/default constraints in CachedStore

2020-10-16 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-23695?focusedWorklogId=501518=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-501518
 ]

ASF GitHub Bot logged work on HIVE-23695:
-

Author: ASF GitHub Bot
Created on: 16/Oct/20 11:00
Start Date: 16/Oct/20 11:00
Worklog Time Spent: 10m 
  Work Description: sankarh commented on a change in pull request #1527:
URL: https://github.com/apache/hive/pull/1527#discussion_r506285109



##
File path: 
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/SharedCache.java
##
@@ -749,6 +797,52 @@ public void 
refreshUniqueConstraints(List constraints) {
   }
 }
 
+public void refreshDefaultConstraints(List 
constraints) {
+  Map newConstraints = new 
ConcurrentHashMap<>();
+  try {
+tableLock.writeLock().lock();
+int size = 0;
+for (SQLDefaultConstraint constraint : constraints) {
+  if 
(compareAndSetMemberCacheUpdated(MemberName.DEFAULT_CONSTRAINT_CACHE, true, 
false)) {
+LOG.debug("Skipping default constraint cache update for table: " + 
getTable().getTableName()
++ "; the default constraint are already refreshed.");
+return;
+  }
+  newConstraints.put(constraint.getDc_name().toLowerCase(), 
constraint);
+  size += getObjectSize(SQLUniqueConstraint.class, constraint);
+}
+defaultConstraintCache = newConstraints;
+updateMemberSize(MemberName.DEFAULT_CONSTRAINT_CACHE, size, 
SizeMode.Snapshot);
+LOG.debug("Default constraints refresh in cache was successful for 
{}.{}.{}", this.getTable().getCatName(),
+this.getTable().getDbName(), this.getTable().getTableName());
+  } finally {
+tableLock.writeLock().unlock();
+  }
+}
+
+public void refreshCheckConstraints(List constraints) {
+  Map newConstraints = new 
ConcurrentHashMap<>();
+  try {
+tableLock.writeLock().lock();
+int size = 0;
+for (SQLCheckConstraint constraint : constraints) {
+  if 
(compareAndSetMemberCacheUpdated(MemberName.CHECK_CONSTRAINT_CACHE, true, 
false)) {
+LOG.debug("Skipping check constraint cache update for table: " + 
getTable().getTableName()
++ "; the check constraint are already refreshed.");
+return;
+  }
+  newConstraints.put(constraint.getDc_name().toLowerCase(), 
constraint);
+  size += getObjectSize(SQLCheckConstraint.class, constraint);
+}
+checkConstraintCache = newConstraints;
+updateMemberSize(MemberName.CHECK_CONSTRAINT_CACHE, size, 
SizeMode.Snapshot);
+LOG.debug("Unique constraints refresh in cache was successful for 
{}.{}.{}", this.getTable().getCatName(),

Review comment:
   Copy paste error: "Check constraints .."

##
File path: 
itests/hive-unit/src/test/java/org/apache/hadoop/hive/metastore/cache/TestCachedStoreUpdateUsingEvents.java
##
@@ -314,160 +308,129 @@ public void testConstraintsForUpdateUsingEvents() 
throws Exception {
 hmsHandler.create_database(db);
 db = rawStore.getDatabase(DEFAULT_CATALOG_NAME, dbName);
 
-String foreignDbName = "test_table_ops_foreign";
-Database foreignDb = createTestDb(foreignDbName, dbOwner);
-hmsHandler.create_database(foreignDb);
-foreignDb = rawStore.getDatabase(DEFAULT_CATALOG_NAME, foreignDbName);
 // Add a table via rawStore
+String parentTableName = "ftbl";
 String tblName = "tbl";
 String tblOwner = "user1";
 FieldSchema col1 = new FieldSchema("col1", "int", "integer column");
 FieldSchema col2 = new FieldSchema("col2", "string", "string column");
+FieldSchema col3 = new FieldSchema("col3", "int", "integer column");
 List cols = new ArrayList();
 cols.add(col1);
 cols.add(col2);
+cols.add(col3);
 List ptnCols = new ArrayList();
+Table parentTable = createTestTbl(dbName, parentTableName, tblOwner, cols, 
ptnCols);
 Table tbl = createTestTbl(dbName, tblName, tblOwner, cols, ptnCols);
-String foreignTblName = "ftbl";
-Table foreignTbl = createTestTbl(foreignDbName, foreignTblName, tblOwner, 
cols, ptnCols);
-
-SQLPrimaryKey key = new SQLPrimaryKey(dbName, tblName, col1.getName(), 1, 
"pk1",
-false, false, false);
-SQLUniqueConstraint uC = new SQLUniqueConstraint(DEFAULT_CATALOG_NAME, 
dbName, tblName,
-col1.getName(), 2, "uc1", false, false, false);
-SQLNotNullConstraint nN = new SQLNotNullConstraint(DEFAULT_CATALOG_NAME, 
dbName, tblName,
-col1.getName(), "nn1", false, false, false);
-SQLForeignKey foreignKey = new SQLForeignKey(key.getTable_db(), 
key.getTable_name(), key.getColumn_name(),
-foreignDbName, foreignTblName, key.getColumn_name(), 2, 1,2,
-"fk1",

[jira] [Work logged] (HIVE-23695) [CachedStore] Add check/default constraints in CachedStore

2020-10-11 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-23695?focusedWorklogId=499072=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-499072
 ]

ASF GitHub Bot logged work on HIVE-23695:
-

Author: ASF GitHub Bot
Created on: 11/Oct/20 15:39
Start Date: 11/Oct/20 15:39
Worklog Time Spent: 10m 
  Work Description: ashish-kumar-sharma commented on a change in pull 
request #1527:
URL: https://github.com/apache/hive/pull/1527#discussion_r502931524



##
File path: 
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/TableCacheObjects.java
##
@@ -3,6 +3,9 @@
 import org.apache.hadoop.hive.metastore.api.AggrStats;
 import org.apache.hadoop.hive.metastore.api.ColumnStatistics;
 import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.SQLAllTableConstraints;
+import org.apache.hadoop.hive.metastore.api.SQLCheckConstraint;
+import org.apache.hadoop.hive.metastore.api.SQLDefaultConstraint;

Review comment:
   removed





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 499072)
Time Spent: 40m  (was: 0.5h)

> [CachedStore] Add check/default constraints in CachedStore
> --
>
> Key: HIVE-23695
> URL: https://issues.apache.org/jira/browse/HIVE-23695
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Adesh Kumar Rao
>Assignee: Ashish Sharma
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> This is blocked by HIVE-23618 (notification events are not generated for 
> default/unique constraints, hence created a separate sub-task from 
> HIVE-22015).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (HIVE-23695) [CachedStore] Add check/default constraints in CachedStore

2020-10-11 Thread ASF GitHub Bot (Jira)



 [ 
https://issues.apache.org/jira/browse/HIVE-23695?focusedWorklogId=499067=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-499067
 ]

ASF GitHub Bot logged work on HIVE-23695:
-

Author: ASF GitHub Bot
Created on: 11/Oct/20 11:09
Start Date: 11/Oct/20 11:09
Worklog Time Spent: 10m 
  Work Description: adesh-rao commented on a change in pull request #1527:
URL: https://github.com/apache/hive/pull/1527#discussion_r502898228



##
File path: 
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/TableCacheObjects.java
##
@@ -3,6 +3,9 @@
 import org.apache.hadoop.hive.metastore.api.AggrStats;
 import org.apache.hadoop.hive.metastore.api.ColumnStatistics;
 import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.SQLAllTableConstraints;
+import org.apache.hadoop.hive.metastore.api.SQLCheckConstraint;
+import org.apache.hadoop.hive.metastore.api.SQLDefaultConstraint;

Review comment:
   not required?

##
File path: 
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/cache/TableCacheObjects.java
##
@@ -3,6 +3,9 @@
 import org.apache.hadoop.hive.metastore.api.AggrStats;
 import org.apache.hadoop.hive.metastore.api.ColumnStatistics;
 import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.SQLAllTableConstraints;
+import org.apache.hadoop.hive.metastore.api.SQLCheckConstraint;

Review comment:
   not required?





This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 499067)
Time Spent: 0.5h  (was: 20m)

> [CachedStore] Add check/default constraints in CachedStore
> --
>
> Key: HIVE-23695
> URL: https://issues.apache.org/jira/browse/HIVE-23695
> Project: Hive
>  Issue Type: Sub-task
>  Components: Standalone Metastore
>Reporter: Adesh Kumar Rao
>Assignee: Ashish Sharma
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> This is blocked by HIVE-23618 (notification events are not generated for 
> default/unique constraints, hence created a separate sub-task from 
> HIVE-22015).



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Work logged] (HIVE-23695) [CachedStore] Add check/default constraints in CachedStore

[jira] [Work logged] (HIVE-23695) [CachedStore] Add check/default constraints in CachedStore

[jira] [Work logged] (HIVE-23695) [CachedStore] Add check/default constraints in CachedStore

[jira] [Work logged] (HIVE-23695) [CachedStore] Add check/default constraints in CachedStore

[jira] [Work logged] (HIVE-23695) [CachedStore] Add check/default constraints in CachedStore

[jira] [Work logged] (HIVE-23695) [CachedStore] Add check/default constraints in CachedStore

[jira] [Work logged] (HIVE-23695) [CachedStore] Add check/default constraints in CachedStore

[jira] [Work logged] (HIVE-23695) [CachedStore] Add check/default constraints in CachedStore

[jira] [Work logged] (HIVE-23695) [CachedStore] Add check/default constraints in CachedStore

9 matches

Site Navigation

Mail list logo

Footer information