kasakrisz commented on a change in pull request #2612:
URL: https://github.com/apache/hive/pull/2612#discussion_r699351152
##########
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
##########
@@ -93,6 +93,11 @@ public CompactionTxnHandler() {
try {
dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
stmt = dbConn.createStatement();
+ // De-duplicate
+ stmt.execute("DELETE FROM \"COMPLETED_TXN_COMPONENTS\" WHERE
CTC_WRITEID\" NOT IN ("+
Review comment:
As far as I know compaction is also deletes records from the
`COMPLETED_TXN_COMPONENTS` table. So after a source table of a materialized
view is compacted incremental rebuild of the MV is no longer possible.
Currently the only way to exploit the benefits fo incremental MV rebuild is
synchronising compaction and MV rebuild schedule.
If `AcidHouseKeeperService` runs more frequent than compaction it can lower
the chances of an incremental MV rebuild.
If I understand correctly this extra delete statement will be run only when
a compaction is in progress. Can we keep it here? @pvary
##########
File path:
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
##########
@@ -93,6 +93,11 @@ public CompactionTxnHandler() {
try {
dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
stmt = dbConn.createStatement();
+ // De-duplicate
+ stmt.execute("DELETE FROM \"COMPLETED_TXN_COMPONENTS\" WHERE
CTC_WRITEID\" NOT IN ("+
Review comment:
A default config value for `AcidHouseKeeperService` which equals the
default compaction schedule is also good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
