ashutosh-bapat commented on a change in pull request #735: HIVE-21960 : Avoid
running stats updater and partition management task on a replicated table.
URL: https://github.com/apache/hive/pull/735#discussion_r308324905
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/stats/StatsUpdaterThread.java
##########
@@ -220,6 +221,16 @@ private void stopWorkers() {
String skipParam =
table.getParameters().get(SKIP_STATS_AUTOUPDATE_PROPERTY);
if ("true".equalsIgnoreCase(skipParam)) return null;
+ // If the table is being replicated into,
+ // 1. the stats are also replicated from the source, so we don't need
those to be calculated
+ // on the target again
+ // 2. updating stats requires a writeId to be created. Hence writeIds on
source and target
+ // can get out of sync when stats are updated. That can cause
consistency issues.
+ String replTrgtParam =
table.getParameters().get(ReplConst.REPL_TARGET_PROPERTY);
Review comment:
That window is too small, but nevertheless finite. Should we add
last.repl.id with value 0 or 1- when creating the table similar to what we are
doing for the db?
The other option is to check db level property e.g. checkpoint, but that
means every time we assess whether a table requires stats updater to work or
partition mgmt task to work, it fetches the Database object. For a small window
corner case this looks pretty expensive.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
