klcopp commented on a change in pull request #2827:
URL: https://github.com/apache/hive/pull/2827#discussion_r763128180
##########
File path: common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
##########
@@ -3190,6 +3190,24 @@ private static void
populateLlapDaemonVarsSet(Set<String> llapDaemonVarsSetLocal
"has had a transaction done on it since the last major compaction. So
decreasing this\n" +
"value will increase the load on the NameNode."),
+
HIVE_COMPACTOR_INITIATOR_DURATION_UPDATE_INTERVAL("hive.compactor.initiator.duration.update.interval",
"1s",
Review comment:
I think the default update interval should be at least 10s, but probably
more like 60s. There should be a balance of keeping users updated without
burdening the HMS
##########
File path:
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
##########
@@ -494,6 +494,18 @@ public static ConfVars getMetaConf(String name) {
12, TimeUnit.HOURS,
"Age of oldest initiated compaction in the compaction queue after
which an error will be logged. " +
"Default time unit is: hours"),
+ COMPACTOR_LONG_RUNNING_INITIATOR_THRESHOLD_WARNING(
+ "metastore.compactor.long.running.initiator.threshold.warning",
+ "hive.compactor.long.running.initiator.threshold.warning",
+ 6, TimeUnit.HOURS,
+ "Initiator duration after which a warning will be logged. " +
+ "Default time unit is: hours"),
+ COMPACTOR_LONG_RUNNING_INITIATOR_THRESHOLD_ERROR(
+ "metastore.compactor.long.running.initiator.threshold.error",
+ "hive.compactor.long.running.initiator.threshold.error",
+ 12, TimeUnit.HOURS,
+ "Initiator duration after which an wrror will be logged. " +
Review comment:
nit: spelling
##########
File path: common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
##########
@@ -3190,6 +3190,24 @@ private static void
populateLlapDaemonVarsSet(Set<String> llapDaemonVarsSetLocal
"has had a transaction done on it since the last major compaction. So
decreasing this\n" +
"value will increase the load on the NameNode."),
+
HIVE_COMPACTOR_INITIATOR_DURATION_UPDATE_INTERVAL("hive.compactor.initiator.duration.update.interval",
"1s",
+ new TimeValidator(TimeUnit.SECONDS),
+ "Time in seconds that drives the update interval of
compaction_initiator_duration metric.\n" +
+ "Smaller value results in a fine grained metric update.\n" +
+ "This updater can be turned off if its value less than or equals
to zero.\n"+
+ "In this case the above metric will be update only after the
initiator completed one cycle.\n" +
+ "The hive.compactor.initiator.on must be turned on (true) in-order
to enable the Initiator,\n" +
+ "otherwise this setting has no effect."),
+
+
HIVE_COMPACTOR_CLEANER_DURATION_UPDATE_INTERVAL("hive.compactor.cleaner.duration.update.interval",
"1s",
+ new TimeValidator(TimeUnit.SECONDS),
+ "Time in seconds that drives the update interval of
compaction_cleaner_duration metric.\n" +
+ "Smaller value results in a fine grained metric update.\n" +
+ "This updater can be turned off if its value less than or equals
to zero.\n"+
+ "In this case the above metric will be update only after the
initiator completed one cycle.\n" +
Review comment:
nit: Initiator is mentioned here and in the lines below.
##########
File path:
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java
##########
@@ -494,6 +494,18 @@ public static ConfVars getMetaConf(String name) {
12, TimeUnit.HOURS,
"Age of oldest initiated compaction in the compaction queue after
which an error will be logged. " +
"Default time unit is: hours"),
+ COMPACTOR_LONG_RUNNING_INITIATOR_THRESHOLD_WARNING(
+ "metastore.compactor.long.running.initiator.threshold.warning",
+ "hive.compactor.long.running.initiator.threshold.warning",
+ 6, TimeUnit.HOURS,
+ "Initiator duration after which a warning will be logged. " +
Review comment:
I would say, "Initiator cycle duration" here just to be clear that it's
the whole cycle we're talking about
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
