snleee commented on a change in pull request #7368:
URL: https://github.com/apache/pinot/pull/7368#discussion_r719800597
##########
File path:
pinot-plugins/pinot-minion-tasks/pinot-minion-builtin-tasks/src/main/java/org/apache/pinot/plugin/minion/tasks/mergerollup/MergeRollupTaskGenerator.java
##########
@@ -90,15 +92,29 @@
*/
@TaskGenerator
public class MergeRollupTaskGenerator implements PinotTaskGenerator {
+ private static final Logger LOGGER =
LoggerFactory.getLogger(MergeRollupTaskGenerator.class);
private static final int DEFAULT_MAX_NUM_RECORDS_PER_TASK = 50_000_000;
private static final String REFRESH = "REFRESH";
- private static final Logger LOGGER =
LoggerFactory.getLogger(MergeRollupTaskGenerator.class);
+ // This is the metric that keeps track of the task delay in the number of
time buckets. For example, if we see this
Review comment:
@mcvsubbu We thought about this approach (initially we wanted to make
the delay metric as `n days`). But, it will not work well with the case of
multiple merge levels. The time bucket size for each merge level can be pretty
flexible (it can be 1 day, 30days, or even 1yr).
For instance, if we are 1 window behind for a 30days window, the time-based
metric will quickly go up to 30 days of delay. So, we decided to normalize this
delay as `# of windows` so that we can put one global threshold.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
