snleee commented on code in PR #10037:
URL: https://github.com/apache/pinot/pull/10037#discussion_r1057996304
##########
pinot-minion/src/main/java/org/apache/pinot/minion/taskfactory/TaskFactoryRegistry.java:
##########
@@ -87,55 +87,91 @@ public TaskResult run() {
_minionMetrics
.addPhaseTiming(taskType, MinionQueryPhase.TASK_QUEUEING,
jobDequeueTimeMs - jobInQueueTimeMs,
TimeUnit.MILLISECONDS);
+ String tableName = null;
try {
// Set taskId in MDC so that one may config logger to route
task logs to separate file.
MDC.put("taskId", _taskConfig.getId());
+ PinotTaskConfig pinotTaskConfig =
PinotTaskConfig.fromHelixTaskConfig(_taskConfig);
+ tableName = pinotTaskConfig.getTableName();
_minionMetrics.addValueToGlobalGauge(MinionGauge.NUMBER_OF_TASKS, 1L);
+ if (tableName != null) {
+ _minionMetrics
+ .addPhaseTiming(tableName + "." + taskType,
MinionQueryPhase.TASK_QUEUEING,
+ jobDequeueTimeMs - jobInQueueTimeMs,
TimeUnit.MILLISECONDS);
+ _minionMetrics.addValueToTableGauge(tableName,
MinionGauge.NUMBER_OF_TASKS, 1L);
+ }
MinionEventObservers.getInstance().addMinionEventObserver(_taskConfig.getId(),
_eventObserver);
- return runInternal();
+ return runInternal(pinotTaskConfig);
} finally {
MinionEventObservers.getInstance().removeMinionEventObserver(_taskConfig.getId());
_minionMetrics.addValueToGlobalGauge(MinionGauge.NUMBER_OF_TASKS, -1L);
long executionTimeMs = System.currentTimeMillis() -
jobDequeueTimeMs;
_minionMetrics
.addPhaseTiming(taskType, MinionQueryPhase.TASK_EXECUTION,
executionTimeMs, TimeUnit.MILLISECONDS);
+ if (tableName != null) {
+ _minionMetrics.addValueToTableGauge(tableName,
MinionGauge.NUMBER_OF_TASKS, -1L);
+ _minionMetrics
+ .addPhaseTiming(tableName + "." + taskType,
MinionQueryPhase.TASK_EXECUTION,
+ executionTimeMs, TimeUnit.MILLISECONDS);
+ }
LOGGER.info("Task: {} completed in: {}ms",
_taskConfig.getId(), executionTimeMs);
// Clear taskId from MDC to reset it.
MDC.remove("taskId");
}
}
- private TaskResult runInternal() {
- PinotTaskConfig pinotTaskConfig =
PinotTaskConfig.fromHelixTaskConfig(_taskConfig);
+ private TaskResult runInternal(PinotTaskConfig pinotTaskConfig) {
if
(StringUtils.isBlank(pinotTaskConfig.getConfigs().get(MinionConstants.AUTH_TOKEN)))
{
pinotTaskConfig.getConfigs().put(MinionConstants.AUTH_TOKEN,
AuthProviderUtils.toStaticToken(MinionContext.getInstance().getTaskAuthProvider()));
}
+ String tableName = pinotTaskConfig.getTableName();
+
_eventObserver.notifyTaskStart(pinotTaskConfig);
_minionMetrics.addMeteredTableValue(taskType,
MinionMeter.NUMBER_TASKS_EXECUTED, 1L);
+ if (tableName != null) {
+ _minionMetrics.addMeteredTableValue(tableName + "." + taskType,
Review Comment:
I think that we have been using `tableName` as the key for the metrics. We
probably need to provide an extra input `key` to the function in case they want
some other dimensions. As we discussed, we need sth like the following:
```
// metric key will be `<tableNameWithType>.<key>.<meter name>`
public void addMeteredTableValue(final String tableName, final String key,
final M meter, final long unitCount) {
}
```
We can add this function and start to use the new function for this PR. Or,
we can at least file the issue and address it all together.
##########
docker/images/pinot/etc/jmx_prometheus_javaagent/configs/minion.yml:
##########
@@ -1,4 +1,22 @@
rules:
+- pattern: "\"org.apache.pinot.common.metrics\"<type=\"MinionMetrics\",
name=\"pinot.minion.version.(\\w+)\"><>(\\w+)"
Review Comment:
discussed offline.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
