kumachanz opened a new issue #11645:
URL: https://github.com/apache/druid/issues/11645
In a cluster using Indexer instead of middle-managers, with metrics emitter
enabled, task metrics are emitted forever.
### Affected Version
0.21.1
### Description
Please include as much detailed information about the problem as possible.
Cluster runs with combined coordinator and overlord. There are 10 indexers.
Indexers have been running for 3 days configured with
`druid.emitter=logging`, the indexers run a mixture of kafka ingestion and
index_parallel ingestion tasks.
We noticed the amount of log output was increasing in volume as each hour
passed.
Looking at the metrics logs we saw this as an example:
```
2021-08-30T00:01:27,351 INFO [MonitorScheduler-0]
org.apache.druid.java.util.emitter.core.LoggingEmitter -
{
"feed": "metrics",
"timestamp": "2021-08-30T00:01:27.351Z",
"service": "druid/indexer",
"host": "10.209.6.173:8091",
"version": "0.21.1",
"metric": "ingest/events/thrownAway",
"value": 0,
"dataSource": "events",
"taskId": [
"partial_index_generate_events_mbbnmdcl_2021-08-27T09:28:56.347Z"
]
}
```
Notice date of the event is 3 days after the time this partial index was
generated. Searching the logs we found this same metric for the same taskId is
emitted every minute since the first time it was generated.
We have worked around this problem by disabling metrics logging on the
indexer as there doesn't seem to be a way of having the task metrics emitted
once.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
