abhishekrb19 opened a new issue, #19021: URL: https://github.com/apache/druid/issues/19021
### Motivation The LoggingEmitter currently emits _all_ metrics it receives without filtering. On busy clusters with high query throughput, this can result in a very large volume of metric logs - on some of our clusters, we've seen ~3M metric logs in 15 minutes. ### Proposed changes - Add a file to capture the default metrics for the LoggingEmitter, similar to the Prometheus and Dropwizard emitters (e.g., `defaultMetricDimensions.json`). - This file can contain the list of allowed metrics. - In the future, we can extend this file to annotate additional user-defined dimensions, if needed. - In the LoggingEmitter `emit(Event event)` method, check the allowlist metrics map and decide whether to include the event or not. ### Rationale Typically, the LoggingEmitter is used in conjunction with a composite emitter that handles low- to medium-cardinality metrics, such as Prometheus, Dropwizard, or Kafka forwarding to a downstream system. Filtering out metrics that are already covered by these emitters would help reduce logging costs while continuing to rely on the appropriate emitter for low- to medium-cardinality metrics. ### Operational impact None. By default, all metrics will continue to be emitted for backwards compatibility with a runtime property. Operators can opt-in to enable filtering with the property. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
