michaelschiff commented on pull request #10412: URL: https://github.com/apache/druid/pull/10412#issuecomment-760648631
@Tiaaa awesome! I see peon metrics in pushgateway as well. <img width="1780" alt="Screen Shot 2021-01-14 at 8 42 38 PM" src="https://user-images.githubusercontent.com/1669670/104682480-a0131900-56a9-11eb-8817-3dea4d0b3b0d.png";> Couple things I've noticed while testing: * Successful pushes in peon tasks still log an exception: ``` 2021-01-15T04:38:20,671 ERROR [main] org.apache.druid.emitter.prometheus.PrometheusEmitter - Unable to push prometheus metrics to pushGateway java.io.IOException: Response code from http://pushgateway:9091/metrics/job/druid/druid/index_parallel_test_data_pbppbchf_2021-01-15T04%3A38%3A07.325Z was 200 at io.prometheus.client.exporter.PushGateway.doRequest(PushGateway.java:325) ~[prometheus_simpleclient_pushgateway_0.7.0.jar:?] at io.prometheus.client.exporter.PushGateway.push(PushGateway.java:138) ~[prometheus_simpleclient_pushgateway_0.7.0.jar:?] at io.prometheus.client.exporter.PushGateway.push(PushGateway.java:151) ~[prometheus_simpleclient_pushgateway_0.7.0.jar:?] at org.apache.druid.emitter.prometheus.PrometheusEmitter.flush(PrometheusEmitter.java:134) [prometheus_emitter_0.18.0-SNAPSHOT.jar:0.18.0-SNAPSHOT] at org.apache.druid.emitter.prometheus.PrometheusEmitter.close(PrometheusEmitter.java:150) [prometheus_emitter_0.18.0-SNAPSHOT.jar:0.18.0-SNAPSHOT] at org.apache.druid.java.util.emitter.core.ComposingEmitter.close(ComposingEmitter.java:93) [druid-core-0.18.1.jar:0.18.1] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_242] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_242] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_242] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_242] at org.apache.druid.java.util.common.lifecycle.Lifecycle$AnnotationBasedHandler.stop(Lifecycle.java:465) [druid-core-0.18.1.jar:0.18.1] at org.apache.druid.java.util.common.lifecycle.Lifecycle.stop(Lifecycle.java:368) [druid-core-0.18.1.jar:0.18.1] at org.apache.druid.cli.CliPeon.run(CliPeon.java:296) [druid-services-0.18.1.jar:0.18.1] at org.apache.druid.cli.Main.main(Main.java:113) [druid-services-0.18.1.jar:0.18.1] ``` * the label we're using right now is the task ID. I think this it going to be too high cardinality for Prometheus ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
![https://user-images.githubusercontent.com/1669670/104682480-a0131900-56a9-11eb-8817-3dea4d0b3b0d.png"](https://user-images.githubusercontent.com/1669670/104682480-a0131900-56a9-11eb-8817-3dea4d0b3b0d.png"){kind=link}