yuecong commented on issue #26060: [SPARK-29400][CORE] Improve PrometheusResource to use labels URL: https://github.com/apache/spark/pull/26060#issuecomment-539813010 > That is a general issue on Apache Spark monitoring instead of this PR, isn't it? So, I have three questions for you. > > 1. Do you use a custom Sink to monitor Apache Spark? > 2. Do you collect only a cluster-wide metrics? > 3. Is it helpful for long-running app monitoring like structured streamings? I agree with it is a general challenge for Apache Spark monitoring using normal Prometheus server. I would suggest just make it clear about its high-cardinality. Maybe this is orthogonal to your PR. just my two cents. People use a highly scalable Prometheus(e.g. M3, Cotext, etc) to handle Spark metrics. Also if we could have one custom exporter to allow users to use a push model to expose it to some distributed time serials database or a pub-sub system(e.g. kafka), it can solve this high cardinality issue as well
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
