[GitHub] spark issue #18690: [SPARK-21334][CORE] Add metrics reporting service to Ext...
Github user raajay commented on the issue: https://github.com/apache/spark/pull/18690 I understand. My previous comment was just a clarification to your question: "I'm not sure how does this code work in your changes?". I will close this PR. The JIRA is already closed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18690: [SPARK-21334][CORE] Add metrics reporting service to Ext...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/18690 So I think if you want to connect your custom sink to Spark Metrics System, then you should at least follow what Spark and codahale metrics library did. Adding a feature in Spark specifically works only for your own sink seems not so reasonable, unless it is a general requirement. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18690: [SPARK-21334][CORE] Add metrics reporting service to Ext...
Github user raajay commented on the issue: https://github.com/apache/spark/pull/18690 @jerryshao My CustomSInk has the report function defined. What I did not have was an equivalent of JmxReporter defined in my CustomSink. The reporter essentially periodically invokes the report function defined in CustomSink --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18690: [SPARK-21334][CORE] Add metrics reporting service to Ext...
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/18690
If your custom sink doesn't have a "reporter", then I'm not sure how does
this code work in your changes? Since `MetricsSystem#report` will call report
API of each sink. I'm still not clearly sure about your scenario.
```
override def run(): Unit = {
metricsSystem.report()
}
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18690: [SPARK-21334][CORE] Add metrics reporting service to Ext...
Github user raajay commented on the issue: https://github.com/apache/spark/pull/18690 We were using a custom sink rather than the JmxSink for gathering metrics. The sink did NOT have a "reporter" like the ones JmxSink or CsvSink have. I guess a cleaner design is to implement a metrics reporter in the Sink and not have a reporting service as part of external shuffle service. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18690: [SPARK-21334][CORE] Add metrics reporting service to Ext...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/18690 I think it is because you don't configure metrics sink here, so by default none of the metrics are reported. For example if I enabled jmx sink (`*.sink.jmx.class=org.apache.spark.metrics.sink.JmxSink`). Then using jconsole to connect to shuffle service, then you will see it on jconsole:  --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18690: [SPARK-21334][CORE] Add metrics reporting service to Ext...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18690 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
