Github user mageru commented on the issue:
https://github.com/apache/spark/pull/19775
Did we really miss out on Prometheus metric functionality because people
couldn't just be cool about it?
---
-
To unsubscribe,
Github user andrusha commented on the issue:
https://github.com/apache/spark/pull/19775
You can use Kubernetes service discovery with conjunction with Prometheus
to find all the available nodes for scraping. Then you just add in your
annotation the host and port.
Github user lony commented on the issue:
https://github.com/apache/spark/pull/19775
@andrusha Do you have a tutorial how to set this up. From my understanding
if I have multiple executers pulling it is harder, as prometheus has to have
all the hostnames. Am I right or wrong?
---
Github user andrusha commented on the issue:
https://github.com/apache/spark/pull/19775
You can also try using https://github.com/andrusha/dropwizard-prometheus,
which implements pull instead of push.
---
-
To
Github user matyix commented on the issue:
https://github.com/apache/spark/pull/19775
For those who are still interested using Prometheus you can get the
standalone package and source code from here:
https://github.com/banzaicloud/spark-metrics . Happy monitoring, try to catch
the
Github user matyix commented on the issue:
https://github.com/apache/spark/pull/19775
@GaalDornick @erikerlandson @jerryshao @felixcheung et all
We gave up this - we have made the requested changes several times and I am
not willing to put more time on this and get in the
Github user GaalDornick commented on the issue:
https://github.com/apache/spark/pull/19775
So, where did we land on the discussion for this PR. Is this change getting
in or is it out?
---
-
To unsubscribe, e-mail:
Github user stoader commented on the issue:
https://github.com/apache/spark/pull/19775
@erikerlandson we tested this on Kubernetes using
https://github.com/prometheus/pushgateway/tree/v0.3.1 and
https://github.com/prometheus/pushgateway/tree/v0.4.0
---
Github user erikerlandson commented on the issue:
https://github.com/apache/spark/pull/19775
Although this is not kube-specific, kubernetes deployment is a major
prometheus use case. Has it been tested in a kube environment?
---
Github user erikerlandson commented on the issue:
https://github.com/apache/spark/pull/19775
I agree w/ @jerryshao that adding new deps to core isn't ideal. (Also that
having #11994 would be really nice)
New deps on a sub-project seems more palatable, but interested in what
Github user stoader commented on the issue:
https://github.com/apache/spark/pull/19775
@smurakozi thank you for reviewing. The PR has been updated based on your
comments.
---
-
To unsubscribe, e-mail:
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/19775
Regardless of the discussion about whether this should live in Spark, does
it need to live in core?
Can it be kept in a separate module like the Ganglia stuff (even though
that one is for
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/19775
My original intention is to expose MetricsSystem related interface in
#11994 , so that users can leverage such interface to build their own metrics
sink/source out of Spark. Unfortunately I'm
Github user matyix commented on the issue:
https://github.com/apache/spark/pull/19775
Hello @erikerlandson @felixcheung @jerryshao - any feedback on this PR?
Shall I close it and not worry about this being merged upstream anymore? We've
been using this in production for the last 3
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19775
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19775
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/19775
Can one of the admins verify this patch?
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
Github user matyix commented on the issue:
https://github.com/apache/spark/pull/19775
Hello @felixcheung @jerryshao
The PR #11994 generally looks good for adding extensibility into the
Metrics system. This PR (Prometheus) works with the changes proposed in PR
#11994 out of
Github user felixcheung commented on the issue:
https://github.com/apache/spark/pull/19775
I agree this is useful to have, @jerryshao is probably right though, it is
likely better to add extensibility into the Metrics system.
@jerryshao I'll review your PR
@matyix could
Github user matyix commented on the issue:
https://github.com/apache/spark/pull/19775
@jerryshao this PR is not Kubernetes specific, it's an extension of the
Spark Metrics system which is part of the core already. We could externalize it
the PR #11994 above ever gets merged.
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/19775
Do we have to put this in Spark, is it a necessary part of k8s? I think if
we pull in that PR(https://github.com/apache/spark/pull/11994), then this can
be stayed out of Spark as a package. Even
Github user erikerlandson commented on the issue:
https://github.com/apache/spark/pull/19775
@matyix thanks for re-submitting!
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional
22 matches
Mail list logo
