[GitHub] spark issue #19775: [SPARK-22343][core] Add support for publishing Spark met...

[mageru]

Github user mageru commented on the issue:

https://github.com/apache/spark/pull/19775
  
Did we really miss out on Prometheus metric functionality because people 
couldn't just be cool about it? 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19775: [SPARK-22343][core] Add support for publishing Spark met...

[andrusha]

Github user andrusha commented on the issue:

https://github.com/apache/spark/pull/19775
  
You can use Kubernetes service discovery with conjunction with Prometheus 
to find all the available nodes for scraping. Then you just add in your 
annotation the host and port.

https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Ckubernetes_sd_config%3E
 

 is a good place to start.
I think it requires a blog post, but not sure if will ever do it.

> On 16 May 2018, at 13:10, Goetz Epperlein  
wrote:
> 
> @andrusha  Do you have a tutorial how to set 
this up. From my understanding if I have multiple executers pulling it is 
harder, as prometheus has to have all the hostnames. Am I right or wrong?
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub 
, or mute 
the thread 
.
> 




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19775: [SPARK-22343][core] Add support for publishing Spark met...

[lony]

Github user lony commented on the issue:

https://github.com/apache/spark/pull/19775
  
@andrusha Do you have a tutorial how to set this up. From my understanding 
if I have multiple executers pulling it is harder, as prometheus has to have 
all the hostnames. Am I right or wrong?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19775: [SPARK-22343][core] Add support for publishing Spark met...

[andrusha]

Github user andrusha commented on the issue:

https://github.com/apache/spark/pull/19775
  
You can also try using https://github.com/andrusha/dropwizard-prometheus, 
which implements pull instead of push.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19775: [SPARK-22343][core] Add support for publishing Spark met...

[matyix]

Github user matyix commented on the issue:

https://github.com/apache/spark/pull/19775
  
For those who are still interested using Prometheus you can get the 
standalone package and source code from here: 
https://github.com/banzaicloud/spark-metrics . Happy monitoring, try to catch 
the issues and avoid those PagerDuty notifications beforehand :).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19775: [SPARK-22343][core] Add support for publishing Spark met...

[matyix]

Github user matyix commented on the issue:

https://github.com/apache/spark/pull/19775
  
@GaalDornick @erikerlandson @jerryshao @felixcheung  et all 

We gave up this - we have made the requested changes several times and I am 
not willing to put more time on this and get in the middle of a debate which is 
not my concern. Currently the Spark monitoring architecture it is how it is - 
and we have made the PR to align with the current architecture of the existing 
sinks and metrics subsystem. What did happen is that now the debate is not 
about whether this is good, needed or not but whether it should be part of 
Spark core, be pluggable, we should refactor the whole metrics subsystem, etc. 
Most likely this will still be the case later and once these will be changed, 
nailed down or agreed by all parties I can rework and resend the PR... 

Anyways, we (and our customers) are using this in production for months - 
we have externalized this into a separate jar which we put it on the CP and 
does not need to be part of Spark (though it should I believe, as Prometheus is 
one of the best open source monitoring framework). 

Should anybody need help how to use this sink with Spark drop me a mail at 
ja...@banzaicloud.com happy to help anybody who'd like to use Prometheus with 
Spark. We do pretty advanced scenarios with this sink and all open source - you 
can read more about [Monitoring Spark with 
Prometheus](https://banzaicloud.com/blog/spark-monitoring/) and [Federated 
monitoring of multiple Spark 
clusters](https://banzaicloud.com/blog/prometheus-application-monitoring/). 

Thanks for all the support.
Janos


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19775: [SPARK-22343][core] Add support for publishing Spark met...

[GaalDornick]

Github user GaalDornick commented on the issue:

https://github.com/apache/spark/pull/19775
  
So, where did we land on the discussion for this PR. Is this change getting 
in or is it out?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19775: [SPARK-22343][core] Add support for publishing Spark met...

[stoader]

Github user stoader commented on the issue:

https://github.com/apache/spark/pull/19775
  
@erikerlandson we tested this on Kubernetes using 
https://github.com/prometheus/pushgateway/tree/v0.3.1 and 
https://github.com/prometheus/pushgateway/tree/v0.4.0


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19775: [SPARK-22343][core] Add support for publishing Spark met...

[erikerlandson]

Github user erikerlandson commented on the issue:

https://github.com/apache/spark/pull/19775
  
Although this is not kube-specific, kubernetes deployment is a major 
prometheus use case. Has it been tested in a kube environment?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19775: [SPARK-22343][core] Add support for publishing Spark met...

[erikerlandson]

Github user erikerlandson commented on the issue:

https://github.com/apache/spark/pull/19775
  
I agree w/ @jerryshao that adding new deps to core isn't ideal.  (Also that 
having #11994 would be really nice)
New deps on a sub-project seems more palatable, but interested in what 
other Sparkers think.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19775: [SPARK-22343][core] Add support for publishing Spark met...

[stoader]

Github user stoader commented on the issue:

https://github.com/apache/spark/pull/19775
  
@smurakozi thank you for reviewing. The PR has been updated based on your 
comments.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19775: [SPARK-22343][core] Add support for publishing Spark met...

[vanzin]

Github user vanzin commented on the issue:

https://github.com/apache/spark/pull/19775
  
Regardless of the discussion about whether this should live in Spark, does 
it need to live in core?

Can it be kept in a separate module like the Ganglia stuff (even though 
that one is for licensing reasons)?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19775: [SPARK-22343][core] Add support for publishing Spark met...

[jerryshao]

Github user jerryshao commented on the issue:

https://github.com/apache/spark/pull/19775
  
My original intention is to expose MetricsSystem related interface in 
#11994 , so that users can leverage such interface to build their own metrics 
sink/source out of Spark. Unfortunately I'm stuck on the #11994 , but still I 
think it is better to leave this as a package out of Spark, pulling to much 
dependencies for non-core functionalities seems not so reasonable (just my 
thoughts).


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19775: [SPARK-22343][core] Add support for publishing Spark met...

[matyix]

Github user matyix commented on the issue:

https://github.com/apache/spark/pull/19775
  
Hello @erikerlandson @felixcheung @jerryshao - any feedback on this PR? 
Shall I close it and not worry about this being merged upstream anymore? We've 
been using this in production for the last 3 months and it's a bit awkward that 
our CI/CD system needs to `patch` the upstream version all the time but we can 
live with that (since it's automated). Please advise. Happy to help to get it 
merge or eventually just close it.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19775: [SPARK-22343][core] Add support for publishing Spark met...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19775
  
Can one of the admins verify this patch?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19775: [SPARK-22343][core] Add support for publishing Spark met...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19775
  
Can one of the admins verify this patch?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19775: [SPARK-22343][core] Add support for publishing Spark met...

[AmplabJenkins]

Github user AmplabJenkins commented on the issue:

https://github.com/apache/spark/pull/19775
  
Can one of the admins verify this patch?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19775: [SPARK-22343][core] Add support for publishing Spark met...

[matyix]

Github user matyix commented on the issue:

https://github.com/apache/spark/pull/19775
  
Hello @felixcheung @jerryshao

The PR #11994 generally looks good for adding extensibility into the 
Metrics system. This PR (Prometheus) works with the changes proposed in PR 
#11994 out of the box, so no modification needed on our side.

As Prometheus is (becoming) a widely used monitoring system personally I 
see a lot of value including this into the Spark code base, same as the current 
metrics. I see PR #11994 rather a solution for those who build custom 
specialised metrics sink.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19775: [SPARK-22343][core] Add support for publishing Spark met...

[felixcheung]

Github user felixcheung commented on the issue:

https://github.com/apache/spark/pull/19775
  
I agree this is useful to have, @jerryshao is probably right though, it is 
likely better to add extensibility into the Metrics system.

@jerryshao I'll review your PR
@matyix could you also review #11994 to see if that suits the need to build 
everything you have here to connect to Prometheus, but external to Spark and on 
top of #11994? I think your feedback will be very valuable.

We can then come back to this PR as needed.



---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19775: [SPARK-22343][core] Add support for publishing Spark met...

[matyix]

Github user matyix commented on the issue:

https://github.com/apache/spark/pull/19775
  
@jerryshao this PR is not Kubernetes specific, it's an extension of the 
Spark Metrics system which is part of the core already. We could externalize it 
the PR #11994 above ever gets merged. Re-factoring and externalizing it after 
(for all the other metrics subsystem is not a big work). Although I submitted 
this PR first on the K8S fork, actually this feature might be beneficial for 
all using the (becoming) de-facto monitoring solution, Prometheus. 


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19775: [SPARK-22343][core] Add support for publishing Spark met...

[jerryshao]

Github user jerryshao commented on the issue:

https://github.com/apache/spark/pull/19775
  
Do we have to put this in Spark, is it a necessary part of k8s? I think if 
we pull in that PR(https://github.com/apache/spark/pull/11994), then this can 
be stayed out of Spark as a package. Even without #11994 , I believe users can 
still add their own Metrics source/sink via exposed SparkEnv/MetricsSystem. My 
concern is that this unnecessarily increases the code base of spark core.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #19775: [SPARK-22343][core] Add support for publishing Spark met...

[erikerlandson]

Github user erikerlandson commented on the issue:

https://github.com/apache/spark/pull/19775
  
@matyix thanks for re-submitting!


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org