Re: Custom metrics sink

[Silvio Fiorito]

Just set your custom sink in the org.apache.spark.metrics.sink namespace and 
configure metrics.properties. Use ConsoleSink as an example. Obviously since 
it’s private the API may change, but in the meantime that should work…

https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/metrics/sink/ConsoleSink.scala#L28


From: Christopher Piggott 
Date: Friday, March 16, 2018 at 4:09 PM
To: "user@spark.apache.org" 
Subject: Custom metrics sink

Just for fun, i want to make a stupid program that makes different frequency 
chimes as each worker becomes active.  That way you can 'hear' what the cluster 
is doing and how it's distributing work.

I thought to do this I would make a custom Sink, but the Sink and everything 
else in org.apache.spark.metrics.sink is private to spark.  What I was hoping 
to do was to just pick up the # of active workers in semi real time (once a 
second?) and have them send a UDP message somewhere... then each worker would 
be assigned to a different frequency chime.  It's just a toy, for fun.

How do you add a custom Sink when these classes don't seem to be exposed?

--C

Re: Custom metrics sink

[Felix Cheung]

There is a proposal to expose them. See SPARK-14151


From: Christopher Piggott 
Sent: Friday, March 16, 2018 1:09:38 PM
To: user@spark.apache.org
Subject: Custom metrics sink

Just for fun, i want to make a stupid program that makes different frequency 
chimes as each worker becomes active.  That way you can 'hear' what the cluster 
is doing and how it's distributing work.

I thought to do this I would make a custom Sink, but the Sink and everything 
else in org.apache.spark.metrics.sink is private to spark.  What I was hoping 
to do was to just pick up the # of active workers in semi real time (once a 
second?) and have them send a UDP message somewhere... then each worker would 
be assigned to a different frequency chime.  It's just a toy, for fun.

How do you add a custom Sink when these classes don't seem to be exposed?

--C

Re: Custom Metrics Sink

[dgoldenberg]

Hi,

I was wondering if there've been any responses to this?

Thanks.



--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/Custom-Metrics-Sink-tp10068p23425.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org