Thanks Takeshi, that's exactly what I was looking for. On Fri, Feb 5, 2016 at 12:32 PM, Takeshi Yamamuro <linguin....@gmail.com> wrote:
> How about using `spark.jars` to send jars into a cluster? > > On Sat, Feb 6, 2016 at 12:00 AM, Matt K <matvey1...@gmail.com> wrote: > >> Yes. And what I'm trying to figure out if there's a way to package the >> jar in such a way that I don't have to install it on every Executor node. >> >> >> On Wed, Feb 3, 2016 at 7:46 PM, Yiannis Gkoufas <johngou...@gmail.com> >> wrote: >> >>> Hi Matt, >>> >>> does the custom class you want to package reports metrics of each >>> Executor? >>> >>> Thanks >>> >>> On 3 February 2016 at 15:56, Matt K <matvey1...@gmail.com> wrote: >>> >>>> Thanks for sharing Yiannis, looks very promising! >>>> >>>> Do you know if I can package a custom class with my application, or >>>> does it have to be pre-deployed on all Executor nodes? >>>> >>>> On Wed, Feb 3, 2016 at 10:36 AM, Yiannis Gkoufas <johngou...@gmail.com> >>>> wrote: >>>> >>>>> Hi Matt, >>>>> >>>>> there is some related work I recently did in IBM Research for >>>>> visualizing the metrics produced. >>>>> You can read about it here >>>>> http://www.spark.tc/sparkoscope-enabling-spark-optimization-through-cross-stack-monitoring-and-visualization-2/ >>>>> We recently opensourced it if you are interested to have a deeper look >>>>> to it: https://github.com/ibm-research-ireland/sparkoscope >>>>> >>>>> Thanks, >>>>> Yiannis >>>>> >>>>> On 3 February 2016 at 13:32, Matt K <matvey1...@gmail.com> wrote: >>>>> >>>>>> Hi guys, >>>>>> >>>>>> I'm looking to create a custom sync based on Spark's Metrics System: >>>>>> >>>>>> https://github.com/apache/spark/blob/9f603fce78fcc997926e9a72dec44d48cbc396fc/core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala >>>>>> >>>>>> If I want to collect metrics from the Driver, Master, and Executor >>>>>> nodes, should the jar with the custom class be installed on Driver, >>>>>> Master, >>>>>> and Executor nodes? >>>>>> >>>>>> Also, on Executor nodes, does the MetricsSystem run inside the >>>>>> Executor's JVM? >>>>>> >>>>>> Thanks, >>>>>> -Matt >>>>>> >>>>> >>>>> >>>> >>>> >>>> -- >>>> www.calcmachine.com - easy online calculator. >>>> >>> >>> >> >> >> -- >> www.calcmachine.com - easy online calculator. >> > > > > -- > --- > Takeshi Yamamuro > -- www.calcmachine.com - easy online calculator.
