One thing I've done before is to install datadogs statsd agent on the nodes. Then you can emit metrics and stats to it and build dashboards on datadog.
Sent from my iPhone > On Dec 5, 2016, at 8:17 PM, Chawla,Sumit <sumitkcha...@gmail.com> wrote: > > Hi Manish > > I am specifically looking for something similar to following: > > > https://ci.apache.org/projects/flink/flink-docs-release-1.1/apis/common/index.html#accumulators--counters. > > > Flink has this concept of Accumulators, where user can keep its custom > counters etc. While the application is executing these counters are > queryable through REST API provided by Flink Monitoring Backend. This way > you don't have to wait for the program to complete. > > > > Regards > Sumit Chawla > > >> On Mon, Dec 5, 2016 at 5:53 PM, manish ranjan <cse1.man...@gmail.com> wrote: >> http://spark.apache.org/docs/latest/monitoring.html >> >> You can even install tools like dstat, iostat, and iotop, collectd can >> provide fine-grained profiling on individual nodes. >> >> If you are using Mesos as Resource Manager , mesos exposes metrics as well >> for the running job. >> >> Manish >> >> ~Manish >> >> >> >>> On Mon, Dec 5, 2016 at 4:17 PM, Chawla,Sumit <sumitkcha...@gmail.com> wrote: >>> Hi All >>> >>> I have a long running job which takes hours and hours to process data. How >>> can i monitor the operational efficency of this job? I am interested in >>> something like Storm\Flink style User metrics/aggregators, which i can >>> monitor while my job is running. Using these metrics i want to monitor, >>> per partition performance in processing items. As of now, only way for me >>> to get these metrics is when the job finishes. >>> >>> One possibility is that spark can flush the metrics to external system >>> every few seconds, and thus use an external system to monitor these >>> metrics. However, i wanted to see if the spark supports any such use case >>> OOB. >>> >>> >>> Regards >>> Sumit Chawla >>> >> >
