Hello All,
I am using spark streaming 1.3 . I want to capture few custom metrics based
on accumulators, I followed somewhat similar to this approach ,
val instrumentation = new SparkInstrumentation("example.metrics")
* val numReqs = sc.accumulator(0L)
* instrumentation.source.registerDailyAccumulator(numReqs, "numReqs")
* instrumentation.register()
https://gist.github.com/ibuenros/9b94736c2bad2f4b8e23
After registering metrics via accumulator , I am not able to see values of
these metrics on sink. I tried console sink , but still no luck
Do I need to set any properties in metrics.conf to enable this custom
source?
Also I noticed accumulators are displayed only in the task stages , It is
difficult to identify which task/stage on ui , that information would be
available .
Is there a way accumulators can be displayed on overall Job stats page ?
Any pointers/examples to achieve this would be helpful , spark monitoring
documentation is not very helpful,
Thanks in advance,
- Snehal
