Re: spark streaming counter metrics

Thanks Mayur. I will take a look at StreamingListener. Is there any example you have handy? On Wed, Jul 2, 2014 at 2:32 AM, Mayur Rustagi wrote: > You may be able to mix StreamingListener & SparkListener to get meaningful > information about your task. however you need to connect a lot of piec

Re: spark streaming counter metrics

You may be able to mix StreamingListener & SparkListener to get meaningful information about your task. however you need to connect a lot of pieces to make sense of the flow.. Mayur Rustagi Ph: +1 (760) 203 3257 http://www.sigmoidanalytics.com @mayur_rustagi

spark streaming counter metrics

I am new to spark streaming and wondering if spark streaming tracks counters (e.g., how many rows in each consumer, how many rows routed to an individual reduce task, etc.) in any form so I can get an idea of how data is skewed? I checked spark job page but don't seem to find any. -- Chen Song