You can see the task details in the Spark UI to see how many bytes do each of the tasks in the skewed stages process. That would be good place to start.
On Mon, Sep 28, 2015 at 7:59 PM, 이기석 <ks881...@gmail.com> wrote: > Hi this is a graduate student studying Spark streaming for research > purpose. > > I want to know whether there is a task skew in my streaming application. > But as far as I found out, the Spark UI does not provide any useful > information to figure this. > > I found a related work from Spark Summit 2014: > > *Sparkling: Identification of Task Skew and Speculative Partition of Data > for Spark applications* > ( > https://spark-summit.org/2014/talk/sparkling-identification-of-task-skew-and-speculative-partition-of-data-for-spark-applications > ) > > However it does not seem to be opened for public use. > Is there any useful tool that I can use to find a task skew? > > > Thanks in advance. >
