Re: Get size of intermediate results

2016-10-20 Thread Egor Pahomov
I needed the same for debugging and I just added "count" action in debug mode for every step I was interested in. It's very time-consuming, but I debug not very often. 2016-10-20 2:17 GMT-07:00 Andreas Hechenberger : > Hey awesome Spark-Dev's :) > > i am new to spark and i read a lot but now i am

Get size of intermediate results

2016-10-20 Thread Andreas Hechenberger
Hey awesome Spark-Dev's :) i am new to spark and i read a lot but now i am stuck :( so please be kind, if i ask silly questions. I want to analyze some algorithms and strategies in spark and for one experiment i want to know the size of the intermediate results between iterations/jobs. Some of th