Re: Performance drop / unstable in 0.8 release

2013-11-07 Thread Wenlei Xie
Hi, I have all the code for the previous 0.8 version. But how I can find out the SNAPSHOT version there? (in project/SparkBuild.scala it just says version := 0.8.0-SNAPSHOT) Best, Wenlei On Wed, Nov 6, 2013 at 12:09 AM, Reynold Xin r...@apache.org wrote: I don't even think task stealing /

Performance drop / unstable in 0.8 release

2013-11-04 Thread Wenlei Xie
Hi, I have some iterative program written in Spark and have been tested under a snapshot version of spark 0.8 before. After I ported it to the 0.8 release version, I see performance drops in large datasets. I am wondering if there is any clue? I monitored the number of partitions on each