Re: ClosureCleaner slowing down Spark SQL queries

2015-05-29 Thread Josh Rosen
Hey, want to file a JIRA for this? This will make it easier to track progress on this issue. Definitely upload the profiler screenshots there, too, since that's helpful information. https://issues.apache.org/jira/browse/SPARK On Wed, May 27, 2015 at 11:12 AM, Nitin Goyal

Re: ClosureCleaner slowing down Spark SQL queries

2015-05-29 Thread Yin Huai
For Spark SQL internal operations, probably we can just create MapPartitionsRDD directly (like https://github.com/apache/spark/commit/5287eec5a6948c0c6e0baaebf35f512324c0679a ). On Fri, May 29, 2015 at 11:04 AM, Josh Rosen rosenvi...@gmail.com wrote: Hey, want to file a JIRA for this? This

Re: ClosureCleaner slowing down Spark SQL queries

2015-05-27 Thread Nitin Goyal
Hi Ted, Thanks a lot for replying. First of all, moving to 1.4.0 RC2 is not easy for us as migration cost is big since lot has changed in Spark SQL since 1.2. Regarding SPARK-7233, I had already looked at it few hours back and it solves the problem for concurrent queries but my problem is just