Re: java.io.NotSerializableException on RDD count() in Java

2013-11-07 Thread Patrick Wendell
No problem - thanks for helping us diagnose this! On Tue, Nov 5, 2013 at 5:04 AM, Yadid Ayzenberg ya...@media.mit.edu wrote: Ah, I see. Thanks very much for you assistance Patrick and Reynold. As a workaround for now, I implemented the SC field as transient and its working fine. Yadid On

Re: java.io.NotSerializableException on RDD count() in Java

2013-11-03 Thread Yadid Ayzenberg
code is below. in the code rdd.count() works, but rdd2.count() fails. public class AnalyticsEngine implements Serializable { private static AnalyticsEngine engine=null; private JavaSparkContext sc; final Logger logger = LoggerFactory.getLogger(AnalyticsEngine.class); private

Re: java.io.NotSerializableException on RDD count() in Java

2013-11-03 Thread Patrick Wendell
Hm, I think you are triggering a bug in the Java API where closures may not be properly cleaned. I think @rxin has reproduced this, deferring to him. - Patrick On Sun, Nov 3, 2013 at 5:25 PM, Yadid Ayzenberg ya...@media.mit.edu wrote: code is below. in the code rdd.count() works, but