This could happen if variable is defined in such a way that it pulls its own class reference into the closure. Hence serilization tries to serialize the whole outer class reference which is not serializable and whole thing failed.
On Wed, Apr 23, 2014 at 3:15 PM, randylu <randyl...@gmail.com> wrote: > my code is like: > rdd2 = rdd1.filter(_._2.length > 1) > rdd2.collect() > it works well, but if i use a variable /num/ instead of 1: > var num = 1 > rdd2 = rdd1.filter(_._2.length > num) > rdd2.collect() > it fails at rdd2.collect() > so strange? > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/about-rdd-filter-tp4657.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > -- Sourav Chandra Senior Software Engineer · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · sourav.chan...@livestream.com o: +91 80 4121 8723 m: +91 988 699 3746 skype: sourav.chandra Livestream "Ajmera Summit", First Floor, #3/D, 68 Ward, 3rd Cross, 7th C Main, 3rd Block, Koramangala Industrial Area, Bangalore 560034 www.livestream.com
