[ https://issues.apache.org/jira/browse/SPARK-7121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Patrick Wendell resolved SPARK-7121. ------------------------------------ Resolution: Fixed Fix Version/s: 1.4.0 > ClosureCleaner does not handle nesting properly > ----------------------------------------------- > > Key: SPARK-7121 > URL: https://issues.apache.org/jira/browse/SPARK-7121 > Project: Spark > Issue Type: Bug > Components: Spark Core > Reporter: Andrew Or > Assignee: Andrew Or > Fix For: 1.4.0 > > > For instance, in SparkContext, I tried to do the following: > {code} > def scope[T](body: => T): T = body // no-op > def myCoolMethod(path: String): RDD[String] = scope { > parallelize(1 to 10).map { _ => path } > } > {code} > and I got an exception complaining that SparkContext is not serializable. The > issue here is that the inner closure is getting its path from the outer > closure (the scope), but the outer closure actually references the > SparkContext object itself to get the `parallelize` method. > Note, however, that the inner closure doesn't actually need the SparkContext; > it just needs a field from the outer closure. If we modify ClosureCleaner to > clean the outer closure recursively while using the fields accessed by the > inner closure, then we can serialize the inner closure. > This is blocking my effort on a visualization task. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org