[
https://issues.apache.org/jira/browse/SPARK-4170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron Davidson resolved SPARK-4170.
-----------------------------------
Resolution: Fixed
Assignee: Sean Owen
> Closure problems when running Scala app that "extends App"
> ----------------------------------------------------------
>
> Key: SPARK-4170
> URL: https://issues.apache.org/jira/browse/SPARK-4170
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.1.0
> Reporter: Sean Owen
> Assignee: Sean Owen
> Priority: Minor
>
> Michael Albert noted this problem on the mailing list
> (http://apache-spark-user-list.1001560.n3.nabble.com/BUG-when-running-as-quot-extends-App-quot-closures-don-t-capture-variables-td17675.html):
> {code}
> object DemoBug extends App {
> val conf = new SparkConf()
> val sc = new SparkContext(conf)
> val rdd = sc.parallelize(List("A","B","C","D"))
> val str1 = "A"
> val rslt1 = rdd.filter(x => { x != "A" }).count
> val rslt2 = rdd.filter(x => { str1 != null && x != "A" }).count
>
> println("DemoBug: rslt1 = " + rslt1 + " rslt2 = " + rslt2)
> }
> {code}
> This produces the output:
> {code}
> DemoBug: rslt1 = 3 rslt2 = 0
> {code}
> If instead there is a proper "main()", it works as expected.
> I also this week noticed that in a program which "extends App", some values
> were inexplicably null in a closure. When changing to use main(), it was fine.
> I assume there is a problem with variables not being added to the closure
> when main() doesn't appear in the standard way.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
