Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/4696#discussion_r25101943
--- Diff: docs/programming-guide.md ---
@@ -728,6 +728,63 @@ def doStuff(self, rdd):
</div>
+### Understanding closures
+One of the harder things about Spark is understanding the scope and life
cycle of variables and methods when executing code across a cluster. A frequent
source of confusion is shown below - where we perform a common task
(incrementing a counter from inside of a for-loop). In our example, we look at
`foreach()` but this same scenario will apply to any other RDD operations that
modify variables outside of their scope.
+
+#### Example
+
+Consider the naiive RDD element sum below which, behaves completely
differently when running spark in `local` mode (e.g. via the shell) and when
deploying a Spark application to a cluster (e.g. via spark-submit to YARN):
+
+<div class="codetabs">
+
+<div data-lang="scala" markdown="1">
+{% highlight scala %}
+var counter = 0
+var rdd = sc.parallelize(data)
+rdd.foreach(x => counter += x)
--- End diff --
Can we add code comments to these examples that say `// wrong: don't do
this!` or something similar to indicate that this is an anti-pattern?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
