Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/4696#discussion_r25101857
--- Diff: docs/programming-guide.md ---
@@ -728,6 +728,63 @@ def doStuff(self, rdd):
</div>
+### Understanding closures
+One of the harder things about Spark is understanding the scope and life
cycle of variables and methods when executing code across a cluster. A frequent
source of confusion is shown below - where we perform a common task
(incrementing a counter from inside of a for-loop). In our example, we look at
`foreach()` but this same scenario will apply to any other RDD operations that
modify variables outside of their scope.
--- End diff --
If you drop the parenthetical from the second sentence, then I don't think
it makes much sense anymore:
> A frequent source of confusion is shown below - where we perform a common
task.
This paragraph might read a little better if we move the general subject of
the example a little bit farther forward, to say something like
> RDD operations that modify variables outside of their scope can be a
frequent source of confusion. In the example below we'll look at code that
uses `foreach()` to increment a counter, but similar issues can occur for other
operations as well.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
