[GitHub] spark pull request: SPARK-3642. Document the nuances of shared var...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/2490 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3642. Document the nuances of shared var...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2490#issuecomment-78117663 [Test build #28433 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28433/consoleFull) for PR 2490 at commit [`aae3340`](https://github.com/apache/spark/commit/aae33403f82f4e88ec14894dd2f00244297f2ca3). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3642. Document the nuances of shared var...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2490#issuecomment-78133971 [Test build #28433 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/28433/consoleFull) for PR 2490 at commit [`aae3340`](https://github.com/apache/spark/commit/aae33403f82f4e88ec14894dd2f00244297f2ca3). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3642. Document the nuances of shared var...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2490#issuecomment-78133978 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/28433/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3642. Document the nuances of shared var...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/2490#issuecomment-78158596 LGTM since the PR now represents the just the first paragraph that Josh alludes to above, and he approved that much. I'll leave it open a little while for more comments just in case. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3642. Document the nuances of shared var...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/2490#issuecomment-73857356 Sandy do you want to remove or update the 2nd paragraph? can be merged then, it looks like. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3642. Document the nuances of shared var...
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/2490#discussion_r22244391 --- Diff: docs/programming-guide.md --- @@ -1183,6 +1189,10 @@ running on the cluster can then add to it using the `add` method or the `+=` ope However, they cannot read its value. Only the driver program can read the accumulator's value, using its `value` method. +An operation referencing an accumulator may run multiple times, either when parts of its output +data become lost or when multiple actions make use of the RDD it produces. In these cases, only the +additions reported by the first successful execution contribute to the accumulator's value. --- End diff -- I was going to suggest that this might not be true, but it looks like this is partially-true now that #2524 has been merged. That PR added a paragraph to the programming guide which clarifies that we guard against duplicate updates only for updates performed inside of actions and _not_ for ones performed in transformations: https://github.com/apache/spark/commit/66cc2431462a5354bb50c196a59da0ffc258c466?diff=unified#diff-3 In light of this, do we still need this paragraph? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3642. Document the nuances of shared var...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2490#issuecomment-68020326 Sorry for letting this sit for so long; I'm working my way through the backlog now, though. I think that the second addition, RE: accumulator updates, may no longer be necessary / may be subsumed by changes in other PRs, but the first paragraph RE: broadcast variables still looks good. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3642. Document the nuances of shared var...
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/2490#issuecomment-63148833 Thanks for the review @Ishiihara . Updated the PR to clarify these points. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3642. Document the nuances of shared var...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2490#issuecomment-63149160 [Test build #23400 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23400/consoleFull) for PR 2490 at commit [`2a81019`](https://github.com/apache/spark/commit/2a81019bee4be16623ed56f3678b8316a663f01b). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3642. Document the nuances of shared var...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2490#issuecomment-63154619 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23400/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3642. Document the nuances of shared var...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2490#issuecomment-63154615 [Test build #23400 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23400/consoleFull) for PR 2490 at commit [`2a81019`](https://github.com/apache/spark/commit/2a81019bee4be16623ed56f3678b8316a663f01b). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3642. Document the nuances of shared var...
Github user Ishiihara commented on a diff in the pull request: https://github.com/apache/spark/pull/2490#discussion_r17959346 --- Diff: docs/programming-guide.md --- @@ -1121,6 +1121,11 @@ than shipping a copy of it with tasks. They can be used, for example, to give ev large input dataset in an efficient manner. Spark also attempts to distribute broadcast variables using efficient broadcast algorithms to reduce communication cost. +Spark automatically broadcasts the common data needed by tasks within each stage. The data +broadcasted this way is cached in serialized form and deserialized before running each task. This +means that explicitly creating broadcast variables is only useful when tasks across multiple stages --- End diff -- The concept of stage is mentioned only in the two added paragraphs. Users new to Spark may not know the internals and the execution mmechanism. It would be nice to if some background is introduced here. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3642. Document the nuances of shared var...
Github user Ishiihara commented on a diff in the pull request: https://github.com/apache/spark/pull/2490#discussion_r17959656 --- Diff: docs/programming-guide.md --- @@ -1183,6 +1188,10 @@ running on the cluster can then add to it using the `add` method or the `+=` ope However, they cannot read its value. Only the driver program can read the accumulator's value, using its `value` method. +The same task may run multiple times, either when its output data becomes lost or when multiple --- End diff -- The same task can mean the same task id or the same computation in a stage. Two tasks that have the some computation may have different task id. It would be nice if some backgrounds is introduced here, eg like the relationship between stage and task sets. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3642. Document the nuances of shared var...
Github user CodingCat commented on a diff in the pull request: https://github.com/apache/spark/pull/2490#discussion_r17949828 --- Diff: docs/programming-guide.md --- @@ -1183,6 +1188,10 @@ running on the cluster can then add to it using the `add` method or the `+=` ope However, they cannot read its value. Only the driver program can read the accumulator's value, using its `value` method. +The same task may run multiple times, either when its output data becomes lost or when multiple +actions make use of the same stage. In these cases, only the additions reported by the first +successful task contribute to the accumulator's value. --- End diff -- @sryza Actually this is not true scala val acc = sc.accumulator(0) scala val data = sc.parallelize(List(1, 2, 3)) data: org.apache.spark.rdd.RDD[Int] = ParallelCollectionRDD[0] at parallelize at console:12 scala val a = data.map(x = acc += 1) scala a.count scala acc.value res5: Int = 3 scala a.count scala acc.value res7: Int = 6 I will resubmit https://github.com/apache/spark/pull/228 tonight or tomorrow, --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3642. Document the nuances of shared var...
GitHub user sryza opened a pull request: https://github.com/apache/spark/pull/2490 SPARK-3642. Document the nuances of shared variables. You can merge this pull request into a Git repository by running: $ git pull https://github.com/sryza/spark sandy-spark-3642 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2490.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2490 commit cb9ffad7db198917420ef694b7fc7ddf91a7b9bc Author: Sandy Ryza sa...@cloudera.com Date: 2014-09-22T16:53:22Z SPARK-3642. Document the nuances of shared variables. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3642. Document the nuances of shared var...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2490#issuecomment-56404364 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20656/consoleFull) for PR 2490 at commit [`cb9ffad`](https://github.com/apache/spark/commit/cb9ffad7db198917420ef694b7fc7ddf91a7b9bc). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-3642. Document the nuances of shared var...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2490#issuecomment-56414048 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/20656/consoleFull) for PR 2490 at commit [`cb9ffad`](https://github.com/apache/spark/commit/cb9ffad7db198917420ef694b7fc7ddf91a7b9bc). * This patch **passes** unit tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
