[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user aarondav commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-64813367 @mateiz @CodingCat Apologies, but can I confirm that the scope of this change is strictly to ensure that actions/result stages never duplicate accumulator updates? The PR title and description are more general than this, but the associated JIRAs suggest the restricted scope. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user CodingCat commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-64813653 yes, originally, I tried to do it for both shuffletask and resultask, later, @mateiz convinced me that we actually cannot handle transformation case so the current change only involves result task, apologize for not changing the PR title on time --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-64841516 Yes, it should be only SPARK-3628. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-64603187 [Test build #23893 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23893/consoleFull) for PR 2524 at commit [`b233737`](https://github.com/apache/spark/commit/b23373746407de3a7c05fbeaf00f67fe78860623). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-64654772 [Test build #23893 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23893/consoleFull) for PR 2524 at commit [`b233737`](https://github.com/apache/spark/commit/b23373746407de3a7c05fbeaf00f67fe78860623). * This patch **fails MiMa tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-64654780 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23893/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user CodingCat commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-64655766 Hey, @mateiz , thank you very much for the review, I addressed all of them except the lastId one, as MIMA wants me to keep that since it's public. also, a question for you, shall I submit the patch to the old version branches, since there are some merge conflicts preventing the patch directly to there --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-64655857 [Test build #23895 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23895/consoleFull) for PR 2524 at commit [`1433e6f`](https://github.com/apache/spark/commit/1433e6f191e321241a1daef3f70dcbf329ed7079). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-64668708 [Test build #23895 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23895/consoleFull) for PR 2524 at commit [`1433e6f`](https://github.com/apache/spark/commit/1433e6f191e321241a1daef3f70dcbf329ed7079). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-64668716 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23895/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-64717507 Can you just not change Accumulator.scala then? That change isn't fixing any kind of bug, it's just a small optimization. Just remove it from this patch. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-64722162 [Test build #23908 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23908/consoleFull) for PR 2524 at commit [`701a1e8`](https://github.com/apache/spark/commit/701a1e8d9e9f5d415bbaa8782a790012859a329a). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user CodingCat commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-64723290 @mateiz sure, just rollback the changes...how about the question to apply the patch to other branches? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-64728890 Don't worry about the other branches now, we can figure it out if we want to backport it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-64728951 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23908/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-64728943 [Test build #23908 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23908/consoleFull) for PR 2524 at commit [`701a1e8`](https://github.com/apache/spark/commit/701a1e8d9e9f5d415bbaa8782a790012859a329a). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/2524 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user CodingCat commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-64484066 @mateiz mind taking further review? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user mateiz commented on a diff in the pull request:
https://github.com/apache/spark/pull/2524#discussion_r20910810
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -21,7 +21,7 @@ import java.io.NotSerializableException
import java.util.Properties
import java.util.concurrent.atomic.AtomicInteger
-import scala.collection.mutable.{ArrayBuffer, HashMap, HashSet, Map, Stack}
+import scala.collection.mutable.{ArrayBuffer, HashMap, HashSet, Map,
Stack, ListBuffer}
--- End diff --
Why was ListBuffer added?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user mateiz commented on a diff in the pull request:
https://github.com/apache/spark/pull/2524#discussion_r20910947
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -901,6 +900,33 @@ class DAGScheduler(
}
}
+ private def updateAccumulator(event: CompletionEvent): Unit = {
--- End diff --
Call this updateAccumulators and add a comment saying ```/** Merge updates
from a task to our local accumulator values */```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user mateiz commented on a diff in the pull request: https://github.com/apache/spark/pull/2524#discussion_r20911158 --- Diff: docs/programming-guide.md --- @@ -1228,6 +1228,11 @@ interface to accumulate data where the resulting type is not the same as the ele a list by collecting together elements), and the `SparkContext.accumulableCollection` method for accumulating common Scala collection types. +bOnly when the accumulator operation is executed within an +action/b, Spark guarantees that the operation will only be applied when the task is successfully finished for +the first time, i.e. the restarted task will not update the value. In transformations, users should be aware of that +the accumulator value would be updated as long as the task is executed. --- End diff -- Thanks for adding this, but it's better to tweak the wording a bit like this: ``` For accumulator updates performed inside bactions only/b, Spark guarantees that each task's update to the accumulator will only be applied once, i.e. restarted tasks will not update the value. In transformations, users should be aware of that each task's update may be applied more than once if tasks or job stages are re-executed. ``` Also, move this paragraph below the language-specific `div`s in the text. Right now it's only going to show up in the Scala version of the docs. Note that there are `div`s below this for Java and Python. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user mateiz commented on a diff in the pull request:
https://github.com/apache/spark/pull/2524#discussion_r20911191
--- Diff: core/src/main/scala/org/apache/spark/Accumulators.scala ---
@@ -252,10 +254,9 @@ private object Accumulators {
val localAccums = Map[Thread, Map[Long, Accumulable[_, _]]]()
var lastId: Long = 0
--- End diff --
You can delete the lastId variable if you use AtomicLong.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user mateiz commented on a diff in the pull request:
https://github.com/apache/spark/pull/2524#discussion_r20911221
--- Diff: core/src/main/scala/org/apache/spark/Accumulators.scala ---
@@ -252,10 +254,9 @@ private object Accumulators {
val localAccums = Map[Thread, Map[Long, Accumulable[_, _]]]()
var lastId: Long = 0
- def newId: Long = synchronized {
-lastId += 1
-lastId
- }
+ private val nextAccumID = new AtomicLong(0)
+
+ def newId: Long = nextAccumID.getAndIncrement
--- End diff --
Nit: this should be called `newId()` since it has side-effects (I know it
wasn't called that before but might as well fix it)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-64502369 @CodingCat thanks for the update, this looks good. I just made a few small comments. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-64203092 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23783/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-64203082 [Test build #23783 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23783/consoleFull) for PR 2524 at commit [`02261b8`](https://github.com/apache/spark/commit/02261b80bf0e0c36cb9756c1fad6cce03b46651a). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user pwendell commented on a diff in the pull request:
https://github.com/apache/spark/pull/2524#discussion_r20774638
--- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
---
@@ -409,6 +409,31 @@ class DAGScheduler(
updateJobIdStageIdMapsList(List(stage))
}
+ def removeStage(stageId: Int) {
--- End diff --
can this be put back where it was before? We should try to minimize the
surface area of the change.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-64091985 [Test build #23745 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23745/consoleFull) for PR 2524 at commit [`881bf52`](https://github.com/apache/spark/commit/881bf52b41624d727eae1b1107f350925f9dafe9). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-64093397 [Test build #23746 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23746/consoleFull) for PR 2524 at commit [`21b6840`](https://github.com/apache/spark/commit/21b6840d2727305b2c758abfffb73f5179d75f0f). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-64094285 [Test build #23748 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23748/consoleFull) for PR 2524 at commit [`1e9e14d`](https://github.com/apache/spark/commit/1e9e14de882560ec3b4338e61abeeb0a05905963). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-64095991 **[Test build #23745 timed out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23745/consoleFull)** for PR 2524 at commit [`881bf52`](https://github.com/apache/spark/commit/881bf52b41624d727eae1b1107f350925f9dafe9) after a configured wait of `120m`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-64095993 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23745/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-64096538 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23748/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-64096534 [Test build #23748 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23748/consoleFull) for PR 2524 at commit [`1e9e14d`](https://github.com/apache/spark/commit/1e9e14de882560ec3b4338e61abeeb0a05905963). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-64097342 **[Test build #23746 timed out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23746/consoleFull)** for PR 2524 at commit [`21b6840`](https://github.com/apache/spark/commit/21b6840d2727305b2c758abfffb73f5179d75f0f) after a configured wait of `120m`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-64097346 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23746/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-64097507 [Test build #23749 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23749/consoleFull) for PR 2524 at commit [`84570d2`](https://github.com/apache/spark/commit/84570d24c546af44282e15d86227fe6c0ccfb56b). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-64097551 [Test build #23749 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23749/consoleFull) for PR 2524 at commit [`84570d2`](https://github.com/apache/spark/commit/84570d24c546af44282e15d86227fe6c0ccfb56b). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-64097552 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23749/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-64097773 [Test build #23750 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23750/consoleFull) for PR 2524 at commit [`83b75f8`](https://github.com/apache/spark/commit/83b75f8564aa6d50e05a0a9b8bfd66bd65277af0). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-64099094 [Test build #23751 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23751/consoleFull) for PR 2524 at commit [`2b2e8cf`](https://github.com/apache/spark/commit/2b2e8cf14ab18b80ea37250fb020af2fbaf9cf32). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-64099557 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23750/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-64099556 [Test build #23750 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23750/consoleFull) for PR 2524 at commit [`83b75f8`](https://github.com/apache/spark/commit/83b75f8564aa6d50e05a0a9b8bfd66bd65277af0). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-64101196 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23751/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-64101193 [Test build #23751 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23751/consoleFull) for PR 2524 at commit [`2b2e8cf`](https://github.com/apache/spark/commit/2b2e8cf14ab18b80ea37250fb020af2fbaf9cf32). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-64104262 [Test build #23757 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23757/consoleFull) for PR 2524 at commit [`6b0aff9`](https://github.com/apache/spark/commit/6b0aff9eedd5186e3d148589f071d7b70c9cd15e). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-64105981 [Test build #23757 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23757/consoleFull) for PR 2524 at commit [`6b0aff9`](https://github.com/apache/spark/commit/6b0aff9eedd5186e3d148589f071d7b70c9cd15e). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-64105983 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/23757/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user mateiz commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-63862394 @CodingCat I think we discussed in https://issues.apache.org/jira/browse/SPARK-3628 that it would be best to do this only for result stages first. Can you do that? The reason is that we can't fully guarantee these semantics for transformations, for two reasons: * A shuffle stage may be resubmitted once the old one is garbage-collected (if periodic cleanup is on) * If you use an accumulator in a pipelined transformation like a map(), and then you make a new RDD built on top of that (e.g. apply another map() to it), it won't count as the same stage so you'll still get the updates twice I think we can clarify our documentation to say accumulators offer this guarantee only in actions, and should be used more as counters in other settings. It would also lead to a *much* simpler patch, which is highly preferred for a bug fix. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user mateiz commented on a diff in the pull request:
https://github.com/apache/spark/pull/2524#discussion_r20670036
--- Diff: core/src/main/scala/org/apache/spark/Accumulators.scala ---
@@ -282,7 +285,6 @@ private object Accumulators {
return ret
}
- // Add values to the original accumulators with some given IDs
--- End diff --
Why was this comment removed?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user mateiz commented on a diff in the pull request:
https://github.com/apache/spark/pull/2524#discussion_r20670065
--- Diff: core/src/main/scala/org/apache/spark/Accumulators.scala ---
@@ -226,9 +227,12 @@ GrowableAccumulableParam[R % Growable[T] with
TraversableOnce[T] with Serializa
* @param param helper object defining how to add elements of type `T`
* @tparam T result type
*/
-class Accumulator[T](@transient initialValue: T, param:
AccumulatorParam[T], name: Option[String])
+class Accumulator[T](@transient initialValue: T, param:
AccumulatorParam[T],
+ name: Option[String])
extends Accumulable[T,T](initialValue, param, name) {
- def this(initialValue: T, param: AccumulatorParam[T]) =
this(initialValue, param, None)
+
+ def this(initialValue: T, param: AccumulatorParam[T]) =
+this(initialValue, param, None)
--- End diff --
Why was formatting changed here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user mateiz commented on the pull request:
https://github.com/apache/spark/pull/2524#issuecomment-63863153
BTW when doing this only for result stages, my suggestion is to use the
data structures within the stage instead of having a second HashMap. I believe
I mentioned this before too (maybe on the previous PR): all you need to do is
move the accumulator update code within the `if (!job.finished(rt.outputId)) {`
for such stages, similar to how it only fetches results once for each task.
Again the point is to avoid adding a new data structure in DAGScheduler that we
must then carefully manage and clean up.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user CodingCat commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-63866302 @mateiz , I see...I had the impression that we agreed on still support shuffle stage deduplication finally... OK, I can shrink this patch to only support result stage --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user CodingCat commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-61878006 ping --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-61476278 [Test build #22814 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22814/consoleFull) for PR 2524 at commit [`f317724`](https://github.com/apache/spark/commit/f317724c77ccb56267844a6d6d70a062c1f2a391). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-61481128 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22814/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-61481119 [Test build #22814 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22814/consoleFull) for PR 2524 at commit [`f317724`](https://github.com/apache/spark/commit/f317724c77ccb56267844a6d6d70a062c1f2a391). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class Accumulator[T](@transient initialValue: T, param: AccumulatorParam[T],` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-61484146 [Test build #22819 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22819/consoleFull) for PR 2524 at commit [`d35d59b`](https://github.com/apache/spark/commit/d35d59b0bc1d32302fb0f696004e76cfebc4ab64). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-61494370 [Test build #22819 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22819/consoleFull) for PR 2524 at commit [`d35d59b`](https://github.com/apache/spark/commit/d35d59b0bc1d32302fb0f696004e76cfebc4ab64). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class Accumulator[T](@transient initialValue: T, param: AccumulatorParam[T],` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user CodingCat commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-61494925 still flaky test...@mateiz, shall we get this merged before 1.2 release? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-61494386 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22819/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-61560296 [Test build #22830 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22830/consoleFull) for PR 2524 at commit [`f33df88`](https://github.com/apache/spark/commit/f33df888b5bcc27df45dad839dea6bb74b62898e). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-61570767 [Test build #22830 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22830/consoleFull) for PR 2524 at commit [`f33df88`](https://github.com/apache/spark/commit/f33df888b5bcc27df45dad839dea6bb74b62898e). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class Accumulator[T](@transient initialValue: T, param: AccumulatorParam[T],` * `class NullType(PrimitiveType):` * ` case class ScalaUdfBuilder[T: TypeTag](f: AnyRef) ` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-61570773 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22830/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user CodingCat commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-60382812 Hi, @mateiz @markhamstra , you want to take further review? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-60452609 [Test build #22168 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22168/consoleFull) for PR 2524 at commit [`a2e0ad7`](https://github.com/apache/spark/commit/a2e0ad7daf8c9c3c27114265174ad34dc6da460a). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-60457714 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22168/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-732][SPARK-3628][CORE][RESUBMIT] elimin...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2524#issuecomment-60457705 [Test build #22168 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22168/consoleFull) for PR 2524 at commit [`a2e0ad7`](https://github.com/apache/spark/commit/a2e0ad7daf8c9c3c27114265174ad34dc6da460a). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
