[GitHub] spark issue #21656: [SPARK-24677][Core]MedianHeap is empty when speculation ...
Github user squito commented on the issue: https://github.com/apache/spark/pull/21656 > Ideally I think for speculation we want to look at the task time for all stage attempts. But that is probably a bigger change then this yeah I agree, on both points. One thing which is a little tricky is that you probably want to make sure you're only counting times from different partitions -- you might times from the same partition from multiple attempts, but that shouldn't count. (or maybe we don't really care that much as its just a heuristic anyway ...) --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21656: [SPARK-24677][Core]MedianHeap is empty when speculation ...
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/21656 In this case one of the older stage attempts (that is a zombie) marked the task as successful but then the newest stage attempt checked to see if it needed to speculate. Is that correct? Ideally I think for speculation we want to look at the task time for all stage attempts. But that is probably a bigger change then this. If we aren't doing that then I think ignoring it for speculation is ok. Otherwise how hard is it to send the actual task info into here so it could use the real time the successful task took? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21656: [SPARK-24677][Core]MedianHeap is empty when speculation ...
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/21656 I assume this is really that it isn't updating successfulTaskDurations? MedianHeap is a collection, can you please update description and title to be more explicit --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21656: [SPARK-24677][Core]MedianHeap is empty when speculation ...
Github user squito commented on the issue: https://github.com/apache/spark/pull/21656 Thanks for finding this and suggesting a fix @cxzl25. But, I'm not sure it makes sense to use this duration. its not how long the task actually took to complete. I think it might make more sense to just ignore this task for speculation. I will think about it some more. cc @markhamstra @tgravescs --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21656: [SPARK-24677][Core]MedianHeap is empty when speculation ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21656 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/92631/ Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21656: [SPARK-24677][Core]MedianHeap is empty when speculation ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21656 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21656: [SPARK-24677][Core]MedianHeap is empty when speculation ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21656 **[Test build #92631 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92631/testReport)** for PR 21656 at commit [`55ddbeb`](https://github.com/apache/spark/commit/55ddbeb26085c9d8cd9c1768479d9b9acdacda2b). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21656: [SPARK-24677][Core]MedianHeap is empty when speculation ...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21656 cc @jiangxb1987 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21656: [SPARK-24677][Core]MedianHeap is empty when speculation ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21656 **[Test build #92631 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/92631/testReport)** for PR 21656 at commit [`55ddbeb`](https://github.com/apache/spark/commit/55ddbeb26085c9d8cd9c1768479d9b9acdacda2b). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21656: [SPARK-24677][Core]MedianHeap is empty when speculation ...
Github user squito commented on the issue: https://github.com/apache/spark/pull/21656 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21656: [SPARK-24677][Core]MedianHeap is empty when speculation ...
Github user cxzl25 commented on the issue: https://github.com/apache/spark/pull/21656 @maropu @cloud-fan @squito Can you trigger a test for this? This is the exception stack in the log: ``` ERROR Utils: uncaught error in thread task-scheduler-speculation, stopping SparkContext java.util.NoSuchElementException: MedianHeap is empty. at org.apache.spark.util.collection.MedianHeap.median(MedianHeap.scala:83) at org.apache.spark.scheduler.TaskSetManager.checkSpeculatableTasks(TaskSetManager.scala:968) at org.apache.spark.scheduler.Pool$$anonfun$checkSpeculatableTasks$1.apply(Pool.scala:94) at org.apache.spark.scheduler.Pool$$anonfun$checkSpeculatableTasks$1.apply(Pool.scala:93) at scala.collection.Iterator$class.foreach(Iterator.scala:742) at scala.collection.AbstractIterator.foreach(Iterator.scala:1194) at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at scala.collection.AbstractIterable.foreach(Iterable.scala:54) at org.apache.spark.scheduler.Pool.checkSpeculatableTasks(Pool.scala:93) at org.apache.spark.scheduler.Pool$$anonfun$checkSpeculatableTasks$1.apply(Pool.scala:94) at org.apache.spark.scheduler.Pool$$anonfun$checkSpeculatableTasks$1.apply(Pool.scala:93) ``` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21656: [SPARK-24677][Core]MedianHeap is empty when speculation ...
Github user cxzl25 commented on the issue: https://github.com/apache/spark/pull/21656 @maropu I have added a unit test. Can you trigger a test for this? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21656: [SPARK-24677][Core]MedianHeap is empty when speculation ...
Github user maropu commented on the issue: https://github.com/apache/spark/pull/21656 Can you add a test to check if no exception thrown in that condition with this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21656: [SPARK-24677][Core]MedianHeap is empty when speculation ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21656 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21656: [SPARK-24677][Core]MedianHeap is empty when speculation ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21656 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21656: [SPARK-24677][Core]MedianHeap is empty when speculation ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21656 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
