[GitHub] spark issue #18594: [SPARK-20904][core] Don't report task failures to driver...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18594 LGTM, merging to master/2.2! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18594: [SPARK-20904][core] Don't report task failures to driver...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18594 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79852/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18594: [SPARK-20904][core] Don't report task failures to driver...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18594 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18594: [SPARK-20904][core] Don't report task failures to driver...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18594 **[Test build #79852 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79852/testReport)** for PR 18594 at commit [`a68c2f2`](https://github.com/apache/spark/commit/a68c2f2478f190ac56a491801c98ebda862605a6). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18594: [SPARK-20904][core] Don't report task failures to driver...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18594 **[Test build #79852 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79852/testReport)** for PR 18594 at commit [`a68c2f2`](https://github.com/apache/spark/commit/a68c2f2478f190ac56a491801c98ebda862605a6). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18594: [SPARK-20904][core] Don't report task failures to driver...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/18594 @squito --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18594: [SPARK-20904][core] Don't report task failures to driver...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/18594 I don't think you understand what the change is doing. The task will still fail, because the executor is dying. The only thing that changes is the failure reason, which will now be "Executor lost", which is actually more correct (any failure caused by races in shutdown are basically because the executor is dying). That allows the driver to ignore the failure in certain cases like it already does (e.g. YARN preempting executors). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18594: [SPARK-20904][core] Don't report task failures to driver...
Github user jiangxb1987 commented on the issue: https://github.com/apache/spark/pull/18594 I'm hesitant to support the change. If we don't notify the failure to driver, the status of the failed task would not be updated, thus not rescheduled, perhaps it's not the behavior we expect to see? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18594: [SPARK-20904][core] Don't report task failures to driver...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18594 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79485/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18594: [SPARK-20904][core] Don't report task failures to driver...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18594 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18594: [SPARK-20904][core] Don't report task failures to driver...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18594 **[Test build #79485 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79485/testReport)** for PR 18594 at commit [`76de32a`](https://github.com/apache/spark/commit/76de32a22cda3edab5f6e7baa12af80112715051). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18594: [SPARK-20904][core] Don't report task failures to driver...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18594 **[Test build #79485 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79485/testReport)** for PR 18594 at commit [`76de32a`](https://github.com/apache/spark/commit/76de32a22cda3edab5f6e7baa12af80112715051). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18594: [SPARK-20904][core] Don't report task failures to driver...
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/18594
Here's the hacky test code for the interested:
```
sc.parallelize(1 to 100, 100).foreach { _ =>
var shuttingDown = false
while (!shuttingDown) {
try {
val hook = new Thread {
override def run() {}
}
// scalastyle:off runtimeaddshutdownhook
Runtime.getRuntime.addShutdownHook(hook)
// scalastyle:on runtimeaddshutdownhook
Runtime.getRuntime.removeShutdownHook(hook)
Thread.sleep(10)
} catch {
case ise: IllegalStateException => shuttingDown = true
}
}
throw new Exception("Task failure during shutdown.")
}
```
Ran that in two shells, one in a low priority queue and one in a high
priority one, restarting the high priority one to force several rounds of
executors being killed by preemption in the low priority queue.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
