GitHub user srowen opened a pull request:
https://github.com/apache/spark/pull/4791
SPARK-4545 [STREAMING] [WIP] If first Spark Streaming batch fails, it waits
10x batch duration before stopping
Consider failed jobs completed too, to avoid excessive waiting dur...ing
shutdown for jobs that will never succeed
@tdas I may need your help on this. You can see some more detail in the
JIRA (https://issues.apache.org/jira/browse/SPARK-4545). Basically, if an error
occurs in a streaming job and the initial job fails, and will never complete,
and the `StreamingContext` is stopped, then it will wait a long time (10x batch
duration by default) for all jobs to finish, when it's already known that this
job failed.
I do not know whether this is a valid fix, but wanted to propose this for a
look. Basically: should failed jobs also be considered completed?
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/srowen/spark SPARK-4545
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/4791.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4791
----
commit e3b4f5162d0bdbe5dd12716a86adc54a1e98292a
Author: Sean Owen <[email protected]>
Date: 2015-02-26T14:51:01Z
(WIP) Consider failed jobs completed too, to avoid excessive waiting during
shutdown for jobs that will never succeed
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
