GitHub user tmgstevens opened a pull request:
https://github.com/apache/spark/pull/19645
[SPARK-22429][STREAMING] Streaming checkpointing code does not retry after
failure
## What changes were proposed in this pull request?
SPARK-14930/SPARK-13693 put in a change to set the fs object to null after
a failure, however the retry loop does not include initialization. Moved fs
initialization inside the retry while loop to aid recoverability.
## How was this patch tested?
Passes all existing unit tests.
(Please explain how this patch was tested. E.g. unit tests, integration
tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise,
remove this)
Please review http://spark.apache.org/contributing.html before opening a
pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tmgstevens/spark SPARK-22429
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19645.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19645
----
commit 7b26f5f1cd56b65622afea6fac215d23d815c67e
Author: Tristan Stevens <[email protected]>
Date: 2017-11-02T19:30:55Z
Fix to pull fs initialization into the retry logic
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
