GitHub user tdas opened a pull request:
https://github.com/apache/spark/pull/5732
[SPARK-7139][Streaming] Allow received block metadata to be saved to WAL
and recovered on driver failure
- Enabled ReceivedBlockTracker WAL by default
- Stored block metadata in the WAL
- Optimized WALBackedBlockRDD by skipping block fetch when the block is
known to not exist in Spark
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tdas/spark SPARK-7139
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/5732.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #5732
----
commit d06fa21f80eeda04c11e454a461902717a2e5c7d
Author: Tathagata Das <[email protected]>
Date: 2015-04-28T03:33:59Z
Enabled ReceivedBlockTracker by default, stored block metadata and
optimized block fetching in WALBackedBlockRDD
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
