GitHub user tdas opened a pull request:
https://github.com/apache/spark/pull/11925
[SPARK-14109][SQL] Fix HDFSMetadataLog to fallback to FileSystem
## What changes were proposed in this pull request?
HDFSMetadataLog uses newer FileContext API to achieve atomic renaming.
However, FileContext implementations may not exist for many scheme for which
there may be FileSystem implementations. In those cases, rather than failing
completely, we should fallback to the FileSystem based implementation, and log
warning that there may be file consistency issues in case the log directory is
concurrently modified.
## How was this patch tested?
Unit test.
Cluster test pending.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/tdas/spark SPARK-14109
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/11925.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #11925
----
commit f3dade7127a7ccde96c002abc6ad75dac5337d6d
Author: Tathagata Das <[email protected]>
Date: 2016-03-24T01:40:00Z
Fix HDFSMetadataLog to fallback to FileSystem
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
