GitHub user pierre-borckmans opened a pull request:
https://github.com/apache/spark/pull/10392
[SPARK-12440] - [core] - Avoid setCheckpoint warning when directory is not
local
In SparkContext method `setCheckpointDir`, a warning is issued when spark
master is not local and the passed directory for the checkpoint dir appears to
be local.
In practice, when relying on HDFS configuration file and using relative
path (incomplete URI without HDFS scheme, ...), this warning should not be
issued and might be confusing.
This PR creates a `FileSystem` with the given directory, and check whether
it is local or not.
The warning is only issued if the directory is not local.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/pierre-borckmans/spark
SPARK-12440_CheckpointDir_Warning_NonLocal
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/10392.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10392
----
commit bace5b98d20a9f7d6876b65120e6668aa064c948
Author: pierre-borckmans <[email protected]>
Date: 2015-12-19T16:55:00Z
AVOID setCheckpoint warning when file system is not local
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
