Assaf Mendelson created SPARK-19045:
---------------------------------------
Summary: irrelevant warning when creating a checkpoint dir
Key: SPARK-19045
URL: https://issues.apache.org/jira/browse/SPARK-19045
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 2.1.0
Reporter: Assaf Mendelson
Priority: Trivial
When I do:
sc.setCheckpointDir("/mydir") I get a warning:
17/01/01 06:26:42 WARN SparkContext: Spark is not running in local mode,
therefore the checkpoint directory must not be on the local filesystem.
Directory '/mydir' appears to be on the local filesystem.
This occurs even though I use a single computer with local filesystem (using
spark standalone). Reading the code it seems the same error would occur if I
would use a cluster and an NFS share.
Checkpoint should work on a local directory as long as there is a single node
(even when using a resource manager rather than a single node) and should work
in a cluster with locally mounted NFS share.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
