GitHub user mgaido91 opened a pull request:
https://github.com/apache/spark/pull/18329
[SPARK-19909][SS] Disabling the usage of a temporary directory for the
checkpoint location if the temporary directory is on a filesystem different
from the default one.
## What changes were proposed in this pull request?
As stated in
[SPARK-19909](https://issues.apache.org/jira/browse/SPARK-19909), if the
`checkpointLocation` is not set, for some formats Spark falls back using a
directory in the `java.io.tmp` dir to store the metadata. This fails with a
weird exception (permission denied) if the default filesystem is different from
the temp one. This is the case if the default filesystem is HDFS, for instance,
and the temp directory is written on the local filesystem.
The change proposed in this PR disables the usage of the temp directory in
this case. Thus, a meaningful `AnalysisException` is thrown, suggesting the
proper fix to the user, i.e. to set a value for `checkpointLocation`, instead
of a `PermissionDenied` exception which is hard to understand and fix.
This behavior is consistent with what happens in all the cases in which the
`checkpointLocation` needs to be set.
## How was this patch tested?
A unit test was added to check the patch and all the unit tests have been
run.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mgaido91/spark master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/18329.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #18329
----
commit 1fc4a954c08296ed24ca90b7dde0f1cfaedf572f
Author: Marco Gaido <[email protected]>
Date: 2017-06-16T14:22:25Z
[SPARK-19909][SS] Disabling the usage of a temporary directory for the
checkpoint location if the temporary directory is on a filesystem different
from the default one.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
