Andrew Or created SPARK-8584:
--------------------------------
Summary: Better exception message if invalid checkpoint dir is
specified
Key: SPARK-8584
URL: https://issues.apache.org/jira/browse/SPARK-8584
Project: Spark
Issue Type: Bug
Components: Spark Core
Affects Versions: 1.0.0
Reporter: Andrew Or
Assignee: Andrew Or
If we're running Spark on a cluster, the checkpoint dir must be a non-local
path. Otherwise, the attempt to read from a checkpoint will fail because the
checkpoint files are written on the executors, not on the driver.
Currently, the error message that you get looks something like the following,
which is not super intuitive:
{code}
Checkpoint RDD 3 (0) has different number of partitions than original RDD 2
(100)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
