[
https://issues.apache.org/jira/browse/SPARK-58881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Jie updated SPARK-58881:
-----------------------------
Description:
{code:none}
$ ls /data/cp/rdd-3
part-00000 part-00000.bak part-00001
$ sc.checkpointFile[Int]("/data/cp/rdd-3").partitions
java.lang.NumberFormatException: For input string: "00000.bak"
{code}
{{ReliableCheckpointRDD.getPartitions}} filters the directory listing with
{{startsWith("part-")}} and then sorts with
{{sortBy(_.getName.stripPrefix("part-").toInt)}}. A file whose name starts with
{{part-}} but does not parse as an integer, such as a leftover
{{part-00000.bak}}, passes the filter and throws a raw
{{NumberFormatException}} out of the sort, before the contiguity check that
would raise {{INVALID_CHECKPOINT_DIRECTORY}}.
The exception carries no indication of which directory or which file is at
fault, and one unrelated leftover file makes an otherwise complete checkpoint
unreadable.
Spark never writes such a name itself: partition files come from
{{"part-%05d".format(i)}} and in-progress writes use a leading dot
({{.part-00000-attempt-7}}), which the filter already excludes. So these names
only appear when a user or an external tool leaves something behind.
Proposal: tighten the filter to accept only {{part-}} followed by digits, so a
non-numeric name is ignored rather than fatal. Note the width is not fixed at
five: {{%05d}} is a minimum width, so a job with more than 100000 partitions
produces {{part-100000}}. A pattern anchored to exactly five digits would break
those.
Names that are numeric but not zero-padded to five digits, such as {{part-1}},
already reach {{INVALID_CHECKPOINT_DIRECTORY}} today and should keep doing so.
Found while reviewing SPARK-58770 (https://github.com/apache/spark/pull/58004),
which named this condition. Left out of that PR because changing which files
fail a read is a behaviour decision rather than a naming one.
> Ignore non-numeric part-* files when reading a checkpoint directory
> -------------------------------------------------------------------
>
> Key: SPARK-58881
> URL: https://issues.apache.org/jira/browse/SPARK-58881
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 5.0.0
> Reporter: Yang Jie
> Priority: Major
>
> {code:none}
> $ ls /data/cp/rdd-3
> part-00000 part-00000.bak part-00001
> $ sc.checkpointFile[Int]("/data/cp/rdd-3").partitions
> java.lang.NumberFormatException: For input string: "00000.bak"
> {code}
> {{ReliableCheckpointRDD.getPartitions}} filters the directory listing with
> {{startsWith("part-")}} and then sorts with
> {{sortBy(_.getName.stripPrefix("part-").toInt)}}. A file whose name starts
> with {{part-}} but does not parse as an integer, such as a leftover
> {{part-00000.bak}}, passes the filter and throws a raw
> {{NumberFormatException}} out of the sort, before the contiguity check that
> would raise {{INVALID_CHECKPOINT_DIRECTORY}}.
> The exception carries no indication of which directory or which file is at
> fault, and one unrelated leftover file makes an otherwise complete checkpoint
> unreadable.
> Spark never writes such a name itself: partition files come from
> {{"part-%05d".format(i)}} and in-progress writes use a leading dot
> ({{.part-00000-attempt-7}}), which the filter already excludes. So these
> names only appear when a user or an external tool leaves something behind.
> Proposal: tighten the filter to accept only {{part-}} followed by digits, so
> a non-numeric name is ignored rather than fatal. Note the width is not fixed
> at five: {{%05d}} is a minimum width, so a job with more than 100000
> partitions produces {{part-100000}}. A pattern anchored to exactly five
> digits would break those.
> Names that are numeric but not zero-padded to five digits, such as
> {{part-1}}, already reach {{INVALID_CHECKPOINT_DIRECTORY}} today and should
> keep doing so.
> Found while reviewing SPARK-58770
> (https://github.com/apache/spark/pull/58004), which named this condition.
> Left out of that PR because changing which files fail a read is a behaviour
> decision rather than a naming one.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
