This is an automated email from the ASF dual-hosted git repository.
uros-b pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new ffb268f6c57f [SPARK-58881][CORE][FOLLOWUP] Document checkpoint
part-file filtering
ffb268f6c57f is described below
commit ffb268f6c57fab88b82e8f13af8ab7bcd42ec614
Author: Charlie Chen <[email protected]>
AuthorDate: Mon Oct 5 13:03:02 2026 +0000
[SPARK-58881][CORE][FOLLOWUP] Document checkpoint part-file filtering
### What changes were proposed in this pull request?
Add one entry to the Spark Core 4.3-to-4.4 migration guide describing the
RDD checkpoint-file filtering change from #58535.
The note explains that names such as `part-00000.bak`, `part-backup`, and
`part-` are now ignored instead of causing `NumberFormatException`. Validation
of numeric partition names and the contiguous partition sequence is unchanged.
### Why are the changes needed?
This documents the behavior change already merged for
[SPARK-58881](https://issues.apache.org/jira/browse/SPARK-58881), following the
[post-merge review
request](https://github.com/apache/spark/pull/58535#issuecomment-5870625252).
The fix was merged into master and branch-4.x; the latter targets Spark 4.4.
### Does this PR introduce _any_ user-facing change?
No. Documentation only; no additional runtime behavior change.
### How was this patch tested?
- Checked the wording against the merged
`ReliableCheckpointRDD.getPartitions` / `isCheckpointFile` implementation and
the SPARK-58881 regression in `CheckpointStorageSuite`.
- `git diff --check` passed; the only changed file is
`docs/core-migration-guide.md`.
- Parsed the page's YAML front matter and checked for unresolved conflict
markers.
- Did not rerun runtime tests because no code changes.
- The full Jekyll documentation build was not run locally: the machine has
Ruby 2.6 and does not have the required Bundler 2.4.22 / Ruby 3 documentation
environment. Remote documentation CI is pending.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-5)
This contribution is submitted under the project's Apache License 2.0.
Closes #59191 from charliec05/codex/spark-58881-checkpoint-migration.
Authored-by: Charlie Chen <[email protected]>
Signed-off-by: Uros Bojanic <[email protected]>
---
docs/core-migration-guide.md | 2 ++
1 file changed, 2 insertions(+)
diff --git a/docs/core-migration-guide.md b/docs/core-migration-guide.md
index 6650f4f02c4a..8fac8b3d4902 100644
--- a/docs/core-migration-guide.md
+++ b/docs/core-migration-guide.md
@@ -24,6 +24,8 @@ license: |
## Upgrading from Core 4.3 to 4.4
+- Since Spark 4.4, reading an RDD checkpoint ignores names beginning with
`part-` unless the suffix consists of one or more ASCII digits. Previously,
names such as `part-00000.bak`, `part-backup`, or `part-` caused checkpoint
recovery to fail with `NumberFormatException`. Validation of numeric partition
names and the contiguous partition sequence is unchanged.
+
- Since Spark 4.4, an executor pod template that names a service account in
`serviceAccountName` keeps it: Spark applies
`spark.kubernetes.authenticate.executor.serviceAccountName`, or
`spark.kubernetes.authenticate.driver.serviceAccountName` as a fallback, only
when the template names no account in either `serviceAccount` or
`serviceAccountName`. Earlier versions decided by reading the deprecated
`serviceAccount` field alone, so with either configuration set, a template that
named the a [...]
- Since Spark 4.4, an empty service account name in an executor pod template
counts as naming no account, which is how Kubernetes itself defaults the two
fields. A template whose deprecated `serviceAccount` field is present but
empty, with `serviceAccountName` empty or absent, used to count as naming an
account: the configured account was dropped and the executor pods ran as the
namespace's default account. They now run as
`spark.kubernetes.authenticate.executor.serviceAccountName` or, f [...]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
