dongjoon-hyun opened a new pull request, #59837:
URL: https://github.com/apache/spark/pull/59837
### What changes were proposed in this pull request?
This PR aims to delete the `LogInfo` listing entries of the deleted attempts
in `FsHistoryProvider.deleteAttemptLogs` again. They were removed accidentally
by SPARK-55793.
The key is the qualified path because `LogInfo` is keyed by
`reader.rootPath`, which is the qualified path returned by
`FileSystem.listStatus`, while `resolveLogPath` returns a path built from the
configured log directory string.
```scala
val (logFs, logPath) = resolveLogPath(attempt.logPath,
attempt.logSourceFullPath)
+ // LogInfo is keyed by the qualified path returned by
FileSystem.listStatus.
+ listing.synchronized {
+ listing.delete(classOf[LogInfo],
logFs.makeQualified(logPath).toString())
+ }
cleanAppData(app.id, attempt.info.attemptId, logPath.toString())
```
### Why are the changes needed?
Since SPARK-55793
([v4.2.0](https://github.com/apache/spark/releases/tag/v4.2.0), 2026-07-11),
`deleteAttemptLogs` deletes the event log files and `ApplicationInfoWrapper`,
but keeps their `LogInfo` entries until the next `checkForLogs` removes them as
stale. Since `cleanLogs` counts `LogInfo` entries for
`spark.history.fs.cleaner.maxNum` right after the max age cleanup in the same
run, the event logs deleted by the max age policy are counted again, and the
cleaner deletes more completed event logs than
`spark.history.fs.cleaner.maxNum` requires.
For example, with `spark.history.fs.cleaner.maxNum=2` and three completed
applications `app1`, `app2`, and `app3` where only `app1` exceeds
`spark.history.fs.cleaner.maxAge`, the cleaner deletes `app1` by the max age
policy, and then deletes `app2` by the max number policy although only two
event logs remain.
In addition, when an event log cannot be deleted, its `LogInfo` entry is
kept and refreshed by every scan. So the application disappears from the
listing permanently and its deletion is not retried.
Note that the original key, `new Path(logDir, attempt.logPath).toString()`,
only matched when `spark.history.fs.logDirectory` was fully-qualified. For a
log directory without a scheme or authority, such as `/path` or `hdfs:///path`
(`new Path("hdfs:///path", "app1")` is `hdfs:/path/app1` while `listStatus`
returns `hdfs://<namenode>/path/app1`), the same over-deletion has existed
since SPARK-28294 added `spark.history.fs.cleaner.maxNum` in
[v3.0.0](https://github.com/apache/spark/releases/tag/v3.0.0) (2020-06-06).
### Does this PR introduce _any_ user-facing change?
Yes. The SHS cleaner keeps `spark.history.fs.cleaner.maxNum` event logs when
the max age policy also deletes event logs in the same run.
- For a fully-qualified `spark.history.fs.logDirectory`, this fixes a
regression in Apache Spark 4.2.0 (SPARK-55793).
- For a log directory without a scheme or authority, such as `/path` or
`hdfs:///path`, this fixes a bug that has existed since Apache Spark 3.0.0
(SPARK-28294).
### How was this patch tested?
Pass the CIs with the newly added test case.
```
$ build/sbt 'core/testOnly *RocksDBBackendFsHistoryProviderSuite -- -z
"SPARK-60581"'
```
**BEFORE**
```
[info] - SPARK-60581: log cleaner with both the max age and the maximum
number of log files *** FAILED *** (708 milliseconds)
[info] List("app3") did not contain the same elements as List("app2",
"app3") (FsHistoryProviderSuite.scala:1594)
```
**AFTER**
```
[info] - SPARK-60581: log cleaner with both the max age and the maximum
number of log files (736 milliseconds)
[info] Tests: succeeded 1, failed 0, canceled 0, ignored 0, pending 0
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
