This is an automated email from the ASF dual-hosted git repository.
uros-b pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new fd3169438dbd [MINOR][SS] Use filterNot in HDFSBackedStateStoreProvider
snapshot filtering
fd3169438dbd is described below
commit fd3169438dbdada4623d310380b7c300543833af
Author: Uros Bojanic <[email protected]>
AuthorDate: Sun Jul 26 10:44:25 2026 +0000
[MINOR][SS] Use filterNot in HDFSBackedStateStoreProvider snapshot filtering
### What changes were proposed in this pull request?
Replaces `filter(_.isSnapshot == false)` with the equivalent, idiomatic
`filterNot(_.isSnapshot)` in `HDFSBackedStateStoreProvider.doSnapshot`.
### Why are the changes needed?
`StoreFile.isSnapshot` is a plain `Boolean` field, so comparing it to
`false` is redundant. `filterNot` is the idiomatic form used throughout Spark
and reads more clearly.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Behavior-preserving one-line change; existing state-store tests cover the
path. No new tests needed.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
Closes #57519 from uros-b/cleanup-hdfs-state-filternot.
Lead-authored-by: Uros Bojanic <[email protected]>
Co-authored-by: Uros Bojanic <[email protected]>
Signed-off-by: Uros Bojanic <[email protected]>
---
.../sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala
index e64e790bd21f..39ac1331c610 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala
@@ -1077,7 +1077,7 @@ private[sql] class HDFSBackedStateStoreProvider extends
StateStoreProvider with
if (files.nonEmpty) {
val lastVersion = files.last.version
val deltaFilesForLastVersion =
- filesForVersion(files, lastVersion).filter(_.isSnapshot == false)
+ filesForVersion(files, lastVersion).filterNot(_.isSnapshot)
synchronized { Option(loadedMaps.get(lastVersion)) } match {
case Some(map) =>
if (deltaFilesForLastVersion.size >
storeConf.minDeltasForSnapshot) {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
