This is an automated email from the ASF dual-hosted git repository.
uros-b pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new c2e0950dba90 [SPARK-58567][SQL] Use nonEmpty/isEmpty instead of length
comparisons in StateDataSource
c2e0950dba90 is described below
commit c2e0950dba90219add482b818d5e79d1faa25def
Author: Uros Bojanic <[email protected]>
AuthorDate: Tue Aug 4 19:04:55 2026 +0000
[SPARK-58567][SQL] Use nonEmpty/isEmpty instead of length comparisons in
StateDataSource
### What changes were proposed in this pull request?
Aligns two emptiness checks on the `storeMetadata` array in
`StateDataSource` to `.nonEmpty`/`.isEmpty`, matching how the same variable is
tested elsewhere in the file.
### Why are the changes needed?
For a Scala `Array` these are exact equivalents of the `length`
comparisons; the nearby `require(... == 1)` count-check is left unchanged.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Stylistic, behavior-identical change. No new tests needed.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.8)
Closes #57767 from uros-b/consistency-statedatasource.
Authored-by: Uros Bojanic <[email protected]>
Signed-off-by: Uros Bojanic <[email protected]>
(cherry picked from commit 0388c74bc5371f5770e88718d42356f25baa5aa6)
Signed-off-by: Uros Bojanic <[email protected]>
---
.../spark/sql/execution/datasources/v2/state/StateDataSource.scala | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/state/StateDataSource.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/state/StateDataSource.scala
index bcf9c5db367a..5dfe2c717b8d 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/state/StateDataSource.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/state/StateDataSource.scala
@@ -320,7 +320,7 @@ class StateDataSource extends TableProvider with
DataSourceRegister with Logging
// Read the schema file path from operator metadata version v2 onwards
// for the transformWithState operator
- val oldSchemaFilePaths = if (storeMetadata.length > 0 &&
storeMetadata.head.version == 2) {
+ val oldSchemaFilePaths = if (storeMetadata.nonEmpty &&
storeMetadata.head.version == 2) {
val opName = storeMetadata.head.operatorName
if
(StatefulOperatorsUtils.TRANSFORM_WITH_STATE_OP_NAMES.exists(opName.contains)) {
val storeMetadataEntry = storeMetadata.head
@@ -442,7 +442,7 @@ class StateDataSource extends TableProvider with
DataSourceRegister with Logging
storeMetadata: Array[StateMetadataTableEntry]): KeyStateEncoderSpec = {
// If operator metadata is not found, then log a warning and continue with
using the no-prefix
// key state encoder
- val keyStateEncoderSpec = if (storeMetadata.length == 0) {
+ val keyStateEncoderSpec = if (storeMetadata.isEmpty) {
logWarning("Metadata for state store not found, possible cause is this
checkpoint " +
"is created by older version of spark. If the query has session window
aggregation, " +
"the state can't be read correctly and runtime exception will be
thrown. " +
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
