This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-connect-swift.git
The following commit(s) were added to refs/heads/main by this push:
new 3a5f20c [SPARK-52746] Test `isLocal` behavior explicitly for Spark
4.0 and 4.1+ servers
3a5f20c is described below
commit 3a5f20c118bc3fbac0a85168e1e22cf2bb33aff1
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sat Aug 8 20:24:53 2026 -0700
[SPARK-52746] Test `isLocal` behavior explicitly for Spark 4.0 and 4.1+
servers
### What changes were proposed in this pull request?
This PR updates the `isLocal` test in `DataFrameTests` to verify the
version-specific expected behavior explicitly, replacing the previous
`TODO(SPARK-52746)` guard that simply skipped the checks on Spark 4.1/4.2
servers.
- Spark 4.0.x servers: `SHOW DATABASES`/`SHOW TABLES` plans are expected to
be local (`isLocal() == true`).
- Spark 4.1+ servers: they are expected to be non-local (`isLocal() ==
false`), which is now asserted instead of skipped.
- The version branch uses `version < "4.1"` (following the existing
string-comparison pattern in this file) instead of enumerating `"4.1"`/`"4.2"`
prefixes, so it keeps working for Spark 4.3+.
### Why are the changes needed?
The behavior difference is not a server bug awaiting a fix, but an
intentional change in Spark 4.1 by
[SPARK-51818](https://issues.apache.org/jira/browse/SPARK-51818) ("Move
QueryExecution creation to AnalyzeHandler and don't Execute for
AnalyzePlanRequests"). Up to 4.0, `AnalyzePlan` executed commands eagerly, so
the analyzed plan became a `CommandResult` and `Dataset.isLocal` returned
`true`. Since 4.1, commands are no longer executed during analysis
(`CommandExecutionMode.SKIP`), s [...]
- https://github.com/apache/spark/pull/50605
### Does this PR introduce _any_ user-facing change?
No. This is a test-only change.
### How was this patch tested?
Pass the CIs.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Fable 5
Closes #492 from dongjoon-hyun/dongjoon/magical-archimedes-157671.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
Tests/SparkConnectTests/DataFrameTests.swift | 9 +++++----
1 file changed, 5 insertions(+), 4 deletions(-)
diff --git a/Tests/SparkConnectTests/DataFrameTests.swift
b/Tests/SparkConnectTests/DataFrameTests.swift
index 98ef0ee..2ebd0db 100644
--- a/Tests/SparkConnectTests/DataFrameTests.swift
+++ b/Tests/SparkConnectTests/DataFrameTests.swift
@@ -539,10 +539,11 @@ struct DataFrameTests {
func isLocal() async throws {
let spark = try await SparkSession.builder.getOrCreate()
let version = await spark.version
- if !(version.starts(with: "4.1") || version.starts(with: "4.2")) { //
TODO(SPARK-52746)
- #expect(try await spark.sql("SHOW DATABASES").isLocal())
- #expect(try await spark.sql("SHOW TABLES").isLocal())
- }
+ // Since Spark 4.1, `AnalyzePlan` no longer executes commands eagerly
(SPARK-51818),
+ // so command plans are not local anymore.
+ let expected = version < "4.1"
+ #expect(try await spark.sql("SHOW DATABASES").isLocal() == expected)
+ #expect(try await spark.sql("SHOW TABLES").isLocal() == expected)
#expect(try await spark.range(1).isLocal() == false)
await spark.stop()
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
