This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-connect-swift.git
The following commit(s) were added to refs/heads/main by this push:
new 571058f [SPARK-60477] Fix `DataFrameStatFunctions.approxQuantile` to
throw instead of crashing on empty results
571058f is described below
commit 571058fa0c9dffdcc67023aaadce6b9376ee6a8a
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Thu Oct 22 10:03:58 2026 -0700
[SPARK-60477] Fix `DataFrameStatFunctions.approxQuantile` to throw instead
of crashing on empty results
### What changes were proposed in this pull request?
This PR aims to fix the single-column
`DataFrameStatFunctions.approxQuantile(_:_:_:)` to throw
`SparkConnectError.invalidState` instead of crashing when the server returns an
empty list of quantiles. It uses the existing `Array.firstOrThrow()` helper
instead of `[0]`.
### Why are the changes needed?
The single-column version returns the first element of the multi-column
result with `[0]`, which has no bounds check. So, an empty result crashes the
client process.
```
Swift/ContiguousArrayBuffer.swift:695: Fatal error: Index out of range
```
A Spark Connect server returns one list of quantiles per requested column
([SparkConnectPlanner.transformStatApproxQuantile](https://github.com/apache/spark/blob/v4.2.0/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/planner/SparkConnectPlanner.scala#L576-L587)
in [v4.2.0](https://github.com/apache/spark/releases/tag/v4.2.0)
(2026-07-11)), so this happens only with a misbehaving or incompatible server.
As described in SPARK-59193, such a server should cause a catchable [...]
### Does this PR introduce _any_ user-facing change?
No for normal usage. When the server returns an empty list of quantiles,
the single-column `approxQuantile` now throws `SparkConnectError.invalidState`
instead of crashing.
```
The server returned an empty response for approxQuantile(_:_:_:).
```
### How was this patch tested?
Pass the CIs. No test case is added because a Spark Connect server returns
an empty list of quantiles only for an empty list of columns, so the changed
code path cannot be tested without a mock server. `firstOrThrow()` on an empty
array is covered by `ExtensionTests.firstOrThrowOnEmptyArray`, and the existing
`DataFrameStatFunctionsTests.approxQuantile` covers the normal and empty
`DataFrame` cases.
In addition, this was verified manually against an `apache/spark:4.2.0`
Spark Connect server with a temporary client interceptor which removes the
column from the `StatApproxQuantile` request. The single-column
`approxQuantile` crashed with `Fatal error: Index out of range` before this
fix, and throws `SparkConnectError.invalidState` after this fix.
`DataFrameStatFunctionsTests` and `ExtensionTests` also passed against
`apache/spark:4.2.0` and `apache/spark:4.0.4-java21-scala` Spark C [...]
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #620 from dongjoon-hyun/SPARK-60477.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
Sources/SparkConnect/DataFrameStatFunctions.swift | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/Sources/SparkConnect/DataFrameStatFunctions.swift
b/Sources/SparkConnect/DataFrameStatFunctions.swift
index f80002e..98934ae 100644
--- a/Sources/SparkConnect/DataFrameStatFunctions.swift
+++ b/Sources/SparkConnect/DataFrameStatFunctions.swift
@@ -77,7 +77,7 @@ public actor DataFrameStatFunctions: Sendable {
public func approxQuantile(
_ col: String, _ probabilities: [Double], _ relativeError: Double
) async throws -> [Double] {
- return try await approxQuantile([col], probabilities, relativeError)[0]
+ return try await approxQuantile([col], probabilities,
relativeError).firstOrThrow()
}
/// Calculates the approximate quantiles of numerical columns of a
``DataFrame``.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]