This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-connect-swift.git
The following commit(s) were added to refs/heads/main by this push:
new 91e7b8d [SPARK-60054] Fix `DataFrame.hint` to not crash on
out-of-range integers and unsupported parameter types
91e7b8d is described below
commit 91e7b8dc3c3a4cb31cae03b920607737404348ca
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Wed Oct 7 13:45:09 2026 -0700
[SPARK-60054] Fix `DataFrame.hint` to not crash on out-of-range integers
and unsupported parameter types
### What changes were proposed in this pull request?
This PR aims to fix `DataFrame.hint` so that it no longer crashes on
out-of-range integers or unsupported parameter types. The public API stays
non-throwing.
`SparkConnectClient.getHint` now converts hint parameters as follows:
- `Int64` and `Int` values within the `Int32` range are still sent as
integer literals, because Spark's `ResolveHints` expects an `IntegerLiteral`
for partition numbers. Values outside that range are sent as long literals, so
the Spark Connect server reports the error. This matches how PySpark infers
integer literals.
- Any other type goes through the existing `ExpressionLiteral.init(_:)`
conversion, so `Double`, `Float`, `Decimal` and so on are now supported.
- Types that cannot be converted are sent as a string literal of their
description instead of being force-cast with `as! String`.
### Why are the changes needed?
`getHint` used `Int32(value)` and `$0 as! String`, which trap and crash the
whole client process. For example:
```swift
df.hint("rebalance", 3_000_000_000) // crash: Int32 overflow
df.hint("x", 1.5) // crash: Could not cast value of type
'Double' to 'String'
```
### Does this PR introduce _any_ user-facing change?
Yes, it fixes a bug. These calls used to crash the process. Now they build
a valid plan, and invalid hints are reported by the Spark Connect server
instead. The API signature does not change.
### How was this patch tested?
Pass the CIs with the newly added test cases
(`SparkConnectClientTests.getHintWithIntegers` and
`SparkConnectClientTests.getHintWithOtherTypes`).
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #585 from dongjoon-hyun/SPARK-60054.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
Sources/SparkConnect/SparkConnectClient.swift | 18 ++++++++++++++---
.../SparkConnectClientTests.swift | 23 ++++++++++++++++++++++
2 files changed, 38 insertions(+), 3 deletions(-)
diff --git a/Sources/SparkConnect/SparkConnectClient.swift
b/Sources/SparkConnect/SparkConnectClient.swift
index a7cf05c..db28eb9 100644
--- a/Sources/SparkConnect/SparkConnectClient.swift
+++ b/Sources/SparkConnect/SparkConnectClient.swift
@@ -1311,13 +1311,25 @@ public actor SparkConnectClient {
case let value as Int32:
literal.integer = value
case let value as Int64: // Hint parameter raises exceptions for Int64
- literal.integer = Int32(value)
+ if let value = Int32(exactly: value) {
+ literal.integer = value
+ } else {
+ literal.long = value
+ }
case let value as Int:
- literal.integer = Int32(value)
+ if let value = Int32(exactly: value) {
+ literal.integer = value
+ } else {
+ literal.long = Int64(value)
+ }
case let value as String:
literal.string = value
default:
- literal.string = $0 as! String
+ if let value = try? ExpressionLiteral($0) {
+ literal = value
+ } else {
+ literal.string = "\($0)"
+ }
}
var expr = Spark_Connect_Expression()
expr.literal = literal
diff --git a/Tests/SparkConnectTests/SparkConnectClientTests.swift
b/Tests/SparkConnectTests/SparkConnectClientTests.swift
index f5a6597..bb155df 100644
--- a/Tests/SparkConnectTests/SparkConnectClientTests.swift
+++ b/Tests/SparkConnectTests/SparkConnectClientTests.swift
@@ -257,4 +257,27 @@ struct SparkConnectClientTests {
}
await client.stop()
}
+
+ @Test
+ func getHintWithIntegers() async throws {
+ let parameters = SparkConnectClient.getHint(
+ Relation(), "rebalance",
+ [Int64(10), 10, Int64(3_000_000_000), -3_000_000_000]
+ ).root.hint.parameters.map { $0.literal.literalType }
+ #expect(parameters[0] == .integer(10))
+ #expect(parameters[1] == .integer(10))
+ #expect(parameters[2] == .long(3_000_000_000))
+ #expect(parameters[3] == .long(-3_000_000_000))
+ }
+
+ @Test
+ func getHintWithOtherTypes() async throws {
+ let parameters = SparkConnectClient.getHint(
+ Relation(), "x", [1.5, Float(2.5), UInt(1), [1, 2]]
+ ).root.hint.parameters.map { $0.literal.literalType }
+ #expect(parameters[0] == .double(1.5))
+ #expect(parameters[1] == .float(2.5))
+ #expect(parameters[2] == .string("1"))
+ #expect(parameters[3] == .string("[1, 2]"))
+ }
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
