This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/spark-connect-swift.git
The following commit(s) were added to refs/heads/main by this push:
new 71f2d3b [SPARK-58563] Support server-side error chain in
`SparkConnectError` via `FetchErrorDetails`
71f2d3b is described below
commit 71f2d3b82107efe2f01a790541d245449cb77fc9
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Tue Aug 4 09:29:40 2026 -0700
[SPARK-58563] Support server-side error chain in `SparkConnectError` via
`FetchErrorDetails`
### What changes were proposed in this pull request?
This PR improves `SparkConnectError` to carry the server-side error chain
by calling the
`FetchErrorDetails` RPC, like the PySpark (`_fetch_enriched_error`) and
Scala
(`GrpcExceptionConverter.fetchEnrichedError`) clients.
1. A new `SparkConnectError.ServerError` struct carries the un-truncated
error message, the
exception class hierarchy, the server-side stack trace frames, and the
per-error
`errorClass`/`sqlState`/`messageParameters`. `SparkConnectError.Details`
gains a `serverErrors`
array (the thrown error first, followed by the cause chain flattened via
`causeIdx`) and a
`serverStackTrace` convenience property that renders the chain in the JVM
`Caused by:`/`at ...` style.
2. When the `google.rpc.ErrorInfo` metadata contains an `errorId`,
`GrpcErrorConverter` sends one
`FetchErrorDetails` request over the already-open gRPC connection. The
call is best-effort
at-most-once without retries, and any failure silently falls back to the
previous
`ErrorInfo`-only conversion so that the original error is never masked.
3. The `errorClass` and `sqlState` from `ErrorInfo` take precedence to keep
the error
classification stable, while the truncated `message` and
`messageParameters` are replaced by
the un-truncated values of the root error.
The server-side stack trace is populated only when the SQL configuration
`spark.sql.connect.serverStacktrace.enabled` is true, like the other
clients.
```swift
} catch SparkConnectError.tableOrViewNotFound(let details) {
print(details.serverErrors.first?.errorTypeHierarchy ?? [])
// ["org.apache.spark.sql.catalyst.ExtendedAnalysisException",
// "org.apache.spark.sql.AnalysisException", ...]
print(details.serverStackTrace ?? "")
}
```
### Why are the changes needed?
Users could not access the server-side stack trace, the cause chain, or the
un-truncated error
message from a caught error although the server provides them via the
`FetchErrorDetails` RPC.
### Does this PR introduce _any_ user-facing change?
Yes, `SparkConnectError` additionally provides the server-side error chain.
The existing code
continues to compile and behave the same because the new
`Details.serverErrors` field has a
default value and the error classification is unchanged.
### How was this patch tested?
Pass the CIs with the newly added `serverErrors` and `serverStackTrace`
tests in
`SparkConnectErrorTests`. All existing test suites pass unmodified.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Fable 5
Closes #479 from dongjoon-hyun/SPARK-58563.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
---
Sources/SparkConnect/DataFrame.swift | 4 +-
Sources/SparkConnect/GrpcErrorConverter.swift | 124 +++++++++++++++++++--
Sources/SparkConnect/SparkConnectClient.swift | 4 +-
Sources/SparkConnect/SparkConnectError.swift | 65 ++++++++++-
Sources/SparkConnect/TypeAliases.swift | 2 +
.../SparkConnectTests/SparkConnectErrorTests.swift | 32 ++++++
6 files changed, 216 insertions(+), 15 deletions(-)
diff --git a/Sources/SparkConnect/DataFrame.swift
b/Sources/SparkConnect/DataFrame.swift
index 26ee5cd..1dfa024 100644
--- a/Sources/SparkConnect/DataFrame.swift
+++ b/Sources/SparkConnect/DataFrame.swift
@@ -318,7 +318,9 @@ public actor DataFrame: Sendable {
do {
return try await f(client)
} catch let error as RPCError where error.code == .internalError {
- throw GrpcErrorConverter.convert(error) ?? error
+ throw await GrpcErrorConverter.convert(
+ error, fetchingDetailsWith: client, sessionID:
spark.client.sessionID,
+ userContext: spark.client.userContext, clientType:
spark.client.clientType) ?? error
}
}
}
diff --git a/Sources/SparkConnect/GrpcErrorConverter.swift
b/Sources/SparkConnect/GrpcErrorConverter.swift
index b6cb2f4..0421a1d 100644
--- a/Sources/SparkConnect/GrpcErrorConverter.swift
+++ b/Sources/SparkConnect/GrpcErrorConverter.swift
@@ -30,28 +30,128 @@ import GRPCProtobuf
/// It parses the `google.rpc.ErrorInfo` from the gRPC status details to
classify errors by
/// the Spark error class and to enrich
``SparkConnectError/Details-swift.struct`` with
/// the error class, SQLSTATE, and message parameters. When the status details
are missing,
-/// it falls back to classifying by the error message.
+/// it falls back to classifying by the error message. When the `ErrorInfo`
provides an
+/// `errorId`, it additionally fetches the server-side error chain with the
un-truncated
+/// messages and optional stack traces by the `FetchErrorDetails` RPC.
enum GrpcErrorConverter {
- /// Convert an ``RPCError`` into a ``SparkConnectError``.
- /// - Parameter error: An ``RPCError`` from the server.
+ /// Convert an ``RPCError`` into a ``SparkConnectError`` after enriching it
with the
+ /// server-side error chain fetched by an additional `FetchErrorDetails` RPC
+ /// like
`org.apache.spark.sql.connect.client.GrpcExceptionConverter.fetchEnrichedError`.
+ /// The RPC call is best-effort at-most-once without retries, and any
failure silently
+ /// falls back to the un-enriched conversion in order not to mask the
original error.
+ /// - Parameters:
+ /// - error: An ``RPCError`` from the server.
+ /// - client: A connected gRPC client to send the `FetchErrorDetails`
request.
+ /// - sessionID: A session ID string. The enrichment is skipped if nil.
+ /// - userContext: A ``UserContext`` of the session.
+ /// - clientType: A client type string.
/// - Returns: A ``SparkConnectError`` or nil if unclassifiable.
- static func convert(_ error: RPCError) -> SparkConnectError? {
- var details = SparkConnectError.Details(message: error.message)
- if let status = try? error.unpackGoogleRPCStatus(),
- let info = status.details.compactMap({ $0.errorInfo }).first
+ static func convert(
+ _ error: RPCError,
+ fetchingDetailsWith client: GRPCClient<some ClientTransport>,
+ sessionID: String?,
+ userContext: UserContext,
+ clientType: String
+ ) async -> SparkConnectError? {
+ var (details, errorID) = parse(error)
+ if let errorID, let sessionID,
+ let response = await fetchErrorDetails(
+ errorID, with: client, sessionID: sessionID, userContext: userContext,
+ clientType: clientType)
{
- details = SparkConnectError.Details(
- message: status.message.isEmpty ? error.message : status.message,
- errorClass: info.metadata["errorClass"],
- sqlState: info.metadata["sqlState"],
- messageParameters: messageParameters(from:
info.metadata["messageParameters"]))
+ details = enrich(details, with: response)
}
+ return classify(details)
+ }
+
+ /// Build ``SparkConnectError/Details-swift.struct`` from the
`google.rpc.ErrorInfo` of
+ /// the gRPC status details, together with the server-side `errorId` if
provided.
+ private static func parse(_ error: RPCError) -> (SparkConnectError.Details,
errorID: String?) {
+ guard let status = try? error.unpackGoogleRPCStatus(),
+ let info = status.details.compactMap({ $0.errorInfo }).first
+ else {
+ return (SparkConnectError.Details(message: error.message), nil)
+ }
+ let details = SparkConnectError.Details(
+ message: status.message.isEmpty ? error.message : status.message,
+ errorClass: info.metadata["errorClass"],
+ sqlState: info.metadata["sqlState"],
+ messageParameters: messageParameters(from:
info.metadata["messageParameters"]))
+ return (details, info.metadata["errorId"])
+ }
+
+ /// Classify ``SparkConnectError/Details-swift.struct`` into a
``SparkConnectError``
+ /// by the Spark error class first and by the error message as a fallback.
+ private static func classify(_ details: SparkConnectError.Details) ->
SparkConnectError? {
if let errorClass = details.errorClass, let converted =
convert(errorClass, details) {
return converted
}
return convert(byMessage: details.message, details)
}
+ /// Send a `FetchErrorDetails` request, returning nil on any failure.
+ private static func fetchErrorDetails(
+ _ errorID: String,
+ with client: GRPCClient<some ClientTransport>,
+ sessionID: String,
+ userContext: UserContext,
+ clientType: String
+ ) async -> FetchErrorDetailsResponse? {
+ let service = SparkConnectService.Client(wrapping: client)
+ var request = FetchErrorDetailsRequest()
+ request.sessionID = sessionID
+ request.userContext = userContext
+ request.clientType = clientType
+ request.errorID = errorID
+ return try? await service.fetchErrorDetails(request)
+ }
+
+ /// Merge a `FetchErrorDetailsResponse` into
``SparkConnectError/Details-swift.struct``.
+ /// The `errorClass` and `sqlState` from `ErrorInfo` take precedence to keep
the
+ /// error classification stable, while the un-truncated message and the
message
+ /// parameters of the root error replace the truncated ones.
+ private static func enrich(
+ _ details: SparkConnectError.Details, with response:
FetchErrorDetailsResponse
+ ) -> SparkConnectError.Details {
+ guard response.hasRootErrorIdx else { return details }
+ let serverErrors = flatten(response)
+ guard let root = serverErrors.first else { return details }
+ return SparkConnectError.Details(
+ message: root.message.isEmpty ? details.message : root.message,
+ errorClass: details.errorClass ?? root.errorClass,
+ sqlState: details.sqlState ?? root.sqlState,
+ messageParameters: root.messageParameters.isEmpty
+ ? details.messageParameters : root.messageParameters,
+ serverErrors: serverErrors)
+ }
+
+ /// Flatten the `causeIdx`-linked errors of a `FetchErrorDetailsResponse`
into an array
+ /// starting from the root error, guarding against out-of-range indices and
cycles.
+ private static func flatten(
+ _ response: FetchErrorDetailsResponse
+ ) -> [SparkConnectError.ServerError] {
+ var serverErrors: [SparkConnectError.ServerError] = []
+ var visited = Set<Int>()
+ var index = Int(response.rootErrorIdx)
+ while response.errors.indices.contains(index),
visited.insert(index).inserted {
+ let error = response.errors[index]
+ let throwable = error.hasSparkThrowable ? error.sparkThrowable : nil
+ serverErrors.append(
+ SparkConnectError.ServerError(
+ message: error.message,
+ errorTypeHierarchy: error.errorTypeHierarchy,
+ stackTrace: error.stackTrace.map {
+
"\($0.declaringClass).\($0.methodName)(\($0.fileName):\($0.lineNumber))"
+ },
+ errorClass: throwable?.hasErrorClass == true ? throwable?.errorClass
: nil,
+ sqlState: throwable?.hasSqlState == true ? throwable?.sqlState : nil,
+ messageParameters: throwable?.messageParameters ?? [:]))
+ guard error.hasCauseIdx else { break }
+ index = Int(error.causeIdx)
+ }
+ return serverErrors
+ }
+
/// Classify by the Spark error class from `ErrorInfo`.
private static func convert(
_ errorClass: String, _ details: SparkConnectError.Details
diff --git a/Sources/SparkConnect/SparkConnectClient.swift
b/Sources/SparkConnect/SparkConnectClient.swift
index 8449f4c..caf8ca9 100644
--- a/Sources/SparkConnect/SparkConnectClient.swift
+++ b/Sources/SparkConnect/SparkConnectClient.swift
@@ -147,7 +147,9 @@ public actor SparkConnectClient {
do {
return try await f(client)
} catch let error as RPCError where error.code == .internalError {
- throw GrpcErrorConverter.convert(error) ?? error
+ throw await GrpcErrorConverter.convert(
+ error, fetchingDetailsWith: client, sessionID: self.sessionID,
+ userContext: self.userContext, clientType: self.clientType) ??
error
}
}
}
diff --git a/Sources/SparkConnect/SparkConnectError.swift
b/Sources/SparkConnect/SparkConnectError.swift
index f20459b..5c62180 100644
--- a/Sources/SparkConnect/SparkConnectError.swift
+++ b/Sources/SparkConnect/SparkConnectError.swift
@@ -25,6 +25,40 @@
/// Note that the equality of ``SparkConnectError`` compares only the error
kind and
/// ignores the attached ``SparkConnectError/Details-swift.struct``.
public enum SparkConnectError: Error, Sendable, Equatable {
+ /// A server-side error fetched by the `FetchErrorDetails` RPC.
+ public struct ServerError: Sendable, Equatable {
+ /// The un-truncated error message.
+ public let message: String
+ /// The fully qualified names of the exception class and its parent
classes.
+ public let errorTypeHierarchy: [String]
+ /// The server-side stack trace frames formatted as
`declaringClass.methodName(fileName:line)`.
+ /// This is non-empty only if the SQL configuration
+ /// `spark.sql.connect.serverStacktrace.enabled` is true.
+ public let stackTrace: [String]
+ /// A Spark error class of this error if it is a `SparkThrowable`.
+ public let errorClass: String?
+ /// A SQLSTATE of this error if it is a `SparkThrowable`.
+ public let sqlState: String?
+ /// The message parameters of the error class.
+ public let messageParameters: [String: String]
+
+ public init(
+ message: String = "",
+ errorTypeHierarchy: [String] = [],
+ stackTrace: [String] = [],
+ errorClass: String? = nil,
+ sqlState: String? = nil,
+ messageParameters: [String: String] = [:]
+ ) {
+ self.message = message
+ self.errorTypeHierarchy = errorTypeHierarchy
+ self.stackTrace = stackTrace
+ self.errorClass = errorClass
+ self.sqlState = sqlState
+ self.messageParameters = messageParameters
+ }
+ }
+
/// Detailed error information.
public struct Details: Sendable, Equatable {
/// An error message.
@@ -35,17 +69,40 @@ public enum SparkConnectError: Error, Sendable, Equatable {
public let sqlState: String?
/// The message parameters of the error class.
public let messageParameters: [String: String]
+ /// The server-side error chain fetched by the `FetchErrorDetails` RPC.
+ /// The first element is the thrown error and the rest follow the cause
chain in order.
+ /// This is empty if the server does not support the RPC or the fetch
fails.
+ public let serverErrors: [ServerError]
public init(
message: String = "",
errorClass: String? = nil,
sqlState: String? = nil,
- messageParameters: [String: String] = [:]
+ messageParameters: [String: String] = [:],
+ serverErrors: [ServerError] = []
) {
self.message = message
self.errorClass = errorClass
self.sqlState = sqlState
self.messageParameters = messageParameters
+ self.serverErrors = serverErrors
+ }
+
+ /// A multi-line string of the server-side error chain in the JVM stack
trace style,
+ /// or nil if ``serverErrors`` is empty.
+ public var serverStackTrace: String? {
+ guard !serverErrors.isEmpty else { return nil }
+ var lines: [String] = []
+ for (index, error) in serverErrors.enumerated() {
+ let className = error.errorTypeHierarchy.first ?? ""
+ if index == 0 {
+ lines.append(className)
+ } else {
+ lines.append("Caused by: \(className): \(error.message)")
+ }
+ lines.append(contentsOf: error.stackTrace.map { "\tat \($0)" })
+ }
+ return lines.joined(separator: "\n")
}
}
@@ -119,6 +176,12 @@ public enum SparkConnectError: Error, Sendable, Equatable {
/// The message parameters of the error class.
public var messageParameters: [String: String] { details.messageParameters }
+ /// The server-side error chain fetched by the `FetchErrorDetails` RPC.
+ public var serverErrors: [ServerError] { details.serverErrors }
+
+ /// A multi-line string of the server-side error chain in the JVM stack
trace style.
+ public var serverStackTrace: String? { details.serverStackTrace }
+
private var caseName: String {
switch self {
case .catalogNotFound: return "catalogNotFound"
diff --git a/Sources/SparkConnect/TypeAliases.swift
b/Sources/SparkConnect/TypeAliases.swift
index 1099931..712ba2c 100644
--- a/Sources/SparkConnect/TypeAliases.swift
+++ b/Sources/SparkConnect/TypeAliases.swift
@@ -32,6 +32,8 @@ typealias ExecutePlanResponse =
Spark_Connect_ExecutePlanResponse
typealias ExplainMode = AnalyzePlanRequest.Explain.ExplainMode
typealias ExpressionLiteral = Spark_Connect_Expression.Literal
typealias ExpressionString = Spark_Connect_Expression.ExpressionString
+typealias FetchErrorDetailsRequest = Spark_Connect_FetchErrorDetailsRequest
+typealias FetchErrorDetailsResponse = Spark_Connect_FetchErrorDetailsResponse
typealias Filter = Spark_Connect_Filter
typealias GroupType = Spark_Connect_Aggregate.GroupType
typealias Join = Spark_Connect_Join
diff --git a/Tests/SparkConnectTests/SparkConnectErrorTests.swift
b/Tests/SparkConnectTests/SparkConnectErrorTests.swift
index b5e3cce..cfeeaed 100644
--- a/Tests/SparkConnectTests/SparkConnectErrorTests.swift
+++ b/Tests/SparkConnectTests/SparkConnectErrorTests.swift
@@ -68,6 +68,38 @@ struct SparkConnectErrorTests {
await spark.stop()
}
+ @Test
+ func serverErrors() async throws {
+ let spark = try await SparkSession.builder.getOrCreate()
+ let error = try await #require(throws: SparkConnectError.self) {
+ try await spark.sql("SELECT * FROM
nonexistent_table_for_error_tests").count()
+ }
+ #expect(error == .TableOrViewNotFound)
+ let root = try #require(error.serverErrors.first)
+
#expect(root.errorTypeHierarchy.contains("org.apache.spark.sql.AnalysisException"))
+ #expect(root.message.contains("nonexistent_table_for_error_tests"))
+ #expect(root.errorClass == "TABLE_OR_VIEW_NOT_FOUND")
+ #expect(
+
root.messageParameters["relationName"]?.contains("nonexistent_table_for_error_tests")
+ ?? false)
+ await spark.stop()
+ }
+
+ @Test
+ func serverStackTrace() async throws {
+ let spark = try await SparkSession.builder.getOrCreate()
+ try await spark.conf.set("spark.sql.connect.serverStacktrace.enabled",
"true")
+ let error = try await #require(throws: SparkConnectError.self) {
+ try await spark.sql("SELECT * FROM
nonexistent_table_for_error_tests").count()
+ }
+ let root = try #require(error.serverErrors.first)
+ #expect(!root.stackTrace.isEmpty)
+ let stackTrace = try #require(error.serverStackTrace)
+ #expect(stackTrace.contains("AnalysisException"))
+ #expect(stackTrace.contains("\tat org.apache.spark."))
+ await spark.stop()
+ }
+
@Test
func detailsPatternMatching() async throws {
let spark = try await SparkSession.builder.getOrCreate()
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
