dongjoon-hyun opened a new pull request, #484:
URL: https://github.com/apache/spark-connect-swift/pull/484
### What changes were proposed in this pull request?
This PR aims to support `Observation` to retrieve the observed metrics of
`DataFrame.observe`.
```swift
let observation = Observation("my_metrics")
let observedDf = try await df.observe(observation, count(col("*")),
max(col("id")))
try await observedDf.count()
let metrics = try await observation.get // ["count(1)": Int64(10),
"max(id)": ...]
```
- Add a new `Observation` actor whose `get` property returns the observed
metrics keyed by the metric column names, or throws
`SparkConnectError.invalidState` if no action has been performed yet.
- Add a `DataFrame.observe(_ observation: Observation, _ expr: Column, _
exprs: Column...)` overload which registers the observation to
`SparkConnectClient`.
- Collect `ExecutePlanResponse.observed_metrics` inside
`executePlanWithReattach` and deliver the values to the registered observations
by matching `ObservedMetrics.name`, like the PySpark Connect client. Since a
reattached stream resumes after `last_response_id`, no response is delivered
twice.
- Add an `ExpressionLiteral.toSwiftValue` helper converting metric value
literals back to Swift values.
### Why are the changes needed?
SPARK-58425 added `DataFrame.observe` which builds the `CollectMetrics`
relation, but the observed metric values returned by the server via
`ExecutePlanResponse.observed_metrics` were discarded, so users had no way to
read them. This provides the `Observation` API with the same usage as Scala and
PySpark.
### Does this PR introduce _any_ user-facing change?
No. This is a new feature addition.
### How was this patch tested?
Pass the CIs with a newly added test case.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Fable 5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
