venkata91 opened a new pull request, #57016: URL: https://github.com/apache/spark/pull/57016
### What changes were proposed in this pull request? Spark Connect server session and operation (execution) data is collected by `SparkConnectServerListener` and shown in the "Connect" Spark UI tab, but unlike SQL ([SPARK-27142](https://issues.apache.org/jira/browse/SPARK-27142)) and Streaming, it is not reachable through the `/api/v1` REST API. This adds a REST resource mirroring the SQL REST API. New endpoints: - `GET /applications/[app-id]/connect/sessions` — list sessions - `GET /applications/[app-id]/connect/sessions/[session-id]` — one session - `GET /applications/[app-id]/connect/operations` — list operations - `GET /applications/[app-id]/connect/operations/[job-tag]` — one operation (and the `[attempt-id]` variants). Implementation notes for reviewers: - The resource classes live in package `org.apache.spark.status.api.v1.connect`, so they are auto-discovered by the existing Jersey package scan (`ApiRootResource` registers `org.apache.spark.status.api.v1` recursively) exactly like the SQL (`...v1.sql`) and Streaming (`...v1.streaming`) resources. No wiring changes are needed. - `ConnectResource` reads from `ui.store.store` — the same `KVStore` that backs the Connect UI tab — via the existing `SparkConnectServerAppStatusStore`. It adds no new data collection. - `SessionInfo`, `ExecutionInfo` and `ExecutionState` are widened from `private[connect]` to `private[spark]` so the resource package can read them. They remain internal (not public API). - The public API surface is the new `SessionData` / `ExecutionData` DTOs in `api.scala`, following the SQL/Streaming convention (no `@since` on these status DTOs). Because the Connect listener already persists its events to the event log, these endpoints also work in the History Server through the existing `SparkConnectServerHistoryServerPlugin`. ### Why are the changes needed? The Connect UI tab exposes per-session and per-operation information (user, timings, state, job/SQL ids, errors) that is only reachable by scraping HTML. Jobs, stages, SQL and Streaming are all queryable as JSON under `/api/v1`; Spark Connect should be too, so operators can build monitoring and automation on top of the Connect server the same way they do for the rest of Spark. ### Does this PR introduce _any_ user-facing change? Yes. It adds new read-only REST endpoints under `/api/v1/applications/[app-id]/connect/...` (listed above) and documents them in `docs/monitoring.md`. There is no change to existing endpoints or behavior. ### How was this patch tested? - `ConnectResourceSuite` — unit tests for the `SessionData` / `ExecutionData` mapping (exercises the private `prepare*` methods directly, mirroring `SqlResourceSuite`). - `ConnectResourceWithActualDataSuite` — end-to-end test that boots the real `SparkConnectService` with the UI enabled, runs a query through a Connect session so the listener populates the KVStore, then hits the endpoints over HTTP and asserts on the JSON (including 404s for unknown ids). - Manual verification against a running Connect server: started the server, ran a query from a Connect client, and confirmed `curl .../connect/sessions` and `.../connect/operations` return the expected JSON. - `./dev/lint-scala` scalafmt check passes for the connect module. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 4.8) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
