This is an automated email from the ASF dual-hosted git repository.
sunchao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new d8d08b6c3170 [SPARK-58097][CONNECT] Preserve composite (userId,
sessionId) session identity in Connect UI/status store
d8d08b6c3170 is described below
commit d8d08b6c3170739ed22737af1e81bada846b9085
Author: Venkata krishnan Sowrirajan <[email protected]>
AuthorDate: Mon Jul 27 17:28:13 2026 -0700
[SPARK-58097][CONNECT] Preserve composite (userId, sessionId) session
identity in Connect UI/status store
### What changes were proposed in this pull request?
Spark Connect identifies a session by `(userId, sessionId)`, and two users
may share the same session UUID. The Connect UI listener/store keyed sessions
by `sessionId` alone, so such sessions collapsed into one record (merged
execution counts; one user's close finished/removed the other's row). History
Server replay reproduced the same bug.
This keys `SessionInfo`'s KVStore natural key and the listener's
live-session map by a synthesized composite `uniqueId = userId/sessionId`, and
threads `userId` through the UI session page, its links, and the REST
`sessions/{sessionId}` endpoint (added in SPARK-57941). Specifically:
- **Composite identity:** `SessionInfo` is keyed on `(userId, sessionId)`;
the session page filters operations on both fields so one user's operations
don't leak into another's page.
- **Opaque user ids through the UI:** `userId` is arbitrary and part of the
key, but the UI wraps requests in `XssSafeRequest` (strips apostrophes,
HTML-escapes) and `PagedTable` re-echoes decoded params. So `userId` is carried
as an unpadded base64url token, whose alphabet (`A-Za-z0-9-_`) survives both
untouched. The REST endpoint takes the same token; an empty token is a valid
empty user id (the protobuf default), so only an absent parameter is rejected.
- **Legacy History Server stores:** `getSession` falls back to a field scan
on a composite-key miss, so disk stores cached by an older Spark (keyed on
`sessionId` alone) still resolve after an upgrade with no replay.
### Why are the changes needed?
Two users sharing a session UUID are distinct sessions in Spark Connect
(see `SparkConnectServiceE2ESuite`), but the UI/status store merged them,
showing incorrect counts and lifecycle state. Surfaced during review of
SPARK-57941.
### Does this PR introduce _any_ user-facing change?
Yes, within the unreleased master/branch line (the REST API and this fix
are unreleased):
- The Connect session detail page and the REST `sessions/{sessionId}`
endpoint now require a `userId` parameter (a base64url token) alongside the
session id.
- **Trade-off on legacy persistent stores:** the natural key changes
without bumping `AppStatusStore.CURRENT_VERSION`. A version bump is the only
replay knob and is global, forcing every application's disk store (core, SQL,
Hive Thrift, Connect) to rebuild, which is disproportionate for a Connect-only
key change. Instead the `getSession` fallback resolves legacy rows in place.
Its one limitation: rows where two users genuinely shared a UUID *before* the
upgrade were already merged und [...]
### How was this patch tested?
Added to `SparkConnectServerListenerSuite`: same-UUID/different-user
coverage (live + History Server replay) and a legacy `sessionId`-keyed row that
the fallback resolves. Added to `SparkConnectServerPageSuite`: per-user
operation isolation and a base64url round-trip through the real
`XssSafeRequest.stripXSS` transform for values containing `'`, `+`, `=`. Added
to `ConnectResourceWithActualDataSuite`: REST lookup by the `userId` token over
the real HTTP filter, plus an empty-user sess [...]
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 4.8)
Closes #57219 from venkata91/SPARK-58097-session-identity.
Authored-by: Venkata krishnan Sowrirajan <[email protected]>
Signed-off-by: Chao Sun <[email protected]>
---
docs/monitoring.md | 6 +-
.../ui/SparkConnectServerAppStatusStore.scala | 21 ++++--
.../connect/ui/SparkConnectServerListener.scala | 18 ++---
.../sql/connect/ui/SparkConnectServerPage.scala | 22 ++++--
.../connect/ui/SparkConnectServerSessionPage.scala | 15 ++--
.../status/api/v1/connect/ConnectResource.scala | 20 +++++-
.../ui/SparkConnectServerListenerSuite.scala | 81 ++++++++++++++++++++--
.../connect/ui/SparkConnectServerPageSuite.scala | 56 +++++++++++++++
.../ConnectResourceWithActualDataSuite.scala | 33 ++++++++-
9 files changed, 238 insertions(+), 34 deletions(-)
diff --git a/docs/monitoring.md b/docs/monitoring.md
index 0c178110e8f6..64552ba28512 100644
--- a/docs/monitoring.md
+++ b/docs/monitoring.md
@@ -722,8 +722,10 @@ can be identified by their `[attempt-id]`. In the API
listed below, when running
</td>
</tr>
<tr>
- <td><code>/applications/[app-id]/connect/sessions/[session-id]</code></td>
- <td>Details for the given Spark Connect session.</td>
+
<td><code>/applications/[app-id]/connect/sessions/[session-id]?userId=[user-id]</code></td>
+ <td>Details for the given Spark Connect session. A session is identified
by the composite
+ <code>(userId, sessionId)</code>, so <code>userId</code> is required and
must be passed as an
+ unpadded base64url-encoded string.</td>
</tr>
<tr>
<td><code>/applications/[app-id]/connect/operations</code></td>
diff --git
a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerAppStatusStore.scala
b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerAppStatusStore.scala
index b5211b6c3072..523edb446080 100644
---
a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerAppStatusStore.scala
+++
b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerAppStatusStore.scala
@@ -47,11 +47,14 @@ class SparkConnectServerAppStatusStore(store: KVStore) {
KVUtils.count(store.view(classOf[SessionInfo]))(_.finishTimestamp == 0)
}
- def getSession(sessionId: String): Option[SessionInfo] = {
+ def getSession(userId: String, sessionId: String): Option[SessionInfo] = {
try {
- Some(store.read(classOf[SessionInfo], sessionId))
+ Some(store.read(classOf[SessionInfo], SessionInfo.uniqueId(userId,
sessionId)))
} catch {
- case _: NoSuchElementException => None
+ case _: NoSuchElementException =>
+ // Fall back for legacy stores keyed on sessionId alone (e.g. a
History Server disk store
+ // from an older Spark). Collapsed same-UUID rows cannot be recovered.
+ getSessionList.find(s => s.userId == userId && s.sessionId ==
sessionId)
}
}
@@ -82,11 +85,16 @@ class SparkConnectServerAppStatusStore(store: KVStore) {
}
private[spark] class SessionInfo(
- @KVIndexParam val sessionId: String,
+ val sessionId: String,
val startTimestamp: Long,
val userId: String,
val finishTimestamp: Long,
val totalExecution: Long) {
+ // Natural key. A session is identified by (userId, sessionId), since two
users may share the
+ // same session UUID; keying on sessionId alone would merge them into one
record.
+ @KVIndexParam
+ def uniqueId: String = SessionInfo.uniqueId(userId, sessionId)
+
@JsonIgnore @KVIndex("finishTime")
private def finishTimeIndex: Long = if (finishTimestamp > 0L)
finishTimestamp else -1L
def totalTime: Long = {
@@ -98,6 +106,11 @@ private[spark] class SessionInfo(
}
}
+private[connect] object SessionInfo {
+ // sessionId is a UUID, so joining with '/' yields a key that is unique per
(userId, sessionId).
+ def uniqueId(userId: String, sessionId: String): String =
s"$userId/$sessionId"
+}
+
private[spark] class ExecutionInfo(
@KVIndexParam val jobTag: String,
val statement: String,
diff --git
a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerListener.scala
b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerListener.scala
index 01a775f89744..dfc36470a7d5 100644
---
a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerListener.scala
+++
b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerListener.scala
@@ -180,7 +180,7 @@ private[connect] class SparkConnectServerListener(
updateLiveStore(executionData) { executionData =>
executionData.state = ExecutionState.STARTED
}
- Option(sessionList.get(e.sessionId)) match {
+ Option(sessionList.get(SessionInfo.uniqueId(e.userId, e.sessionId))) match
{
case Some(sessionData) =>
updateLiveStore(sessionData) { sessionData =>
sessionData.totalExecution += 1 }
case None =>
@@ -282,7 +282,7 @@ private[connect] class SparkConnectServerListener(
private def onSessionClosed(e: SparkListenerConnectSessionClosed) = {
sessionList.compute(
- e.sessionId,
+ SessionInfo.uniqueId(e.userId, e.sessionId),
(_, sessionData) => {
if (sessionData != null) {
updateStoreWithTriggerEnabled(sessionData) { sessionData =>
@@ -319,11 +319,11 @@ private[connect] class SparkConnectServerListener(
private def getOrCreateSession(
sessionId: String,
- userName: String,
+ userId: String,
startTime: Long): LiveSessionData = {
sessionList.computeIfAbsent(
- sessionId,
- _ => new LiveSessionData(sessionId, startTime, userName))
+ SessionInfo.uniqueId(userId, sessionId),
+ _ => new LiveSessionData(sessionId, startTime, userId))
}
private def getOrCreateExecution(
@@ -369,7 +369,9 @@ private[connect] class SparkConnectServerListener(
j.finishTimestamp != 0L
}
- toDelete.foreach { j => kvstore.delete(j.getClass, j.sessionId) }
+ toDelete.foreach { j =>
+ kvstore.delete(j.getClass, SessionInfo.uniqueId(j.userId, j.sessionId))
+ }
}
/**
@@ -431,14 +433,14 @@ private[connect] class LiveExecutionData(
private[connect] class LiveSessionData(
val sessionId: String,
val startTimestamp: Long,
- val userName: String)
+ val userId: String)
extends LiveEntity {
var finishTimestamp: Long = 0L
var totalExecution: Int = 0
override protected def doUpdate(): Any = {
- new SessionInfo(sessionId, startTimestamp, userName, finishTimestamp,
totalExecution)
+ new SessionInfo(sessionId, startTimestamp, userId, finishTimestamp,
totalExecution)
}
def totalTime: Long = {
if (finishTimestamp == 0L) {
diff --git
a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerPage.scala
b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerPage.scala
index b34cf8d5cfa7..ea78b2dc59f6 100644
---
a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerPage.scala
+++
b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerPage.scala
@@ -19,6 +19,7 @@ package org.apache.spark.sql.connect.ui
import java.net.URLEncoder
import java.nio.charset.StandardCharsets.UTF_8
+import java.util.Base64
import scala.xml.Node
@@ -30,6 +31,17 @@ import org.apache.spark.ui._
import org.apache.spark.ui.UIUtils._
import org.apache.spark.util.Utils
+// userId is part of a session's natural key but is opaque and arbitrary. It
is carried through UI
+// links as unpadded base64url, whose alphabet (A-Za-z0-9-_) survives
XssSafeRequest sanitization
+// and the PagedTable parameter re-echo unchanged.
+private[spark] object ConnectUiUtils {
+ def encodeUserId(userId: String): String =
+ Base64.getUrlEncoder.withoutPadding.encodeToString(userId.getBytes(UTF_8))
+
+ def decodeUserId(token: String): String =
+ new String(Base64.getUrlDecoder.decode(token), UTF_8)
+}
+
/** Page for Spark UI that shows statistics for a Spark Connect Server. */
private[ui] class SparkConnectServerPage(parent: SparkConnectServerTab)
extends WebUIPage("")
@@ -276,10 +288,11 @@ private[ui] class SqlStatsPagedTable(
<a href={sqlURL(request, sqlExecId)}>[{sqlExecId}]</a>
}
}
- val sessionLink = "%s/%s/session/?id=%s".format(
+ val sessionLink = "%s/%s/session/?id=%s&userId=%s".format(
UIUtils.prependBaseUri(request, parent.basePath),
parent.prefix,
- info.sessionId)
+ URLEncoder.encode(info.sessionId, UTF_8.name()),
+ ConnectUiUtils.encodeUserId(info.userId))
<tr>
<td>
@@ -406,10 +419,11 @@ private[ui] class SessionStatsPagedTable(
}
override def row(session: SessionInfo): Seq[Node] = {
- val sessionLink = "%s/%s/session/?id=%s".format(
+ val sessionLink = "%s/%s/session/?id=%s&userId=%s".format(
UIUtils.prependBaseUri(request, parent.basePath),
parent.prefix,
- session.sessionId)
+ URLEncoder.encode(session.sessionId, UTF_8.name()),
+ ConnectUiUtils.encodeUserId(session.userId))
<tr>
<td> {session.userId} </td>
<td> <a href={sessionLink}> {session.sessionId} </a> </td>
diff --git
a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerSessionPage.scala
b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerSessionPage.scala
index 391fc6c09fd1..292b0946382f 100644
---
a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerSessionPage.scala
+++
b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/ui/SparkConnectServerSessionPage.scala
@@ -38,10 +38,14 @@ private[ui] class SparkConnectServerSessionPage(parent:
SparkConnectServerTab)
def render(request: HttpServletRequest): Seq[Node] = {
val sessionId = request.getParameter("id")
require(sessionId != null && sessionId.nonEmpty, "Missing id parameter")
+ // userId arrives as a base64url token; see ConnectUiUtils.
+ val userIdParam = request.getParameter("userId")
+ require(userIdParam != null, "Missing userId parameter")
+ val userId = ConnectUiUtils.decodeUserId(userIdParam)
val content = store.synchronized { // make sure all parts in this page are
consistent
store
- .getSession(sessionId)
+ .getSession(userId, sessionId)
.map { sessionStat =>
generateBasicStats(sessionId) ++
<br/> ++
@@ -56,7 +60,7 @@ private[ui] class SparkConnectServerSessionPage(parent:
SparkConnectServerTab)
{sessionStat.totalExecution}
Request(s)
</h4> ++
- generateSQLStatsTable(request, sessionStat.sessionId)
+ generateSQLStatsTable(request, sessionStat.userId,
sessionStat.sessionId)
}
.getOrElse(<div>No information to display for session
{sessionId}</div>)
}
@@ -80,9 +84,12 @@ private[ui] class SparkConnectServerSessionPage(parent:
SparkConnectServerTab)
}
/** Generate stats of batch statements of the Spark Connect server */
- private def generateSQLStatsTable(request: HttpServletRequest, sessionID:
String): Seq[Node] = {
+ private def generateSQLStatsTable(
+ request: HttpServletRequest,
+ userId: String,
+ sessionId: String): Seq[Node] = {
val executionList = store.getExecutionList
- .filter(_.sessionId == sessionID)
+ .filter(exec => exec.userId == userId && exec.sessionId == sessionId)
val numStatement = executionList.size
val table = if (numStatement > 0) {
diff --git
a/sql/connect/server/src/main/scala/org/apache/spark/status/api/v1/connect/ConnectResource.scala
b/sql/connect/server/src/main/scala/org/apache/spark/status/api/v1/connect/ConnectResource.scala
index 6e85ab5ead2b..8185eac4a074 100644
---
a/sql/connect/server/src/main/scala/org/apache/spark/status/api/v1/connect/ConnectResource.scala
+++
b/sql/connect/server/src/main/scala/org/apache/spark/status/api/v1/connect/ConnectResource.scala
@@ -20,7 +20,7 @@ package org.apache.spark.status.api.v1.connect
import jakarta.ws.rs._
import jakarta.ws.rs.core.MediaType
-import org.apache.spark.sql.connect.ui.{ExecutionInfo, SessionInfo,
SparkConnectServerAppStatusStore}
+import org.apache.spark.sql.connect.ui.{ConnectUiUtils, ExecutionInfo,
SessionInfo, SparkConnectServerAppStatusStore}
import org.apache.spark.status.api.v1.{BadParameterException, BaseAppResource,
NotFoundException}
@Produces(Array(MediaType.APPLICATION_JSON))
@@ -37,12 +37,26 @@ private[v1] class ConnectResource extends BaseAppResource {
sessions.map(prepareSessionData)
}
+ // Sessions are keyed by (userId, sessionId), so userId is required. It is a
base64url token (see
+ // ConnectUiUtils); an empty token is a valid empty user id, so only an
absent param is rejected.
@GET
@Path("sessions/{sessionId}")
- def session(@PathParam("sessionId") sessionId: String): SessionData = withUI
{ ui =>
+ def session(
+ @PathParam("sessionId") sessionId: String,
+ @QueryParam("userId") userId: String): SessionData = withUI { ui =>
+ if (userId == null) {
+ throw new BadParameterException("userId is required.")
+ }
+ val decodedUserId =
+ try {
+ ConnectUiUtils.decodeUserId(userId)
+ } catch {
+ case _: IllegalArgumentException =>
+ throw new BadParameterException("userId must be a base64url-encoded
string.")
+ }
val store = new SparkConnectServerAppStatusStore(ui.store.store)
store
- .getSession(sessionId)
+ .getSession(decodedUserId, sessionId)
.map(prepareSessionData)
.getOrElse(throw new NotFoundException("unknown session id: " +
sessionId))
}
diff --git
a/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/ui/SparkConnectServerListenerSuite.scala
b/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/ui/SparkConnectServerListenerSuite.scala
index e9cd7c692373..be31462f87a8 100644
---
a/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/ui/SparkConnectServerListenerSuite.scala
+++
b/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/ui/SparkConnectServerListenerSuite.scala
@@ -51,7 +51,7 @@ class SparkConnectServerListenerSuite
val (statusStore: SparkConnectServerAppStatusStore, listener:
SparkConnectServerListener) =
createAppStatusStore(live)
listener.onOtherEvent(
- SparkListenerConnectSessionStarted("sessionId", "user",
System.currentTimeMillis()))
+ SparkListenerConnectSessionStarted("sessionId", "userId",
System.currentTimeMillis()))
listener.onOtherEvent(
SparkListenerConnectOperationStarted(
jobTag,
@@ -113,16 +113,16 @@ class SparkConnectServerListenerSuite
createAppStatusStore(live)
var time = 0
listener.onOtherEvent(
- SparkListenerConnectSessionStarted("sessionId1", "user",
System.currentTimeMillis()))
+ SparkListenerConnectSessionStarted("sessionId1", "userId",
System.currentTimeMillis()))
time += 1
listener.onOtherEvent(
- SparkListenerConnectSessionStarted("sessionId2", "user",
System.currentTimeMillis()))
+ SparkListenerConnectSessionStarted("sessionId2", "userId",
System.currentTimeMillis()))
time += 1
listener.onOtherEvent(SparkListenerConnectSessionClosed("sessionId1",
"userId", time))
time += 1
listener.onOtherEvent(SparkListenerConnectSessionClosed("sessionId2",
"userId", time))
listener.onOtherEvent(
- SparkListenerConnectSessionStarted("sessionId3", "user",
System.currentTimeMillis()))
+ SparkListenerConnectSessionStarted("sessionId3", "userId",
System.currentTimeMillis()))
time += 1
listener.onOtherEvent(SparkListenerConnectSessionClosed("sessionId3",
"userId", time))
@@ -131,11 +131,78 @@ class SparkConnectServerListenerSuite
}
assert(statusStore.getOnlineSessionNum === 0)
assert(statusStore.getSessionCount === 1)
- assert(statusStore.getSession("sessionId1") === None)
+ assert(statusStore.getSession("userId", "sessionId1") === None)
assert(listener.noLiveData())
}
}
+ Seq(true, false).foreach { live =>
+ test(s"SPARK-58097: sessions sharing a UUID across users stay distinct
(live = $live)") {
+ val (statusStore: SparkConnectServerAppStatusStore, listener:
SparkConnectServerListener) =
+ createAppStatusStore(live, sessionLimit = 10)
+ val sessionId = "shared-session-uuid"
+ // Two users share the same session UUID; Spark Connect must keep them
distinct.
+ listener.onOtherEvent(
+ SparkListenerConnectSessionStarted(sessionId, "userA",
System.currentTimeMillis()))
+ listener.onOtherEvent(
+ SparkListenerConnectSessionStarted(sessionId, "userB",
System.currentTimeMillis()))
+ // One operation against userA's session only.
+ listener.onOtherEvent(
+ SparkListenerConnectOperationStarted(
+ ExecuteJobTag("userA", sessionId, "operationId"),
+ "operationId",
+ System.currentTimeMillis(),
+ sessionId,
+ "userA",
+ "userName",
+ "dummy query",
+ Set()))
+ listener.onOtherEvent(SparkListenerConnectSessionClosed(sessionId,
"userA", 1000L))
+ if (live) {
+ assert(statusStore.getOnlineSessionNum === 1)
+ assert(statusStore.getSession("userB",
sessionId).exists(_.finishTimestamp === 0))
+ }
+ // Close userB at a distinct time so the history store also records both.
+ listener.onOtherEvent(SparkListenerConnectSessionClosed(sessionId,
"userB", 2000L))
+
+ if (!live) {
+ kvstore.close(false)
+ }
+
+ // The sessions stay separate: distinct counts and close times, no
collapse.
+ assert(statusStore.getSessionCount === 2)
+ val sessionA = statusStore.getSession("userA", sessionId)
+ val sessionB = statusStore.getSession("userB", sessionId)
+ assert(sessionA.isDefined && sessionB.isDefined)
+ assert(sessionA.get.totalExecution === 1)
+ assert(sessionB.get.totalExecution === 0)
+ assert(sessionA.get.finishTimestamp === 1000L)
+ assert(sessionB.get.finishTimestamp === 2000L)
+ }
+ }
+
+ test("SPARK-58097: getSession falls back to a scan for legacy
sessionId-keyed rows") {
+ // A legacy store keys SessionInfo on sessionId alone, so the
composite-key read misses while
+ // the row is still enumerable via the view. Force that read to fail to
reproduce it.
+ val legacyStore = new InMemoryStore {
+ override def read[T](klass: Class[T], naturalKey: Object): T = {
+ if (klass == classOf[SessionInfo]) throw new NoSuchElementException()
+ else super.read(klass, naturalKey)
+ }
+ }
+ val store = new SparkConnectServerAppStatusStore(legacyStore)
+ val sessionId = "legacy-session-uuid"
+ legacyStore.write(new SessionInfo(sessionId, 1000L, "userA", 2000L, 3L))
+
+ val session = store.getSession("userA", sessionId)
+ assert(session.isDefined)
+ assert(session.get.userId === "userA")
+ assert(session.get.sessionId === sessionId)
+ assert(session.get.totalExecution === 3L)
+ // A miss with no matching row still returns None.
+ assert(store.getSession("userB", sessionId).isEmpty)
+ }
+
test(
"update execution info when event reordering causes job and sql" +
" start to come after operation closed") {
@@ -240,12 +307,12 @@ class SparkConnectServerListenerSuite
properties
}
- private def createAppStatusStore(live: Boolean) = {
+ private def createAppStatusStore(live: Boolean, sessionLimit: Int = 1) = {
val sparkConf = new SparkConf()
sparkConf
.set(ASYNC_TRACKING_ENABLED, false)
.set(LIVE_ENTITY_UPDATE_PERIOD, 0L)
- .set(CONNECT_UI_SESSION_LIMIT, 1)
+ .set(CONNECT_UI_SESSION_LIMIT, sessionLimit)
.set(CONNECT_UI_STATEMENT_LIMIT, 10)
kvstore = new ElementTrackingStore(new InMemoryStore, sparkConf)
if (live) {
diff --git
a/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/ui/SparkConnectServerPageSuite.scala
b/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/ui/SparkConnectServerPageSuite.scala
index 61d416328d0e..7f6af17bc41b 100644
---
a/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/ui/SparkConnectServerPageSuite.scala
+++
b/sql/connect/server/src/test/scala/org/apache/spark/sql/connect/ui/SparkConnectServerPageSuite.scala
@@ -20,6 +20,7 @@ package org.apache.spark.sql.connect.ui
import java.util.{Calendar, Locale}
import jakarta.servlet.http.HttpServletRequest
+import org.apache.commons.text.StringEscapeUtils
import org.mockito.Mockito.{mock, when, RETURNS_SMART_NULLS}
import org.scalatest.BeforeAndAfter
@@ -110,6 +111,7 @@ class SparkConnectServerPageSuite
val request = mock(classOf[HttpServletRequest])
when(request.getParameter("id")).thenReturn("sessionId")
+
when(request.getParameter("userId")).thenReturn(ConnectUiUtils.encodeUserId("userId"))
val tab = mock(classOf[SparkConnectServerTab], RETURNS_SMART_NULLS)
when(tab.startTime).thenReturn(Calendar.getInstance().getTime)
when(tab.store).thenReturn(store)
@@ -131,4 +133,58 @@ class SparkConnectServerPageSuite
html.contains("collapse-table\" data-bs-toggle=\"collapse\"" +
" data-bs-target=\"#aggregated-sqlsessionstat\""))
}
+
+ test("SPARK-58097: session page only shows the requested user's operations")
{
+ // Two users share the same session UUID, each running a distinct query.
+ kvstore = new ElementTrackingStore(new InMemoryStore, new SparkConf())
+ val listener = new SparkConnectServerListener(kvstore, new SparkConf)
+ val store = new SparkConnectServerAppStatusStore(kvstore)
+ Seq(("userA", "query from A"), ("userB", "query from B")).foreach { case
(user, query) =>
+ val jobTag = s"jobTag-$user"
+ listener.onOtherEvent(
+ SparkListenerConnectSessionStarted("sharedSession", user,
System.currentTimeMillis()))
+ listener.onOtherEvent(
+ SparkListenerConnectOperationStarted(
+ jobTag,
+ "operationId",
+ System.currentTimeMillis(),
+ "sharedSession",
+ user,
+ "userName",
+ query,
+ Set()))
+ listener.onOtherEvent(
+ SparkListenerConnectOperationClosed(jobTag, "operationId",
System.currentTimeMillis()))
+ }
+
+ val request = mock(classOf[HttpServletRequest])
+ when(request.getParameter("id")).thenReturn("sharedSession")
+
when(request.getParameter("userId")).thenReturn(ConnectUiUtils.encodeUserId("userA"))
+ val tab = mock(classOf[SparkConnectServerTab], RETURNS_SMART_NULLS)
+ when(tab.startTime).thenReturn(Calendar.getInstance().getTime)
+ when(tab.store).thenReturn(store)
+ when(tab.appName).thenReturn("testing")
+ when(tab.headerTabs).thenReturn(Seq.empty)
+ val page = new SparkConnectServerSessionPage(tab)
+ val html = page.render(request).toString().toLowerCase(Locale.ROOT)
+
+ // Only userA's operation is listed; userB's must not leak into userA's
session page.
+ assert(html.contains("query from a"))
+ assert(!html.contains("query from b"))
+ }
+
+ test("SPARK-58097: encoded userId survives UI request sanitization") {
+ // Mirror XssSafeRequest.stripXSS: strip newlines/apostrophes, then
HTML-escape (version 4.0).
+ val newlineAndQuote = raw"(?i)(\r\n|\n|\r|%0D%0A|%0A|%0D|'|%27)".r
+ def stripXSS(s: String): String =
+ StringEscapeUtils.escapeHtml4(newlineAndQuote.replaceAllIn(s, ""))
+
+ // Opaque user ids with characters the sanitizer would otherwise strip or
escape.
+ Seq("a'b", "alice+tag", "user=name", "plain", "a/b&c").foreach { userId =>
+ val token = ConnectUiUtils.encodeUserId(userId)
+ // The base64url token passes through request sanitization unchanged, so
it round-trips.
+ assert(stripXSS(token) === token, s"token for '$userId' was altered by
sanitization")
+ assert(ConnectUiUtils.decodeUserId(stripXSS(token)) === userId)
+ }
+ }
}
diff --git
a/sql/connect/server/src/test/scala/org/apache/spark/status/api/v1/connect/ConnectResourceWithActualDataSuite.scala
b/sql/connect/server/src/test/scala/org/apache/spark/status/api/v1/connect/ConnectResourceWithActualDataSuite.scala
index 2b3c9344b1de..c350d768acf1 100644
---
a/sql/connect/server/src/test/scala/org/apache/spark/status/api/v1/connect/ConnectResourceWithActualDataSuite.scala
+++
b/sql/connect/server/src/test/scala/org/apache/spark/status/api/v1/connect/ConnectResourceWithActualDataSuite.scala
@@ -25,6 +25,7 @@ import org.json4s.jackson.JsonMethods
import org.apache.spark.SparkConf
import org.apache.spark.sql.connect.SparkConnectServerTest
+import org.apache.spark.sql.connect.service.SparkListenerConnectSessionStarted
import org.apache.spark.sql.connect.ui.ExecutionState
import org.apache.spark.util.Utils
@@ -73,6 +74,9 @@ class ConnectResourceWithActualDataSuite extends
SparkConnectServerTest {
private def enc(s: String): String =
java.net.URLEncoder.encode(s, java.nio.charset.StandardCharsets.UTF_8)
+ private def encodeUserId(s: String): String =
+ org.apache.spark.sql.connect.ui.ConnectUiUtils.encodeUserId(s)
+
test("Connect REST API exposes sessions and operations") {
// Run a real query through a Connect session; this posts the
session/operation listener
// events that the SparkConnectServerListener writes into the KVStore.
@@ -95,8 +99,11 @@ class ConnectResourceWithActualDataSuite extends
SparkConnectServerTest {
assert(op.get.userId === defaultUserId)
assert(ExecutionState.values.map(_.toString).contains(op.get.state))
- val one =
JsonMethods.parse(getOk(s"/sessions/$defaultSessionId")).extract[TestSessionData]
+ val one = JsonMethods
+
.parse(getOk(s"/sessions/$defaultSessionId?userId=${encodeUserId(defaultUserId)}"))
+ .extract[TestSessionData]
assert(one.sessionId === defaultSessionId)
+ assert(one.userId === defaultUserId)
val oneOp = JsonMethods
.parse(getOk(s"/operations/detail?jobTag=${enc(op.get.jobTag)}"))
@@ -106,6 +113,25 @@ class ConnectResourceWithActualDataSuite extends
SparkConnectServerTest {
}
}
+ test("SPARK-58097: session with an empty user id is addressable via an empty
userId token") {
+ // An empty user id is valid (the protobuf default), and its base64url
token is the empty
+ // string. The JVM client rejects it, so post the event directly to
populate the store.
+ val sessionId = java.util.UUID.randomUUID.toString
+ spark.sparkContext.listenerBus.post(
+ SparkListenerConnectSessionStarted(sessionId, "",
System.currentTimeMillis()))
+ spark.sparkContext.listenerBus.waitUntilEmpty(eventuallyTimeout.toMillis)
+
+ eventuallyWithTimeout {
+ val one = JsonMethods
+ .parse(getOk(s"/sessions/$sessionId?userId="))
+ .extract[TestSessionData]
+ assert(one.sessionId === sessionId)
+ assert(one.userId === "")
+ }
+ // An absent userId parameter is still rejected.
+ assert(get(new URI(s"$baseUrl/sessions/$sessionId").toURL)._1 === 400)
+ }
+
test("SPARK-57941: operation with a user id containing '/' is fetchable via
the jobTag query") {
// The job tag embeds the raw user id, so a '/' in it would break a path
segment. The detail
// endpoint takes the job tag as a query parameter, which handles such
user ids.
@@ -130,7 +156,10 @@ class ConnectResourceWithActualDataSuite extends
SparkConnectServerTest {
}
test("Connect REST API returns 404 for unknown ids") {
- assert(get(new URI(s"$baseUrl/sessions/does-not-exist").toURL)._1 === 404)
+ assert(
+ get(
+ new URI(
+
s"$baseUrl/sessions/does-not-exist?userId=${encodeUserId("nobody")}").toURL)._1
=== 404)
assert(get(new
URI(s"$baseUrl/operations/detail?jobTag=does-not-exist").toURL)._1 === 404)
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]