This is an automated email from the ASF dual-hosted git repository.
zhengruifeng pushed a commit to branch branch-4.3
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.3 by this push:
new 40857dafa1b6 [SPARK-60383][CORE] Keep streaming shuffle implementation
details out of the public API
40857dafa1b6 is described below
commit 40857dafa1b64dbd82d929fa296e4efd05e9bac8
Author: Ruifeng Zheng <[email protected]>
AuthorDate: Thu Oct 22 18:50:21 2026 +0800
[SPARK-60383][CORE] Keep streaming shuffle implementation details out of
the public API
### What changes were proposed in this pull request?
Mark three streaming shuffle implementation APIs as `private[spark]`:
- `ShuffleLocationResponse`
- `BarrierTaskContext.addPostStatusUpdateListener`
- `SparkEnv.streamingShuffleOutputTracker`
Please backport this fix to `branch-4.x` and `branch-4.3`.
### Why are the changes needed?
These public declarations expose internal types that are already
`private[spark]`. Restricting their visibility before the 4.3.0 release avoids
establishing unintended public API commitments.
### Does this PR introduce _any_ user-facing change?
The three internal declarations are no longer public Scala APIs. Their
implementations and runtime behavior are unchanged.
### How was this patch tested?
All 62 tests passed across `TaskContextSuite`, `BarrierTaskContextSuite`,
and `StreamingShuffleOutputTrackerSuite` using `core/testOnly` with Java 17.
The subsequent documentation and deprecation restoration was checked against
the original source, and all three visibility fixes were verified to remain
intact. `git diff --check` passed.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-6)
Closes #59633 from zhengruifeng/core-streaming-shuffle-api-visibility-dev-5.
Authored-by: Ruifeng Zheng <[email protected]>
Signed-off-by: Ruifeng Zheng <[email protected]>
(cherry picked from commit 42230274f919789a86bbe1c37d5cb022c5294dec)
Signed-off-by: Ruifeng Zheng <[email protected]>
---
core/src/main/scala/org/apache/spark/BarrierTaskContext.scala | 3 ++-
core/src/main/scala/org/apache/spark/SparkEnv.scala | 2 +-
.../main/scala/org/apache/spark/StreamingShuffleOutputTracker.scala | 2 +-
3 files changed, 4 insertions(+), 3 deletions(-)
diff --git a/core/src/main/scala/org/apache/spark/BarrierTaskContext.scala
b/core/src/main/scala/org/apache/spark/BarrierTaskContext.scala
index f9cf2c4bc0c9..cdbc01d3b183 100644
--- a/core/src/main/scala/org/apache/spark/BarrierTaskContext.scala
+++ b/core/src/main/scala/org/apache/spark/BarrierTaskContext.scala
@@ -302,7 +302,8 @@ class BarrierTaskContext private[spark] (
taskContext.createResourceUninterruptibly(resourceBuilder)
}
- override def addPostStatusUpdateListener(listener:
PostStatusUpdateListener): TaskContext = {
+ override private[spark] def addPostStatusUpdateListener(
+ listener: PostStatusUpdateListener): TaskContext = {
taskContext.addPostStatusUpdateListener(listener)
this
}
diff --git a/core/src/main/scala/org/apache/spark/SparkEnv.scala
b/core/src/main/scala/org/apache/spark/SparkEnv.scala
index 877a26994376..395dced003fe 100644
--- a/core/src/main/scala/org/apache/spark/SparkEnv.scala
+++ b/core/src/main/scala/org/apache/spark/SparkEnv.scala
@@ -438,7 +438,7 @@ class SparkEnv (
@volatile private var _streamingShuffleOutputTracker:
Option[StreamingShuffleOutputTracker] =
None
- def streamingShuffleOutputTracker: Option[StreamingShuffleOutputTracker] =
+ private[spark] def streamingShuffleOutputTracker:
Option[StreamingShuffleOutputTracker] =
_streamingShuffleOutputTracker
/**
diff --git
a/core/src/main/scala/org/apache/spark/StreamingShuffleOutputTracker.scala
b/core/src/main/scala/org/apache/spark/StreamingShuffleOutputTracker.scala
index df4e76b017db..7b62346d1f72 100644
--- a/core/src/main/scala/org/apache/spark/StreamingShuffleOutputTracker.scala
+++ b/core/src/main/scala/org/apache/spark/StreamingShuffleOutputTracker.scala
@@ -125,7 +125,7 @@ private[spark] class
StreamingShuffleOutputTrackerMasterEndpoint(
}
}
-case class ShuffleLocationResponse(
+private[spark] case class ShuffleLocationResponse(
shuffleTaskLocations: Map[Long, StreamingShuffleTaskLocation],
numShuffleWriterTasks: Int)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]