This is an automated email from the ASF dual-hosted git repository.
yao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 1359c1327345 [SPARK-45776][CORE] Remove the defensive null check for
`MapOutputTrackerMaster#unregisterShuffle` added in SPARK-39553
1359c1327345 is described below
commit 1359c1327345efdf9a35c46a355b5f928ac33e6d
Author: yangjie01 <[email protected]>
AuthorDate: Fri Nov 3 17:56:29 2023 +0800
[SPARK-45776][CORE] Remove the defensive null check for
`MapOutputTrackerMaster#unregisterShuffle` added in SPARK-39553
### What changes were proposed in this pull request?
This pr Remove the defensive null check for
`MapOutputTrackerMaster#unregisterShuffle` added in SPARK-39553.
### Why are the changes needed?
https://github.com/scala/bug/issues/12613 has been fixed in Scala 2.13.9.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Existing test like `SPARK-39553: Multi-thread unregister shuffle shouldn't
throw NPE` in `MapOutputTrackerSuite`
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #43644 from LuciferYang/remove-39553-null-check.
Lead-authored-by: yangjie01 <[email protected]>
Co-authored-by: YangJie <[email protected]>
Signed-off-by: Kent Yao <[email protected]>
---
core/src/main/scala/org/apache/spark/MapOutputTracker.scala | 8 ++------
1 file changed, 2 insertions(+), 6 deletions(-)
diff --git a/core/src/main/scala/org/apache/spark/MapOutputTracker.scala
b/core/src/main/scala/org/apache/spark/MapOutputTracker.scala
index 6d5c04635ad7..a787cdefe808 100644
--- a/core/src/main/scala/org/apache/spark/MapOutputTracker.scala
+++ b/core/src/main/scala/org/apache/spark/MapOutputTracker.scala
@@ -906,12 +906,8 @@ private[spark] class MapOutputTrackerMaster(
/** Unregister shuffle data */
def unregisterShuffle(shuffleId: Int): Unit = {
shuffleStatuses.remove(shuffleId).foreach { shuffleStatus =>
- // SPARK-39553: Add protection for Scala 2.13 due to
https://github.com/scala/bug/issues/12613
- // We should revert this if Scala 2.13 solves this issue.
- if (shuffleStatus != null) {
- shuffleStatus.invalidateSerializedMapOutputStatusCache()
- shuffleStatus.invalidateSerializedMergeOutputStatusCache()
- }
+ shuffleStatus.invalidateSerializedMapOutputStatusCache()
+ shuffleStatus.invalidateSerializedMergeOutputStatusCache()
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
