[jira] [Commented] (SPARK-39553) Failed to remove shuffle ${shuffleId} - null

[Yang Jie (Jira)]

Title: Message Title

Yang Jie commented on SPARK-39553   Re: Failed to remove shuffle ${shuffleId} - null chang org.apache.spark.MapOutputTrackerMaster#unregisterShuffle method from   def unregisterShuffle(shuffleId: Int): Unit = { shuffleStatuses.remove(shuffleId).foreach { shuffleStatus => shuffleStatus.invalidateSerializedMapOutputStatusCache() shuffleStatus.invalidateSerializedMergeOutputStatusCache() } }     to     def unregisterShuffle(shuffleId: Int): Unit = { shuffleStatuses.remove(shuffleId) match { case Some(shuffleStatus) => if (shuffleStatus == null) { logWarning( s"shuffleId = $shuffleId, shuffleStatuses = ${shuffleStatuses.mkString(",")}") } else { shuffleStatus.invalidateSerializedMapOutputStatusCache() shuffleStatus.invalidateSerializedMergeOutputStatusCache() } case _ => // do nothing } }     then test sql/core module I found the following logs, there is really `Some(null)`     20:40:51.705 WARN org.apache.spark.MapOutputTrackerMaster: shuffleId = 8, shuffleStatuses = 6 -> org.apache.spark.ShuffleStatus@921c8d9,7 -> org.apache.spark.ShuffleStatus@5fde57c7,9 -> org.apache.spark.ShuffleStatus@242654eb,10 -> org.apache.spark.ShuffleStatus@460561a4,11 -> org.apache.spark.ShuffleStatus@41650ebd   Add Comment   This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)