attilapiros commented on a change in pull request #31763:
URL: https://github.com/apache/spark/pull/31763#discussion_r589060306
##########
File path: core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala
##########
@@ -52,6 +52,13 @@ private[spark] sealed trait MapStatus {
* partitionId of the task or taskContext.taskAttemptId is used.
*/
def mapId: Long
+
+ /**
+ * Extra metadata for map status. This could be used by different
ShuffleManager implementation
+ * to store information they need. For example, a Remote Shuffle Service
ShuffleManager could
+ * store shuffle server information and let reducer task know where to fetch
shuffle data.
+ */
+ def metadata: Option[Serializable]
Review comment:
@Ngone51 I agree with you that finishing the design laid out in
SPARK-25299 would be much better.
This is why I opened https://github.com/apache/spark/pull/30763 as a copy of
Matthew Cheah's original PR for SPARK-31801 (because he is busy with other
projects) and kept it up-to-date several times with the master.
But it haven't got enough reviews and I wouldn't want to block @hiboyang
further, https://github.com/apache/spark/pull/30004#issuecomment-782915655.
I am sure with your help we can complete SPARK-31801 and be on the road of
SPARK-25299.
So next week I will do the conflict resolution and ping you when the PR is
ready for review. Is this okay?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
