mridulm commented on code in PR #43280:
URL: https://github.com/apache/spark/pull/43280#discussion_r1350809807
##########
core/src/main/java/org/apache/spark/shuffle/api/ShuffleDriverComponents.java:
##########
@@ -66,8 +66,19 @@ default void removeShuffle(int shuffleId, boolean blocking)
{}
* Does this shuffle component support reliable storage - external to the
lifecycle of the
* executor host ? For example, writing shuffle data to a distributed
filesystem or
* persisting it in a remote shuffle service.
+ *
+ * Note: This method is for compatibility with older implementations,
+ * the newer implementation should not use this method.
Review Comment:
This comment addition is incorrect.
A shuffle implementation can either choose to be reliable irrespective of
executor id, or choose to be reliable only for specific subset of executor id's
- that is an implementation choice of the implementation, and not a function of
newer vs older
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
