Ngone51 commented on a change in pull request #31876:
URL: https://github.com/apache/spark/pull/31876#discussion_r615927199
##########
File path: core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala
##########
@@ -52,8 +53,20 @@ private[spark] sealed trait MapStatus {
* partitionId of the task or taskContext.taskAttemptId is used.
*/
def mapId: Long
-}
+ protected def loadLocation(in: ObjectInput): Location = {
+ val conf = SparkEnv.get.conf
+ conf.get(config.SHUFFLE_LOCATION_PLUGIN_CLASS).map { locClass =>
+ val loc = Utils.loadExtensions(
+ classOf[Location],
+ Seq(locClass),
+ conf
+ ).head
+ loc.readExternal(in)
+ loc
Review comment:
I may not understand well about the usage of `LocationFactory` here.
But, the problem is, we need a fresh `Location` instance here every time and
then make it to a specific one by `readExternal(...)`. So, I'm not sure what do
we want to cache here? Or, you mean we can cache the reflected `class`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
