vanzin commented on a change in pull request #24554: [SPARK-27622][Core]
Avoiding the network when block manager fetches disk persisted RDD blocks from
the same host
URL: https://github.com/apache/spark/pull/24554#discussion_r288346640
##########
File path:
core/src/main/scala/org/apache/spark/storage/BlockManagerMessages.scala
##########
@@ -93,11 +94,21 @@ private[spark] object BlockManagerMessages {
case class GetLocations(blockId: BlockId) extends ToBlockManagerMaster
- case class GetLocationsAndStatus(blockId: BlockId) extends
ToBlockManagerMaster
+ case class GetLocationsAndStatus(blockId: BlockId, requesterHost: String)
+ extends ToBlockManagerMaster
- // The response message of `GetLocationsAndStatus` request.
- case class BlockLocationsAndStatus(locations: Seq[BlockManagerId], status:
BlockStatus) {
- assert(locations.nonEmpty)
+ /**
+ * The response message of `GetLocationsAndStatus` request.
+ *
+ * @param localDirs if it is persisted-to-disk on the same host as the
requester executor is
+ * running on then localDirs will be Some and the cached
data will be in a file
+ * in one of those dirs, otherwise it is None.
+ */
+ case class BlockLocationsAndStatus(
+ locations: Seq[BlockManagerId],
+ status: BlockStatus,
+ localDirs: Option[Array[String]]) {
+ assert(locations.nonEmpty || localDirs.isDefined)
Review comment:
The assert can be reverted to the old one, right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
