attilapiros commented on a change in pull request #24499: [SPARK-27677][Core]
Serve local disk persisted blocks by the external service after releasing
executor by dynamic allocation
URL: https://github.com/apache/spark/pull/24499#discussion_r285038949
##########
File path: core/src/main/scala/org/apache/spark/storage/BlockManager.scala
##########
@@ -414,8 +395,9 @@ private[spark] class BlockManager(
*/
def initialize(appId: String): Unit = {
blockTransferService.init(this)
- shuffleClient.init(appId)
-
+ externalShuffleClient.foreach { shuffleClient =>
+ shuffleClient.init(appId)
Review comment:
There were two `init` methods for `BlockTransferService`:
1) One with the `appID` was inherited from `ShuffleClient` and had an empty
body:
https://github.com/apache/spark/blob/5476f6c078a05d8d2dd31acb513d312c9f28dd2c/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ShuffleClient.java#L28-L33
And **I have removed this one** (as it was probably just to avoid an if
condition and a casting of `externalShuffleClient` to `ExternalShuffleClient`
but now we have the if condition within the `Option[ExternalShuffleClient]`).
2) The other `init` is: `org.apache.spark.network.BlockTransferService#init`
which gets a `BlockDataManager`. This one is called right before your comment
in the line 397 (new numbering):
https://github.com/apache/spark/blob/a849554eba22db846bc00efc5b36e2438b481f6a/core/src/main/scala/org/apache/spark/storage/BlockManager.scala#L396-L400
These double `init` was a bit misleading, I am positive this even improves
the readability. Otherwise the `available only after [[init]] is invoked`
appearing in many comments before the `BlockTransferService` methods should
even mention by which of the `init` method exactly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
