prakharjain09 commented on a change in pull request #27864: [SPARK-20732][CORE]
Decommission cache blocks to other executors when an executor is decommissioned
URL: https://github.com/apache/spark/pull/27864#discussion_r405545766
##########
File path: core/src/main/scala/org/apache/spark/internal/config/package.scala
##########
@@ -413,6 +413,21 @@ package object config {
.intConf
.createWithDefault(1)
+ private[spark] val STORAGE_DECOMMISSION_ENABLED =
+ ConfigBuilder("spark.storage.decommission.enabled")
+ .doc("Whether to decommission the block manager when decommissioning
executor")
Review comment:
@dongjoon-hyun Thanks for the detailed review.
I couldn't understand your question with respect to dynamic allocation
completely (as per my understanding dynamic allocation doesn't initiates
decommissioning). This change works together with SPARK-20628
[PR](https://github.com/apache/spark/pull/26440). After SPARK-20628, Spark
scheduler starts receiving decommissionExecutor message (because of AWS spot
loss, Azure low-priority VMs etc). In such case, we should offload cache data
from that executor to other active executors so that we don't eventually loose
those cache blocks. This is done in a best effort way i.e. move cache blocks if
space is available on active executors. If space is not available on other
active executors, then keep the cache blocks.
Next step after this change is to initiate decommissioning from
DynamicAllocation. Let say minExecutor=2, maxExecutor=50. A spark application
is running at max 50 executors and all of them have little bit of cache data
(say 2GB out of total available 5GB capacity). So DynamicAllocation is not able
to downscale any of the executors (although they are idle). In future,
DynamicAllocation can leverage changes in this PR to do defragmentation of
cache data to fewer set of executors so that some of the executors can be freed
up.
Hope this answers your question.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
