attilapiros commented on a change in pull request #29226:
URL: https://github.com/apache/spark/pull/29226#discussion_r460504301
##########
File path:
core/src/test/scala/org/apache/spark/storage/BlockManagerDecommissionIntegrationSuite.scala
##########
@@ -107,6 +115,21 @@ class BlockManagerDecommissionIntegrationSuite extends
SparkFunSuite with LocalS
}
override def onBlockUpdated(blockUpdated: SparkListenerBlockUpdated):
Unit = {
+ if (blockUpdated.blockUpdatedInfo.blockId.isRDD && persist) {
+ // Persisted RDD blocks are a bit weirder than shuffle blocks: Even
though
+ // the tasks are run say on executors (0, 1, 2), the RDD blocks
might end up only
+ // on executors 0 and 1. So we cannot just indiscriminately
decommission any executor.
Review comment:
Please investigate why this placement is happening (as at first it
sounds a bit strange).
You might find a more important bug to fix or very good lesson about the
internals of RDD persistence.
Tipp: please double check "though the tasks are run say on executors (0, 1,
2)" part!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
