agrawaldevesh commented on a change in pull request #29211:
URL: https://github.com/apache/spark/pull/29211#discussion_r465266535
##########
File path:
core/src/test/scala/org/apache/spark/storage/BlockManagerDecommissionIntegrationSuite.scala
##########
@@ -266,18 +266,17 @@ class BlockManagerDecommissionIntegrationSuite extends
SparkFunSuite with LocalS
val execIdToBlocksMapping = storageStatus.map(
status => (status.blockManagerId.executorId, status.blocks)).toMap
// No cached blocks should be present on executor which was decommissioned
-
assert(execIdToBlocksMapping(execToDecommission).keys.filter(_.isRDD).toSeq ===
Seq(),
+ assert(
+ !execIdToBlocksMapping.contains(execToDecommission) ||
+ execIdToBlocksMapping(execToDecommission).keys.filter(_.isRDD).toSeq ===
Seq(),
"Cache blocks should be migrated")
if (persist) {
// There should still be all the RDD blocks cached
assert(execIdToBlocksMapping.values.flatMap(_.keys).count(_.isRDD) ===
numParts)
}
- // Make the executor we decommissioned exit
- sched.client.killExecutors(List(execToDecommission))
-
- // Wait for the executor to be removed
- executorRemovedSem.acquire(1)
+ // Wait for the executor to be removed automatically after migration.
+ assert(executorRemovedSem.tryAcquire(1, 5L, TimeUnit.MINUTES))
Review comment:
I get that it can take up to 5 minutes, but I would like to understand
better whether it should take 5 minutes ? The executor is supposed to exit as
soon as the migration is over, right ? Or if there is some polling there --
that polling should ideally be configurable. So the only thing we are really
waiting for here is the migration : That migration shouldn't take 5 minutes --
we don't have that much data to migrate.
Perhaps you can greatly reduce the executor heartbeat interval such that
this can be sped up ? (I use this trick in the other block manager
decommissioning test)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
