attilapiros commented on pull request #28708:
URL: https://github.com/apache/spark/pull/28708#issuecomment-638174017
I think we should test the offloading of the shuffle blocks at the unit test
level so I suggest to add new test to the `BlockManagerSuite`:
```scala
test("test migration of shuffle blocks during decommissioning") {
val store1 = makeBlockManager(3500, "exec1")
val store2 = makeBlockManager(3500, "exec2")
val blockSize = 5
val shuffleDataBlockContent = Array[Byte](0, 1, 2, 3, 4)
val shuffleData = ShuffleDataBlockId(0, 0, 0)
Files.write(store1.diskBlockManager.getFile(shuffleData).toPath(),
shuffleDataBlockContent)
val shuffleIndexBlockContent = Array[Byte](5, 6, 7, 8, 9)
val shuffleIndex = ShuffleIndexBlockId(0, 0, 0)
Files.write(store1.diskBlockManager.getFile(shuffleIndex).toPath(),
shuffleIndexBlockContent)
mapOutputTracker.registerShuffle(0, 1)
mapOutputTracker.registerMapOutput(0, 0,
MapStatus(store1.blockManagerId, Array(blockSize), 0))
assert(mapOutputTracker.shuffleStatuses(0).mapStatuses(0).location ===
store1.blockManagerId)
val env = mock(classOf[SparkEnv])
when(env.blockManager).thenReturn(store1)
when(env.conf).thenReturn(conf)
SparkEnv.set(env)
store1.offloadShuffleBlocks()
eventually(timeout(1.second), interval(10.milliseconds)) {
assert(mapOutputTracker.shuffleStatuses(0).mapStatuses(0).location ===
store2.blockManagerId)
}
assert(Files.readAllBytes(store2.diskBlockManager.getFile(shuffleData).toPath())
=== shuffleDataBlockContent)
assert(Files.readAllBytes(store2.diskBlockManager.getFile(shuffleIndex).toPath())
=== shuffleIndexBlockContent)
}
```
I think this could be a good starting point for any other tests at this
level. This way we could check the streaming upload here and remove it from the
`BlockManagerReplicationSuite` which is a higher level test (gravitate towards
the integration testing). What do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
