This is an automated email from the ASF dual-hosted git repository.
wuyi pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new f077759460c9 [SPARK-48631][CORE][TEST] Fix test "error during
accessing host local dirs for executors"
f077759460c9 is described below
commit f077759460c9801ac46956b6c0e34a52d0a23ff4
Author: Bo Zhang <[email protected]>
AuthorDate: Fri Jun 21 10:37:50 2024 +0800
[SPARK-48631][CORE][TEST] Fix test "error during accessing host local dirs
for executors"
### What changes were proposed in this pull request?
This change fixes test "error during accessing host local dirs for
executors" in ShuffleBlockFetcherIteratorSuite, by setting up `blocksByAddress`
correctly for the `ShuffleBlockFetcherIterator` used in the test.
### Why are the changes needed?
The test is for host local blocks, but the `BlockManagerId` was not set up
correctly, so `ShuffleBlockFetcherIterator` would treat those blocks as remote
blocks instead.
This change is to make the test valid to prevent further breaking changes.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Added an additional check in the test. Also tested locally and verified
that the correct code path was reached.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #46989 from bozhang2820/spark-48631.
Authored-by: Bo Zhang <[email protected]>
Signed-off-by: Yi Wu <[email protected]>
---
.../org/apache/spark/storage/ShuffleBlockFetcherIteratorSuite.scala | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git
a/core/src/test/scala/org/apache/spark/storage/ShuffleBlockFetcherIteratorSuite.scala
b/core/src/test/scala/org/apache/spark/storage/ShuffleBlockFetcherIteratorSuite.scala
index 02900e14b1f6..ca77d2c7b709 100644
---
a/core/src/test/scala/org/apache/spark/storage/ShuffleBlockFetcherIteratorSuite.scala
+++
b/core/src/test/scala/org/apache/spark/storage/ShuffleBlockFetcherIteratorSuite.scala
@@ -392,9 +392,11 @@ class ShuffleBlockFetcherIteratorSuite extends
SparkFunSuite with PrivateMethodT
configureMockTransfer(Map())
val iterator = createShuffleBlockIteratorWithDefaults(
- Map(hostLocalBmId -> toBlockList(hostLocalBlocks.keys, 1L, 1))
+ Map(hostLocalBmId -> toBlockList(hostLocalBlocks.keys, 1L, 1)),
+ blockManager = Some(blockManager)
)
intercept[FetchFailedException] { iterator.next() }
+ verify(mockExternalBlockStoreClient, times(1)).getHostLocalDirs(any(),
any(), any(), any())
}
test("Hit maxBytesInFlight limitation before maxBlocksInFlightPerAddress") {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
