This is an automated email from the ASF dual-hosted git repository.
mridulm80 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 8fabd5e [SPARK-38428][SHUFFLE] Check the FetchShuffleBlocks message
only once to improve iteration in external shuffle service
8fabd5e is described below
commit 8fabd5efe311347a4d1c441813dd0bd3148655df
Author: weixiuli <[email protected]>
AuthorDate: Tue Mar 8 12:59:34 2022 -0600
[SPARK-38428][SHUFFLE] Check the FetchShuffleBlocks message only once to
improve iteration in external shuffle service
### What changes were proposed in this pull request?
Currently, the FetchShuffleBlocks is checked in each element of a
ShuffleManagedBufferIterator, which is unnecessary and it only needs to be
checked once in the ShuffleManagedBufferIterator constructor.
### Why are the changes needed?
To improve performance.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Existing unittests.
Closes #35743 from weixiuli/SPARK-38428-improve-iteration.
Authored-by: weixiuli <[email protected]>
Signed-off-by: Mridul Muralidharan <mridul<at>gmail.com>
---
.../org/apache/spark/network/shuffle/ExternalBlockHandler.java | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git
a/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalBlockHandler.java
b/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalBlockHandler.java
index 1e413f6..52bc0f9c 100644
---
a/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalBlockHandler.java
+++
b/common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalBlockHandler.java
@@ -512,14 +512,14 @@ public class ExternalBlockHandler extends RpcHandler
mapIds = msg.mapIds;
reduceIds = msg.reduceIds;
batchFetchEnabled = msg.batchFetchEnabled;
- }
-
- @Override
- public boolean hasNext() {
// mapIds.length must equal to reduceIds.length, and the passed in
FetchShuffleBlocks
// must have non-empty mapIds and reduceIds, see the checking logic in
// OneForOneBlockFetcher.
assert(mapIds.length != 0 && mapIds.length == reduceIds.length);
+ }
+
+ @Override
+ public boolean hasNext() {
return mapIdx < mapIds.length && reduceIdx < reduceIds[mapIdx].length;
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
