This is an automated email from the ASF dual-hosted git repository.
mridulm80 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 96bcb04 [SPARK-38344][SHUFFLE] Avoid to submit task when there are no
requests to push up in push-based shuffle
96bcb04 is described below
commit 96bcb0406ea229b1c5ecbb98a1de6ba0d6a238b4
Author: weixiuli <[email protected]>
AuthorDate: Tue Mar 1 20:20:30 2022 -0600
[SPARK-38344][SHUFFLE] Avoid to submit task when there are no requests to
push up in push-based shuffle
### What changes were proposed in this pull request?
Avoid to submit task when there are no requests to push up in push-based
shuffle.
### Why are the changes needed?
This is a performance improvement to the existing functionality.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Pass GA.
Existing unittests.
Closes #35675 from weixiuli/SPARK-38344-SBS.
Authored-by: weixiuli <[email protected]>
Signed-off-by: Mridul Muralidharan <mridul<at>gmail.com>
---
.../main/scala/org/apache/spark/shuffle/ShuffleBlockPusher.scala | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git
a/core/src/main/scala/org/apache/spark/shuffle/ShuffleBlockPusher.scala
b/core/src/main/scala/org/apache/spark/shuffle/ShuffleBlockPusher.scala
index d6972cd..230ec7e 100644
--- a/core/src/main/scala/org/apache/spark/shuffle/ShuffleBlockPusher.scala
+++ b/core/src/main/scala/org/apache/spark/shuffle/ShuffleBlockPusher.scala
@@ -118,11 +118,11 @@ private[spark] class ShuffleBlockPusher(conf: SparkConf)
extends Logging {
pushRequests ++= Utils.randomize(requests)
if (pushRequests.isEmpty) {
notifyDriverAboutPushCompletion()
+ } else {
+ submitTask(() => {
+ tryPushUpToMax()
+ })
}
-
- submitTask(() => {
- tryPushUpToMax()
- })
}
private[shuffle] def tryPushUpToMax(): Unit = {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
