This is an automated email from the ASF dual-hosted git repository.
rexxiong pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/celeborn.git
The following commit(s) were added to refs/heads/main by this push:
new 14d9cd130 [CELEBORN-1685][FOLLOWUP]
CelebornShuffleFallbackPolicyRunner should compute shuffleFallbackCounts via
class name of ShuffleFallbackPolicy implementation
14d9cd130 is described below
commit 14d9cd130dd2c18cea60674406c124096800fe58
Author: SteNicholas <[email protected]>
AuthorDate: Tue Nov 12 20:56:27 2024 +0800
[CELEBORN-1685][FOLLOWUP] CelebornShuffleFallbackPolicyRunner should
compute shuffleFallbackCounts via class name of ShuffleFallbackPolicy
implementation
### What changes were proposed in this pull request?
`CelebornShuffleFallbackPolicyRunner` should compute
`shuffleFallbackCounts` via class name of `ShuffleFallbackPolicy`
implementation.
Follow up #2891.
### Why are the changes needed?
`CelebornShuffleFallbackPolicyRunner` computes `shuffleFallbackCounts` via
class name of `Option` instead of class name of `ShuffleFallbackPolicy`
implementation at present.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
CI.
Closes #2909 from SteNicholas/CELEBORN-1685.
Authored-by: SteNicholas <[email protected]>
Signed-off-by: Shuang <[email protected]>
---
.../spark/shuffle/celeborn/CelebornShuffleFallbackPolicyRunner.scala | 2 +-
.../spark/shuffle/celeborn/CelebornShuffleFallbackPolicyRunner.scala | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git
a/client-spark/spark-2/src/main/scala/org/apache/spark/shuffle/celeborn/CelebornShuffleFallbackPolicyRunner.scala
b/client-spark/spark-2/src/main/scala/org/apache/spark/shuffle/celeborn/CelebornShuffleFallbackPolicyRunner.scala
index 3ba52ef4d..8b1b51628 100644
---
a/client-spark/spark-2/src/main/scala/org/apache/spark/shuffle/celeborn/CelebornShuffleFallbackPolicyRunner.scala
+++
b/client-spark/spark-2/src/main/scala/org/apache/spark/shuffle/celeborn/CelebornShuffleFallbackPolicyRunner.scala
@@ -45,7 +45,7 @@ class CelebornShuffleFallbackPolicyRunner(conf: CelebornConf)
extends Logging {
"Fallback to spark built-in shuffle implementation is prohibited.")
} else {
lifecycleManager.shuffleFallbackCounts.compute(
- fallbackPolicy.getClass.getName,
+ fallbackPolicy.get.getClass.getName,
new BiFunction[String, java.lang.Long, java.lang.Long] {
override def apply(k: String, v: java.lang.Long): java.lang.Long =
{
if (v == null) {
diff --git
a/client-spark/spark-3/src/main/scala/org/apache/spark/shuffle/celeborn/CelebornShuffleFallbackPolicyRunner.scala
b/client-spark/spark-3/src/main/scala/org/apache/spark/shuffle/celeborn/CelebornShuffleFallbackPolicyRunner.scala
index 8a2b7c889..c01a67218 100644
---
a/client-spark/spark-3/src/main/scala/org/apache/spark/shuffle/celeborn/CelebornShuffleFallbackPolicyRunner.scala
+++
b/client-spark/spark-3/src/main/scala/org/apache/spark/shuffle/celeborn/CelebornShuffleFallbackPolicyRunner.scala
@@ -43,7 +43,7 @@ class CelebornShuffleFallbackPolicyRunner(conf: CelebornConf)
extends Logging {
"Fallback to spark built-in shuffle implementation is prohibited.")
} else {
lifecycleManager.shuffleFallbackCounts.compute(
- fallbackPolicy.getClass.getName,
+ fallbackPolicy.get.getClass.getName,
(_, v) => {
if (v == null) {
1L
