This is an automated email from the ASF dual-hosted git repository.
rexxiong pushed a commit to branch branch-0.3
in repository https://gitbox.apache.org/repos/asf/incubator-celeborn.git
The following commit(s) were added to refs/heads/branch-0.3 by this push:
new e5bd8073b [CELEBORN-1026] Optimize registerShuffle fallback log
e5bd8073b is described below
commit e5bd8073b0b2e8b6a1177798ce72c6e5c5b0fe66
Author: sychen <[email protected]>
AuthorDate: Wed Oct 11 20:25:44 2023 +0800
[CELEBORN-1026] Optimize registerShuffle fallback log
### What changes were proposed in this pull request?
### Why are the changes needed?
According to https://github.com/apache/incubator-celeborn/pull/1955 , when
celeborn is not available, `spark.dynamicAllocation.enabled=true` and
`spark.shuffle.service. enabled=true`, shuffle data should not be lost.
### Does this PR introduce _any_ user-facing change?
### How was this patch tested?
Closes #1963 from cxzl25/CELEBORN-1026.
Authored-by: sychen <[email protected]>
Signed-off-by: Shuang <[email protected]>
(cherry picked from commit 62ba44d8daf39ab13726a3579e0adc2e47fa226f)
Signed-off-by: Shuang <[email protected]>
---
.../java/org/apache/spark/shuffle/celeborn/SparkShuffleManager.java | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git
a/client-spark/spark-3/src/main/java/org/apache/spark/shuffle/celeborn/SparkShuffleManager.java
b/client-spark/spark-3/src/main/java/org/apache/spark/shuffle/celeborn/SparkShuffleManager.java
index eb530555f..e93a6b40a 100644
---
a/client-spark/spark-3/src/main/java/org/apache/spark/shuffle/celeborn/SparkShuffleManager.java
+++
b/client-spark/spark-3/src/main/java/org/apache/spark/shuffle/celeborn/SparkShuffleManager.java
@@ -127,7 +127,8 @@ public class SparkShuffleManager implements ShuffleManager {
if (fallbackPolicyRunner.applyAllFallbackPolicy(
lifecycleManager, dependency.partitioner().numPartitions())) {
- if (conf.getBoolean("spark.dynamicAllocation.enabled", false)) {
+ if (conf.getBoolean("spark.dynamicAllocation.enabled", false)
+ && !conf.getBoolean("spark.shuffle.service.enabled", false)) {
logger.error(
"DRA is enabled but we fallback to vanilla Spark
SortShuffleManager for "
+ "shuffle: {} due to fallback policy. It may cause block can
not found when reducer "
