cfmcgrady commented on code in PR #2494:
URL: https://github.com/apache/celeborn/pull/2494#discussion_r1592256952
##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -4370,10 +4369,13 @@ object CelebornConf extends Logging {
buildConf("celeborn.client.spark.shuffle.fallback.policy")
.categories("client")
.version("0.5.0")
- .doc(
- s"Celeborn supports the following kind of fallback policies. 1.
${FallbackPolicy.ALWAYS.name}: force fallback shuffle to Spark's default; " +
- s"2. ${FallbackPolicy.AUTO.name}: consider other factors like
availability of enough workers and quota, or whether shuffle of partition
number is lower than
celeborn.client.spark.shuffle.forceFallback.numPartitionsThreshold; " +
- s"3. ${FallbackPolicy.NEVER.name}: the job will fail if it is
concluded that fallback is required based on factors above.")
+ .doc(s"Celeborn supports the following kind of fallback policies. " +
+ s"1. ${FallbackPolicy.ALWAYS.name}: always use spark built-in shuffle
implementation; " +
+ s"2. ${FallbackPolicy.AUTO.name}: prefer to use celeborn shuffle
implementation, and fallback to use spark " +
+ s"built-in shuffle implementation based on certain factors, e.g.
availability of enough workers and quota, " +
+ s"shuffle partition number; " +
+ s"3. ${FallbackPolicy.NEVER.name}: always use celeborn shuffle
implementation, and fail fast when it it is " +
+ s"concluded that fallback is required based on factors above.")
Review Comment:
```suggestion
.doc("Celeborn supports the following kind of fallback policies. " +
s"1. ${FallbackPolicy.ALWAYS.name}: always use spark built-in
shuffle implementation; " +
s"2. ${FallbackPolicy.AUTO.name}: prefer to use celeborn shuffle
implementation, and fallback to use spark " +
"built-in shuffle implementation based on certain factors, e.g.
availability of enough workers and quota, " +
"shuffle partition number; " +
s"3. ${FallbackPolicy.NEVER.name}: always use celeborn shuffle
implementation, and fail fast when it it is " +
"concluded that fallback is required based on factors above.")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
