mridulm commented on code in PR #2444:
URL: https://github.com/apache/celeborn/pull/2444#discussion_r1550976377
##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -4172,6 +4173,14 @@ object CelebornConf extends Logging {
.booleanConf
.createWithDefault(true)
+ val SPARK_SHUFFLE_FALLBACK_ENABLED: ConfigEntry[Boolean] =
+ buildConf("celeborn.client.spark.shuffle.fallback.enabled")
+ .categories("client")
+ .version("0.5.0")
+ .doc("When false, job failed immediately instead of trying to fall back
to Spark's default shuffle.")
Review Comment:
nit:
```suggestion
.doc("When false, job will not attempt to fallback to Spark's default
shuffle.")
```
##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -4172,6 +4173,14 @@ object CelebornConf extends Logging {
.booleanConf
.createWithDefault(true)
+ val SPARK_SHUFFLE_FALLBACK_ENABLED: ConfigEntry[Boolean] =
Review Comment:
This config seems to be in conflict with
`celeborn.client.spark.shuffle.forceFallback.enabled` and
`celeborn.client.spark.shuffle.forceFallback.numPartitionsThreshold`.
Add to the documentation about this interaction as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
