littlexyw commented on code in PR #2444:
URL: https://github.com/apache/celeborn/pull/2444#discussion_r1554808574
##########
common/src/main/scala/org/apache/celeborn/common/CelebornConf.scala:
##########
@@ -4172,6 +4173,14 @@ object CelebornConf extends Logging {
.booleanConf
.createWithDefault(true)
+ val SPARK_SHUFFLE_FALLBACK_ENABLED: ConfigEntry[Boolean] =
Review Comment:
> If we accept breaking change, how about
>
> ```
> celeborn.client.spark.shuffle.fallback.policy=[ALWAYS|NEVER|AUTO]
> celeborn.client.spark.shuffle.fallback.numPartitionsThreshold
> ```
Are you saying that celeborn.client.spark.shuffle.forceFallback.enabled and
celeborn.client.spark.shuffle.fallback.enabled need to be replaced with
celeborn.client.spark.shuffle.fallback.policy? What I want is to provide an
option to cancel any fallback to external shuffle service for the sake of
safety. Like insufficient quota or workers,
celeborn.client.spark.shuffle.forceFallback.enabled=true is just one cause for
fallback and can be disabled by
celeborn.client.spark.shuffle.fallback.enabled=false.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
