dongjoon-hyun commented on code in PR #39703:
URL: https://github.com/apache/spark/pull/39703#discussion_r1083692686
##########
core/src/main/scala/org/apache/spark/scheduler/SchedulableBuilder.scala:
##########
@@ -86,10 +87,17 @@ private[spark] class FairSchedulableBuilder(val rootPool:
Pool, sc: SparkContext
logInfo(s"Creating Fair Scheduler pools from default file:
$DEFAULT_SCHEDULER_FILE")
Some((is, DEFAULT_SCHEDULER_FILE))
} else {
- logWarning("Fair Scheduler configuration file not found so jobs will
be scheduled in " +
- s"FIFO order. To use fair scheduling, configure pools in
$DEFAULT_SCHEDULER_FILE or " +
- s"set ${SCHEDULER_ALLOCATION_FILE.key} to a file that contains the
configuration.")
- None
+ val is =
Utils.getSparkClassLoader.getResourceAsStream(DEFAULT_SCHEDULER_TEMPLATE_FILE)
+ if (is != null) {
+ logInfo("Creating Fair Scheduler pools from default template file:
" +
+ s"$DEFAULT_SCHEDULER_TEMPLATE_FILE.")
+ Some((is, DEFAULT_SCHEDULER_TEMPLATE_FILE))
+ } else {
+ logWarning("Fair Scheduler configuration file not found so jobs
will be scheduled in " +
+ s"FIFO order. To use fair scheduling, configure pools in
$DEFAULT_SCHEDULER_FILE " +
+ s"or set ${SCHEDULER_ALLOCATION_FILE.key} to a file that
contains the configuration.")
+ None
+ }
Review Comment:
First of all, this is not a testing issue. As I wrote in the PR description,
our documentation is wrong. It says `spark.scheduler.mode=FAIR` will return a
FAIR scheduler. However, we are getting `FIFO` scheduler now.
> Note - if this is only for testing, we can special case it that way via
spark.testing
`None` is the previous behavior which ends up with `FIFO` scheduler with the
WARNING message, `23/01/22 14:47:38 WARN FairSchedulableBuilder: Fair Scheduler
configuration file not found so jobs will be scheduled in FIFO order. To use
fair scheduling, configure pools in fairscheduler.xml or set
spark.scheduler.allocation.file to a file that contains the configuration.`
> Instead, why not simply rely on returning None here ?
Got it. I understand your point about the `template` file. The reason why I
tried to use template file is that I cannot put the real `fairscheduler.xml`
file because it can be used already in the production.
> We should not be relying on template file - in deployments, template file
can be invalid - admin's are not expecting it to be read by spark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
