GitHub user erenavsarogullari opened a pull request:
https://github.com/apache/spark/pull/16813
[SPARK-19466][CORE][SCHEDULER] Improve Fair Scheduler Logging
Fair Scheduler Logging for the following cases can be useful for the user.
1. If **valid** `spark.scheduler.allocation.file` property is set, user can
be informed and aware which scheduler file is processed when `SparkContext`
initializes.
2. If **invalid** `spark.scheduler.allocation.file` property is set,
currently, the following stacktrace is shown to user. In addition to this, more
meaningful message can be shown to user by emphasizing the problem at building
level of fair scheduler. Also other potential issues can be covered at this
level as **Fair Scheduler can not be built. + exception stacktrace**
```
Exception in thread "main" java.io.FileNotFoundException: INVALID_FILE (No
such file or directory)
at java.io.FileInputStream.open0(Native Method)
at java.io.FileInputStream.open(FileInputStream.java:195)
at java.io.FileInputStream.<init>(FileInputStream.java:138)
at java.io.FileInputStream.<init>(FileInputStream.java:93)
at
org.apache.spark.scheduler.FairSchedulableBuilder$$anonfun$buildPools$1.apply(SchedulableBuilder.scala:76)
at
org.apache.spark.scheduler.FairSchedulableBuilder$$anonfun$buildPools$1.apply(SchedulableBuilder.scala:75)
```
3. If `spark.scheduler.allocation.file` property is not set and **default**
fair scheduler file (**fairscheduler.xml**) is found in classpath, it will be
loaded but currently, user is not informed for using default file so logging
can be useful as **Fair Scheduler file: fairscheduler.xml is found successfully
and will be parsed.**
4. If **spark.scheduler.allocation.file** property is not set and
**default** fair scheduler file does not exist in classpath, currently, user is
not informed so logging can be useful as **No Fair Scheduler file found.**
Also this PR is related with https://github.com/apache/spark/pull/15237 to
emphasize fileName in warning logs when fair scheduler file has invalid
minShare, weight or schedulingMode values.
## How was this patch tested?
Added new Unit Tests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/erenavsarogullari/spark SPARK-19466
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/16813.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #16813
----
commit 631354c93c294b6820f112c31e27c0ee35a8ab57
Author: erenavsarogullari <[email protected]>
Date: 2017-02-05T20:59:40Z
Improve Fair Scheduler Logging
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
