GitHub user vanzin opened a pull request:
https://github.com/apache/spark/pull/10205
[RFC] Add type-safe config keys to SparkConf.
This is, in a way, the basics to enable SPARK-529 (which was closed as
won't fix but I think is still valuable). In fact, Spark SQL created
something for that, and this change basically factors out that code
and inserts it into SparkConf, with some extra bells and whistles.
To showcase the usage of this pattern, I modified the YARN backend
to use the new config keys (defined in the new YarnConfigKeys object).
Most of the changes are mechanic, although logic had to be slightly
modified in a handful of places.
I did not port the conf entry registry that SQLConf keeps back to
core, although in the future that could be a useful addition (e.g.
to programatically list all known config options). That required
some code duplication in SQLConf, although it shares the actual
implementation of the entries with SparkConf.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/vanzin/spark conf-opts
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/10205.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #10205
----
commit 8b00d76d06012b7777b461c5ac8cb93b5aa928ef
Author: Marcelo Vanzin <[email protected]>
Date: 2015-12-07T19:54:00Z
[RFC] Add type-safe config keys to SparkConf.
This is, in a way, the basics to enable SPARK-529 (which was closed as
won't fix but I think is still valuable). In fact, Spark SQL created
something for that, and this change basically factors out that code
and inserts it into SparkConf, with some extra bells and whistles.
To showcase the usage of this pattern, I modified the YARN backend
to use the new config keys (defined in the new YarnConfigKeys object).
Most of the changes are mechanic, although logic had to be slightly
modified in a handful of places.
I did not port the conf entry registry that SQLConf keeps back to
core, although in the future that could be a useful addition (e.g.
to programatically list all known config options). That required
some code duplication in SQLConf, although it shares the actual
implementation of the entries with SparkConf.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
