GitHub user pwendell opened a pull request:
https://github.com/apache/spark/pull/880
Organize configuration options
This PR improves and organizes the config option page
and makes a few other changes to config docs. See a preview here:
http://people.apache.org/~pwendell/config-improvements/configuration.html
The biggest changes are:
1. The configs for the standalone master/workers were moved to the
standalone page and out of the general config doc.
2. SPARK_LOCAL_DIRS was missing from the standalone docs.
3. Expanded discussion of injecting configs with spark-submit, including an
example.
4. Config options were organized into the following categories:
- Runtime Environment
- Shuffle Behavior
- Spark UI
- Compression and Serialization
- Execution Behavior
- Networking
- Scheduling
- Security
- Spark Streaming
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/pwendell/spark config-cleanup
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/880.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #880
----
commit 204b2480028a1a4256ed248f4dbf689b60723ac3
Author: Patrick Wendell <[email protected]>
Date: 2014-05-25T03:05:19Z
Small fixes
commit 4af9e07494b4de99e1e099ff9c04a74fa3f02951
Author: Patrick Wendell <[email protected]>
Date: 2014-05-25T03:25:58Z
Adding SPARK_LOCAL_DIRS docs
commit 2d719efd9f68563119be1f527e97a19df4aa7485
Author: Patrick Wendell <[email protected]>
Date: 2014-05-25T03:26:30Z
Small fix
commit 29b54461e07557d66cfa7128f6c222106ce5a5e8
Author: Patrick Wendell <[email protected]>
Date: 2014-05-25T03:56:05Z
Better discussion of spark-submit in configuration docs
commit 592e94ac20f4d209c9e2334875f33d811f5e1a64
Author: Patrick Wendell <[email protected]>
Date: 2014-05-25T07:28:10Z
Stash
commit 54b184d4a3c10386fd73cf8b8d0db7800d4ac560
Author: Patrick Wendell <[email protected]>
Date: 2014-05-25T04:40:10Z
Adding standalone configs to the standalone page
commit f7e79bc42c1635686c3af01eef147dae92de2529
Author: Patrick Wendell <[email protected]>
Date: 2014-05-26T04:43:11Z
Re-organizing config options.
This uses the following categories:
- Runtime Environment
- Shuffle Behavior
- Spark UI
- Compression and Serialization
- Execution Behavior
- Networking
- Scheduling
- Security
- Spark Streaming
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
