[GitHub] spark pull request: add spark.driver.memory to config docs
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/2410#issuecomment-58470754 Ok I merged this into master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: add spark.driver.memory to config docs
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/2410 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: add spark.driver.memory to config docs
Github user nartz commented on the pull request: https://github.com/apache/spark/pull/2410#issuecomment-58398280 i can, but it seems maybe the scope of this pull-request is too small, as it needs to encapsulate and explain a bunch of new parameters popping up, and nuances about when to use spark.driver.memory (as i see many people having problems on the mailing list) - maybe we should just turn it into an issue that can be more comprehensive? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: add spark.driver.memory to config docs
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/2410#issuecomment-58418926 Ok, fair enough. Though none of these configs are actually new; they have been around since 1.0 but just never documented. I can merge this and file a separate JIRA for these other configs. Thanks @nartz. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: add spark.driver.memory to config docs
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/2410#issuecomment-58420097 Also could you add `[Docs]` to the title? It will help us organize our PRs. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: add spark.driver.memory to config docs
Github user ash211 commented on the pull request: https://github.com/apache/spark/pull/2410#issuecomment-55862389 One note about this setting is that I'm not sure it works in all settings -- if you pass it to a driver as a parameter then it's too late to take effect (the JVM has already started). ```./bin/spark-shell --driver-java-options -Dspark.driver.memory=1576m``` doesn't actually change driver memory. Maybe the docs should mention where it does and doesn't work? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: add spark.driver.memory to config docs
Github user nartz commented on the pull request: https://github.com/apache/spark/pull/2410#issuecomment-55909584 using `./bin/spark-submit --driver-memory 3g myscript.py` on the command line works for me --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: add spark.driver.memory to config docs
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/2410#issuecomment-55921974 @ash211 I believe it'll work actually, because we treat `--driver-*` arguments specially in bash before we launch the driver JVM. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: add spark.driver.memory to config docs
Github user ash211 commented on the pull request: https://github.com/apache/spark/pull/2410#issuecomment-55926912 @andrewor14 the `--driver-memory` parameter works but `--driver-java-options -Dspark.driver.memory=3g` doesn't. Do you expect both to work? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: add spark.driver.memory to config docs
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/2410#issuecomment-55927721 Ah sorry, it will work only if it's `--driver-java-options -Xms3g -Xmx3g`, because those will be passed directly to the java command when launching the JVM. Though in general `--driver-java-options` (and likewise `spark.driver.extraJavaOptions`) should not take in Spark properties, so I'm not sure if that's worth a mention in the docs. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: add spark.driver.memory to config docs
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/2410#issuecomment-55967503 Hey @nartz, can you also add `spark.driver.extraClassPath`, `spark.driver.extraLibraryPath` and `spark.driver.extraJavaOptions`? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: add spark.driver.memory to config docs
GitHub user nartz opened a pull request: https://github.com/apache/spark/pull/2410 add spark.driver.memory to config docs It took me a minute to track this down, so I thought it could be useful to have it in the docs. I'm unsure if 512mb is the default for spark.driver.memory? Also - there could be a better value for the 'description' to differentiate it from spark.executor.memory. You can merge this pull request into a Git repository by running: $ git pull https://github.com/nartz/spark docs/add-spark-driver-memory-to-config-docs Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/2410.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2410 commit 74521b814bb6c47ce0c060d2dfc320b8d8454021 Author: Nathan Artz nathanartz@nathans-macbook-pro.local Date: 2014-09-16T14:20:19Z add spark.driver.memory to config docs --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: add spark.driver.memory to config docs
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2410#issuecomment-55749884 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: add spark.driver.memory to config docs
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/2410#issuecomment-55813836 LGTM. I'm surprised this isn't documented. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: add spark.driver.memory to config docs
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/2410#discussion_r17630588 --- Diff: docs/configuration.md --- @@ -104,6 +104,14 @@ of the most common options to set are: /td /tr tr + tdcodespark.driver.memory/code/td + td512m/td + td +Amount of memory to use for the driver process, i.e. where spark context is initialized. --- End diff -- Can you call this `SparkContext` to be consistent with other places in the docs? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
