[GitHub] spark pull request: [SPARK-4344][DOCS] adding documentation on spa...
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/3209#issuecomment-64405811 test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4344][DOCS] adding documentation on spa...
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/3209#issuecomment-64405912 @vanzin I think I'll pull this in and you will have to remove it again in https://github.com/apache/spark/pull/3233 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4344][DOCS] adding documentation on spa...
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/3209#issuecomment-64406285 I pulled this into both master and branch-1.2 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4344][DOCS] adding documentation on spa...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/3209 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4344][DOCS] adding documentation on spa...
Github user arahuja commented on the pull request: https://github.com/apache/spark/pull/3209#issuecomment-64003170 @vanzin that sounds reasonable, though confusing if #3233 does not go in soon, but anyways, sounds fine to me. Is there something I should do for that? Reopen this against 1.2 or something the comitters can do? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4344][DOCS] adding documentation on spa...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/3209#issuecomment-64009711 I don't know if it's possible to move a PR to a different branch (or whether you need to create a new one). In any case, it's not a big deal if this goes into master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4344][DOCS] adding documentation on spa...
Github user arahuja commented on the pull request: https://github.com/apache/spark/pull/3209#issuecomment-63507703 Updated it to the one line change on userClassPathFirst, but just as happy to close it if #3233 goes in --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4344][DOCS] adding documentation on spa...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/3209#issuecomment-63518490 I'm ok with the message, although if you only add it on branch-1.2 it would save me the work of having to delete it as part of my other PR. :-) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4344][DOCS] adding documentation on spa...
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/3209#issuecomment-63078583 so @vanzin put up a pull request to implement it on yarn (https://github.com/apache/spark/pull/3233). Can we just close this one in favor of that? I'm fine if you plan to make the one line change soon for 1.2 and then the other pull request can remove it though too. But we would need this change soon. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4344][DOCS] adding documentation on spa...
GitHub user arahuja opened a pull request: https://github.com/apache/spark/pull/3209 [SPARK-4344][DOCS] adding documentation on spark.yarn.user.classpath.first The documentation for the two parameters is the same with a pointer from the standalone parameter to the yarn parameter You can merge this pull request into a Git repository by running: $ git pull https://github.com/arahuja/spark yarn-classpath-first-param Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/3209.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3209 commit 821862b79e6447b4a56f788bdad35f85fe351fbc Author: arahuja aahuj...@gmail.com Date: 2014-11-11T17:40:24Z [SPARK-4344][DOCS] adding documentation on spark.yarn.user.classpath.first --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4344][DOCS] adding documentation on spa...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/3209#issuecomment-62590575 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4344][DOCS] adding documentation on spa...
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/3209#discussion_r20172562 --- Diff: docs/configuration.md --- @@ -224,6 +224,7 @@ Apart from these, the following properties are also available, and may be useful (Experimental) Whether to give user-added jars precedence over Spark's own jars when loading classes in Executors. This feature can be used to mitigate conflicts between Spark's dependencies and user dependencies. It is currently an experimental feature. +(On Yarn use spark.yarn.user.classpath.first) --- End diff -- Actually, the two config options do not work the same way. I'm working on implementing `userClassPathFirst` for yarn mode and, in the process, will deprecate `spark.yarn.user.classpath.first`, since the semantics of the former are less dangerous. (If curious check out SPARK-2966, although the bug doesn't have a whole lot of information.) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4344][DOCS] adding documentation on spa...
Github user arahuja commented on a diff in the pull request: https://github.com/apache/spark/pull/3209#discussion_r20178399 --- Diff: docs/configuration.md --- @@ -224,6 +224,7 @@ Apart from these, the following properties are also available, and may be useful (Experimental) Whether to give user-added jars precedence over Spark's own jars when loading classes in Executors. This feature can be used to mitigate conflicts between Spark's dependencies and user dependencies. It is currently an experimental feature. +(On Yarn use spark.yarn.user.classpath.first) --- End diff -- Great - I think that will be clearer. Is it worth having a comment in 1.1 at least though that (correctly) states how and when to use which one? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4344][DOCS] adding documentation on spa...
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/3209#discussion_r20181813 --- Diff: docs/configuration.md --- @@ -224,6 +224,7 @@ Apart from these, the following properties are also available, and may be useful (Experimental) Whether to give user-added jars precedence over Spark's own jars when loading classes in Executors. This feature can be used to mitigate conflicts between Spark's dependencies and user dependencies. It is currently an experimental feature. +(On Yarn use spark.yarn.user.classpath.first) --- End diff -- Personally, I think `spark.yarn.user.classpath.first` is a little too dangerous for us to publicly document - and we've lived so far without documenting it, and asking people to try it on a as-needed basis. But let's see if others (@tgravescs @andrewor14) have a different opinion. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4344][DOCS] adding documentation on spa...
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/3209#discussion_r20182510 --- Diff: docs/configuration.md --- @@ -224,6 +224,7 @@ Apart from these, the following properties are also available, and may be useful (Experimental) Whether to give user-added jars precedence over Spark's own jars when loading classes in Executors. This feature can be used to mitigate conflicts between Spark's dependencies and user dependencies. It is currently an experimental feature. +(On Yarn use spark.yarn.user.classpath.first) --- End diff -- it was left undocumented on purpose (expert config) used as last resort type thing. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4344][DOCS] adding documentation on spa...
Github user arahuja commented on a diff in the pull request: https://github.com/apache/spark/pull/3209#discussion_r20183100 --- Diff: docs/configuration.md --- @@ -224,6 +224,7 @@ Apart from these, the following properties are also available, and may be useful (Experimental) Whether to give user-added jars precedence over Spark's own jars when loading classes in Executors. This feature can be used to mitigate conflicts between Spark's dependencies and user dependencies. It is currently an experimental feature. +(On Yarn use spark.yarn.user.classpath.first) --- End diff -- OK - perhaps than just a note that the other configuration won't work on Yarn? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4344][DOCS] adding documentation on spa...
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/3209#discussion_r20186306 --- Diff: docs/configuration.md --- @@ -224,6 +224,7 @@ Apart from these, the following properties are also available, and may be useful (Experimental) Whether to give user-added jars precedence over Spark's own jars when loading classes in Executors. This feature can be used to mitigate conflicts between Spark's dependencies and user dependencies. It is currently an experimental feature. +(On Yarn use spark.yarn.user.classpath.first) --- End diff -- Yeah I think it's sufficient to add a one line doc that the other config doesn't work on Yarn. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
