[GitHub] spark pull request: SPARK-4628: Put all external projects behind a...
Github user arahuja commented on a diff in the pull request: https://github.com/apache/spark/pull/3485#discussion_r20958627 --- Diff: pom.xml --- @@ -1201,6 +1196,18 @@ /dependencies /profile +!-- External projects are not built in less this flag is enabled. -- +profile + idexternal/id --- End diff -- Should this be `external-projects`? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4344][DOCS] adding documentation on spa...
Github user arahuja commented on the pull request: https://github.com/apache/spark/pull/3209#issuecomment-64003170 @vanzin that sounds reasonable, though confusing if #3233 does not go in soon, but anyways, sounds fine to me. Is there something I should do for that? Reopen this against 1.2 or something the comitters can do? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4480] Avoid many small spills in extern...
Github user arahuja commented on the pull request: https://github.com/apache/spark/pull/3353#issuecomment-63700762 Was this not going into master? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4344][DOCS] adding documentation on spa...
Github user arahuja commented on the pull request: https://github.com/apache/spark/pull/3209#issuecomment-63507703 Updated it to the one line change on userClassPathFirst, but just as happy to close it if #3233 goes in --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4281][Build] Package Yarn shuffle servi...
Github user arahuja commented on the pull request: https://github.com/apache/spark/pull/3147#issuecomment-62834492 I had that as well when I did provide -Pyarn to `make-distribution.sh` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4344][DOCS] adding documentation on spa...
GitHub user arahuja opened a pull request: https://github.com/apache/spark/pull/3209 [SPARK-4344][DOCS] adding documentation on spark.yarn.user.classpath.first The documentation for the two parameters is the same with a pointer from the standalone parameter to the yarn parameter You can merge this pull request into a Git repository by running: $ git pull https://github.com/arahuja/spark yarn-classpath-first-param Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/3209.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3209 commit 821862b79e6447b4a56f788bdad35f85fe351fbc Author: arahuja aahuj...@gmail.com Date: 2014-11-11T17:40:24Z [SPARK-4344][DOCS] adding documentation on spark.yarn.user.classpath.first --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4344][DOCS] adding documentation on spa...
Github user arahuja commented on a diff in the pull request: https://github.com/apache/spark/pull/3209#discussion_r20178399 --- Diff: docs/configuration.md --- @@ -224,6 +224,7 @@ Apart from these, the following properties are also available, and may be useful (Experimental) Whether to give user-added jars precedence over Spark's own jars when loading classes in Executors. This feature can be used to mitigate conflicts between Spark's dependencies and user dependencies. It is currently an experimental feature. +(On Yarn use spark.yarn.user.classpath.first) --- End diff -- Great - I think that will be clearer. Is it worth having a comment in 1.1 at least though that (correctly) states how and when to use which one? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-4344][DOCS] adding documentation on spa...
Github user arahuja commented on a diff in the pull request: https://github.com/apache/spark/pull/3209#discussion_r20183100 --- Diff: docs/configuration.md --- @@ -224,6 +224,7 @@ Apart from these, the following properties are also available, and may be useful (Experimental) Whether to give user-added jars precedence over Spark's own jars when loading classes in Executors. This feature can be used to mitigate conflicts between Spark's dependencies and user dependencies. It is currently an experimental feature. +(On Yarn use spark.yarn.user.classpath.first) --- End diff -- OK - perhaps than just a note that the other configuration won't work on Yarn? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2878]: Fix custom spark.kryo.registrato...
Github user arahuja commented on the pull request: https://github.com/apache/spark/pull/1890#issuecomment-52446389 I think this may be the issue I have been wrangling with the last couple days. I see a variety of odd Kryo related errors, slightly different each time: 14/08/17 22:52:01 ERROR Executor: Exception in task ID 17061 com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 12763 at com.esotericsoftware.kryo.util.DefaultClassResolver.readCl 14/08/17 22:52:01 ERROR Executor: Exception in task ID 17051 java.lang.IndexOutOfBoundsException: Index: 5927, Size: 0 at java.util.ArrayList.rangeCheck(ArrayList.java:635) Looking through the executor logs I do see 14/08/17 22:52:00 ERROR KryoSerializer: Failed to run spark.kryo.registrator java.lang.ClassNotFoundException: org.bdgenomics.guacamole.GuacamoleKryoRegistrator Also, Sandy we are running on YARN and still seem to see this, is there a workaround you know of? Or is there any known workaround in general? Thanks, Arun On Sat, Aug 16, 2014 at 6:05 PM, Patrick Wendell notificati...@github.com wrote: Hey @GrahamDennis https://github.com/GrahamDennis thanks for an extremely thorough analysis of this issue here and on the JIRA. I think that @rxin https://github.com/rxin was able to solve this in a PR that improves the way we deal with passing classloaders to our serializers. Do you mind trying that fix (which has now been merged) and seeing if it fixes your issue? #1972 https://github.com/apache/spark/pull/1972 â Reply to this email directly or view it on GitHub https://github.com/apache/spark/pull/1890#issuecomment-52407167. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
