[GitHub] spark pull request: [SPARK-5052] Add common/base classes to fix gu...
Github user elmer-garduno commented on the pull request: https://github.com/apache/spark/pull/3874#issuecomment-71600458 Thanks for merging! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5052] Add common/base classes to fix gu...
Github user elmer-garduno commented on the pull request: https://github.com/apache/spark/pull/3874#issuecomment-71411925 Jenkins test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5052] Add common/base classes to fix gu...
Github user elmer-garduno commented on the pull request: https://github.com/apache/spark/pull/3874#issuecomment-69871141 Ping? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5052] Add common/base classes to fix gu...
Github user elmer-garduno commented on the pull request: https://github.com/apache/spark/pull/3874#issuecomment-68891252 Thanks all, anything else left to be done here? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5052] Add common/base classes to fix gu...
Github user elmer-garduno commented on the pull request: https://github.com/apache/spark/pull/3874#issuecomment-68785782 Thanks, that worked, I updated the PR to reflect those changes. And here is a list of the actual classes that get included into the jar: jar tf assembly/target/scala-2.10/spark-assembly-1.3.0-SNAPSHOT-hadoop1.2.1.jar |grep com.google.common.base com/google/common/base/ com/google/common/base/Absent.class com/google/common/base/Function.class com/google/common/base/Optional$1$1.class com/google/common/base/Optional$1.class com/google/common/base/Optional.class com/google/common/base/Present.class com/google/common/base/Supplier.class --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5052] Add common/base classes to fix gu...
Github user elmer-garduno commented on the pull request: https://github.com/apache/spark/pull/3874#issuecomment-68774936 In fact what happens is the following: If you embed guava on your jar you still get `java.lang.NoSuchMethodError: com.google.common.base.Optional.transform(Lcom/google/common/base/Function;)Lcom/google/common/base/Optional;` as it seems to be loading the Optional from Spark first. It is when you specify `spark.files.userClassPathFirst=true` that you get `java.lang.NoClassDefFoundError: org/apache/spark/Partition;` given that option is marked as experimental, I don't know to which extent it is usable. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5052] Add common/base classes to fix gu...
Github user elmer-garduno commented on the pull request: https://github.com/apache/spark/pull/3874#issuecomment-68765892 I tried adding Function before, but it had some runtime dependencies. Please see the last part of the third comment on https://issues.apache.org/jira/plugins/servlet/mobile#issue/SPARK-5052. On Mon, Jan 5, 2015, 10:27 AM Marcelo Vanzin notificati...@github.com wrote: Damn, exposing Optional in the API turned out to be a minefield... I see the problem and I don't see a good solution other than adding Function to the list of exclusions. That being said, I think only Function should be added. The source https://code.google.com/p/guava-libraries/source/browse/guava/src/com/google/common/base/Function.java?name=master indicates it has no references to other types, so hopefully it shouldn't cause any other issues... â Reply to this email directly or view it on GitHub https://github.com/apache/spark/pull/3874#issuecomment-68751824. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5052] Add common/base classes to fix gu...
Github user elmer-garduno commented on the pull request: https://github.com/apache/spark/pull/3874#issuecomment-68777890 Let me try just adding `Function` without `Function*` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5052] Add common/base classes to fix gu...
Github user elmer-garduno commented on the pull request: https://github.com/apache/spark/pull/3874#issuecomment-68607699 The problem seems to be fixed when using `spark-submit` instead of `spark-class`, that seems to set the appropriate precedence on the classpaths, so no need for any special flags or changes here. Thanks --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5052] Add common/base classes to fix gu...
Github user elmer-garduno closed the pull request at: https://github.com/apache/spark/pull/3874 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5052] Add common/base classes to fix gu...
GitHub user elmer-garduno reopened a pull request: https://github.com/apache/spark/pull/3874 [SPARK-5052] Add common/base classes to fix guava methods signatures. Fixes problems with incorrect method signatures related to shaded classes. For discussion see the jira issue. You can merge this pull request into a Git repository by running: $ git pull https://github.com/elmer-garduno/spark fix_guava_signatures Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/3874.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3874 commit 596ddbaa8c2f9bf6d2099e799b682d2b15ad9dc6 Author: Elmer Garduno elm...@google.com Date: 2015-01-02T03:45:35Z Add common/base classes to fix guava methods signatures. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5052] Add common/base classes to fix gu...
Github user elmer-garduno commented on the pull request: https://github.com/apache/spark/pull/3874#issuecomment-68616943 Reopening, the problem is still there when running on the standalone cluster. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5052] Add common/base classes to fix gu...
Github user elmer-garduno commented on the pull request: https://github.com/apache/spark/pull/3874#issuecomment-68617200 @vanzin, any input here? Running with the patched code seems to solve the problem. Running on 1.2.0 standalone cluster generates the following errors: a) `java.lang.NoSuchMethodError: com.google.common.base.Optional.transform(Lcom/google/common/base/Function;)Lcom/google/common/base/Optional;` as explained on the issue. b) `java.lang.NoClassDefFoundError: org/apache/spark/Partition;` when running with `spark.files.userClassPathFirst=true`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5052] Add common/base classes to fix gu...
Github user elmer-garduno commented on the pull request: https://github.com/apache/spark/pull/3874#issuecomment-68535926 I tried that before using spark.files.userClassPathFirst, but it resulted in a java.lang.NoClassDefFoundError: org/apache/spark/Partition ([full stack trace](https://gist.github.com/elmer-garduno/e65e3d992357253c6111)), which seemed bad enough to not go that way, but maybe someone else here knows the correct way to achieve it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5052] Add common/base classes to fix gu...
GitHub user elmer-garduno opened a pull request: https://github.com/apache/spark/pull/3874 [SPARK-5052] Add common/base classes to fix guava methods signatures. Fixes problems with incorrect method signatures related to shaded classes. For discussion see the jira issue. You can merge this pull request into a Git repository by running: $ git pull https://github.com/elmer-garduno/spark fix_guava_signatures Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/3874.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3874 commit 596ddbaa8c2f9bf6d2099e799b682d2b15ad9dc6 Author: Elmer Garduno elm...@google.com Date: 2015-01-02T03:45:35Z Add common/base classes to fix guava methods signatures. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
