[GitHub] spark issue #18901: [SPARK-21689][YARN] Download user jar from remote in cas...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18901 Build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18901: [SPARK-21689][YARN] Download user jar from remote in cas...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18901 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80559/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18901: [SPARK-21689][YARN] Download user jar from remote in cas...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18901 **[Test build #80559 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80559/testReport)** for PR 18901 at commit [`7bbd1ad`](https://github.com/apache/spark/commit/7bbd1ad7f4a4e282fda78b6f9dfdf2ebdba98a65). * This patch passes all tests. * This patch **does not merge cleanly**. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18901: [SPARK-21689][YARN] Download user jar from remote in cas...
Github user caneGuy commented on the issue: https://github.com/apache/spark/pull/18901 All right, i will close this pr.Thanks for your time @vanzin @jerryshao @tgravescs . --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18901: [SPARK-21689][YARN] Download user jar from remote in cas...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/18901 As Tom suggested, if you add hbase to your gateway's Spark installation, you won't need to download it every time you submit an application. This change, the way it is, is really not something that should go into Spark, for the reasons already mentioned. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18901: [SPARK-21689][YARN] Download user jar from remote in cas...
Github user caneGuy commented on the issue: https://github.com/apache/spark/pull/18901 i execute `mvn checkstyle:checkstyle` locally with success status, but i can not find more logs in jenkins since i want to find which file failed with style check. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18901: [SPARK-21689][YARN] Download user jar from remote in cas...
Github user caneGuy commented on the issue: https://github.com/apache/spark/pull/18901 Yes i have a workaround solution,such as add local jar in "--jars".But i do not think this is a very edge case. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18901: [SPARK-21689][YARN] Download user jar from remote in cas...
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/18901 If this is a common problem for your users why not just install the hbase jars on the launcher box? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18901: [SPARK-21689][YARN] Download user jar from remote in cas...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18901 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80562/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18901: [SPARK-21689][YARN] Download user jar from remote in cas...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18901 **[Test build #80562 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80562/testReport)** for PR 18901 at commit [`3b07797`](https://github.com/apache/spark/commit/3b07797ce767093ef385c5abca7ed1d8e5784451). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18901: [SPARK-21689][YARN] Download user jar from remote in cas...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18901 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18901: [SPARK-21689][YARN] Download user jar from remote in cas...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18901 **[Test build #80562 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80562/testReport)** for PR 18901 at commit [`3b07797`](https://github.com/apache/spark/commit/3b07797ce767093ef385c5abca7ed1d8e5784451). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18901: [SPARK-21689][YARN] Download user jar from remote in cas...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18901 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18901: [SPARK-21689][YARN] Download user jar from remote in cas...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18901 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80561/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18901: [SPARK-21689][YARN] Download user jar from remote in cas...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18901 **[Test build #80561 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80561/testReport)** for PR 18901 at commit [`ae785d9`](https://github.com/apache/spark/commit/ae785d93ba26428f2a01fb64d8ce53c3f88cb6af). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18901: [SPARK-21689][YARN] Download user jar from remote in cas...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18901 **[Test build #80561 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80561/testReport)** for PR 18901 at commit [`ae785d9`](https://github.com/apache/spark/commit/ae785d93ba26428f2a01fb64d8ce53c3f88cb6af). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18901: [SPARK-21689][YARN] Download user jar from remote in cas...
Github user caneGuy commented on the issue: https://github.com/apache/spark/pull/18901 But i think this case should be fixed since many users of our inner branch has suffered from this problem. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18901: [SPARK-21689][YARN] Download user jar from remote in cas...
Github user caneGuy commented on the issue: https://github.com/apache/spark/pull/18901 @vanzin i have also thought about what you mentioned above.But since i do not have enough background knowledge ,i can not think about how to check user need hbase class first.And my original idea is only download primarysource jar,since most user will package their hbase client into primarysource jar. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18901: [SPARK-21689][YARN] Download user jar from remote in cas...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18901 **[Test build #80559 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80559/testReport)** for PR 18901 at commit [`7bbd1ad`](https://github.com/apache/spark/commit/7bbd1ad7f4a4e282fda78b6f9dfdf2ebdba98a65). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18901: [SPARK-21689][YARN] Download user jar from remote in cas...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/18901 I am not a fan of this change. It makes submission unnecessarily more expensive for everybody to fix an edge case. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18901: [SPARK-21689][YARN] Download user jar from remote in cas...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18901 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80542/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18901: [SPARK-21689][YARN] Download user jar from remote in cas...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18901 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18901: [SPARK-21689][YARN] Download user jar from remote in cas...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18901 **[Test build #80542 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80542/testReport)** for PR 18901 at commit [`24c80d1`](https://github.com/apache/spark/commit/24c80d1cbce1b63951a70a73645deea21850043d). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18901: [SPARK-21689][YARN] Download user jar from remote in cas...
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/18901 Note I'll file a separate jira for this but even in client mode it will download and then reupload the file to hdfs for yarn mode, thus defeating the purpose of it being in hdfs. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18901: [SPARK-21689][YARN] Download user jar from remote in cas...
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/18901 Generally its expected that where you launch your job from has all the necessary jars already, thus all the jar would be available for the Client. here we download all jars that the user explicitly put into hdfs. If that is a lot of jars this could take some time. I see I missed someone put something in to this in client mode, but client mode explicitly needs those to be able to run. Here we are downloading them all for a token in one specific case. The majority of cases this is just going to add overhead and take up disk space. I'm not sure I agree with always doing this. Is this going to download the file and then just reupload it to the staging directory and not use the original file in HDFS? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18901: [SPARK-21689][YARN] Download user jar from remote in cas...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18901 **[Test build #80542 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80542/testReport)** for PR 18901 at commit [`24c80d1`](https://github.com/apache/spark/commit/24c80d1cbce1b63951a70a73645deea21850043d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18901: [SPARK-21689][YARN] Download user jar from remote in cas...
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/18901 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18901: [SPARK-21689][YARN] Download user jar from remote in cas...
Github user caneGuy commented on the issue: https://github.com/apache/spark/pull/18901 @vanzin take a look at this pr?Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18901: [SPARK-21689][YARN] Download user jar from remote in cas...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/18901 LGTM generally, CC @vanzin to review. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18901: [SPARK-21689][YARN] Download user jar from remote in cas...
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/18901 Can you please add a unit test to verify the classpath with your changes? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #18901: [SPARK-21689][YARN] Download user jar from remote in cas...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18901 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
