[GitHub] spark pull request: [SPARK-2290] Worker should directly use its ow...
Github user YanTangZhai closed the pull request at: https://github.com/apache/spark/pull/1392 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2290] Worker should directly use its ow...
Github user YanTangZhai commented on the pull request: https://github.com/apache/spark/pull/1392#issuecomment-51190110 @pwendell Sorry, I'm late. Please disregard this PR since #1734 has been closed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2290] Worker should directly use its ow...
Github user YanTangZhai closed the pull request at: https://github.com/apache/spark/pull/1244 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-2290] Worker should directly use its ow...
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/1392#issuecomment-50691533 @YanTangZhai can you merge this up to master and make sure it is passing tests? Thanks --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2290] Worker should directly use its ow...
Github user YanTangZhai commented on the pull request: https://github.com/apache/spark/pull/1392#issuecomment-49584362 Hi @andrewor14 , that's ok. Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2290] Worker should directly use its ow...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/1392#issuecomment-49353609 Hey @YanTangZhai, on second thought I think we should keep the config, but not set it by default like we do currently. The user may have multiple installations of Spark on the same Worker machine, and spark.home previously provided them a way to do that. We should keep this functionality, but make it optional as opposed to forcing it on them. Also, since we no longer need spark.home anymore, it would be good to remove all occurrences of it to remove confusion (exception for backwards compatibility). However, this is slightly tricky because Mesos handles this differently from other modes. So I suggest this: I will take over from here, because this change seems a little more involved than we originally imagined it to be. How does that sound? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2290] Worker should directly use its ow...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/1244#issuecomment-49126716 We only want to do this if the driver shares the same directory structure as the executors. This is an assumption that is incorrect in many deployment settings. Really, we should have something like `spark.executor.home` that is not the same as `SPARK_HOME`. I am not 100% sure if we can just rip this functionality out actually. I am under the impression that Mesos still depends on something like this, so we should double check before we remove it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2290] Worker should directly use its ow...
Github user CodingCat commented on the pull request: https://github.com/apache/spark/pull/1244#issuecomment-49153988 @andrewor14 yeah, I agree with you, I just thought in somewhere (document in the earlier versions? I cannot find it now), the user has to set this env variable? so I said prioritizing worker side SPARK_HOME, if this is not set, Spark will try to read application setup about SPARK_HOME (which may generates error if the directory structure is not the same) I also noticed this JIRA https://issues.apache.org/jira/browse/SPARK-2454 (left some comments there) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2290] Worker should directly use its ow...
Github user CodingCat commented on the pull request: https://github.com/apache/spark/pull/1244#issuecomment-49122687 instead of discard sparkHome parameter entirely, shall we just prioritizing local SPARK_HOME env and pass the applicationDesc.sparkHome only if SPARK_HOME is not set in worker side? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2290] Worker should directly use its ow...
GitHub user YanTangZhai opened a pull request: https://github.com/apache/spark/pull/1392 [SPARK-2290] Worker should directly use its own sparkHome instead of appDesc.sparkHome when LaunchExecutor Worker should directly use its own sparkHome instead of appDesc.sparkHome when LaunchExecutor You can merge this pull request into a Git repository by running: $ git pull https://github.com/YanTangZhai/spark SPARK-2290 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/1392.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1392 commit d3072fc05c7c20ec9d90732db2b9b26a4d27e290 Author: YanTangZhai tyz0...@163.com Date: 2014-07-13T11:50:14Z Update ApplicationDescription.scala commit 78ec6bc8c5d1af64ca21e1a231b47911df6d4f90 Author: YanTangZhai tyz0...@163.com Date: 2014-07-13T11:52:34Z Update JsonProtocol.scala commit 95e6ccc354167117430ce4cb7b2f5063a454ff1d Author: YanTangZhai tyz0...@163.com Date: 2014-07-13T11:54:55Z Update TestClient.scala commit 508dcb65d04e3f12f99e03572a1cc277e7f1aeca Author: YanTangZhai tyz0...@163.com Date: 2014-07-13T11:58:01Z Update SparkDeploySchedulerBackend.scala commit 6d6700aaad941779485eee2c35c4ab0cd278529e Author: YanTangZhai tyz0...@163.com Date: 2014-07-13T12:01:40Z Update Worker.scala commit c360154ae5b03e7854d63573494fc6113295a7ec Author: YanTangZhai tyz0...@163.com Date: 2014-07-13T12:04:16Z Update JsonProtocolSuite.scala commit 6febb215fb73735760fae957a4e71e2a61c17c77 Author: YanTangZhai tyz0...@163.com Date: 2014-07-13T12:07:35Z Update ExecutorRunnerTest.scala --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2290] Worker should directly use its ow...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1392#issuecomment-48839494 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2290] Worker should directly use its ow...
Github user YanTangZhai commented on the pull request: https://github.com/apache/spark/pull/1392#issuecomment-48839557 #1244 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2290] Worker should directly use its ow...
Github user YanTangZhai commented on the pull request: https://github.com/apache/spark/pull/1392#issuecomment-48839668 fix #1244 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2290] Worker should directly use its ow...
Github user YanTangZhai commented on the pull request: https://github.com/apache/spark/pull/1244#issuecomment-48839912 I've fixed the compile problem. Please review and test again. Thanks very much. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2290] Worker should directly use its ow...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/1392#issuecomment-48853888 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2290] Worker should directly use its ow...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1392#issuecomment-48854002 QA tests have started for PR 1392. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16609/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2290] Worker should directly use its ow...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1392#issuecomment-48855862 QA results for PR 1392:br- This patch FAILED unit tests.br- This patch merges cleanlybr- This patch adds no public classesbrbrFor more information see test ouptut:brhttps://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16609/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2290] Worker should directly use its ow...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/1244#issuecomment-48664105 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2290] Worker should directly use its ow...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/1244#issuecomment-48664165 Changes look reasonable to me. There were a few questions from the mailing list about this, so it'll be good to get this in. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2290] Worker should directly use its ow...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1244#issuecomment-48664391 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2290] Worker should directly use its ow...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1244#issuecomment-48664408 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2290] Worker should directly use its ow...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1244#issuecomment-48664567 QA tests have started for PR 1244. This patch merges cleanly. brView progress: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16518/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2290] Worker should directly use its ow...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1244#issuecomment-48672993 QA results for PR 1244:br- This patch FAILED unit tests.br- This patch merges cleanlybr- This patch adds no public classesbrbrFor more information see test ouptut:brhttps://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16518/consoleFull --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2290] Worker should directly use its ow...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1244#issuecomment-48673000 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16518/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2290] Worker should directly use its ow...
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/1244#issuecomment-48677179 @YanTangZhai This doesn't compile... could you fix it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2290] Worker should directly use its ow...
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/1244#issuecomment-48283271 LGTM pending tests, this is something that has confused people before, so I think it's best to just leave it out. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2290] Worker should directly use its ow...
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/1244#issuecomment-48283315 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2290] Worker should directly use its ow...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1244#issuecomment-48283504 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2290] Worker should directly use its ow...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1244#issuecomment-48298659 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2290] Worker should directly use its ow...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1244#issuecomment-48298667 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16401/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2290] Worker should directly use its ow...
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/1244#issuecomment-48419256 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2290] Worker should directly use its ow...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1244#issuecomment-48419483 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2290] Worker should directly use its ow...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1244#issuecomment-48419494 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2290] Worker should directly use its ow...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1244#issuecomment-48420065 Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16440/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2290] Worker should directly use its ow...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1244#issuecomment-48420063 Merged build finished. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2290] Worker should directly use its ow...
GitHub user YanTangZhai opened a pull request: https://github.com/apache/spark/pull/1244 [SPARK-2290] Worker should directly use its own sparkHome instead of appDesc.sparkHome when LaunchExecutor Worker should directly use its own sparkHome instead of appDesc.sparkHome when LaunchExecutor You can merge this pull request into a Git repository by running: $ git pull https://github.com/YanTangZhai/spark master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/1244.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1244 commit 05c3a789a00996a5502b78711b44d80e8812fdbb Author: hakeemzhai hakeemzhai@hakeemzhai.(none) Date: 2014-06-27T07:42:18Z [SPARK-2290] Worker should directly use its own sparkHome instead of appDesc.sparkHome when LaunchExecutor --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2290] Worker should directly use its ow...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1244#issuecomment-47317532 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2290] Worker should directly use its ow...
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/1244#issuecomment-47318509 If we are going to remove this feature, we should just take the sparkHome field out of `ApplicationDescription` entirely. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
