Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/3112
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3112#issuecomment-65693674
Thanks for testing this out! I also tested it with my own integration test
and it now passes, so this looks good to me. I'm going to merge this into
`master` and
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3112#issuecomment-65694771
I've also backported this to `branch-1.1`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3112#issuecomment-63248280
I haven't forgotten about reviewing this; while attempting to run my
integration test to see whether this fixed my bug, I uncovered a potential
regression (unrelated
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/3112#issuecomment-62904870
@JoshRosen Is it ok?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3112#issuecomment-62905316
[Test build #23313 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23313/consoleFull)
for PR 3112 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3112#issuecomment-62923544
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3112#issuecomment-62923529
[Test build #23313 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23313/consoleFull)
for PR 3112 at commit
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3112#issuecomment-62972127
I have an integration test framework that has a reproduction for the case
where I originally observed this problem, so I'd like to try to run your PR
there to verify
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3112#issuecomment-62998382
[Test build #23347 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23347/consoleFull)
for PR 3112 at commit
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/3112#issuecomment-62998569
@JoshRosen I have reverted the dot which I think is produced in modify
comments. And the blank between `!` and `args.isEmpty` in
`ApplicationMasterArguments` is
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3112#issuecomment-63004493
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3112#issuecomment-63004489
[Test build #23347 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23347/consoleFull)
for PR 3112 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3112#issuecomment-62520250
[Test build #23213 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23213/consoleFull)
for PR 3112 at commit
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/3112#issuecomment-62522970
Now we ignore it in SparkSubmit.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3112#issuecomment-62529493
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3112#issuecomment-62529488
[Test build #23213 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23213/consoleFull)
for PR 3112 at commit
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/3112#issuecomment-62392939
Ok, I understand the use case for it in yarn client mode.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/3112#issuecomment-62505153
@JoshRosen Sorry for just checking the 1.1.0 version(where only client mode
is supported in standalone mode) not the latest documents. Now I ignore
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3112#issuecomment-62505211
[Test build #23198 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23198/consoleFull)
for PR 3112 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3112#issuecomment-62505972
[Test build #23200 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23200/consoleFull)
for PR 3112 at commit
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3112#discussion_r20133512
--- Diff: core/src/main/scala/org/apache/spark/deploy/Client.scala ---
@@ -155,6 +155,14 @@ object Client {
}
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3112#discussion_r20133547
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -206,6 +202,14 @@ class SparkContext(config: SparkConf) extends
SparkStatusAPI
Github user JoshRosen commented on a diff in the pull request:
https://github.com/apache/spark/pull/3112#discussion_r20133601
--- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala ---
@@ -206,6 +202,14 @@ class SparkContext(config: SparkConf) extends
SparkStatusAPI
Github user JoshRosen commented on the pull request:
https://github.com/apache/spark/pull/3112#issuecomment-62507240
In Spark Standalone `cluster` deploy mode, I think that `master` will still
point to a regular standalone master URL (e.g. `spark://my-master-hostname`)
but the driver
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/3112#issuecomment-62509266
` then this might cause problems later down the road if the remote
container wants to inject some machine-specific values for these settings` like
what? We do
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3112#issuecomment-62510955
[Test build #23198 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23198/consoleFull)
for PR 3112 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3112#issuecomment-62510961
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/3112#issuecomment-62511253
Ohh..I got what you mean. It is indeed inappropriate to set `spark.master`
to `standalone-master` as we still need the master url.
---
If your project is set up
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3112#issuecomment-62511369
[Test build #23200 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/23200/consoleFull)
for PR 3112 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3112#issuecomment-62511377
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/3112#issuecomment-62302608
Additionally, I think spark.driver.host is useful in all client mode
including standalone, mesos mode ( i don't know it very mych) and yarn-client
mode. When the
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/3112#issuecomment-61987488
does this only occur if user manually sets spark.driver.host ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/3112#issuecomment-61988279
Yep. Considering setting this config item in yarn-client mode is ok and
switching between yarn-client and yarn-cluster mode is convenient, I think it
would be
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/3112#issuecomment-61995720
I'm just curious what is the reason to set it for client mode?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/3112#issuecomment-62026687
If we didn't set it, Yarn cluster would probably not recognize driver's
host (default value of spark.driver.host) because the nodes in yarn could not
just
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/3112#issuecomment-62044782
Sorry perhaps my questions was misleading. Are you as a user setting
spark.driver.host for some reason? The user doesn't need to set it in yarn mode
(either client or
Github user WangTaoTheTonic commented on the pull request:
https://github.com/apache/spark/pull/3112#issuecomment-62090652
Here is the scenario: When a client submit an application on a node which
is not in yarn cluster. The ApplicationMaster may not recognize the client's
hostname
GitHub user WangTaoTheTonic opened a pull request:
https://github.com/apache/spark/pull/3112
[SPARK-4253]Ignore spark.driver.host in yarn-cluster mode
And I see description in configuration document is This is used for
communicating with the executors and the standalone Master.,
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3112#issuecomment-61824158
[Test build #22939 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22939/consoleFull)
for PR 3112 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/3112#issuecomment-61840803
[Test build #22939 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/22939/consoleFull)
for PR 3112 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/3112#issuecomment-61840821
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
42 matches
Mail list logo
