Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-217525101
I have seen cases where driver and executor needs different classpath,
mostly in client mode. gateways/launcher boxes can have different setup then
hadoop nodes. I h
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-217517817
Thanks for bringing this up, Sean. Yeah, I'd like to see this go in too. I
realize there are some merge conflicts here. I will make the semantics clear in
documentati
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-217514791
I still think this is a good idea; just need to clearly define the
semantics (in case they aren't). e.g. `driver.extraClassPath` should be added
before `common.extraClass
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-217511858
Eh ... I didn't see much of a resolution on this. Does anyone feel weakly,
moderately, strongly for/against this?
---
If your project is set up for it, you can reply to
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-165600240
I actually haven't seen a case of users needing different class paths for
the driver and executor; in general, the executor class path is either the same
or a subset of t
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-165547719
> I see the case for this makes it easier for users in some cases but I
also see it as possibly confusing and just more maintenance and complexity in
the spark code.
Github user tgravescs commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-165466946
I'm a bit on the fence about this. Its kind of bringing us full circle to
what SPARK used to have when they had env variables, most of those applied
across everything
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-164900402
Sure, I wasn't sure if this would be updated any time soon.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-164877416
Hey Andrew, thanks for your feedback.
> What happens if I specify different versions of some jar on both common
and executor class path? Which one gets loaded
Github user andrewor14 commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-164873314
I'm not super convinced that we should add something like this. This patch
introduces a config that's not really necessary and has potentially unclear
semantics. E.g.
Github user vanzin commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-143849420
Looks sane but there's some room for cleanup; also, need to resolve
conflicts.
---
If your project is set up for it, you can reply to this email and have your
reply appe
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7739#discussion_r40593778
--- Diff:
core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala ---
@@ -143,6 +144,9 @@ private[deploy] class SparkSubmitArguments(args:
Seq[
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7739#discussion_r40593517
--- Diff:
core/src/test/scala/org/apache/spark/deploy/SparkSubmitSuite.scala ---
@@ -624,3 +639,15 @@ object UserClasspathFirstTest {
}
}
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7739#discussion_r40593338
--- Diff: core/src/main/scala/org/apache/spark/deploy/Client.scala ---
@@ -67,8 +67,15 @@ private class ClientEndpoint(
// people call `addJa
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7739#discussion_r40593175
--- Diff: core/src/main/scala/org/apache/spark/SparkConf.scala ---
@@ -505,6 +510,23 @@ class SparkConf(loadDefaults: Boolean) extends
Cloneable with Logging {
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7739#discussion_r40576432
--- Diff: core/src/main/scala/org/apache/spark/SparkConf.scala ---
@@ -505,6 +510,23 @@ class SparkConf(loadDefaults: Boolean) extends
Cloneable with Logging {
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7739#discussion_r40576297
--- Diff: core/src/main/scala/org/apache/spark/SparkConf.scala ---
@@ -465,6 +466,10 @@ class SparkConf(loadDefaults: Boolean) extends
Cloneable with Logging {
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7739#discussion_r40576243
--- Diff: bin/spark-shell ---
@@ -35,7 +35,8 @@ export _SPARK_CMD_USAGE="Usage: ./bin/spark-shell
[options]"
# so we need to add the "-Dscala.usejavacp=tr
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/7739#discussion_r40576075
--- Diff: bin/spark-shell ---
@@ -35,7 +35,8 @@ export _SPARK_CMD_USAGE="Usage: ./bin/spark-shell
[options]"
# so we need to add the "-Dscala.usejavacp=tr
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-133203764
Thanks Hari! The YarnClusterSuite change is rather benign and not
absolutely necessary. So, based on this I believe this request is ready to be
merged.
And,
Github user harishreedharan commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-133203284
Since most of the changes don't affect the cluster managers directly, and
spark submit changes have tests, I think this is good (I was confused by the
one change
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-133154333
Jenkins, test this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not ha
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-132823658
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-132823659
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-132823574
**[Test build #41277 timed
out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41277/console)**
for PR 7739 at commit
[`bca4fa3`](https://github
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-132821458
Build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does n
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-132821459
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-132821326
[Test build #41275 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41275/console)
for PR 7739 at commit
[`5f62d39`](https://github.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-132802356
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-132802355
Build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does n
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-132802279
[Test build #41272 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41272/console)
for PR 7739 at commit
[`8740751`](https://github.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-132770381
[Test build #41277 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41277/consoleFull)
for PR 7739 at commit
[`bca4fa3`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-132770029
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-13277
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-132769253
[Test build #41275 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41275/consoleFull)
for PR 7739 at commit
[`5f62d39`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-132768154
Build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have thi
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-132768203
Build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this f
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-132752506
Hey @harishreedharan about
> Could you also add some tests to each of the components (YARN/standalone
and mesos to ensure the correct classpaths get passed in the
Github user markgrover commented on a diff in the pull request:
https://github.com/apache/spark/pull/7739#discussion_r37457145
--- Diff: core/src/main/scala/org/apache/spark/SparkConf.scala ---
@@ -479,6 +484,23 @@ class SparkConf(loadDefaults: Boolean) extends
Cloneable with Loggi
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-132751610
[Test build #41272 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/41272/consoleFull)
for PR 7739 at commit
[`8740751`](https://gith
Github user markgrover commented on a diff in the pull request:
https://github.com/apache/spark/pull/7739#discussion_r37456912
--- Diff: core/src/main/scala/org/apache/spark/SparkConf.scala ---
@@ -479,6 +484,23 @@ class SparkConf(loadDefaults: Boolean) extends
Cloneable with Loggi
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-132749838
Build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this f
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-132749781
Build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have thi
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-130161669
Thanks @harishreedharan and @squito for your reviews!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well.
Github user markgrover commented on a diff in the pull request:
https://github.com/apache/spark/pull/7739#discussion_r36826616
--- Diff: core/src/main/scala/org/apache/spark/deploy/Client.scala ---
@@ -68,7 +68,12 @@ private class ClientEndpoint(
val mainClass = "org.ap
Github user markgrover commented on a diff in the pull request:
https://github.com/apache/spark/pull/7739#discussion_r36826563
--- Diff:
core/src/main/scala/org/apache/spark/deploy/rest/StandaloneRestServer.scala ---
@@ -135,7 +135,10 @@ private[rest] class StandaloneSubmitRequestS
Github user markgrover commented on a diff in the pull request:
https://github.com/apache/spark/pull/7739#discussion_r36826589
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackend.scala
---
@@ -88,7 +88,10 @@ private[spark] class MesosSchedu
Github user markgrover commented on a diff in the pull request:
https://github.com/apache/spark/pull/7739#discussion_r36826585
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala
---
@@ -61,8 +61,10 @@ private[spark] class SparkDeploy
Github user markgrover commented on a diff in the pull request:
https://github.com/apache/spark/pull/7739#discussion_r36826580
--- Diff:
core/src/main/scala/org/apache/spark/deploy/rest/mesos/MesosRestServer.scala ---
@@ -88,7 +88,10 @@ private[mesos] class MesosSubmitRequestServle
Github user markgrover commented on a diff in the pull request:
https://github.com/apache/spark/pull/7739#discussion_r36826569
--- Diff:
core/src/main/scala/org/apache/spark/deploy/rest/SubmitRestProtocolRequest.scala
---
@@ -17,6 +17,8 @@
package org.apache.spark.deplo
Github user markgrover commented on a diff in the pull request:
https://github.com/apache/spark/pull/7739#discussion_r36826550
--- Diff: core/src/main/scala/org/apache/spark/SparkConf.scala ---
@@ -479,6 +484,23 @@ class SparkConf(loadDefaults: Boolean) extends
Cloneable with Loggi
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/7739#discussion_r36820995
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/mesos/MesosSchedulerBackend.scala
---
@@ -88,7 +88,10 @@ private[spark] class MesosSchedulerB
Github user squito commented on a diff in the pull request:
https://github.com/apache/spark/pull/7739#discussion_r36820815
--- Diff: core/src/main/scala/org/apache/spark/SparkConf.scala ---
@@ -479,6 +484,23 @@ class SparkConf(loadDefaults: Boolean) extends
Cloneable with Logging {
Github user harishreedharan commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-130092076
This looks pretty good. Could you also add some tests to each of the
components (YARN/standalone and mesos to ensure the correct classpaths get
passed in the cor
Github user harishreedharan commented on a diff in the pull request:
https://github.com/apache/spark/pull/7739#discussion_r36804452
--- Diff:
core/src/main/scala/org/apache/spark/scheduler/cluster/SparkDeploySchedulerBackend.scala
---
@@ -61,8 +61,10 @@ private[spark] class SparkD
Github user harishreedharan commented on a diff in the pull request:
https://github.com/apache/spark/pull/7739#discussion_r36804066
--- Diff:
core/src/main/scala/org/apache/spark/deploy/rest/mesos/MesosRestServer.scala ---
@@ -88,7 +88,10 @@ private[mesos] class MesosSubmitRequestS
Github user harishreedharan commented on a diff in the pull request:
https://github.com/apache/spark/pull/7739#discussion_r36804010
--- Diff:
core/src/main/scala/org/apache/spark/deploy/rest/StandaloneRestServer.scala ---
@@ -135,7 +135,10 @@ private[rest] class StandaloneSubmitReq
Github user harishreedharan commented on a diff in the pull request:
https://github.com/apache/spark/pull/7739#discussion_r36802676
--- Diff:
core/src/main/scala/org/apache/spark/deploy/rest/StandaloneRestServer.scala ---
@@ -135,7 +135,10 @@ private[rest] class StandaloneSubmitReq
Github user harishreedharan commented on a diff in the pull request:
https://github.com/apache/spark/pull/7739#discussion_r36802639
--- Diff:
core/src/main/scala/org/apache/spark/deploy/rest/SubmitRestProtocolRequest.scala
---
@@ -17,6 +17,8 @@
package org.apache.spark.
Github user harishreedharan commented on a diff in the pull request:
https://github.com/apache/spark/pull/7739#discussion_r36802511
--- Diff: core/src/main/scala/org/apache/spark/deploy/Client.scala ---
@@ -68,7 +68,12 @@ private class ClientEndpoint(
val mainClass = "o
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-129715625
Jenkins, retest this please.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-128778513
Hmm, looks like PYTHONPATH is not being set up properly by some python
tests. It turns out I am not the only suffering from this - the build before
mine
(https://amp
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-128773200
I am taking a look.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-128766775
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-128766648
[Test build #40171 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/40171/console)
for PR 7739 at commit
[`e33a404`](https://github.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-128713202
[Test build #40171 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/40171/consoleFull)
for PR 7739 at commit
[`e33a404`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-128712090
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-128712053
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
Github user squito commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-128712008
Jenkins, retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-128127359
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-128127209
[Test build #39860 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/39860/console)
for PR 7739 at commit
[`e33a404`](https://github.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-128119799
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-128119667
[Test build #231 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SlowSparkPullRequestBuilder/231/console)
for PR 7739 at commit
[`e33a404`](https://github.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-128069167
[Test build #39860 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/39860/consoleFull)
for PR 7739 at commit
[`e33a404`](https://gith
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-128068310
[Test build #231 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SlowSparkPullRequestBuilder/231/consoleFull)
for PR 7739 at commit
[`e33a404`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-128068043
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-128068035
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-128019886
Thanks @squito !
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this fe
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-127992200
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-127991853
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
Github user squito commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-127991794
Jenkins, retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user squito commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-127991776
Jenkins, add to whitelist
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have th
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-127762510
If that doesn't work and some can retrigger the test run on my behalf, I'd
appreciate that. cc @squito @sarutak
---
If your project is set up for it, you can reply
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-127762398
Seems unrelated failure. I don't think I am on the whitelist but I will try
this:
Jenkins, test this please.
---
If your project is set up for it, you can r
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-127729550
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-127729397
[Test build #39716 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/39716/console)
for PR 7739 at commit
[`e33a404`](https://github.
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-127675240
[Test build #39716 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/39716/consoleFull)
for PR 7739 at commit
[`e33a404`](https://gith
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-127672773
Merged build started.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-127635012
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not h
Github user markgrover commented on the pull request:
https://github.com/apache/spark/pull/7739#issuecomment-127634581
Hi @sarutak, thanks again for taking a look, I've rebased and fixed up the
history. I have also update the PR with some more info.
---
If your project is set up for
90 matches
Mail list logo
