Github user skonto commented on the issue:
https://github.com/apache/spark/pull/18630
@vanzin I dont forgot I didnt see agreement.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this featur
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/18630
You forgot to address @BryanCutler 's comments; I'll fix the easy ones
during merge.
Merging to master.
---
If your project is set up for it, you can reply to this email and have your
reply
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18630
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18630
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80549/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18630
**[Test build #80549 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80549/testReport)**
for PR 18630 at commit
[`db60b27`](https://github.com/apache/spark/commit/d
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18630
**[Test build #80549 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80549/testReport)**
for PR 18630 at commit
[`db60b27`](https://github.com/apache/spark/commit/db
Github user skonto commented on the issue:
https://github.com/apache/spark/pull/18630
@vanzin fixed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if
Github user skonto commented on the issue:
https://github.com/apache/spark/pull/18630
@vanzin do you think we are ready for a merge?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this featu
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18630
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80468/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18630
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18630
**[Test build #80468 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80468/testReport)**
for PR 18630 at commit
[`c0b0a7d`](https://github.com/apache/spark/commit/c
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/18630
Ok, thanks for checking. It doesn't look like it's coming from your
changes, so I'm sure it's just me.
---
If your project is set up for it, you can reply to this email and have your
reply app
Github user skonto commented on the issue:
https://github.com/apache/spark/pull/18630
This is how I build things:
./build/mvn -Pmesos -Phadoop-2.7 -Dhadoop.version=2.7.0 -DskipTests clean
package
# -DskipTests clean package
export JAVA_HOME=/usr/lib/jvm/java
Github user skonto commented on the issue:
https://github.com/apache/spark/pull/18630
@BryanCutler sure check here, it works:
https://gist.github.com/skonto/dc2070d1529c97ec5de32e99983a834f
---
If your project is set up for it, you can reply to this email and have your
reply appea
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/18630
Maybe it was just something with my env - but I was running it locally, can
you just verify that works too? Just don't specify the `--master` conf and run
out of your spark home dir
---
If yo
Github user skonto commented on the issue:
https://github.com/apache/spark/pull/18630
> spark-2.3.0-SNAPSHOT-bin-18630/bin$ ./spark-shell --verbose --master
spark://ip-10-10-1-79:7077
Using properties file: null
Parsed arguments:
master spark://ip-10-10-
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/18630
Yeah, just by running `bin/spark-shell` it failed immediately with that
error. I double-check by rebuilding and same thing but I'm not sure if was
something from your changes or not. Are you a
Github user skonto commented on the issue:
https://github.com/apache/spark/pull/18630
@BryanCutler you just started spark shell and it failed? How can I
reproduce it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If yo
Github user BryanCutler commented on the issue:
https://github.com/apache/spark/pull/18630
Sure, python support could be added at a later point, I was just thinking
if it was only a small addition to what's already here, but no problem. Btw,
after checking out this PR I tried spark-s
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/18630
I wasn't really expecting python support to be added here. I wonder if
there's a bug open for that.
---
If your project is set up for it, you can reply to this email and have your
reply appear on Gi
Github user skonto commented on the issue:
https://github.com/apache/spark/pull/18630
@BryanCutler @vanzin to make things testable DriverWrapper needs
refactoring from a quick look I took.
py files are resolved in client mode, let's fix it in another PR (I could
do it). The doc
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18630
**[Test build #80468 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80468/testReport)**
for PR 18630 at commit
[`c0b0a7d`](https://github.com/apache/spark/commit/c0
Github user skonto commented on the issue:
https://github.com/apache/spark/pull/18630
@vanzin I fixed these minor issues.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18630
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18630
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80433/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18630
**[Test build #80433 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80433/testReport)**
for PR 18630 at commit
[`a177992`](https://github.com/apache/spark/commit/a
Github user skonto commented on the issue:
https://github.com/apache/spark/pull/18630
@vanzin fixed the issues. Ready for merge.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18630
**[Test build #80433 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80433/testReport)**
for PR 18630 at commit
[`a177992`](https://github.com/apache/spark/commit/a1
Github user skonto commented on the issue:
https://github.com/apache/spark/pull/18630
@srowen or @vanzin can I get a merge pls?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18630
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80208/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18630
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18630
**[Test build #80208 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80208/testReport)**
for PR 18630 at commit
[`70649e2`](https://github.com/apache/spark/commit/7
Github user skonto commented on the issue:
https://github.com/apache/spark/pull/18630
@vanzin fixed the issues. Please give it another try or merge.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18630
**[Test build #80208 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80208/testReport)**
for PR 18630 at commit
[`70649e2`](https://github.com/apache/spark/commit/70
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18630
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79906/
Test PASSed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18630
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18630
**[Test build #79906 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79906/testReport)**
for PR 18630 at commit
[`c559a08`](https://github.com/apache/spark/commit/c
Github user skonto commented on the issue:
https://github.com/apache/spark/pull/18630
@vanzin I did the required refactoring, addressing the issues you mentioned
and added to the description a better message so its available in git log when
this PR is merged.
---
If your project is
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18630
**[Test build #79906 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79906/testReport)**
for PR 18630 at commit
[`c559a08`](https://github.com/apache/spark/commit/c5
Github user skonto commented on the issue:
https://github.com/apache/spark/pull/18630
@vanzin thnx done, I changed the title. I am willing to work on the
distributed cache but needs time for sure.
---
If your project is set up for it, you can reply to this email and have your
reply a
Github user vanzin commented on the issue:
https://github.com/apache/spark/pull/18630
As with the Mesos change, I'm ok with this approach until there's some sort
of distributed cache available for those cluster managers. I still need to
review the code, though.
@skonto please
Github user skonto commented on the issue:
https://github.com/apache/spark/pull/18630
I agree that's why I didnt try to do the refactoring... I thought it would
take a long time... As for the cluster mode some people use it due to the
supervise feature, guess not too many, but dont ha
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/18630
One layer for different cluster managers seems promising, but looks like it
requires lots of refactoring works (trying to build a Spark own distributed
cache and change the existing way to distrib
Github user skonto commented on the issue:
https://github.com/apache/spark/pull/18630
@jerryshao I agree I had shared this thought with @vanzin for utilizing the
distributed cache or whatever is available there in the other PR for mesos. For
mesos I am also going to discuss this with
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/18630
I'm wondering if we can prepare all the resources in `SparkSubmit` and
upload to HDFS's staging folder, then driver could download them from HDFS and
add to classpath. This is what basically YARN#
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18630
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18630
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79670/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18630
**[Test build #79670 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79670/testReport)**
for PR 18630 at commit
[`b75387c`](https://github.com/apache/spark/commit/b
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18630
**[Test build #79670 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79670/testReport)**
for PR 18630 at commit
[`b75387c`](https://github.com/apache/spark/commit/b7
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18630
**[Test build #79669 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79669/testReport)**
for PR 18630 at commit
[`1f92106`](https://github.com/apache/spark/commit/1
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18630
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79669/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18630
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user skonto commented on the issue:
https://github.com/apache/spark/pull/18630
@jerryshao @vanzin I updated the PR, pls review. In standalone cluster
mode with this new update I can resolve jars paths of the form:
/tmp/tmp7255787727862122372/jarname.jar,
file:///home/ubunt
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18630
**[Test build #79669 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79669/testReport)**
for PR 18630 at commit
[`1f92106`](https://github.com/apache/spark/commit/1f
Github user skonto commented on the issue:
https://github.com/apache/spark/pull/18630
One thing I noticed is that if you use https you need to set : --conf
spark.ssl.enabled=true --conf spark.ssl.protocol=TLS but this does nto apply to
the user jar. I think that is an issue.
---
If
Github user skonto commented on the issue:
https://github.com/apache/spark/pull/18630
@jerryshao
>
Are you trying to support --packages in standalone cluster?
Yes that is my intention as stated in the description. I will proceed with
the changes thnx for
Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/18630
Are you trying to support `--packages` in standalone cluster?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18630
**[Test build #79599 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79599/testReport)**
for PR 18630 at commit
[`9816704`](https://github.com/apache/spark/commit/9
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18630
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79599/
Test FAILed.
---
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18630
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18630
**[Test build #79599 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79599/testReport)**
for PR 18630 at commit
[`9816704`](https://github.com/apache/spark/commit/98
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18630
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18630
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79598/
Test FAILed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18630
**[Test build #79598 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79598/testReport)**
for PR 18630 at commit
[`81a5dee`](https://github.com/apache/spark/commit/8
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18630
**[Test build #79598 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79598/testReport)**
for PR 18630 at commit
[`81a5dee`](https://github.com/apache/spark/commit/81
65 matches
Mail list logo
