Github user dmvieira commented on the issue:
https://github.com/apache/spark/pull/15000
Hey guys, How can I do same thing using sparkR?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this fe
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/15000
merged to master
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or i
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15000
**[Test build #3280 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3280/consoleFull)**
for PR 15000 at commit
[`cb13f43`](https://github.com/apache/spark/commit
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15000
**[Test build #3280 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3280/consoleFull)**
for PR 15000 at commit
[`cb13f43`](https://github.com/apache/spark/commit/
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/15000
@davies do you have an opinion?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/15000
It's still probably reasonable to plumb it through but I'll leave it open a
bit for comments.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub a
Github user apetresc commented on the issue:
https://github.com/apache/spark/pull/15000
As requested, removed the Java property and left only the PySpark one.
I'll admit I didn't appreciate that you could access the Scala SparkContext
straight from PySpark originally (I figure
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/15000
I don't see value in exposing it, and many other things aren't exposed via
JSC. It's really only things that need a different API to make sense in Java.
---
If your project is set up for it, you can
Github user zjffdu commented on the issue:
https://github.com/apache/spark/pull/15000
yeah, I just try the following statement and it works. But I think it is no
harm to expose it in JavaSparkContext as well.
```
sc._jsc.sc().uiWebUrl().get()
```
---
If your project
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/15000
`_jsc` is a `JavaSparkContext` and `.sc()` is a `SparkContext`. That's
accessible, but maybe you'll tell me that the Py4J wrapper won't play nice with
Scala `Option`? not sure. See how other similar
Github user zjffdu commented on the issue:
https://github.com/apache/spark/pull/15000
@srowen `_jsc.sc()` is `JavaSparkContext`, I think that's why @apetresc
expose it in `JavaSparkContext` first
---
If your project is set up for it, you can reply to this email and have your
reply
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/15000
Looking at context.py, it seems that it accesses things directly from
SparkContext via _jsc.sc() where possible. I think that means you can just
expose this in Pyspark without exposing it separately
Github user apetresc commented on the issue:
https://github.com/apache/spark/pull/15000
@srowen: Just to make sure I understand, are you asking me to remove the
Java accessor here, and just plumb straight through to the Scala object from
PySpark? Or is it fine as-is?
---
If your pro
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/15000
Ah right dumb question. Yeah I think it makes some sense ... maybe not even
for Java because there are lots of methods we don't plumb through because you
can easily access them directly rom scala. Py
Github user apetresc commented on the issue:
https://github.com/apache/spark/pull/15000
PySpark. I don't think anyone runs Java through Jupyter, haha.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does n
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/15000
Is this Java or Pyspark? In Java you can still get this property directly
from the underlying SparkContext.
---
If your project is set up for it, you can reply to this email and have your
reply appe
Github user apetresc commented on the issue:
https://github.com/apache/spark/pull/15000
Well, here's the use case I want it for: I'm building some plugins for
JupyterHub to make it more Spark-aware, and I want to be able to link the user
out to the right WebUI for their kernel. Short
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/15000
My only hesitation about this is that this property really only exists to
print it in the shell. Is there a good use case for it otherwise? I know it's
minor but want to make sure we're not just doin
Github user zjffdu commented on the issue:
https://github.com/apache/spark/pull/15000
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the featur
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15000
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
e
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15000
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/65057/
Test PASSed.
---
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15000
**[Test build #65057 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65057/consoleFull)**
for PR 15000 at commit
[`9c57eb7`](https://github.com/apache/spark/commit/
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/15000
**[Test build #65057 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/65057/consoleFull)**
for PR 15000 at commit
[`9c57eb7`](https://github.com/apache/spark/commit/9
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/15000
Jenkins test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes
Github user srowen commented on the issue:
https://github.com/apache/spark/pull/15000
That's probably OK; Java users can already pretty easily get this anyway
but not Python users.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/15000
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feat
26 matches
Mail list logo
