Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/9185#issuecomment-167862437
Merging to master
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/9185
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user falaki commented on the pull request:
https://github.com/apache/spark/pull/9185#issuecomment-167664825
ping @marmbrus
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9185#issuecomment-167668475
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9185#issuecomment-167668477
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/9185#issuecomment-167665605
This seems fine to me as a first step. Eventually we will probably want to
make the RBackend multi-session aware.
---
If your project is set up for it, you can reply
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9185#issuecomment-167689398
**[Test build #48386 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48386/consoleFull)**
for PR 9185 at commit
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9185#issuecomment-167703809
**[Test build #48386 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/48386/consoleFull)**
for PR 9185 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9185#issuecomment-167703846
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9185#issuecomment-167703848
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9185#issuecomment-167688404
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/9185#issuecomment-167688501
test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user yhuai commented on the pull request:
https://github.com/apache/spark/pull/9185#issuecomment-167684925
test this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this
Github user felixcheung commented on the pull request:
https://github.com/apache/spark/pull/9185#issuecomment-167688400
As pointed out above, R code actually does not call `createSQLContext`
multiple times:
https://github.com/apache/spark/blob/master/R/pkg/R/sparkR.R#L243
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9185#issuecomment-167688403
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user shivaram commented on the pull request:
https://github.com/apache/spark/pull/9185#issuecomment-167738526
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled
Github user sun-rui commented on the pull request:
https://github.com/apache/spark/pull/9185#issuecomment-155406712
@davies, I don't understand your two cases. SparkR is actually a standalone
spark application in R, the JVM backend is dedicated, won't share with other
applications.
Github user shivaram commented on the pull request:
https://github.com/apache/spark/pull/9185#issuecomment-154490703
Thanks @marmbrus - That helps. Yeah so I think from my perspective it makes
sense to just have one *active* session for SparkR and thus just one active
SQLContext.
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/9185#issuecomment-154524422
I would propose that sparkR.init just has a flag that says if sessions
should be isolated or not. When it connects it can either call
Github user felixcheung commented on the pull request:
https://github.com/apache/spark/pull/9185#issuecomment-154529467
@shivaram I did plan on changing `sparkR.init()` and `sparkRSQL.init()`
I like the idea @marmbrus proposed - we could have
`sparkR.init(new.sql.session = TRUE)`
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/9185#issuecomment-154531048
/cc @davies
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/9185#issuecomment-154548391
R is does not support threading, it's reasonable that SparkR does not
support multiple sessions in the same R process in the same time, as we move to
have a SQLContext a
Github user marmbrus commented on the pull request:
https://github.com/apache/spark/pull/9185#issuecomment-154212591
Sessions allow multiple users to share a Spark SQL Cluster without
clobbering each other. Imagine you have multiple R sessions connected to the
same Spark cluster.
Github user shivaram commented on the pull request:
https://github.com/apache/spark/pull/9185#issuecomment-153782645
Is there an example of how people use sessions ? Or rather can @davies or
you describe what is the API used to support sessions in Scala / Python ?
---
If your
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/9185#issuecomment-153614611
sessions are critical - actually one of the most important features for
spark 1.6.
---
If your project is set up for it, you can reply to this email and have your
reply
Github user shivaram commented on the pull request:
https://github.com/apache/spark/pull/9185#issuecomment-152774944
@rxin can you comment on how important sessions are (esp. with respect to
SparkR) ? If they are not important we can significantly simplify things and
just support one
Github user shivaram commented on the pull request:
https://github.com/apache/spark/pull/9185#issuecomment-151562392
@davies @rxin Any updates on this ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/9185#issuecomment-151567766
@falaki HiveContext has more functionality than SQLContext (Window
functions, ORC files etc.), and a few semantic difference (how to parse decimal
and intervals).
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9185#issuecomment-151568758
**[Test build #1957 has
started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1957/consoleFull)**
for PR 9185 at commit
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/9185#issuecomment-151568092
We could merge this after pass tests.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9185#issuecomment-151610589
**[Test build #1957 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/1957/consoleFull)**
for PR 9185 at commit
Github user felixcheung commented on the pull request:
https://github.com/apache/spark/pull/9185#issuecomment-150318041
I vote for simplicity for SparkR and not have multiple session.
In fact I observe it is already messy to handle DataFrame created by a
different SparkContext
Github user sun-rui commented on the pull request:
https://github.com/apache/spark/pull/9185#issuecomment-150162723
On the R side, there is a cache of created SQLContext/HiveContext, so R
won't call createSQLContext() second time. See
Github user falaki commented on the pull request:
https://github.com/apache/spark/pull/9185#issuecomment-150077277
HiveContext is a subclass of SQLContext. So all of SQLContext functionality
continues to work. cc @marmbrus what do you think?
---
If your project is set up for it, you
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9185#issuecomment-149670410
Merged build triggered.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not
GitHub user falaki opened a pull request:
https://github.com/apache/spark/pull/9185
[SPARKR] [SPARK-11199] Improve R context management story and add
getOrCreate
* Changes api.r.SQLUtils to use ```SQLContext.getOrCreate``` instead of
creating a new context.
* Adds a simple
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9185#issuecomment-149672129
**[Test build #44006 has
started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44006/consoleFull)**
for PR 9185 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9185#issuecomment-149696821
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
Github user SparkQA commented on the pull request:
https://github.com/apache/spark/pull/9185#issuecomment-149696782
**[Test build #44006 has
finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/44006/consoleFull)**
for PR 9185 at commit
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/9185#issuecomment-149696819
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user rxin commented on the pull request:
https://github.com/apache/spark/pull/9185#issuecomment-149764857
cc @davies
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
Github user davies commented on the pull request:
https://github.com/apache/spark/pull/9185#issuecomment-149781581
@falaki SQLContext.getOrCreate could return HiveContext, it's slightly
different than `new SQLContext`, is this what we want?
---
If your project is set up for it, you
42 matches
Mail list logo
