[
https://issues.apache.org/jira/browse/SPARK-13573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15175089#comment-15175089
]
Chip Senkbeil commented on SPARK-13573:
---------------------------------------
In terms of the JVM class whose methods we are invoking, the majority can be
found here:
https://github.com/apache/incubator-toree/blob/master/kernel-api/src/main/scala/org/apache/toree/interpreter/broker/BrokerState.scala#L90
We basically maintain an object that acts as a code queue where the SparkR
process pulls off code to evaluate and then sends back results as strings.
We also had to write a wrapper for the RBackend since it was package protected:
https://github.com/apache/incubator-toree/blob/master/sparkr-interpreter/src/main/scala/org/apache/toree/kernel/interpreter/sparkr/ReflectiveRBackend.scala
> Open SparkR APIs (R package) to allow better 3rd party usage
> ------------------------------------------------------------
>
> Key: SPARK-13573
> URL: https://issues.apache.org/jira/browse/SPARK-13573
> Project: Spark
> Issue Type: Improvement
> Components: SparkR
> Reporter: Chip Senkbeil
>
> Currently, SparkR's R package does not expose enough of its APIs to be used
> flexibly. That I am aware of, SparkR still requires you to create a new
> SparkContext by invoking the sparkR.init method (so you cannot connect to a
> running one) and there is no way to invoke custom Java methods using the
> exposed SparkR API (unlike PySpark).
> We currently maintain a fork of SparkR that is used to power the R
> implementation of Apache Toree, which is a gateway to use Apache Spark. This
> fork provides a connect method (to use an existing Spark Context), exposes
> needed methods like invokeJava (to be able to communicate with our JVM to
> retrieve code to run, etc), and uses reflection to access
> org.apache.spark.api.r.RBackend.
> Here is the documentation I recorded regarding changes we need to enable
> SparkR as an option for Apache Toree:
> https://github.com/apache/incubator-toree/tree/master/sparkr-interpreter/src/main/resources
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
