[jira] [Commented] (SPARK-11524) Support SparkR with Mesos cluster
[ https://issues.apache.org/jira/browse/SPARK-11524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15581268#comment-15581268 ] Sun Rui commented on SPARK-11524: - for cluster mode, the R script needs to transferred to the slave node chosen to run the spark driver. For YARN, this is done by adding it to the local resources for the application. For Mesos, I think the file server mechanism can be utilized. > Support SparkR with Mesos cluster > - > > Key: SPARK-11524 > URL: https://issues.apache.org/jira/browse/SPARK-11524 > Project: Spark > Issue Type: New Feature > Components: SparkR >Affects Versions: 1.5.1 >Reporter: Sun Rui > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11524) Support SparkR with Mesos cluster
[ https://issues.apache.org/jira/browse/SPARK-11524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15581023#comment-15581023 ] Susan X. Huynh commented on SPARK-11524: Thanks for the advice and for breaking down the sub-issues. For mesos _cluster_ mode, is there additional work? Or is it just the same problem to locate the sparkr.zip at slave nodes? > Support SparkR with Mesos cluster > - > > Key: SPARK-11524 > URL: https://issues.apache.org/jira/browse/SPARK-11524 > Project: Spark > Issue Type: New Feature > Components: SparkR >Affects Versions: 1.5.1 >Reporter: Sun Rui > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11524) Support SparkR with Mesos cluster
[ https://issues.apache.org/jira/browse/SPARK-11524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15580960#comment-15580960 ] Sun Rui commented on SPARK-11524: - great, go ahead. Please look at the linking JIRA issues as references. Basically, to enable SparkR with mesos client mode, to problem is to locate the sparkr.zip at slave nodes, and set the environment for R so that it can load the SparkR package. (No need to pass the sparkr.zip from the driver node to slave nodes, because the file is already part of the spark distribution.) When locating sparkr.zip, two cases should be considered: locally installed spark distribution or downloaded from a public-accessible place. (check spark.mesos.executor.home ) I just broke this issue into sub-issues, we can address it one by one > Support SparkR with Mesos cluster > - > > Key: SPARK-11524 > URL: https://issues.apache.org/jira/browse/SPARK-11524 > Project: Spark > Issue Type: New Feature > Components: SparkR >Affects Versions: 1.5.1 >Reporter: Sun Rui > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11524) Support SparkR with Mesos cluster
[ https://issues.apache.org/jira/browse/SPARK-11524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15578137#comment-15578137 ] Susan X. Huynh commented on SPARK-11524: I would like to work on this. What are the missing pieces? What configurations need to be tested? [~sunrui][~felixcheung] > Support SparkR with Mesos cluster > - > > Key: SPARK-11524 > URL: https://issues.apache.org/jira/browse/SPARK-11524 > Project: Spark > Issue Type: New Feature > Components: SparkR >Affects Versions: 1.5.1 >Reporter: Sun Rui > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
