I am trying to setup a Spark stand-alone cluster and run SparkR. I have one master and 2 slaves setup on a Windows Server 2012. I have tried running the sparkR + RStudio example as shown in this blog:[ http://blog.danielemaasit.com/2015/07/26/installing-and-starting-sparkr-locally-on-windows-8-1-and-rstudio/ <http://blog.danielemaasit.com/2015/07/26/installing-and-starting-sparkr-locally-on-windows-8-1-and-rstudio/> ] I was able to execute this seamlessly on my RStudio(in the server) - but what I want to achieve is firing the sparkR driver program remotely(on my laptop), so that it gets connected to my master's IP address and so that the execution part runs on my standalone Spark installation. Is this possible via SparkR? I found an article online where a similar situation is discussed.[ https://qnalist.com/questions/5006092/cannot-submit-to-a-spark-application-to-a-remote-cluster-spark-1-0 <https://qnalist.com/questions/5006092/cannot-submit-to-a-spark-application-to-a-remote-cluster-spark-1-0> ] I'm not sure where and how to include "spark.home" in my case though.Is this the right way to proceed?My code & error below from RStudio(running on laptop):Sys.setenv(SPARK_HOME = "C:/spark-1.6.2-bin-hadoop2.6").libPaths(c(file.path(Sys.getenv("SPARK_HOME"), "R", "lib"), .libPaths()))#load the Sparkr library#library(SparkR)library(SparkR, lib.loc = "C:/spark-1.6.2-bin-hadoop2.6/R/lib")# Create a spark context and a SQL contextsc <- sparkR.init(master = "spark://Master's IP:7077",appName="HelloWorld") sqlContext <- sparkRSQL.init(sc)#create a sparkR DataFrameDF <- createDataFrame(sqlContext, faithful)head(DF)# Create a simple local data.framelocalDF <- data.frame(name=c("John", "Smith", "Sarah"), age=c(19, 23, 18))# Convert local data frame to a SparkR DataFramedf <- createDataFrame(sqlContext, localDF)# Print its schemaprintSchema(df)# root # |-- name: string (nullable = true)# |-- age: double (nullable = true)# Create a DataFrame from a JSON filepath <- file.path(Sys.getenv("SPARK_HOME"),"examples/src/main/resources/people.json") peopleDF <- jsonFile(sqlContext, path)printSchema(peopleDF)# Register this DataFrame as a table.registerTempTable(peopleDF, "people")# SQL statements can be run by using the sql methods provided by sqlContextteenagers <- sql(sqlContext, "SELECT name FROM people WHERE age >= 13 AND age <= 19")# Call collect to get a local data.frameteenagersLocalDF <- collect(teenagers)# Print the teenagers in our dataset print(teenagersLocalDF)# Stop the SparkContext nowsparkR.stop()ERROR:> sc <- sparkR.init(master = "spark://Master's IP:7077", appName="HelloWorld")Launching java with spark-submit command C:/spark-1.6.2-bin-hadoop2.6/bin/spark-submit.cmd sparkr-shell C:\Users\admin\AppData\Local\Temp\RtmpOQea5v\backend_port31d417b617d8 Error in invokeJava(isStatic = TRUE, className, methodName, ...) : java.lang.NullPointerException at org.apache.spark.SparkContext.(SparkContext.scala:583) at org.apache.spark.api.java.JavaSparkContext.(JavaSparkContext.scala:59) at org.apache.spark.api.r.RRDD$.createSparkContext(RRDD.scala:376) at org.apache.spark.api.r.RRDD.createSparkContext(RRDD.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.spark.api.r.RBackendHandler.handleMethodCall(RBackendHandler.scala:141) at org.apache.spark.api.r.RBackendHandler.channelRead0(RBackendHandler.scala:86) at org.apache.spark.api.r.RBackendHandler.channelRead0(RBackendHandler.scala:38) at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) at io.netty.channel.AbstractC
-- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/SparkR-Exception-in-invokeJava-SparkR-Windows-standalone-cluster-tp27298.html Sent from the Apache Spark User List mailing list archive at Nabble.com.
