Perhaps it is with spark.sql.warehouse.dir="E:/Exp/"
That you have in the sparkConfig parameter. Unfortunately the exception stack is fairly far away from the actual error, but from the top of my head spark.sql.warehouse.dir and HADOOP_HOME are the two different pieces that is not set in the Windows tests. _____________________________ From: Md. Rezaul Karim <rezaul.ka...@insight-centre.org<mailto:rezaul.ka...@insight-centre.org>> Sent: Monday, January 2, 2017 7:58 AM Subject: Re: Issue with SparkR setup on RStudio To: Felix Cheung <felixcheun...@hotmail.com<mailto:felixcheun...@hotmail.com>> Cc: spark users <user@spark.apache.org<mailto:user@spark.apache.org>> Hello Cheung, Happy New Year! No, I did not configure Hive on my machine. Even I have tried not setting the HADOOP_HOME but getting the same error. Regards, _________________________________ Md. Rezaul Karim BSc, MSc PhD Researcher, INSIGHT Centre for Data Analytics National University of Ireland, Galway IDA Business Park, Dangan, Galway, Ireland Web:http://www.reza-analytics.eu/index.html<http://139.59.184.114/index.html> On 29 December 2016 at 19:16, Felix Cheung <felixcheun...@hotmail.com<mailto:felixcheun...@hotmail.com>> wrote: Any reason you are setting HADOOP_HOME? >From the error it seems you are running into issue with Hive config likely >with trying to load hive-site.xml. Could you try not setting HADOOP_HOME ________________________________ From: Md. Rezaul Karim <rezaul.ka...@insight-centre.org<mailto:rezaul.ka...@insight-centre.org>> Sent: Thursday, December 29, 2016 10:24:57 AM To: spark users Subject: Issue with SparkR setup on RStudio Dear Spark users, I am trying to setup SparkR on RStudio to perform some basic data manipulations and MLmodeling. However, I am a strange error while creating SparkR session or DataFrame that says:java.lang.IllegalArgumentException Error while instantiating 'org.apache.spark.sql.hive.HiveSessionState. According to Spark documentation athttp://spark.apache.org/docs/latest/sparkr.html#starting-up-sparksession, I don’t need to configure Hive path or related variables. I have the following source code: SPARK_HOME = "C:/spark-2.1.0-bin-hadoop2.7" HADOOP_HOME= "C:/spark-2.1.0-bin-hadoop2.7/bin/" library(SparkR, lib.loc = c(file.path(Sys.getenv("SPARK_HOME"), "R", "lib"))) sparkR.session(appName = "SparkR-DataFrame-example", master = "local[*]", sparkConfig = list(spark.sql.warehouse.dir="E:/Exp/", spark.driver.memory = "8g"), enableHiveSupport = TRUE) # Create a simple local data.frame localDF <- data.frame(name=c("John", "Smith", "Sarah"), age=c(19, 23, 18)) # Convert local data frame to a SparkDataFrame df <- createDataFrame(localDF) print(df) head(df) sparkR.session.stop() Please note that the HADOOP_HOME contains the ‘winutils.exe’ file. The details of the eror is as follows: Error in handleErrors(returnStatus, conn) : java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveSessionState': at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$$reflect(SparkSession.scala:981) at org.apache.spark.sql.SparkSession.sessionState$lzycompute(SparkSession.scala:110) at org.apache.spark.sql.SparkSession.sessionState(SparkSession.scala:109) at org.apache.spark.sql.api.r.SQLUtils$$anonfun$setSparkContextSessionConf$2.apply(SQLUtils.scala:67) at org.apache.spark.sql.api.r.SQLUtils$$anonfun$setSparkContextSessionConf$2.apply(SQLUtils.scala:66) at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733) at scala.collection.Iterator$class.foreach(Iterator.scala:893) at scala.collection.AbstractIterator.foreach(Iterator.scala:1336) at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at scala.collection.AbstractIterable.foreach(Iterable.scala:54) at scala.collection.Traversabl Any kind of help would be appreciated. Regards, _________________________________ Md. Rezaul Karim BSc, MSc PhD Researcher, INSIGHT Centre for Data Analytics National University of Ireland, Galway IDA Business Park, Dangan, Galway, Ireland Web:http://www.reza-analytics.eu/index.html<http://139.59.184.114/index.html>
