Re: sparkR ORC support.

2016-01-12 Thread Sandeep Khurana
; > Is there a reason you want to call stop? If you do, you would need to call > the line hivecontext <- sparkRHive.init(sc) again. > > > _ > From: Sandeep Khurana > Sent: Tuesday, January 12, 2016 5:20 AM > Subject: Re: sparkR ORC support. > To: F

Re: sparkR ORC support.

2016-01-12 Thread Felix Cheung
would need to call the line hivecontext <- sparkRHive.init(sc) again. _ From: Sandeep Khurana Sent: Tuesday, January 12, 2016 5:20 AM Subject: Re: sparkR ORC support. To: Felix Cheung Cc: spark users , Prem Sure , Deepak Sharma , Yanbo Liang It worked for so

Re: sparkR ORC support.

2016-01-12 Thread Sandeep Khurana
k-client") >>> >>> .libPaths(c(file.path(Sys.getenv("SPARK_HOME"), "R", "lib"), >>> .libPaths())) >>> library(SparkR) >>> >>> sc <- sparkR.init() >>> hivecontext <- sparkRHive.init(sc) >>&g

Re: sparkR ORC support.

2016-01-12 Thread Sandeep Khurana
gt;> stop("invalid jobj ", value$id) >> 7 >> writeJobj(con, object) >> 6 >> writeObject(con, a) >> 5 >> writeArgs(rc, args) >> 4 >> invokeJava(isStatic = TRUE, className, methodName, ...) >> 3 >> callJStatic("org.apache.s

Re: sparkR ORC support.

2016-01-12 Thread Sandeep Khurana
kRHive.init(sc) > df <- loadDF(hivecontext, "/data/ingest/sparktest1/", "orc") > > > > -- > Date: Tue, 12 Jan 2016 14:28:58 +0530 > Subject: Re: sparkR ORC support. > From: sand...@infoworks.io > To: felixcheun...@hotmail.com > CC

RE: sparkR ORC support.

2016-01-12 Thread Felix Cheung
Hive.init(sc)df <- loadDF(hivecontext, "/data/ingest/sparktest1/", "orc") Date: Tue, 12 Jan 2016 14:28:58 +0530 Subject: Re: sparkR ORC support. From: sand...@infoworks.io To: felixcheun...@hotmail.com CC: yblia...@gmail.com; user@spark.apache.org; premsure...@gmail.com;

Re: sparkR ORC support.

2016-01-12 Thread Sandeep Khurana
t; > Firstly I don't have ORC data to verify but this should work: > > df <- loadDF(sqlContext, "data/path", "orc") > > Secondly, could you check if sparkR.stop() was called? sparkRHive.init() > should be called after sparkR.init() - please check if there i

Re: sparkR ORC support.

2016-01-06 Thread Felix Cheung
a/path", "orc") >> >> Secondly, could you check if sparkR.stop() was called? sparkRHive.init() >> should be called after sparkR.init() - please check if there is any error >> message there. >> >> _ >> From: Prem Sure

Re: sparkR ORC support.

2016-01-06 Thread Yanbo Liang
data to verify but this should work: >> >> df <- loadDF(sqlContext, "data/path", "orc") >> >> Secondly, could you check if sparkR.stop() was called? sparkRHive.init() >> should be called after sparkR.init() - please check if there is any error

Re: sparkR ORC support.

2016-01-06 Thread Sandeep Khurana
ge there. > > _________________ > From: Prem Sure > Sent: Tuesday, January 5, 2016 8:12 AM > Subject: Re: sparkR ORC support. > To: Sandeep Khurana > Cc: spark users , Deepak Sharma < > deepakmc...@gmail.com> > > > > Yes Sandeep, also copy hive-s

Re: sparkR ORC support.

2016-01-05 Thread Felix Cheung
is any error message there. _ From: Prem Sure Sent: Tuesday, January 5, 2016 8:12 AM Subject: Re: sparkR ORC support. To: Sandeep Khurana Cc: spark users , Deepak Sharma Yes Sandeep, also copy hive-site.xml too to spark conf directory. On Tue, Jan 5, 2016

Re: sparkR ORC support.

2016-01-05 Thread Prem Sure
Yes Sandeep, also copy hive-site.xml too to spark conf directory. On Tue, Jan 5, 2016 at 10:07 AM, Sandeep Khurana wrote: > Also, do I need to setup hive in spark as per the link > http://stackoverflow.com/questions/26360725/accesing-hive-tables-in-spark > ? > > We might need to copy hdfs-site.

Re: sparkR ORC support.

2016-01-05 Thread Sandeep Khurana
Also, do I need to setup hive in spark as per the link http://stackoverflow.com/questions/26360725/accesing-hive-tables-in-spark ? We might need to copy hdfs-site.xml file to spark conf directory ? On Tue, Jan 5, 2016 at 8:28 PM, Sandeep Khurana wrote: > Deepak > > Tried this. Getting this erro

Re: sparkR ORC support.

2016-01-05 Thread Sandeep Khurana
Deepak Tried this. Getting this error now rror in sql(hivecontext, "FROM CATEGORIES SELECT category_id", "") : unused argument ("") On Tue, Jan 5, 2016 at 6:48 PM, Deepak Sharma wrote: > Hi Sandeep > can you try this ? > > results <- sql(hivecontext, "FROM test SELECT id","") > > Thanks > D

Re: sparkR ORC support.

2016-01-05 Thread Deepak Sharma
Hi Sandeep can you try this ? results <- sql(hivecontext, "FROM test SELECT id","") Thanks Deepak On Tue, Jan 5, 2016 at 5:49 PM, Sandeep Khurana wrote: > Thanks Deepak. > > I tried this as well. I created a hivecontext with "hivecontext <<- > sparkRHive.init(sc) " . > > When I tried to r

Re: sparkR ORC support.

2016-01-05 Thread Sandeep Khurana
Thanks Deepak. I tried this as well. I created a hivecontext with "hivecontext <<- sparkRHive.init(sc) " . When I tried to read hive table from this , results <- sql(hivecontext, "FROM test SELECT id") I get below error, Error in callJMethod(sqlContext, "sql", sqlQuery) : Invalid jobj 2.

Re: sparkR ORC support.

2016-01-05 Thread Deepak Sharma
Hi Sandeep I am not sure if ORC can be read directly in R. But there can be a workaround .First create hive table on top of ORC files and then access hive table in R. Thanks Deepak On Tue, Jan 5, 2016 at 4:57 PM, Sandeep Khurana wrote: > Hello > > I need to read an ORC files in hdfs in R using