Ah, sorry for misreading. I don't think there is a way to use UDF in your SQLs only with SparkSQL. You might be able to use with SparkHive, but I'm sorry, I don't know well.
I think you should use the function before convert to SchemaRDD if you can. Thanks. 2014-07-04 18:16 GMT+09:00 Martin Gammelsæter <martingammelsae...@gmail.com> : > Takuya, thanks for your reply :) > I am already doing that, and it is working well. My question is, can I > define arbitrary functions to be used in these queries? > > On Fri, Jul 4, 2014 at 11:12 AM, Takuya UESHIN <ues...@happy-camper.st> > wrote: > > Hi, > > > > You can convert standard RDD of Product class (e.g. case class) to > SchemaRDD > > by SQLContext. > > Load data from Cassandra into RDD of case class, convert it to SchemaRDD > and > > register it, > > then you can use it in your SQLs. > > > > > http://spark.apache.org/docs/latest/sql-programming-guide.html#running-sql-on-rdds > > > > Thanks. > > > > > > > > 2014-07-04 17:59 GMT+09:00 Martin Gammelsæter > > <martingammelsae...@gmail.com>: > > > >> Hi! > >> > >> I have a Spark cluster running on top of a Cassandra cluster, using > >> Datastax' new driver, and one of the fields of my RDDs is an > >> XML-string. In a normal Scala sparkjob, parsing that data is no > >> problem, but I would like to also make that information available > >> through Spark SQL. So, is there any way to write user defined > >> functions for Spark SQL? I know that a HiveContext is available, but I > >> understand that that is for querying data from Hive, and I don't have > >> Hive in my stack (please correct me if I'm wrong). > >> > >> I would love to be able to do something like the following: > >> > >> val casRdd = sparkCtx.cassandraTable("ks", "cf") > >> > >> // registerAsTable etc > >> > >> val res = sql("SELECT id, xmlGetTag(xmlfield, 'sometag') FROM cf") > >> > >> -- > >> Best regards, > >> Martin Gammelsæter > > > > > > > > > > -- > > Takuya UESHIN > > Tokyo, Japan > > > > http://twitter.com/ueshin > > > > -- > Mvh. > Martin Gammelsæter > 92209139 > -- Takuya UESHIN Tokyo, Japan http://twitter.com/ueshin
