Re: Registering custom UDAFs with HiveConetxt in SparkSQL, how?
Hello Shahab, Are you able to read tables created in Hive from Spark SQL ? If yes, how are you referring them ? On Thu, Mar 26, 2015 at 1:11 PM, Takeshi Yamamuro wrote: > I think it is not `sqlContext` but hiveContext because `create temporary > function` is not supported in SQLContext. > > On Wed, Mar 25, 2015 at 5:58 AM, Jon Chase wrote: > >> Shahab - >> >> This should do the trick until Hao's changes are out: >> >> >> sqlContext.sql("create temporary function foobar as >> 'com.myco.FoobarUDAF'"); >> >> sqlContext.sql("select foobar(some_column) from some_table"); >> >> >> This works without requiring to 'deploy' a JAR with the UDAF in it - just >> make sure the UDAF is in your project's classpath. >> >> >> >> >> On Tue, Mar 10, 2015 at 8:21 PM, Cheng, Hao wrote: >> >>> Oh, sorry, my bad, currently Spark SQL doesn’t provide the user >>> interface for UDAF, but it can work seamlessly with Hive UDAF (via >>> HiveContext). >>> >>> >>> >>> I am also working on the UDAF interface refactoring, after that we can >>> provide the custom interface for extension. >>> >>> >>> >>> https://github.com/apache/spark/pull/3247 >>> >>> >>> >>> >>> >>> *From:* shahab [mailto:shahab.mok...@gmail.com] >>> *Sent:* Wednesday, March 11, 2015 1:44 AM >>> *To:* Cheng, Hao >>> *Cc:* user@spark.apache.org >>> *Subject:* Re: Registering custom UDAFs with HiveConetxt in SparkSQL, >>> how? >>> >>> >>> >>> Thanks Hao, >>> >>> But my question concerns UDAF (user defined aggregation function ) not >>> UDTF( user defined type function ). >>> >>> I appreciate if you could point me to some starting point on UDAF >>> development in Spark. >>> >>> >>> >>> Thanks >>> >>> Shahab >>> >>> On Tuesday, March 10, 2015, Cheng, Hao wrote: >>> >>> Currently, Spark SQL doesn’t provide interface for developing the >>> custom UDTF, but it can work seamless with Hive UDTF. >>> >>> >>> >>> I am working on the UDTF refactoring for Spark SQL, hopefully will >>> provide an Hive independent UDTF soon after that. >>> >>> >>> >>> *From:* shahab [mailto:shahab.mok...@gmail.com] >>> *Sent:* Tuesday, March 10, 2015 5:44 PM >>> *To:* user@spark.apache.org >>> *Subject:* Registering custom UDAFs with HiveConetxt in SparkSQL, how? >>> >>> >>> >>> Hi, >>> >>> >>> >>> I need o develop couple of UDAFs and use them in the SparkSQL. While >>> UDFs can be registered as a function in HiveContext, I could not find any >>> documentation of how UDAFs can be registered in the HiveContext?? so far >>> what I have found is to make a JAR file, out of developed UDAF class, and >>> then deploy the JAR file to SparkSQL . >>> >>> >>> >>> But is there any way to avoid deploying the jar file and register it >>> programmatically? >>> >>> >>> >>> >>> >>> best, >>> >>> /Shahab >>> >>> >> > > > -- > --- > Takeshi Yamamuro > -- Deepak
Re: Registering custom UDAFs with HiveConetxt in SparkSQL, how?
I think it is not `sqlContext` but hiveContext because `create temporary function` is not supported in SQLContext. On Wed, Mar 25, 2015 at 5:58 AM, Jon Chase wrote: > Shahab - > > This should do the trick until Hao's changes are out: > > > sqlContext.sql("create temporary function foobar as > 'com.myco.FoobarUDAF'"); > > sqlContext.sql("select foobar(some_column) from some_table"); > > > This works without requiring to 'deploy' a JAR with the UDAF in it - just > make sure the UDAF is in your project's classpath. > > > > > On Tue, Mar 10, 2015 at 8:21 PM, Cheng, Hao wrote: > >> Oh, sorry, my bad, currently Spark SQL doesn’t provide the user >> interface for UDAF, but it can work seamlessly with Hive UDAF (via >> HiveContext). >> >> >> >> I am also working on the UDAF interface refactoring, after that we can >> provide the custom interface for extension. >> >> >> >> https://github.com/apache/spark/pull/3247 >> >> >> >> >> >> *From:* shahab [mailto:shahab.mok...@gmail.com] >> *Sent:* Wednesday, March 11, 2015 1:44 AM >> *To:* Cheng, Hao >> *Cc:* user@spark.apache.org >> *Subject:* Re: Registering custom UDAFs with HiveConetxt in SparkSQL, >> how? >> >> >> >> Thanks Hao, >> >> But my question concerns UDAF (user defined aggregation function ) not >> UDTF( user defined type function ). >> >> I appreciate if you could point me to some starting point on UDAF >> development in Spark. >> >> >> >> Thanks >> >> Shahab >> >> On Tuesday, March 10, 2015, Cheng, Hao wrote: >> >> Currently, Spark SQL doesn’t provide interface for developing the >> custom UDTF, but it can work seamless with Hive UDTF. >> >> >> >> I am working on the UDTF refactoring for Spark SQL, hopefully will >> provide an Hive independent UDTF soon after that. >> >> >> >> *From:* shahab [mailto:shahab.mok...@gmail.com] >> *Sent:* Tuesday, March 10, 2015 5:44 PM >> *To:* user@spark.apache.org >> *Subject:* Registering custom UDAFs with HiveConetxt in SparkSQL, how? >> >> >> >> Hi, >> >> >> >> I need o develop couple of UDAFs and use them in the SparkSQL. While UDFs >> can be registered as a function in HiveContext, I could not find any >> documentation of how UDAFs can be registered in the HiveContext?? so far >> what I have found is to make a JAR file, out of developed UDAF class, and >> then deploy the JAR file to SparkSQL . >> >> >> >> But is there any way to avoid deploying the jar file and register it >> programmatically? >> >> >> >> >> >> best, >> >> /Shahab >> >> > -- --- Takeshi Yamamuro
Re: Registering custom UDAFs with HiveConetxt in SparkSQL, how?
Shahab - This should do the trick until Hao's changes are out: sqlContext.sql("create temporary function foobar as 'com.myco.FoobarUDAF'"); sqlContext.sql("select foobar(some_column) from some_table"); This works without requiring to 'deploy' a JAR with the UDAF in it - just make sure the UDAF is in your project's classpath. On Tue, Mar 10, 2015 at 8:21 PM, Cheng, Hao wrote: > Oh, sorry, my bad, currently Spark SQL doesn't provide the user > interface for UDAF, but it can work seamlessly with Hive UDAF (via > HiveContext). > > > > I am also working on the UDAF interface refactoring, after that we can > provide the custom interface for extension. > > > > https://github.com/apache/spark/pull/3247 > > > > > > *From:* shahab [mailto:shahab.mok...@gmail.com] > *Sent:* Wednesday, March 11, 2015 1:44 AM > *To:* Cheng, Hao > *Cc:* user@spark.apache.org > *Subject:* Re: Registering custom UDAFs with HiveConetxt in SparkSQL, how? > > > > Thanks Hao, > > But my question concerns UDAF (user defined aggregation function ) not > UDTF( user defined type function ). > > I appreciate if you could point me to some starting point on UDAF > development in Spark. > > > > Thanks > > Shahab > > On Tuesday, March 10, 2015, Cheng, Hao wrote: > > Currently, Spark SQL doesn't provide interface for developing the custom > UDTF, but it can work seamless with Hive UDTF. > > > > I am working on the UDTF refactoring for Spark SQL, hopefully will provide > an Hive independent UDTF soon after that. > > > > *From:* shahab [mailto:shahab.mok...@gmail.com] > *Sent:* Tuesday, March 10, 2015 5:44 PM > *To:* user@spark.apache.org > *Subject:* Registering custom UDAFs with HiveConetxt in SparkSQL, how? > > > > Hi, > > > > I need o develop couple of UDAFs and use them in the SparkSQL. While UDFs > can be registered as a function in HiveContext, I could not find any > documentation of how UDAFs can be registered in the HiveContext?? so far > what I have found is to make a JAR file, out of developed UDAF class, and > then deploy the JAR file to SparkSQL . > > > > But is there any way to avoid deploying the jar file and register it > programmatically? > > > > > > best, > > /Shahab > >
RE: Registering custom UDAFs with HiveConetxt in SparkSQL, how?
Oh, sorry, my bad, currently Spark SQL doesn’t provide the user interface for UDAF, but it can work seamlessly with Hive UDAF (via HiveContext). I am also working on the UDAF interface refactoring, after that we can provide the custom interface for extension. https://github.com/apache/spark/pull/3247 From: shahab [mailto:shahab.mok...@gmail.com] Sent: Wednesday, March 11, 2015 1:44 AM To: Cheng, Hao Cc: user@spark.apache.org Subject: Re: Registering custom UDAFs with HiveConetxt in SparkSQL, how? Thanks Hao, But my question concerns UDAF (user defined aggregation function ) not UDTF( user defined type function ). I appreciate if you could point me to some starting point on UDAF development in Spark. Thanks Shahab On Tuesday, March 10, 2015, Cheng, Hao mailto:hao.ch...@intel.com>> wrote: Currently, Spark SQL doesn’t provide interface for developing the custom UDTF, but it can work seamless with Hive UDTF. I am working on the UDTF refactoring for Spark SQL, hopefully will provide an Hive independent UDTF soon after that. From: shahab [mailto:shahab.mok...@gmail.com] Sent: Tuesday, March 10, 2015 5:44 PM To: user@spark.apache.org Subject: Registering custom UDAFs with HiveConetxt in SparkSQL, how? Hi, I need o develop couple of UDAFs and use them in the SparkSQL. While UDFs can be registered as a function in HiveContext, I could not find any documentation of how UDAFs can be registered in the HiveContext?? so far what I have found is to make a JAR file, out of developed UDAF class, and then deploy the JAR file to SparkSQL . But is there any way to avoid deploying the jar file and register it programmatically? best, /Shahab
Re: Registering custom UDAFs with HiveConetxt in SparkSQL, how?
Thanks Hao, But my question concerns UDAF (user defined aggregation function ) not UDTF( user defined type function ). I appreciate if you could point me to some starting point on UDAF development in Spark. Thanks Shahab On Tuesday, March 10, 2015, Cheng, Hao wrote: > Currently, Spark SQL doesn’t provide interface for developing the custom > UDTF, but it can work seamless with Hive UDTF. > > > > I am working on the UDTF refactoring for Spark SQL, hopefully will provide > an Hive independent UDTF soon after that. > > > > *From:* shahab [mailto:shahab.mok...@gmail.com > ] > *Sent:* Tuesday, March 10, 2015 5:44 PM > *To:* user@spark.apache.org > > *Subject:* Registering custom UDAFs with HiveConetxt in SparkSQL, how? > > > > Hi, > > > > I need o develop couple of UDAFs and use them in the SparkSQL. While UDFs > can be registered as a function in HiveContext, I could not find any > documentation of how UDAFs can be registered in the HiveContext?? so far > what I have found is to make a JAR file, out of developed UDAF class, and > then deploy the JAR file to SparkSQL . > > > > But is there any way to avoid deploying the jar file and register it > programmatically? > > > > > > best, > > /Shahab >
RE: Registering custom UDAFs with HiveConetxt in SparkSQL, how?
Currently, Spark SQL doesn’t provide interface for developing the custom UDTF, but it can work seamless with Hive UDTF. I am working on the UDTF refactoring for Spark SQL, hopefully will provide an Hive independent UDTF soon after that. From: shahab [mailto:shahab.mok...@gmail.com] Sent: Tuesday, March 10, 2015 5:44 PM To: user@spark.apache.org Subject: Registering custom UDAFs with HiveConetxt in SparkSQL, how? Hi, I need o develop couple of UDAFs and use them in the SparkSQL. While UDFs can be registered as a function in HiveContext, I could not find any documentation of how UDAFs can be registered in the HiveContext?? so far what I have found is to make a JAR file, out of developed UDAF class, and then deploy the JAR file to SparkSQL . But is there any way to avoid deploying the jar file and register it programmatically? best, /Shahab