Re: write and call UDF in spark dataframe

[Kabeer Ahmed]

Divya:

https://databricks.com/blog/2015/09/16/spark-1-5-dataframe-api-highlights-datetimestring-handling-time-intervals-and-udafs.html

The link gives a complete example of registering a udAf - user defined 
aggregate function. This is a complete example and this example should give you 
a complete idea of registering a UDF. If you still need a hand let us know.

Thanks
Kabeer.

Sent from Nylas 
N1,
 the extensible, open source mail client.

On Jul 21 2016, at 8:13 am, Jacek Laskowski  wrote:

On Thu, Jul 21, 2016 at 5:53 AM, Mich Talebzadeh
 wrote:
> something similar

Is this going to be in Scala?

> def ChangeToDate (word : String) : Date = {
> //return
> TO_DATE(FROM_UNIXTIME(UNIX_TIMESTAMP(word,"dd/MM/"),"-MM-dd"))
> val d1 = Date.valueOf(ReverseDate(word))
> return d1
> }
> sqlContext.udf.register("ChangeToDate", ChangeToDate(_:String))

then...please use lowercase method names and *no* return please ;-)

BTW, no sqlContext as of Spark 2.0. Sorry.../me smiling nicely

Jacek

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: write and call UDF in spark dataframe

[Jacek Laskowski]

On Thu, Jul 21, 2016 at 5:53 AM, Mich Talebzadeh
 wrote:
> something similar

Is this going to be in Scala?

> def ChangeToDate (word : String) : Date = {
>   //return
> TO_DATE(FROM_UNIXTIME(UNIX_TIMESTAMP(word,"dd/MM/"),"-MM-dd"))
>   val d1 = Date.valueOf(ReverseDate(word))
>   return d1
> }
> sqlContext.udf.register("ChangeToDate", ChangeToDate(_:String))

then...please use lowercase method names and *no* return please ;-)

BTW, no sqlContext as of Spark 2.0. Sorry.../me smiling nicely

Jacek

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: write and call UDF in spark dataframe

[Jacek Laskowski]

On Thu, Jul 21, 2016 at 4:53 AM, Divya Gehlot  wrote:

> To be very specific I am looking for UDFs syntax for example which takes
> String as parameter and returns integer .. how do we define the return type

val f: String => Int = ???
val myUDF = udf(f)

or

val myUDF = udf[String, Int] { ??? }

or

val myUDF = udf { (s: String) => ??? }

Jacek

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: write and call UDF in spark dataframe

[Jacek Laskowski]

On Wed, Jul 20, 2016 at 1:22 PM, Rishabh Bhardwaj  wrote:

> val new_df = df.select(from_unixtime($"time").as("newtime"))

or better yet using tick (less typing and more prose than code :))

df.select(from_unixtime('time) as "newtime")

Jacek

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Re: write and call UDF in spark dataframe

[Mich Talebzadeh]

something similar

def ChangeToDate (word : String) : Date = {
  //return
TO_DATE(FROM_UNIXTIME(UNIX_TIMESTAMP(word,"dd/MM/"),"-MM-dd"))
  val d1 = Date.valueOf(ReverseDate(word))
  return d1
}
sqlContext.udf.register("ChangeToDate", ChangeToDate(_:String))

Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
<https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*



http://talebzadehmich.wordpress.com


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.



On 21 July 2016 at 03:53, Divya Gehlot  wrote:

> Hi ,
> To be very specific I am looking for UDFs syntax for example which takes
> String as parameter and returns integer .. how do we define the return type
> .
>
>
> Thanks,
>
> Divya
>
> On 21 July 2016 at 00:24, Andy Davidson 
> wrote:
>
>> Hi Divya
>>
>> In general you will get better performance if you can minimize your use
>> of UDFs. Spark 2.0/ tungsten does a lot of code generation. It will have to
>> treat your UDF as a block box.
>>
>> Andy
>>
>> From: Rishabh Bhardwaj 
>> Date: Wednesday, July 20, 2016 at 4:22 AM
>> To: Rabin Banerjee 
>> Cc: Divya Gehlot , "user @spark" <
>> user@spark.apache.org>
>> Subject: Re: write and call UDF in spark dataframe
>>
>> Hi Divya,
>>
>> There is already "from_unixtime" exists in
>> org.apache.spark.sql.frunctions,
>> Rabin has used that in the sql query,if you want to use it in
>> dataframe DSL you can try like this,
>>
>> val new_df = df.select(from_unixtime($"time").as("newtime"))
>>
>>
>> Thanks,
>> Rishabh.
>>
>> On Wed, Jul 20, 2016 at 4:21 PM, Rabin Banerjee <
>> dev.rabin.baner...@gmail.com> wrote:
>>
>>> Hi Divya ,
>>>
>>> Try,
>>>
>>> val df = sqlContext.sql("select from_unixtime(ts,'-MM-dd') as `ts` from 
>>> mr")
>>>
>>> Regards,
>>> Rabin
>>>
>>> On Wed, Jul 20, 2016 at 12:44 PM, Divya Gehlot 
>>> wrote:
>>>
>>>> Hi,
>>>> Could somebody share example of writing and calling udf which converts
>>>> unix tme stamp to date tiime .
>>>>
>>>>
>>>> Thanks,
>>>> Divya
>>>>
>>>
>>>
>>
>

Re: write and call UDF in spark dataframe

[Divya Gehlot]

Hi ,
To be very specific I am looking for UDFs syntax for example which takes
String as parameter and returns integer .. how do we define the return type
.


Thanks,

Divya

On 21 July 2016 at 00:24, Andy Davidson 
wrote:

> Hi Divya
>
> In general you will get better performance if you can minimize your use of
> UDFs. Spark 2.0/ tungsten does a lot of code generation. It will have to
> treat your UDF as a block box.
>
> Andy
>
> From: Rishabh Bhardwaj 
> Date: Wednesday, July 20, 2016 at 4:22 AM
> To: Rabin Banerjee 
> Cc: Divya Gehlot , "user @spark" <
> user@spark.apache.org>
> Subject: Re: write and call UDF in spark dataframe
>
> Hi Divya,
>
> There is already "from_unixtime" exists in org.apache.spark.sql.frunctions,
> Rabin has used that in the sql query,if you want to use it in
> dataframe DSL you can try like this,
>
> val new_df = df.select(from_unixtime($"time").as("newtime"))
>
>
> Thanks,
> Rishabh.
>
> On Wed, Jul 20, 2016 at 4:21 PM, Rabin Banerjee <
> dev.rabin.baner...@gmail.com> wrote:
>
>> Hi Divya ,
>>
>> Try,
>>
>> val df = sqlContext.sql("select from_unixtime(ts,'-MM-dd') as `ts` from 
>> mr")
>>
>> Regards,
>> Rabin
>>
>> On Wed, Jul 20, 2016 at 12:44 PM, Divya Gehlot 
>> wrote:
>>
>>> Hi,
>>> Could somebody share example of writing and calling udf which converts
>>> unix tme stamp to date tiime .
>>>
>>>
>>> Thanks,
>>> Divya
>>>
>>
>>
>

Re: write and call UDF in spark dataframe

[Andy Davidson]

Hi Divya

In general you will get better performance if you can minimize your use of
UDFs. Spark 2.0/ tungsten does a lot of code generation. It will have to
treat your UDF as a block box.

Andy

From:  Rishabh Bhardwaj 
Date:  Wednesday, July 20, 2016 at 4:22 AM
To:  Rabin Banerjee 
Cc:  Divya Gehlot , "user @spark"

Subject:  Re: write and call UDF in spark dataframe

> Hi Divya,
> 
> There is already "from_unixtime" exists in org.apache.spark.sql.frunctions,
> Rabin has used that in the sql query,if you want to use it in dataframe DSL
> you can try like this,
> 
>> val new_df = df.select(from_unixtime($"time").as("newtime"))
> 
> Thanks,
> Rishabh.
> 
> On Wed, Jul 20, 2016 at 4:21 PM, Rabin Banerjee 
> wrote:
>> Hi Divya ,
>> 
>> Try,
>> 
>> val df = sqlContext.sql("select from_unixtime(ts,'-MM-dd') as `ts` from
>> mr")
>> Regards,
>> Rabin
>> 
>> On Wed, Jul 20, 2016 at 12:44 PM, Divya Gehlot 
>> wrote:
>>> Hi,
>>> Could somebody share example of writing and calling udf which converts unix
>>> tme stamp to date tiime .
>>> 
>>> 
>>> Thanks,
>>> Divya 
>> 
> 

Re: write and call UDF in spark dataframe

[Mich Talebzadeh]

yep something in line of

val df = sqlContext.sql("SELECT FROM_unixtime(unix_timestamp(), 'dd/MM/
HH:mm:ss.ss') as time ")

Note that this does not require a column from an already existing table.

HTH

Dr Mich Talebzadeh



LinkedIn * 
https://www.linkedin.com/profile/view?id=AAEWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
*



http://talebzadehmich.wordpress.com


*Disclaimer:* Use it at your own risk. Any and all responsibility for any
loss, damage or destruction of data or any other property which may arise
from relying on this email's technical content is explicitly disclaimed.
The author will in no case be liable for any monetary damages arising from
such loss, damage or destruction.



On 20 July 2016 at 12:22, Rishabh Bhardwaj  wrote:

> Hi Divya,
>
> There is already "from_unixtime" exists in org.apache.spark.sql.frunctions,
> Rabin has used that in the sql query,if you want to use it in
> dataframe DSL you can try like this,
>
> val new_df = df.select(from_unixtime($"time").as("newtime"))
>
>
> Thanks,
> Rishabh.
>
> On Wed, Jul 20, 2016 at 4:21 PM, Rabin Banerjee <
> dev.rabin.baner...@gmail.com> wrote:
>
>> Hi Divya ,
>>
>> Try,
>>
>> val df = sqlContext.sql("select from_unixtime(ts,'-MM-dd') as `ts` from 
>> mr")
>>
>> Regards,
>> Rabin
>>
>> On Wed, Jul 20, 2016 at 12:44 PM, Divya Gehlot 
>> wrote:
>>
>>> Hi,
>>> Could somebody share example of writing and calling udf which converts
>>> unix tme stamp to date tiime .
>>>
>>>
>>> Thanks,
>>> Divya
>>>
>>
>>
>

Re: write and call UDF in spark dataframe

[Rishabh Bhardwaj]

Hi Divya,

There is already "from_unixtime" exists in org.apache.spark.sql.frunctions,
Rabin has used that in the sql query,if you want to use it in dataframe DSL
you can try like this,

val new_df = df.select(from_unixtime($"time").as("newtime"))


Thanks,
Rishabh.

On Wed, Jul 20, 2016 at 4:21 PM, Rabin Banerjee <
dev.rabin.baner...@gmail.com> wrote:

> Hi Divya ,
>
> Try,
>
> val df = sqlContext.sql("select from_unixtime(ts,'-MM-dd') as `ts` from 
> mr")
>
> Regards,
> Rabin
>
> On Wed, Jul 20, 2016 at 12:44 PM, Divya Gehlot 
> wrote:
>
>> Hi,
>> Could somebody share example of writing and calling udf which converts
>> unix tme stamp to date tiime .
>>
>>
>> Thanks,
>> Divya
>>
>
>

Re: write and call UDF in spark dataframe

[Rabin Banerjee]

Hi Divya ,

Try,

val df = sqlContext.sql("select from_unixtime(ts,'-MM-dd') as `ts` from mr")

Regards,
Rabin
On Wed, Jul 20, 2016 at 12:44 PM, Divya Gehlot 
wrote:

> Hi,
> Could somebody share example of writing and calling udf which converts
> unix tme stamp to date tiime .
>
>
> Thanks,
> Divya
>

write and call UDF in spark dataframe

[Divya Gehlot]

Hi,
Could somebody share example of writing and calling udf which converts unix
tme stamp to date tiime .


Thanks,
Divya