You can invoke exactly the same functions on scala side as well i.e.
http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.sql.functions$
Have you tried them?
On Thu, Mar 24, 2016 at 10:29 PM, Mich Talebzadeh <mich.talebza...@gmail.com
> wrote:
>
> Hi,
>
> Read a CSV in with the following schema
>
> scala> df.printSchema
> root
> |-- Invoice Number: string (nullable = true)
> |-- Payment date: string (nullable = true)
> |-- Net: string (nullable = true)
> |-- VAT: string (nullable = true)
> |-- Total: string (nullable = true)
>
> I use mapping as below
>
> case class Invoices(Invoicenumber: String, Paymentdate: String, Net:
> Double, VAT: Double, Total: Double)
>
> val a = df.filter(col("Total") > "").map(p => Invoices(p(0).toString,
> p(1).toString, p(2).toString.substring(1).replace(",", "").toDouble,
> p(3).toString.substring(1).replace(",", "").toDouble,
> p(4).toString.substring(1).replace(",", "").toDouble))
>
>
> I want to convert p(1).toString to datetime like below when I used in sql
>
> TO_DATE(FROM_UNIXTIME(UNIX_TIMESTAMP(paymentdate,'dd/MM/yyyy'),'yyyy-MM-dd'))
> AS paymentdate
>
>
> Thanks
>
>
> Dr Mich Talebzadeh
>
>
>
> LinkedIn *
> https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>*
>
>
>
> http://talebzadehmich.wordpress.com
>
>
>
--
Regards,
Alexander
