I’m not entirely sure if this is what you’re asking, but you could just use the
datediff function:
val df2 = df.withColumn("ID”, datediff($"end", $"start”))
If you want it formatted as {n}D then:
val df2 = df.withColumn("ID", concat(datediff($"end", $"start"),lit("D")))
Thanks,
Silvio
From: Divya Gehlot <[email protected]<mailto:[email protected]>>
Date: Sunday, March 20, 2016 at 11:42 PM
To: "user @spark" <[email protected]<mailto:[email protected]>>
Subject: Get the number of days dynamically in with Column
I have a time stamping table which has data like
No of Days ID
1 1D
2 2D
and so on till 30 days
Have another Dataframe with
start date and end date
I need to get the difference between these two days and get the ID from Time
Stamping table and do With Column .
The tedious solution is
val dfTimeStamping = df.withColumn("ID",when(Diff between Start date and
Enddate ,"1D").when(Diff between Start date and Enddate ,"2D")).. have to do
till 30 days .
How can I do it dynamically ?
Thanks,
Divya
