Hello,
I have a dataframe, apply from_unixtime seems to expose an anomaly:
*scala> val bhDF4 = bhDF.withColumn("ts1", $"ts" + 28800).withColumn("ts2",
from_unixtime($"ts" + 28800,"YYYYMMddhhmmss"))*
*bhDF4: org.apache.spark.sql.DataFrame = [user_id: int, item_id: int ... 5
more fields]*
*scala> bhDF4.show*
*+-------+-------+-------+--------+----------+----------+--------------+*
*|user_id|item_id| cat_id|behavior| ts| ts1| ts2|*
*+-------+-------+-------+--------+----------+----------+--------------+*
*| 1|2268318|2520377| pv|1511544070|1511572870|20171124082110|*
*| 1|2333346|2520771| pv|1511561733|1511590533|20171125011533|*
*| 1|2576651| 149192| pv|1511572885|1511601685|20171125042125|*
*| 1|3830808|4181361| pv|1511593493|1511622293|20171125100453|*
*| 1|4365585|2520377| pv|1511596146|1511624946|20171125104906|*
*| 1|4606018|2735466| pv|1511616481|1511645281|20171125042801|*
*| 1| 230380| 411153| pv|1511644942|1511673742|20171126122222|*
*| 1|3827899|2920476| pv|1511713473|1511742273|20171126072433|*
*| 1|3745169|2891509| pv|1511725471|1511754271|20171126104431|*
*| 1|1531036|2920476| pv|1511733732|1511762532|20171127010212|*
*| 1|2266567|4145813| pv|1511741471|1511770271|20171127031111|*
*| 1|2951368|1080785| pv|1511750828|1511779628|20171127054708|*
*| 1|3108797|2355072| pv|1511758881|1511787681|20171127080121|*
*| 1|1338525| 149192| pv|1511773214|1511802014|20171127120014|*
*| 1|2286574|2465336| pv|1511797167|1511825967|20171127063927|*
*| 1|5002615|2520377| pv|1511839385|1511868185|20171128062305|*
*| 1|2734026|4145813| pv|1511842184|1511870984|20171128070944|*
*| 1|5002615|2520377| pv|1511844273|1511873073|20171128074433|*
*| 1|3239041|2355072| pv|1511855664|1511884464|20171128105424|*
*| 1|4615417|4145813| pv|1511870864|1511899664|20171128030744|*
*+-------+-------+-------+--------+----------+----------+--------------+*
*only showing top 20 rows------------------------------------------------*
*All ts2 are supposed to show date after 20171125 while there seems to be
at least one anomaly showing 20171124*
*Any thought?*
*Sincerely yours,*
*Raymond*
