[ https://issues.apache.org/jira/browse/SPARK-22364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16220483#comment-16220483 ]
Sean Owen commented on SPARK-22364: ----------------------------------- Is the problem that this hour doesn't exist in the timezone because it's the hour that is skipped when the clocks move forwards? See https://stackoverflow.com/questions/14201469/java-date-and-daylight-saving which actually talks about the exact same date. > unix_timestamp function sets valid dates to null > ------------------------------------------------ > > Key: SPARK-22364 > URL: https://issues.apache.org/jira/browse/SPARK-22364 > Project: Spark > Issue Type: Bug > Components: SQL > Affects Versions: 2.2.0 > Environment: Windows 10, United Kingdom > Reporter: Matthew Sinton-Hewitt > > org.apache.spark.sql.functions.unix_timestamp sets some valid dates to null. > The dates happen to be the at the start of Daylight Savings time (UK and > possibly elsewhere). > {code:java} > val spark = SparkSession.builder.getOrCreate() > import spark.implicits._ > spark.sparkContext.parallelize( > Seq("25/03/2012 00:59", "25/03/2012 01:00", "25/03/2012 01:59", > "25/03/2012 02:01")) > .toDF("date") > .select(unix_timestamp($"date", "dd/MM/yyyy HH:mm")) > .show(false) > // results: > // 1332637140, null, null, 1332637260 > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org