[ https://issues.apache.org/jira/browse/SPARK-39536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-39536. ---------------------------------- Resolution: Invalid > to_date function is returning incorrect value > --------------------------------------------- > > Key: SPARK-39536 > URL: https://issues.apache.org/jira/browse/SPARK-39536 > Project: Spark > Issue Type: Bug > Components: PySpark > Affects Versions: 3.2.1 > Environment: I'm facing this issue in databricks community edition. > I'm using DBR 10.4 LTS. > Reporter: Sridhar Varanasi > Priority: Major > Attachments: to_date_issue.PNG > > > Hi, > > I have a dataframe which has a column containing dates in string format. Now > while converting this to date type using to_date , it's giving incorrect date > format values. Following is the example code. > > > df = spark.createDataFrame( > [("11/25/1991",), ("1/2/1991",), ("11/30/1991",)], > ['date_str'] > ) > > spark.sql("set spark.sql.legacy.timeParserPolicy=LEGACY") > > df = (df > .withColumn('new_date' > ,to_date(col('date_str'),'mm/dd/yyyy'))) > display(df) > > > In the above dataframe we get the date converted correctly for the 2nd row > but for 1st and 3rd row we are getting incorrect dates post conversion. > > > Could you please look into this issue? > > Thanks, > Sridhar -- This message was sent by Atlassian Jira (v8.20.7#820007) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org