[GitHub] [spark] HyukjinKwon edited a comment on pull request #28445: [SPARK-31212][SQL][2.4] Fix Failure of casting the '1000-02-29' string to the date type
HyukjinKwon edited a comment on pull request #28445: URL: https://github.com/apache/spark/pull/28445#issuecomment-624455200 Okay, I had some time to investigate the related items here. I think there are some more places to fix, e.g. [here](https://github.com/apache/spark/blob/branch-2.4/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala#L610). Let's don't fix these in branch-2.4 only because: - These were "fixed" in the master branch by switching the calendar. I think we should take this fix was superseded by it in the master. We can't backport it to branch-2.4 because it's too much breaking change, though. - It's pretty risky. Given the history of fixing it in the master, fixing one caused a bunch of other related issues. These new issues have been fixed and found over one year, and it's still in progress. I would like to avoid to keep fixing by completely different approaches in 2.4 compared to 3.0. - It will causes a bunch of behaviour changes. Yes, these will be bug fixes but I tend to be more conservative. See also [Bug compatibility](https://en.wikipedia.org/wiki/Bug_compatibility). Considering these risks, let's don't land these fixes. We can more conservatively just document these in Spark 2.4 specifically given the potential maintenance overhead and technical difficulty, if we should. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] [spark] HyukjinKwon edited a comment on pull request #28445: [SPARK-31212][SQL][2.4] Fix Failure of casting the '1000-02-29' string to the date type
HyukjinKwon edited a comment on pull request #28445: URL: https://github.com/apache/spark/pull/28445#issuecomment-624455200 Okay, I had some time to investigate the related items here. I think there are some more places to fix, e.g. [here](https://github.com/apache/spark/blob/branch-2.4/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala#L610). Let's don't fix these in branch-2.4 only because: - These were "fixed" in the master branch by switching the calendar. I think we should take this fix was superseded by it in the master. We can't backport it to branch-2.4 because it's too much breaking change, though. - It's pretty risky. Given the history of fixing it in the master, fixing one caused a bunch of other related issues. These new issues have been fixed and found over one year, and it's still in progress. I would like to avoid to keep fixing by completely different approaches in 2.4 compared to 3.0. - It will causes a bunch of behaviour changes. Yes, these will be bug fixes but I tend to be more conservative. See also [Bug compatibility](https://en.wikipedia.org/wiki/Bug_compatibility). Considering these risks, let's don't land these fixes. We can more conservatively just document these in Spark 2.4 specifically given the potential maintenance overhead and technical difficulty. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
