AngersZhuuuu commented on a change in pull request #34616: URL: https://github.com/apache/spark/pull/34616#discussion_r756540118
########## File path: docs/sql-migration-guide.md ########## @@ -338,6 +338,12 @@ license: | - In Spark 3.0, datetime pattern letter `F` is **aligned day of week in month** that represents the concept of the count of days within the period of a week where the weeks are aligned to the start of the month. In Spark version 2.4 and earlier, it is **week of month** that represents the concept of the count of weeks within the month where weeks start on a fixed day-of-week, e.g. `2020-07-30` is 30 days (4 weeks and 2 days) after the first day of the month, so `date_format(date '2020-07-30', 'F')` returns 2 in Spark 3.0, but as a week count in Spark 2.x, it returns 5 because it locates in the 5th week of July 2020, where week one is 2020-07-01 to 07-04. + - In Spark version 2.4 and below, `spark-sql` interface will chop `\` when query match `\;`. In Spark 3.0, `spark-sql` interface will not chop `\` when query match `\;`. See [HIVE-15297](https://issues.apache.org/jira/browse/HIVE-15297) for more details. For example: Review comment: The behavior of bin/spark-shell and bin/spark-sql is inconsisten for all the version ==, but saprk-shell's behavior is same for both spark2 and spark3 In spark-2.4.7 ``` spark-sql> explain extended select split('dawdawdawd','\\\\\\\\;'); == Parsed Logical Plan == 'Project [unresolvedalias('split(dawdawdawd, \\\;), None)] +- OneRowRelation == Analyzed Logical Plan == split(dawdawdawd, \\\;): array<string> Project [split(dawdawdawd, \\\;) AS split(dawdawdawd, \\\;)#2] +- OneRowRelation == Optimized Logical Plan == Project [[dawdawdawd] AS split(dawdawdawd, \\\;)#2] +- OneRowRelation == Physical Plan == *(1) Project [[dawdawdawd] AS split(dawdawdawd, \\\;)#2] +- Scan OneRowRelation[] Time taken: 1.613 seconds, Fetched 1 row(s) ``` ``` Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 2.4.7-sdi-015 /_/ Using Scala version 2.11.12 (OpenJDK 64-Bit Server VM, Java 1.8.0_232) Type in expressions to have them evaluated. Type :help for more information. scala> spark.sql("select split('dawdawdawd','\\\\\\\\;')").explain(true) == Parsed Logical Plan == 'Project [unresolvedalias('split(dawdawdawd, \\;), None)] +- OneRowRelation == Analyzed Logical Plan == split(dawdawdawd, \\;): array<string> Project [split(dawdawdawd, \\;) AS split(dawdawdawd, \\;)#3] +- OneRowRelation == Optimized Logical Plan == Project [[dawdawdawd] AS split(dawdawdawd, \\;)#3] +- OneRowRelation == Physical Plan == *(1) Project [[dawdawdawd] AS split(dawdawdawd, \\;)#3] +- Scan OneRowRelation[] ``` In spark-3.1.2 ``` spark-sql> explain extended select split('dawdawdawd','\\\\\\\\;'); == Parsed Logical Plan == 'Project [unresolvedalias('split(dawdawdawd, \\\\;), None)] +- OneRowRelation == Analyzed Logical Plan == split(dawdawdawd, \\\\;, -1): array<string> Project [split(dawdawdawd, \\\\;, -1) AS split(dawdawdawd, \\\\;, -1)#5] +- OneRowRelation == Optimized Logical Plan == Project [[dawdawdawd] AS split(dawdawdawd, \\\\;, -1)#5] +- OneRowRelation == Physical Plan == *(1) Project [[dawdawdawd] AS split(dawdawdawd, \\\\;, -1)#5] +- *(1) Scan OneRowRelation[] Time taken: 2.468 seconds, Fetched 1 row(s) spark-sql> ``` ``` Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 3.1.2-sdi-011 /_/ Using Scala version 2.12.10 (OpenJDK 64-Bit Server VM, Java 1.8.0_232) Type in expressions to have them evaluated. Type :help for more information. scala> spark.sql("select split('dawdawdawd','\\\\\\\\;')").explain(true) == Parsed Logical Plan == 'Project [unresolvedalias('split(dawdawdawd, \\;), None)] +- OneRowRelation == Analyzed Logical Plan == split(dawdawdawd, \\;, -1): array<string> Project [split(dawdawdawd, \\;, -1) AS split(dawdawdawd, \\;, -1)#0] +- OneRowRelation == Optimized Logical Plan == Project [[dawdawdawd] AS split(dawdawdawd, \\;, -1)#0] +- OneRowRelation == Physical Plan == *(1) Project [[dawdawdawd] AS split(dawdawdawd, \\;, -1)#0] +- *(1) Scan OneRowRelation[] ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
