dilipbiswal commented on a change in pull request #25331: [SPARK-27768][SQL]
Support Infinity/NaN-related float/double literals case-insensitively
URL: https://github.com/apache/spark/pull/25331#discussion_r313221044
##########
File path: docs/sql-migration-guide-upgrade.md
##########
@@ -161,6 +161,95 @@ license: |
- Since Spark 3.0, Dataset query fails if it contains ambiguous column
reference that is caused by self join. A typical example: `val df1 = ...; val
df2 = df1.filter(...);`, then `df1.join(df2, df1("a") > df2("a"))` returns an
empty result which is quite confusing. This is because Spark cannot resolve
Dataset column references that point to tables being self joined, and
`df1("a")` is exactly the same as `df2("a")` in Spark. To restore the behavior
before Spark 3.0, you can set `spark.sql.analyzer.failAmbiguousSelfJoin` to
`false`.
+ - Since Spark 3.0, `Cast` function processes string literals such as
'Infinity', '+Infinity', '-Infinity', 'NaN', 'Inf', '+Inf', '-Inf' in case
insensitive manner when casting the literals to `Double` or `Float` type to
ensure greater compatibility with other database systems. This behaviour change
is illustrated in the table below:
+ <table class="table">
+ <tr>
+ <th>
+ <b>Operation</b>
+ </th>
+ <th>
+ <b>Result prior to spark 3.0</b>
+ </th>
+ <th>
+ <b>Result starting spark 3.0</b>
+ </th>
+ </tr>
+ <tr>
+ <td>
+ CAST('infinity' TO DOUBLE)<br>
+ CAST('+infinity' TO DOUBLE)<br>
+ CAST('inf' TO DOUBLE)<br>
+ CAST('+inf' TO DOUBLE)<br>
+ </td>
+ <td>
+ NULL
+ </td>
+ <td>
+ Double.PositiveInfinity
+ </td>
+ </tr>
+ <tr>
+ <td>
+ CAST('-infinity' TO DOUBLE)<br>
+ CAST('-inf' TO DOUBLE)<br>
+ </td>
+ <td>
+ NULL
+ </td>
+ <td>
+ Double.NegativeInfinity
+ </td>
+ </tr>
+ <tr>
+ <td>
+ CAST('infinity' TO FLOAT)<br>
+ CAST('+infinity' TO FLOAT)<br>
+ CAST('inf' TO FLOAT)<br>
+ CAST('+inf' TO FLOAT)<br>
+ </td>
+ <td>
+ NULL
+ </td>
+ <td>
+ Float.PositiveInfinity
+ </td>
+ </tr>
+ <tr>
+ <td>
+ CAST('-infinity' TO FLOAT)<br>
+ CAST('-inf' TO FLOAT)<br>
+ </td>
+ <td>
+ NULL
+ </td>
+ <td>
+ Float.NegativeInfinity
+ </td>
+ </tr>
+ <tr>
+ <td>
+ CAST('nan' TO DOUBLE)
+ </td>
+ <td>
+ NULL
+ </td>
+ <td>
+ Double.NaN
+ </td>
+ </tr>
+ <tr>
+ <td>
+ CAST('nan' TO FLOAT)
Review comment:
@dongjoon-hyun oops...thanks.. fixed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
