xsystem2022 opened a new pull request, #38230: URL: https://github.com/apache/spark/pull/38230
### **What changes were proposed in this pull request?** Add a clear warning in spark SQL so that users are aware that a decimal containing an exponent will be converted to double, which may lead to a loss of precision. The warning message: `In Spark 3.0, numbers written in scientific notation would be parsed as Double. To restore the behavior before Spark 3.0, you can set spark.sql.legacy.exponentLiteralAsDecimal.enabled to true.` For example, when we want to insert `8.8888888888888888888e9` to a `DECIMAL(20,10)` column of a table, we will be able to see the following warning: ```sql spark-sql> CREATE TABLE t(c0 DECIMAL(20,10)); spark-sql> INSERT INTO t VALUES (8.8888888888888888888e9); 22/10/07 15:36:04 WARN SparkSqlAstBuilder: In Spark 3.0, numbers written in scientific notation would be parsed as Double. To restore the behavior before Spark 3.0, you can set spark.sql.legacy.exponentLiteralAsDecimal.enabled to true. spark-sql> select * from t; 8888888888.8888900000 ``` The warning will not show up for DECIMAL values without exponent. ### **Why are the changes needed?** Users may not be aware that numbers written in scientific notation (for example, `1E2`) would be parsed as Double during insertion. Adding a warning to let them be aware of potential precision loss and options of setting one configuration to restore the behavior before Spark 3.0. ### **Does this PR introduce any user-facing change?** Yes, it adds a warning. ### **How was this patch tested?** There are no new tests added. All existing tests pass. This change was tested by running the Spark SQL shell with decimal values with and without values containing exponent (For example, see "What changes were proposed in this pull request? "). We are happy to also add a unit test if it is necessary. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
