xsys created SPARK-40616:
----------------------------
Summary: Loss of precision using SparkSQL shell on high-precision
DECIMAL types
Key: SPARK-40616
URL: https://issues.apache.org/jira/browse/SPARK-40616
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.2.1
Reporter: xsys
h3. Describe the bug
We are trying to save {{DECIMAL}} values with high precision in a table using
the SparkSQL shell.
When we {{INSERT}} decimal values with precision higher than the standard
double precision.
This seems to be caused by type inference at shell parsing inferring that the
value is a double type.
h3. To reproduce
On Spark 3.2.1 (commit {{{}4f25b3f712{}}}), using {{{}spark-sql{}}}:
{code:java}
$SPARK_HOME/bin/spark-sql{code}
In the shell:
{code:java}
CREATE TABLE t(c0 DECIMAL(20,10));
INSERT INTO t VALUES (8.8888888888888888888e9);
SELECT * FROM t;{code}
Executing the above gives this:
{code:java}
spark-sql> CREATE TABLE t(c0 DECIMAL(20,10));
22/09/29 11:28:41 WARN ResolveSessionCatalog: A Hive serde table will be
created as there is no table provider specified. You can set
spark.sql.legacy.createHiveTableByDefault to false so that native data source
table will be created instead.
Time taken: 0.118 seconds
spark-sql> INSERT INTO t VALUES (8.8888888888888888888e9);
Time taken: 0.392 seconds
spark-sql> SELECT * FROM t;
8888888888.8888900000
Time taken: 0.197 seconds, Fetched 1 row(s){code}
h3. Expected behavior
We expect the inserted value to retain the precision as determined by the
parameters for the {{DECIMAL}} type. For example, we expect the example above
to return {{{}8888888888.8888888888{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
