[
https://issues.apache.org/jira/browse/SPARK-26535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun resolved SPARK-26535.
-----------------------------------
Resolution: Won't Do
Hi, [~mgaido].
First of all, Hive starts to use `Decimal` by default. Also, this introduces
TPCDS-H query result difference between Spark versions. We cannot do this.
{code}
hive> select version();
OK
3.1.1 rf4e0529634b6231a0072295da48af466cf2f10b7
Time taken: 0.089 seconds, Fetched: 1 row(s)
hive> explain select 2.3;
OK
STAGE DEPENDENCIES:
Stage-0 is a root stage
STAGE PLANS:
Stage: Stage-0
Fetch Operator
limit: -1
Processor Tree:
TableScan
alias: _dummy_table
Row Limit Per Split: 1
Statistics: Num rows: 1 Data size: 10 Basic stats: COMPLETE Column
stats: COMPLETE
Select Operator
expressions: 2.3 (type: decimal(2,1))
outputColumnNames: _col0
Statistics: Num rows: 1 Data size: 112 Basic stats: COMPLETE Column
stats: COMPLETE
ListSink
{code}
> Parsing literals as DOUBLE instead of DECIMAL
> ---------------------------------------------
>
> Key: SPARK-26535
> URL: https://issues.apache.org/jira/browse/SPARK-26535
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: Marco Gaido
> Priority: Major
>
> As pointed out by [~dkbiswal]'s comment
> https://github.com/apache/spark/pull/22450#issuecomment-423082389, most of
> other RDBMS (DB2, Presto, Hive, MSSQL) consider literals as DOUBLE by default.
> Spark as of now consider them as DECIMAL. This is quite problematic
> especially in relation with the operations on decimal, for which we base our
> implementation on Hive/MSSQL.
> So this ticket is for moving by default the resolution of literals as DOUBLE,
> but with a config which allows to get back to the previous behavior.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
