Alexander Shkapsky created SPARK-21332:
------------------------------------------
Summary: Incorrect result type inferred for some decimal
expressions
Key: SPARK-21332
URL: https://issues.apache.org/jira/browse/SPARK-21332
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.0.2
Reporter: Alexander Shkapsky
Decimal expressions do not always follow the type inference rules explained in
DecimalPrecision.scala. An incorrect result type is produced when the
expressions contains more than 2 decimals.
For example:
spark-sql> CREATE TABLE Decimals(decimal_26_6 DECIMAL(26,6));
...
spark-sql> describe decimals;
...
decimal_26_6 decimal(26,6) NULL
spark-sql> explain select decimal_26_6 * decimal_26_6 from decimals;
...
== Physical Plan ==
*Project [CheckOverflow((decimal_26_6#99 * decimal_26_6#99),
DecimalType(38,12)) AS (decimal_26_6 * decimal_26_6)#100]
+- HiveTableScan [decimal_26_6#99], MetastoreRelation default, decimals
However:
spark-sql> explain select decimal_26_6 * decimal_26_6 * decimal_26_6 from
decimals;
...
== Physical Plan ==
*Project [CheckOverflow((cast(CheckOverflow((decimal_26_6#104 *
decimal_26_6#104), DecimalType(38,12)) as decimal(26,6)) * decimal_26_6#104),
DecimalType(38,12)) AS ((decimal_26_6 * decimal_26_6) * decimal_26_6)#105]
+- HiveTableScan [decimal_26_6#104], MetastoreRelation default, decimals
The expected result type is DecimalType(38,18).
In Hive 1.1.0:
hive> explain select decimal_26_6 * decimal_26_6 from decimals;
OK
STAGE DEPENDENCIES:
Stage-0 is a root stage
STAGE PLANS:
Stage: Stage-0
Fetch Operator
limit: -1
Processor Tree:
TableScan
alias: decimals
Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats:
NONE
Select Operator
expressions: (decimal_26_6 * decimal_26_6) (type: decimal(38,12))
outputColumnNames: _col0
Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column
stats: NONE
ListSink
Time taken: 0.772 seconds, Fetched: 17 row(s)
hive> explain select decimal_26_6 * decimal_26_6 * decimal_26_6 from decimals;
OK
STAGE DEPENDENCIES:
Stage-0 is a root stage
STAGE PLANS:
Stage: Stage-0
Fetch Operator
limit: -1
Processor Tree:
TableScan
alias: decimals
Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column stats:
NONE
Select Operator
expressions: ((decimal_26_6 * decimal_26_6) * decimal_26_6) (type:
decimal(38,18))
outputColumnNames: _col0
Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column
stats: NONE
ListSink
Time taken: 0.064 seconds, Fetched: 17 row(s)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
