GitHub user aokolnychyi opened a pull request:
https://github.com/apache/spark/pull/18583
[SPARK-21332][SQL] Incorrect result type inferred for some decimal
expressions
## What changes were proposed in this pull request?
This PR changes the direction of expression transformation in the
DecimalPrecision rule. Previously, the expressions were transformed down, which
led to incorrect result types when decimal expressions had other decimal
expressions as their operands. The root cause of this issue was in visiting
outer nodes before their children. Consider the example below:
```
val inputSchema = StructType(StructField("col", DecimalType(26, 6)) ::
Nil)
val sc = spark.sparkContext
val rdd = sc.parallelize(1 to 2).map(_ => Row(BigDecimal(12)))
val df = spark.createDataFrame(rdd, inputSchema)
// Works correctly since no nested decimal expression is involved
// Expected result type: (26, 6) * (26, 6) = (38, 12)
df.select($"col" * $"col").explain(true)
df.select($"col" * $"col").printSchema()
// Gives a wrong result since there is a nested decimal expression that
should be visited first
// Expected result type: ((26, 6) * (26, 6)) * (26, 6) = (38, 12) *
(26, 6) = (38, 18)
df.select($"col" * $"col" * $"col").explain(true)
df.select($"col" * $"col" * $"col").printSchema()
```
The example above gives the following output:
```
// Correct result without sub-expressions
== Parsed Logical Plan ==
'Project [('col * 'col) AS (col * col)#4]
+- LogicalRDD [col#1]
== Analyzed Logical Plan ==
(col * col): decimal(38,12)
Project [CheckOverflow((promote_precision(cast(col#1 as decimal(26,6))) *
promote_precision(cast(col#1 as decimal(26,6)))), DecimalType(38,12)) AS (col *
col)#4]
+- LogicalRDD [col#1]
== Optimized Logical Plan ==
Project [CheckOverflow((col#1 * col#1), DecimalType(38,12)) AS (col *
col)#4]
+- LogicalRDD [col#1]
== Physical Plan ==
*Project [CheckOverflow((col#1 * col#1), DecimalType(38,12)) AS (col *
col)#4]
+- Scan ExistingRDD[col#1]
// Schema
root
|-- (col * col): decimal(38,12) (nullable = true)
// Incorrect result with sub-expressions
== Parsed Logical Plan ==
'Project [(('col * 'col) * 'col) AS ((col * col) * col)#11]
+- LogicalRDD [col#1]
== Analyzed Logical Plan ==
((col * col) * col): decimal(38,12)
Project
[CheckOverflow((promote_precision(cast(CheckOverflow((promote_precision(cast(col#1
as decimal(26,6))) * promote_precision(cast(col#1 as decimal(26,6)))),
DecimalType(38,12)) as decimal(26,6))) * promote_precision(cast(col#1 as
decimal(26,6)))), DecimalType(38,12)) AS ((col * col) * col)#11]
+- LogicalRDD [col#1]
== Optimized Logical Plan ==
Project [CheckOverflow((cast(CheckOverflow((col#1 * col#1),
DecimalType(38,12)) as decimal(26,6)) * col#1), DecimalType(38,12)) AS ((col *
col) * col)#11]
+- LogicalRDD [col#1]
== Physical Plan ==
*Project [CheckOverflow((cast(CheckOverflow((col#1 * col#1),
DecimalType(38,12)) as decimal(26,6)) * col#1), DecimalType(38,12)) AS ((col *
col) * col)#11]
+- Scan ExistingRDD[col#1]
// Schema
root
|-- ((col * col) * col): decimal(38,12) (nullable = true)
```
## How was this patch tested?
This PR was tested with available unit tests. Moreover, there are tests to
cover previously failing scenarios.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/aokolnychyi/spark spark-21332
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/18583.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #18583
----
commit 7b35ed2e497f3660e100ad103244beea767f2a73
Author: aokolnychyi <[email protected]>
Date: 2017-07-09T12:54:48Z
[SPARK-21332][SQL] Fix the incorrect result type inferred for some decimal
expressions
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
