jinwensc opened a new pull request #28278:
URL: https://github.com/apache/spark/pull/28278
… bigdecimal not both cast to double"
### What changes were proposed in this pull request?
Fix zhe bug when compare bigdecimal with string,both will cast to
double,result in precision loss and result error.
this PR promote string to bigdecimal.
### Why are the changes needed?
this bug resulted in inaccurate results.
**example**:
| tb1.id | tb1.value |
| ------ | :--------------------------------- |
| 1 | BigDecimal("12345678901234567891") |
| 2 | BigDecimal("12345678901234567892") |
| tb2.id | tb2.value |
| ------ | :--------------------- |
| 1 | "12345678901234567891" |
| 2 | "12345678901234567892" |
spark.sql("select * from tb1 left join tb2 on tb1.value=tb2.value")
```shell
== Analyzed Logical Plan ==
id: int, value: decimal(38,18), id: int, value: string
Project [id#5, value#6, id#14, value#15]
+- Join LeftOuter, (cast(value#6 as double) = cast(value#15 as double))
:- SubqueryAlias `tb1`
: +- Project [_1#2 AS id#5, _2#3 AS value#6]
: +- LocalRelation [_1#2, _2#3]
+- SubqueryAlias `tb2`
+- Project [_1#11 AS id#14, _2#12 AS value#15]
+- LocalRelation [_1#11, _2#12]
```
result:
| tb1.id | tb1.value | tb2.id | tb2.value
|
| ------ | :--------------------------------- | ------ |
---------------------- |
| 1 | BigDecimal("12345678901234567891") | 1 |
"12345678901234567891" |
| 1 | BigDecimal("12345678901234567891") | 2 |
"12345678901234567892" |
| 2 | BigDecimal("12345678901234567892") | 1 |
"12345678901234567891" |
| 2 | BigDecimal("12345678901234567892") | 2 |
"12345678901234567892" |
Data doubled due to lack of accuracy.
we expected :
| tb1.id | tb1.value | tb2.id | tb2.value
|
| ------ | :--------------------------------- | ------ |
---------------------- |
| 1 | BigDecimal("12345678901234567891") | 1 |
"12345678901234567891" |
| 2 | BigDecimal("12345678901234567892") | 2 |
"12345678901234567892" |
### Does this PR introduce any user-facing change?
no
### How was this patch tested?
Added tests to SQLQuerySuite
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
