GitHub user mateiz opened a pull request:
https://github.com/apache/spark/pull/2983
[SPARK-3930] [SPARK-3933] [WIP] Support fixed-precision decimal in SQL, and
some optimizations
- Adds optional precision and scale to Spark SQL's decimal type, which
behave similarly to those in Hive 13
(https://cwiki.apache.org/confluence/download/attachments/27362075/Hive_Decimal_Precision_Scale_Support.pdf)
- Replaces our internal representation of decimals with a Decimal class
that can store small values in a mutable Long, saving memory in this situation
and letting some operations happen directly on Longs
This is still marked WIP because there are a few TODOs, but I'll remove
that tag when done.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mateiz/spark decimal-1
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/2983.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2983
----
commit df3cb3252451def6c94ff20c42f3cc36a5265554
Author: Matei Zaharia <[email protected]>
Date: 2014-10-02T00:16:15Z
Make the result of AVG on Decimals be Decimal, not Double
commit 629d9860dccccce02833b4f363074ecc29693ab9
Author: Matei Zaharia <[email protected]>
Date: 2014-10-03T06:06:32Z
Add optional precision and scale to DecimalType, but use Unlimited for now
Implement CAST to fixed-precision decimal
Add rules for propagating precision through decimal calculations
These work by casting things to Decimal.Unlimited to do the actual
operation, then adding a cast on the result. They will result in more
casts than needed, but on the other hand they avoid having each
arithmetic operator know about decimal precision rules. We might be able
to add more rules later to eliminate some intermediate casts.
commit 55c664ac5d9d3744bc7559817a73cc90a5c6e5d8
Author: Matei Zaharia <[email protected]>
Date: 2014-10-05T03:14:04Z
Added mutable Decimal that will be more efficient for small precisions
Optimize sums and averages on fixed-precision Decimals
commit 4e4bf3fb020bb8398c73c9f8525214f7ff8487ed
Author: Matei Zaharia <[email protected]>
Date: 2014-10-28T22:59:34Z
Some test and bug fixes
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
