GitHub user heary-cao opened a pull request:
https://github.com/apache/spark/pull/22998
[SPARK-26001][SQL]Reduce memory copy when writing decimal
## What changes were proposed in this pull request?
this PR fix 2 here:
- when writing non-null decimals, we not zero-out all the 16 allocated
bytes. if the number of bytes needed for a decimal is greater than 8. then we
not need zero-out between 0-byte and 8-byte. The first 8-byte will be covered
when writing decimal.
- when writing null decimals, we not zero-out all the 16 allocated bytes.
BitSetMethods.set the label for null and the length of decimal to 0. when we
get the decimal, will not access the 16 byte memory value, so this is safe.
## How was this patch tested?
the existed test cases.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/heary-cao/spark writeDecimal
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22998.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22998
----
commit bab69d426578a009ce7796e14757c6ae79d57f28
Author: caoxuewen <cao.xuewen@...>
Date: 2018-11-10T06:31:52Z
Reduce memory copy when writing decimal
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
