Peifeng Li created SPARK-59443:
----------------------------------
Summary: Reduce allocations when reading small-precision decimals
from ArrowColumnVector
Key: SPARK-59443
URL: https://issues.apache.org/jira/browse/SPARK-59443
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Peifeng Li
ArrowColumnVector.DecimalAccessor converts each non-null Decimal128 value
through Decimal.apply(vector.getObject(rowId), precision, scale).
Arrow Java 19.0.0 constructs a temporary byte array and BigInteger when
materializing the BigDecimal. This also happens for source precision 1–18,
whose valid unscaled integers fit in a signed long.
A small-precision accessor can read the native Decimal128 words, check
sign extension, and construct BigDecimal from the unscaled long and source
scale. Values that do not fit in a long retain the full-width conversion.
Spark still applies the requested precision and scale, and the result
retains BigDecimal-backed Decimal behavior for checked integer casts.
ArrowCachedBatchSerializer already reads native long words for compact
decimals when writing directly to UnsafeRow. This change applies the same
decoding approach to the general ArrowColumnVector interface.
A prototype based on master 15ec0e55b5d5faa9b7844978eddfec696f95f778,
using Arrow Java 19.0.0 and Corretto Java 17.0.16, measured:
- decimal(18,2) -> decimal(18,2): 192 -> 96 allocated bytes per read.
- decimal(18,2) -> decimal(18,1): 256 -> 160 allocated bytes per read.
- decimal(38,2) -> decimal(38,2): 200 -> 200 allocated bytes per read.
The allocation comparison used the original and modified ArrowColumnVector
classes in separate JVMs, with two runs per version and five measured
samples per scenario. Input batches contained 4,096 alternating positive
and negative values; returned Decimal objects were retained in an array.
Related Vortex connector discussion:
https://github.com/vortex-data/vortex/issues/9837
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
