[
https://issues.apache.org/jira/browse/HIVE-30225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Denys Kuzmenko updated HIVE-30225:
----------------------------------
Description:
A vectorized ReduceSink writes a decimal64 value (a decimal of up to 18
digits held as a scaled long) to LazyBinary by first deserializing it into a
HiveDecimalWritable and then serializing
that through the general decimal path, which goes through BigInteger. The
reducer reads it back the same way, building a HiveDecimal from BigInteger
bytes before the vectorized reader turns
it into a long again.
*Fix*
* LazyBinarySerializeWrite.writeDecimal64 writes the same bytes as the
general path directly from the long: the scale with trailing fractional zeroes
trimmed (as HiveDecimal normalizes),
then the unscaled value's minimal big-endian two's-complement bytes (as
BigInteger#toByteArray).
* LazyBinaryDeserializeRead rebuilds a decimal of at most 18 digits and scale
from those bytes as a long (HiveDecimalWritable#deserialize64), and falls back
to the general conversion for any
other decimal.
* The wire format is unchanged.
*Measured*
Cluster (TPC-DS 10 TB Parquet, 10 x r5d, same session). The run also had
HIVE-30224 and HIVE-30226 on, so these figures are for the three together:
* -135.4 s over 26 queries.
* q35 38.6 -> 19.8 s, q78 -24.7 s, q75 -21.9 s.
> LazyBinary: a decimal64 value goes through the general decimal conversion on
> both sides of the exchange
> -------------------------------------------------------------------------------------------------------
>
> Key: HIVE-30225
> URL: https://issues.apache.org/jira/browse/HIVE-30225
> Project: Hive
> Issue Type: Improvement
> Reporter: Denys Kuzmenko
> Priority: Major
>
> A vectorized ReduceSink writes a decimal64 value (a decimal of up to 18
> digits held as a scaled long) to LazyBinary by first deserializing it into a
> HiveDecimalWritable and then serializing
> that through the general decimal path, which goes through BigInteger. The
> reducer reads it back the same way, building a HiveDecimal from BigInteger
> bytes before the vectorized reader turns
> it into a long again.
>
>
>
>
>
> *Fix*
>
>
> * LazyBinarySerializeWrite.writeDecimal64 writes the same bytes as the
> general path directly from the long: the scale with trailing fractional
> zeroes trimmed (as HiveDecimal normalizes),
> then the unscaled value's minimal big-endian two's-complement bytes (as
> BigInteger#toByteArray).
>
> * LazyBinaryDeserializeRead rebuilds a decimal of at most 18 digits and
> scale from those bytes as a long (HiveDecimalWritable#deserialize64), and
> falls back to the general conversion for any
> other decimal.
>
>
> * The wire format is unchanged.
>
>
>
>
>
> *Measured*
>
>
> Cluster (TPC-DS 10 TB Parquet, 10 x r5d, same session). The run also had
> HIVE-30224 and HIVE-30226 on, so these figures are for the three together:
>
> * -135.4 s over 26 queries.
>
>
> * q35 38.6 -> 19.8 s, q78 -24.7 s, q75 -21.9 s.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
