[
https://issues.apache.org/jira/browse/SPARK-59805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dustin Smith updated SPARK-59805:
---------------------------------
Description:
For DECIMAL columns with precision > 18, UnsafeRow stores the unscaled value as
big-endian two's-complement bytes. {{UnsafeRow.getDecimal}},
{{UnsafeRow.setDecimal}} and {{UnsafeRowWriter.write}} always convert through
{{BigInteger}}/{{BigDecimal}}, even when the unscaled value fits in a long.
Every read allocates a {{byte[]}}, a {{BigInteger}} and two {{BigDecimal}}s,
and returns a BigDecimal-backed {{Decimal}}.
This matters most for aggregation. The buffer of {{sum}}/{{avg}} over
{{decimal(p, s)}} has precision {{p + 10}}, so {{sum(decimal(15,2))}} uses a
{{decimal(25,2)}} buffer. Each buffer read comes back BigDecimal-backed, so
{{Decimal.+}} never takes its long fast path. In a CPU profile of TPC-H SF10,
11.4% of samples were in {{BigDecimal}}/{{BigInteger}} or {{Decimal}}.
Proposal: read and write these values directly as a long when they fit, keeping
the stored bytes identical to {{BigInteger.toByteArray}}, and return a compact
(long-backed) {{Decimal}} when {{|unscaled| < 10^18}} and {{0 <= scale <= 18}}.
This also requires {{Decimal.roundToInt}}/{{roundToShort}}/{{roundToByte}} to
truncate exactly for BigDecimal-backed values, as {{roundToLong}} already does.
Their current double-based check rounds a value just below a bound past it, so
under ANSI mode {{CAST(CAST('2147483647.99999999' AS DECIMAL(20,8)) AS INT)}}
fails with CAST_OVERFLOW instead of returning 2147483647. Without that fix, the
same cast would succeed or fail depending on the internal representation.
Measured: {{sum}}/{{avg}} of {{decimal(15,2)}} about 1.7x faster per row; TPC-H
SF10 79.4s -> 73.8s (q1 11.55s -> 8.76s); TPC-DS SF10 unchanged within noise.
> Avoid BigDecimal round trips for wide decimals that fit in a long in UnsafeRow
> ------------------------------------------------------------------------------
>
> Key: SPARK-59805
> URL: https://issues.apache.org/jira/browse/SPARK-59805
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Dustin Smith
> Priority: Major
> Labels: pull-request-available
>
> For DECIMAL columns with precision > 18, UnsafeRow stores the unscaled value
> as big-endian two's-complement bytes. {{UnsafeRow.getDecimal}},
> {{UnsafeRow.setDecimal}} and {{UnsafeRowWriter.write}} always convert through
> {{BigInteger}}/{{BigDecimal}}, even when the unscaled value fits in a long.
> Every read allocates a {{byte[]}}, a {{BigInteger}} and two {{BigDecimal}}s,
> and returns a BigDecimal-backed {{Decimal}}.
> This matters most for aggregation. The buffer of {{sum}}/{{avg}} over
> {{decimal(p, s)}} has precision {{p + 10}}, so {{sum(decimal(15,2))}} uses a
> {{decimal(25,2)}} buffer. Each buffer read comes back BigDecimal-backed, so
> {{Decimal.+}} never takes its long fast path. In a CPU profile of TPC-H SF10,
> 11.4% of samples were in {{BigDecimal}}/{{BigInteger}} or {{Decimal}}.
> Proposal: read and write these values directly as a long when they fit,
> keeping the stored bytes identical to {{BigInteger.toByteArray}}, and return
> a compact (long-backed) {{Decimal}} when {{|unscaled| < 10^18}} and {{0 <=
> scale <= 18}}.
> This also requires {{Decimal.roundToInt}}/{{roundToShort}}/{{roundToByte}} to
> truncate exactly for BigDecimal-backed values, as {{roundToLong}} already
> does. Their current double-based check rounds a value just below a bound past
> it, so under ANSI mode {{CAST(CAST('2147483647.99999999' AS DECIMAL(20,8)) AS
> INT)}} fails with CAST_OVERFLOW instead of returning 2147483647. Without that
> fix, the same cast would succeed or fail depending on the internal
> representation.
> Measured: {{sum}}/{{avg}} of {{decimal(15,2)}} about 1.7x faster per row;
> TPC-H SF10 79.4s -> 73.8s (q1 11.55s -> 8.76s); TPC-DS SF10 unchanged within
> noise.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
