David Mollitor created SPARK-59950:
--------------------------------------
Summary: Fix integer-division truncation in SizeEstimator's
sampled-array extrapolation
Key: SPARK-59950
URL: https://issues.apache.org/jira/browse/SPARK-59950
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 4.1.0
Reporter: David Mollitor
{{SizeEstimator.visitArray}} estimates the size of an object array with more
than
{{ARRAY_SIZE_FOR_SAMPLING}} (400) elements from two samples of
{{ARRAY_SAMPLE_SIZE }}(100) elements, then extrapolates to the array's full
length:
{code:scala}
val size = math.min(s1, s2)
state.size += math.max(s1, s2) +
(size * ((length - ARRAY_SAMPLE_SIZE) / ARRAY_SAMPLE_SIZE))
{code}
{{length}} and {{ARRAY_SAMPLE_SIZE}} are both {{{}Int{}}}, so
{{(length - ARRAY_SAMPLE_SIZE) / ARRAY_SAMPLE_SIZE}} is evaluated as integer
division *before* the multiply. The remainder is discarded, so up to 99
elements are left out of the estimate. A 499-element array is sized as if it
had 400 elements; the estimate is correct only when the length is a multiple of
100.
h3. Impact
{{MemoryStore}} weighs a deserialized cached block by calling
{{SizeEstimator.estimate}} over the block's {{Array[T]}} of rows, so any cached
partition whose row count is not a multiple of 100 is under-reported.
Collections that track their size through {{SizeEstimator}} (e.g. spill
thresholds) are affected the same way. The error is bounded by 99/N of the
elements' size, so it is most pronounced for arrays from 401 up to a few
thousand elements.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
