[
https://issues.apache.org/jira/browse/SPARK-59950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated SPARK-59950:
-----------------------------------
Affects Version/s: 3.5.8
2.4.8
> Fix integer-division truncation in SizeEstimator's sampled-array extrapolation
> ------------------------------------------------------------------------------
>
> Key: SPARK-59950
> URL: https://issues.apache.org/jira/browse/SPARK-59950
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 2.4.8, 4.1.0, 3.5.8, 4.2.0, 4.0.4
> Reporter: David Mollitor
> Priority: Minor
>
> {{SizeEstimator.visitArray}} estimates the size of an object array with more
> than
> {{ARRAY_SIZE_FOR_SAMPLING}} (400) elements from two samples of
> {{ARRAY_SAMPLE_SIZE }}(100) elements, then extrapolates to the array's full
> length:
> {code:scala}
> val size = math.min(s1, s2)
> state.size += math.max(s1, s2) +
> (size * ((length - ARRAY_SAMPLE_SIZE) / ARRAY_SAMPLE_SIZE))
> {code}
> {{length}} and {{ARRAY_SAMPLE_SIZE}} are both {{{}Int{}}}, so
> {{(length - ARRAY_SAMPLE_SIZE) / ARRAY_SAMPLE_SIZE}} is evaluated as integer
> division *before* the multiply. The remainder is discarded, so up to 99
> elements are left out of the estimate. A 499-element array is sized as if it
> had 400 elements; the estimate is correct only when the length is a multiple
> of 100.
> h3. Impact
> {{MemoryStore}} weighs a deserialized cached block by calling
> {{SizeEstimator.estimate}} over the block's {{Array[T]}} of rows, so any
> cached partition whose row count is not a multiple of 100 is under-reported.
> Collections that track their size through {{SizeEstimator}} (e.g. spill
> thresholds) are affected the same way. The error is bounded by 99/N of the
> elements' size, so it is most pronounced for arrays from 401 up to a few
> thousand elements.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
