David Mollitor created SPARK-59653:
--------------------------------------
Summary: Lazily materialize UnsafeMapData key/value arrays to
reduce allocations
Key: SPARK-59653
URL: https://issues.apache.org/jira/browse/SPARK-59653
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.1.0
Reporter: David Mollitor
h2. Problem
{{UnsafeRow.getMap}} / {{UnsafeArrayData.getMap}} create a fresh
{{UnsafeMapData}} per access, and its constructor eagerly allocates two nested
{{UnsafeArrayData}} objects ({{{}keys{}}} and {{{}values{}}}).
For a transient, count-only map access – {{{}size(map){}}},
{{{}cardinality{}}}, and the {{size(x) > 0}} filter that
{{InferFiltersFromGenerate}} inserts below every non-outer
{{{}explode{}}}/{{{}inline{}}} – this nested object generates a wrapper plus
two sub-wrappers are allocated per row purely to read an element count that is
already stored inline in the layout.
By contrast, {{UnsafeArrayData}} is a flat object (primitive fields plus a
base-object reference after {{{}pointTo{}}}), which the JIT already
scalar-replaces – so {{size(array_col)}} allocates nothing. Only maps carry
this per-row garbage.
h2. Change
Make {{UnsafeMapData}} flat so a count-only access is scalar-replaceable by the
JIT:
* Build the {{{}keys{}}}/{{{}values{}}} {{UnsafeArrayData}} views lazily in
{{{}keyArray(){}}}/{{{}valueArray() {}}}instead of eagerly in the constructor.
* Compute {{numElements()}} by reading the key-array header directly
({{{}Platform.getLong(baseObject, baseOffset + 8){}}}), which equals
{{keyArray().numElements() }}without materializing any view.
* Store the key-array byte size as a primitive field in {{pointTo}} and drop
the eager
{{keys.numElements() == values.numElements()}} debug assertion.
There is no storage-format change and no call-site changes:
{{{}keyArray(){}}}/{{{}valueArray(){}}} keep their signatures, and
serialization ({{{}Externalizable{}}}/Kryo), {{{}copy(){}}}, and
{{MapData.foreach}}
are unaffected (they go through the accessors or operate on the raw bytes).
h2. Impact
On a representative {{size(map_col)}} workload (whole-stage codegen on,
assertions disabled to match production), the per-row {{getMap}} wrapper
allocation dropped from ~4022 to ~94 JFR allocation samples (~98% eliminated),
and {{UnsafeMapData}} leaves the top of the allocation-by-class profile.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
