Yuming Wang created SPARK-60832:
-----------------------------------
Summary: Shrink the sparse index array of LongToUnsafeRowMap at
optimize time
Key: SPARK-60832
URL: https://issues.apache.org/jira/browse/SPARK-60832
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 3.5.9, 4.0.5, 4.1.3, 4.1.4, 4.2.1, 4.3.0, 5.0.0
Reporter: Yuming Wang
LongToUnsafeRowMap (the map behind broadcast LongHashedRelation for single long
keys)
sizes its sparse index array by the estimated ROW count and keeps a load factor
<= 0.5
headroom while being built. When many rows share few distinct keys and the key
range is
too wide for optimize()'s dense conversion, the finished map keeps a
mostly-empty array,
and write() serializes it verbatim -- so the headroom is paid for in driver
memory, in
the broadcast payload, and again in every executor's deserialized copy.
Example: 100M rows sharing 10K wide-range keys keep a ~2 GiB index that only
needs
512 KiB (~4096x waste).
Proposal: shrink the sparse array once at optimize() time (when dense mode does
not
apply) by rehashing the occupied (key, address) pairs into the smallest
power-of-two
array that keeps updateIndex's load factor invariant (numKeys * 4 <=
array.size()).
write()/read() stay unchanged, so the wire format is untouched and executors
deserialize
a right-sized index. Right-sized builds (unique keys, or any map that grew
during the
build) are no-ops; at the 1<<30 array-size ceiling the shrink never grows.
ShuffledHashJoinExec passes no size estimate and its growth ladder already
lands on the
minimal size, so this is effectively broadcast-only.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
