David Mollitor created SPARK-59896:
--------------------------------------
Summary: Copy a freqItems value only when storing it in the
counter map
Key: SPARK-59896
URL: https://issues.apache.org/jira/browse/SPARK-59896
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.1.0
Reporter: David Mollitor
h2. Summary
{{{}CollectFrequentItems{}}}, the {{collect_frequent_items}} aggregate behind
{{{}DataFrame.stat.freqItems{}}}, copies every input value with
{{InternalRow.copyValue}} before it looks the value up in its item -> count
map. The copy is only needed when the map stores the value. On a hit, the map
keeps the key it already has, so the copy is thrown away.
h2. Problem
{{CollectFrequentItems.update}} runs once per input row:
{code:scala}
val key = child.eval(input)
if (key != null) {
this.add(buffer, InternalRow.copyValue(key), 1L)
} ...
{code}
The copy exists because the evaluated value may point into a reused input
buffer, so any key the map stores must own its bytes. The copy happens before
{{add}} knows whether the key is already tracked. For a {{{}UTF8String{}}},
{{copyValue}} is {{{}UTF8String.copy(){}}}: a new {{{}byte[]{}}}, a copy of all
the bytes, and a new {{UTF8String}} object. Struct, array and map values, and
the {{BinaryView}} values behind {{GEOMETRY}} / {{{}GEOGRAPHY{}}}, are copied
the same way.
{{freqItems}} is meant for skewed data, where the frequent values keep
recurring. There, most rows are hits on an existing entry, and for each of them
the copy is pure overhead.
h2. Proposed change
Pass the evaluated value to {{add}} uncopied, and copy it only where the map
stores it (the two insert sites in {{{}add{}}}). Incrementing an existing entry
keeps the entry's stored key, so looking up with a value that points into a
reused buffer is safe. {{merge}} adds keys from another buffer that already
owns them, so it keeps not copying. This also removes the null special case in
{{{}update{}}}, since {{InternalRow.copyValue(null)}} is {{{}null{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
