David Mollitor created SPARK-59556:
--------------------------------------
Summary: Compile a lookup-shaped CASE WHEN on a string key to a
hash probe
Key: SPARK-59556
URL: https://issues.apache.org/jira/browse/SPARK-59556
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 4.1.0
Reporter: David Mollitor
h2. Problem
A {{CASE WHEN}} whose branches are all {{key = literal THEN constant}} on a
single key is really a lookup table -- e.g. mapping status/country/category
codes to labels, often generated by BI tools or dbt with tens to hundreds of
arms. Today {{CaseWhen}} whole-stage codegen emits an if/else-if chain
({{CaseWhen.multiBranchesCodegen}}). For a string key that is up to N
{{UTF8String}} comparisons per row, growing linearly with the branch count.
{code:sql}
CASE state_code WHEN 'AL' THEN 'Alabama' WHEN 'AK' THEN 'Alaska' ... ELSE
'Unknown' END
{code}
h2. Change
A codegen-only recognizer in {{CaseWhen.doGenCode}}: when the CASE is a lookup
on a binary-collation string key, it builds a constant key -> value table once
on the driver and emits a boxing-free, constant-size inline hash probe (a
power-of-two open-addressed {{int[]}} bucket table plus parallel key/value
{{ArrayData}}) instead of the O(N) chain. The generated code is O(1) in size
and O(1) per row. Non-lookup CASEs fall back to the existing chain unchanged.
The interpreted {{eval}} path and the {{CaseWhen}} node itself are unchanged,
so there is no plan-shape / golden-file impact -- purely a code-generation
substitution.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
