[
https://issues.apache.org/jira/browse/SPARK-60431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60431:
-----------------------------------
Labels: pull-request-available (was: )
> Split a wide projection's outputs into methods under whole-stage codegen
> ------------------------------------------------------------------------
>
> Key: SPARK-60431
> URL: https://issues.apache.org/jira/browse/SPARK-60431
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Qing Fu
> Priority: Major
> Labels: pull-request-available
>
> Under whole-stage codegen, {{ProjectExec}} generates the code of all its
> outputs into one method. No single output needs to be large: a projection of
> a few dozen ordinary expressions adds up past HotSpot's 8000-byte limit for
> JIT-compiling a method ({{-XX:+DontCompileHugeMethods}}), and the stage then
> runs interpreted. Spark falls back from whole-stage codegen only past
> {{spark.sql.codegen.hugeMethodLimit}} (64KB), so such a stage keeps
> whole-stage codegen and runs several times slower than without it.
> Measured on master (local, JDK 17), 300,000 rows, each column
> {{concat_ws('-', upper(s), lower(s), cast(id + i as string), substring(s, i,
> 3), cast(id * i % 7 as string))}} over {{id}} and {{s = cast(id as string)}}:
> ||Columns||Whole-stage codegen on||Whole-stage codegen off||
> |50|7.7 s|1.7 s|
> |90|13.3 s|2.9 s|
> At 50 columns the stage's largest method is 13,187 bytes.
> SPARK-33301 added a gate that splits expressions under whole-stage codegen
> only in the methods past the JIT limit, for CASE WHEN, and SPARK-60047
> proposes the same for other single large expressions. Neither covers a
> projection whose outputs are each small and only large together.
> Proposed: under the same gate, split each output of a projection whose code
> lands in a method past the limit into a method of its own, which stores the
> output's value in fields, as the subexpression elimination functions do. The
> output's code becomes the call, which the operator consuming it still
> evaluates where it reads the output, so no output is evaluated earlier than
> today (evaluating several outputs in one block would, for example, raise an
> ANSI error for a row that a filter above the projection drops). The inputs an
> output reads that are not evaluated yet are evaluated ahead of its call. A
> stage whose methods are under the limit keeps its code unchanged.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
