Qing Fu created SPARK-60431:
-------------------------------
Summary: Split a wide projection's outputs into methods under
whole-stage codegen
Key: SPARK-60431
URL: https://issues.apache.org/jira/browse/SPARK-60431
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Qing Fu
Under whole-stage codegen, {{ProjectExec}} generates the code of all its
outputs into one method. No single output needs to be large: a projection of a
few dozen ordinary expressions adds up past HotSpot's 8000-byte limit for
JIT-compiling a method ({{-XX:+DontCompileHugeMethods}}), and the stage then
runs interpreted. Spark falls back from whole-stage codegen only past
{{spark.sql.codegen.hugeMethodLimit}} (64KB), so such a stage keeps whole-stage
codegen and runs several times slower than without it.
Measured on master (local, JDK 17), 300,000 rows, each column {{concat_ws('-',
upper(s), lower(s), cast(id + i as string), substring(s, i, 3), cast(id * i % 7
as string))}} over {{id}} and {{s = cast(id as string)}}:
||Columns||Whole-stage codegen on||Whole-stage codegen off||
|50|7.7 s|1.7 s|
|90|13.3 s|2.9 s|
At 50 columns the stage's largest method is 13,187 bytes.
SPARK-33301 added a gate that splits expressions under whole-stage codegen only
in the methods past the JIT limit, for CASE WHEN, and SPARK-60047 proposes the
same for other single large expressions. Neither covers a projection whose
outputs are each small and only large together.
Proposed: under the same gate, split each output of a projection whose code
lands in a method past the limit into a method of its own, which stores the
output's value in fields, as the subexpression elimination functions do. The
output's code becomes the call, which the operator consuming it still evaluates
where it reads the output, so no output is evaluated earlier than today
(evaluating several outputs in one block would, for example, raise an ANSI
error for a row that a filter above the projection drops). The inputs an output
reads that are not evaluated yet are evaluated ahead of its call. A stage whose
methods are under the limit keeps its code unchanged.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
