Yang Jie created SPARK-60510:
--------------------------------
Summary: Check that values handed out as JavaCode.variable are
Java names
Key: SPARK-60510
URL: https://issues.apache.org/jira/browse/SPARK-60510
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: Yang Jie
CodegenContext.addMutableState returns either an inlined field name or a slot
of a compacted array, such as mutableStateArray_0[3], and a producer may wrap
either with JavaCode.variable. A slot is not a Java name, so no method can take
it as a parameter. After SPARK-59589, testing checks this where generated code
may be moved into a method: CodegenContext.collectInputs under the whole stage
split and common expression policies, and assertParametersAreNames for the
subexpression, aggregate and switch-case methods. Two things remain:
- The check runs only where a method is considered, so a mislabelled value that
never reaches one is not caught, and the producer learns about it only once a
consumer starts moving that code into a method. A check at the producer would
catch it where it is made, for example a testing-only check in
JavaCode.variable and JavaCode.isNullVariable, or an addMutableState variant
that returns a GlobalValue. It cannot simply be added to JavaCode.variable:
collectInputs wraps INPUT_ROW with it, and the hand-built values in
WithExpressionEvalSuite and CollectInputsSuite use slot names on purpose.
- From reading the code, CodegenContext.wholeStageSplitArguments appends
INPUT_ROW to the arguments of a whole stage split method without going through
collectInputs, so an INPUT_ROW that is a slot would become a parameter named
arr[i], which splitCall's check does not catch either. Every operator that sets
INPUT_ROW to a field today adds it with forceInline = true.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
