[
https://issues.apache.org/jira/browse/SPARK-44773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17756642#comment-17756642
]
Snoot.io commented on SPARK-44773:
----------------------------------
User 'wankunde' has created a pull request for this issue:
https://github.com/apache/spark/pull/42450
> Code-gen CodegenFallback expression in WholeStageCodegen if possible
> --------------------------------------------------------------------
>
> Key: SPARK-44773
> URL: https://issues.apache.org/jira/browse/SPARK-44773
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 3.5.0
> Reporter: Wan Kun
> Priority: Major
>
> Now both WholeStageCodegen framework and SubExpressionElimination framework
> does not support CodegenFallback expression, but the CodegenFallback
> expression which contains nullSafeEval method could gen-code just like common
> expressions, now they are always be executed in a new
> SpecificUnsafeProjection class, and we can not eliminate the sub expressions.
> For example:
> SQL:
> {code:sql}
> SELECT from_json(regexp_replace(s, 'a', 'x'), 'x INT, b DOUBLE').x,
> from_json(regexp_replace(s, 'a', 'x'), 'x INT, b DOUBLE').b
> FROM values('{"a":1, "b":0.8}') t(s)
> {code}
> plan:
> {code:java}
> *(1) Project [from_json(StructField(x,IntegerType,true),
> regexp_replace(s#218, a, x, 1), Some(America/Los_Angeles)).x AS
> from_json(regexp_replace(s, a, x, 1)).x#219,
> from_json(StructField(b,DoubleType,true), regexp_replace(s#218, a, x, 1),
> Some(America/Los_Angeles)).b AS from_json(regexp_replace(s, a, x, 1)).b#220]
> +- *(1) LocalTableScan [s#218]
> {code}
> Due to expression org.apache.spark.sql.catalyst.expressions.JsonToStructs is
> CodegenFallback expression, so we can not reuse the result of
> {*}regexp_replace(s, 'a', 'x'){*}.
> We can support expression
> org.apache.spark.sql.catalyst.expressions.JsonToStructs code-gen in
> WholeStageCodegen framework, and then reuse the result of
> {*}regexp_replace(s, 'a', 'x'){*}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
