[
https://issues.apache.org/jira/browse/SPARK-35881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-35881:
------------------------------------
Assignee: (was: Apache Spark)
> [SQL] AQE does not support columnar execution for the final query stage
> -----------------------------------------------------------------------
>
> Key: SPARK-35881
> URL: https://issues.apache.org/jira/browse/SPARK-35881
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.0.3, 3.1.2, 3.2.0
> Reporter: Andy Grove
> Priority: Major
>
> In AdaptiveSparkPlanExec, a query is broken down into stages and these stages
> are executed until the entire query has been executed. These stages can be
> row-based or columnar. However, the final stage, produced by the private
> getFinalPhysicalPlan method is always assumed to be row-based. The only way
> to execute the final stage is by calling the various doExecute methods on
> AdaptiveSparkPlanExec, and doExecuteColumnar is not implemented. The
> supportsColumnar method also always returns false.
> In the RAPIDS Accelerator for Apache Spark, we currently call the private
> getFinalPhysicalPlan method using reflection and then determine if that plan
> is columnar or not, and then call the appropriate doExecute method, bypassing
> the doExecute methods on AdaptiveSparkPlanExec. We would like a supported
> mechanism for executing a columnar AQE plan so that we do not need to use
> reflection.
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
