[
https://issues.apache.org/jira/browse/SPARK-52484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kent Yao resolved SPARK-52484.
------------------------------
Fix Version/s: 4.1.0
Resolution: Fixed
Issue resolved by pull request 51183
[https://github.com/apache/spark/pull/51183]
> Skip child.supportsColumnar assertion from driver side in ColumnarToRowExec
> ---------------------------------------------------------------------------
>
> Key: SPARK-52484
> URL: https://issues.apache.org/jira/browse/SPARK-52484
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 4.0.0
> Reporter: Hongze Zhang
> Assignee: Hongze Zhang
> Priority: Major
> Fix For: 4.1.0
>
>
> Starting from https://github.com/apache/spark/pull/25264, an assertion
> assert(child.supportsColumnar) was introduced to fail the instantiation of
> ColumnarToRowExec when the child query plan doesn't support columnar
> processing.
> However this check could become a blocker for 3rd plugins when an
> intermediate state of the query plan is needed, e.g., in the Apache Gluten
> project, a C2R node on top of a dummy row-based query plan node could
> sometimes be generated as an intermediate state upon the final query plan.
> This plan doesn't have to be executed but should be created without error.
> Hence, Gluten had to [fake a task
> context|https://github.com/apache/incubator-gluten/blob/0a1b5c28678653242ab0fd7b28ebba1dca43ccb1/gluten-core/src/main/scala/org/apache/gluten/extension/columnar/enumerated/planner/plan/GlutenPlanModel.scala#L50-L55]
> to bypass the assertion, though it's indeed a workaround.
> This change is to remove the assertion to ensure some flexibilities to the
> 3rd plugins. This should be no harm for the upstream Apache Spark, because
> the query execution will still be failed by [this
> error|https://github.com/apache/spark/blob/5d0b2f41794bf4dd25b3ce19bc4f634082b40876/sql/core/src/main/scala/org/apache/spark/sql/execution/SparkPlan.scala#L343-L351]
> without this assertion on an illegal query plan.
> This could also make ColumnarToRowExec align with RowToColumnarExec - the
> later doesn't do this check during instantiation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
