HyukjinKwon commented on a change in pull request #25008: [SPARK-28213][SQL]
Replace ColumnarBatchScan with equivilant from Columnar
URL: https://github.com/apache/spark/pull/25008#discussion_r307569775
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/test/SQLTestUtils.scala
##########
@@ -392,7 +392,7 @@ private[sql] trait SQLTestUtilsBase
*/
protected def stripSparkFilter(df: DataFrame): DataFrame = {
val schema = df.schema
- val withoutFilters = df.queryExecution.sparkPlan.transform {
+ val withoutFilters = df.queryExecution.executedPlan.transform {
Review comment:
Seems like the existing columnar logic within each plans have moved to
`RowToColumnarExec` and `ColumnarToRowExec` to deduplicate but it's now
dependent on `ApplyColumnarRulesAndInsertTransitions` rule, which, now,
requires execution preparation (`QueryExecution.preparations`).
However, per the doc, `executedPlan` should be only used for execution
ideally. It could have been best to avoid. @revans2 even though it's too late,
can you please describe what does this PR fixes in the PR description
(presumably by listing each item)?
I have no idea what this PR fixes from reading the PR description and JIRA.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
