revans2 commented on a change in pull request #25008: [SPARK-28213][SQL]
Replace ColumnarBatchScan with equivilant from Columnar
URL: https://github.com/apache/spark/pull/25008#discussion_r307770674
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/test/SQLTestUtils.scala
##########
@@ -392,7 +392,7 @@ private[sql] trait SQLTestUtilsBase
*/
protected def stripSparkFilter(df: DataFrame): DataFrame = {
val schema = df.schema
- val withoutFilters = df.queryExecution.sparkPlan.transform {
+ val withoutFilters = df.queryExecution.executedPlan.transform {
Review comment:
This PR replaces ColumnarBatchScan with ColumnarToRowExec. This involved
changes to all subclasses of COlumnarBatchScan and to AdaptiveSparkPlanExec so
it would also execute the needed columnar transition rules. I also made any
fixes needed for tests. I preferred to keep the changes as small as possible
for the tests, which is why I made a small change here in a test utility class.
The issue was that some tests were directly execution the plan returned by
this function, which used to work for some very limited use cases, but did not
work in all cases. I am happy to try and fix issues with this approach for the
tests, just let me know what is the correct way to do it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
