kiszk commented on a change in pull request #24795: [SPARK-27945][SQL] Minimal
changes to support columnar processing
URL: https://github.com/apache/spark/pull/24795#discussion_r295898720
##########
File path:
sql/core/src/test/scala/org/apache/spark/sql/SparkSessionExtensionSuite.scala
##########
@@ -116,6 +121,34 @@ class SparkSessionExtensionSuite extends SparkFunSuite {
}
}
+ test("inject columnar") {
+ val extensions = create { extensions =>
+ extensions.injectColumnar(session =>
+ MyColumarRule(PreRuleReplaceAddWithBrokenVersion(), MyPostRule()))
+ }
+ withSession(extensions) { session =>
+ assert(session.sessionState.columnarRules.contains(
+ MyColumarRule(PreRuleReplaceAddWithBrokenVersion(), MyPostRule())))
+ import session.sqlContext.implicits._
+ // repartitioning avoids having the add operation pushed up into the
LocalTableScan
+ val data = Seq((100L), (200L), (300L)).toDF("vals").repartition(1)
+ val df = data.selectExpr("vals + 1")
+ // Verify that both pre and post processing of the plan worked.
+ val found = df.queryExecution.executedPlan.collect {
+ case rep: ReplacedRowToColumnarExec => 1
Review comment:
How about the following? The summation return the unique value that
corresponds to the combination.
```
case rep: ... => 1
case proj: ... => 4
case c2r: ... => 13
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
