abellina commented on a change in pull request #24795: [SPARK-27945][SQL]
Minimal changes to support columnar processing
URL: https://github.com/apache/spark/pull/24795#discussion_r293939557
##########
File path:
sql/core/src/test/scala/org/apache/spark/sql/SparkSessionExtensionSuite.scala
##########
@@ -116,6 +121,23 @@ class SparkSessionExtensionSuite extends SparkFunSuite {
}
}
+ test("inject columnar") {
+ val extensions = create { extensions =>
+ extensions.injectColumnar(session => MyColumarRule(PreRuleReplaceAdd(),
MyPostRule()))
+ }
+ withSession(extensions) { session =>
+ assert(session.sessionState.columnarRules.contains(
+ MyColumarRule(PreRuleReplaceAdd(), MyPostRule())))
+ import session.sqlContext.implicits._
+ // repartitioning avoids having the add operation pushed up into the
LocalTableScan
+ val data = Seq((100L), (200L), (300L)).toDF("vals").repartition(1)
+ val result = data.selectExpr("vals + 1").collect()
Review comment:
maybe lets make the rule `PreRuleReplaceAddPlusExtraOne` as I see that 100 +
1 is not 102, but you stated this within the rule :)
re:
https://github.com/apache/spark/pull/24795/files#diff-b7face523211086b889c334ae4075ad2R549
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
