andygrove commented on code in PR #892:
URL: https://github.com/apache/datafusion-comet/pull/892#discussion_r1737910733
##########
spark/src/test/scala/org/apache/comet/exec/CometExecSuite.scala:
##########
@@ -1547,6 +1547,55 @@ class CometExecSuite extends CometTestBase {
})
}
+ test("SparkToColumnar over BatchScan (Spark Parquet reader)") {
+ Seq("", "parquet").foreach { v1List =>
+ Seq("true", "false").foreach { parquetVectorized =>
+ Seq(
+ "cast(id as tinyint)",
+ "cast(id as smallint)",
+ "cast(id as integer)",
+ "cast(id as bigint)",
+ "cast(id as float)",
+ "cast(id as double)",
+ "cast(id as decimal)",
+ "cast(id as timestamp)",
+ "cast(id as string)",
+ "cast(id as binary)",
+ "struct(id)").foreach { valueType =>
+ {
+ withSQLConf(
+ SQLConf.USE_V1_SOURCE_LIST.key -> v1List,
+ CometConf.COMET_NATIVE_SCAN_ENABLED.key -> "false",
+ CometConf.COMET_CONVERT_FROM_PARQUET_ENABLED.key -> "true",
+ SQLConf.PARQUET_VECTORIZED_READER_ENABLED.key ->
parquetVectorized) {
+ withTempPath { dir =>
+ var df = spark
+ .range(10000)
+ .selectExpr("id as key", s"$valueType as value")
+ .toDF("key", "value")
+
+ df.write.parquet(dir.toString())
+
+ df = spark.read.parquet(dir.toString())
+ checkSparkAnswerAndOperator(
+ df.select("*").groupBy("key", "value").count(),
+ includeClasses = Seq(classOf[CometSparkToColumnarExec]))
+
+ // Verify that the BatchScanExec nodes supported columnar
output when requested.
+ val leaves = df.queryExecution.executedPlan.collectLeaves()
+ if (parquetVectorized == "true") {
Review Comment:
very minor nit: rather than using strings to represent booleans in
`Seq("true", "false")`, I think it would be better to have booleans here so
that we can just have have `if (parquetVectorized)` here and when we need to
access it as a string when setting configs we can call `.toString`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
