Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22597#discussion_r224984073
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcFilterSuite.scala
---
@@ -383,4 +384,15 @@ class OrcFilterSuite extends OrcTest with
SharedSQLContext {
)).get.toString
}
}
+
+ test("SPARK-25579 ORC PPD should support column names with dot") {
+ import testImplicits._
+
+ withTempDir { dir =>
+ val path = new File(dir, "orc").getCanonicalPath
+ Seq((1, 2), (3, 4)).toDF("col.dot.1", "col.dot.2").write.orc(path)
+ val df = spark.read.orc(path).where("`col.dot.1` = 1 and `col.dot.2`
= 2")
+ checkAnswer(stripSparkFilter(df), Row(1, 2))
--- End diff --
@dongjoon-hyun, technically shouldn't we test if the stripes are filtered?
I added some tests a long ago (`stripSparkFilter` is added by me FWIW as well):
https://github.com/apache/spark/blob/5d572fc7c35f76e27b2ab400674923eb8ba91745/sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcQuerySuite.scala#L445-L459
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
