Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21657#discussion_r202527982
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/csv/CSVSuite.scala
---
@@ -1579,4 +1579,20 @@ class CSVSuite extends QueryTest with
SharedSQLContext with SQLTestUtils with Te
}
}
}
+
+ test("SPARK-24676 project required data from parsed data when
columnPruning disabled") {
+ withSQLConf(SQLConf.CSV_PARSER_COLUMN_PRUNING.key -> "false") {
+ withTempPath { path =>
+ val dir = path.getAbsolutePath
+ spark.range(10).selectExpr("id % 2 AS p", "id AS c0", "id AS
c1").write.partitionBy("p")
+ .option("header", "true").csv(dir)
+ var df = spark.read.option("header",
true).csv(dir).selectExpr("sum(p)", "count(c0)")
--- End diff --
Normally, we do not use `var` for DataFrame even in test cases
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
