Github user dbtsai commented on a diff in the pull request:
https://github.com/apache/spark/pull/22357#discussion_r215853912
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaPruningSuite.scala
---
@@ -155,6 +155,30 @@ class ParquetSchemaPruningSuite
Row(null) :: Row(null) :: Nil)
}
+ testSchemaPruning("select a single complex field and in where clause") {
+ val query = sql("select name.first from contacts where name.first =
'Jane'")
+ checkScan(query, "struct<name:struct<first:string>>")
+ checkAnswer(query, Row("Jane") :: Nil)
--- End diff --
can you add another tests that select `name.first` and `name.last,` and
apply `where clause` on `name.first`. We should only read `name.first` and
`name.last` without name.middle.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
