Github user mallman commented on a diff in the pull request:
https://github.com/apache/spark/pull/21889#discussion_r207718734
--- Diff:
sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaPruningSuite.scala
---
@@ -0,0 +1,205 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql.execution.datasources.parquet
+
+import java.io.File
+
+import org.apache.spark.sql.{QueryTest, Row}
+import org.apache.spark.sql.execution.FileSchemaPruningTest
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSQLContext
+
+class ParquetSchemaPruningSuite
+ extends QueryTest
+ with ParquetTest
+ with FileSchemaPruningTest
+ with SharedSQLContext {
+ case class FullName(first: String, middle: String, last: String)
+ case class Contact(
+ id: Int,
+ name: FullName,
+ address: String,
+ pets: Int,
+ friends: Array[FullName] = Array(),
+ relatives: Map[String, FullName] = Map())
+
+ val janeDoe = FullName("Jane", "X.", "Doe")
+ val johnDoe = FullName("John", "Y.", "Doe")
+ val susanSmith = FullName("Susan", "Z.", "Smith")
+
+ val contacts =
+ Contact(0, janeDoe, "123 Main Street", 1, friends = Array(susanSmith),
+ relatives = Map("brother" -> johnDoe)) ::
+ Contact(1, johnDoe, "321 Wall Street", 3, relatives = Map("sister" ->
janeDoe)) :: Nil
+
+ case class Name(first: String, last: String)
+ case class BriefContact(id: Int, name: Name, address: String)
+
+ val briefContacts =
+ BriefContact(2, Name("Janet", "Jones"), "567 Maple Drive") ::
+ BriefContact(3, Name("Jim", "Jones"), "6242 Ash Street") :: Nil
+
+ case class ContactWithDataPartitionColumn(
+ id: Int,
+ name: FullName,
+ address: String,
+ pets: Int,
+ friends: Array[FullName] = Array(),
+ relatives: Map[String, FullName] = Map(),
+ p: Int)
+
+ case class BriefContactWithDataPartitionColumn(id: Int, name: Name,
address: String, p: Int)
+
+ val contactsWithDataPartitionColumn =
+ contacts.map { case Contact(id, name, address, pets, friends,
relatives) =>
+ ContactWithDataPartitionColumn(id, name, address, pets, friends,
relatives, 1) }
+ val briefContactsWithDataPartitionColumn =
+ briefContacts.map { case BriefContact(id, name, address) =>
+ BriefContactWithDataPartitionColumn(id, name, address, 2) }
+
+ testSchemaPruning("select a single complex field") {
+ val query = sql("select name.middle from contacts order by id")
+ checkScanSchemata(query, "struct<id:int,name:struct<middle:string>>")
+ checkAnswer(query, Row("X.") :: Row("Y.") :: Row(null) :: Row(null) ::
Nil)
+ }
+
+ testSchemaPruning("select a single complex field and its parent struct")
{
+ val query = sql("select name.middle, name from contacts order by id")
+ checkScanSchemata(query,
"struct<id:int,name:struct<first:string,middle:string,last:string>>")
+ checkAnswer(query,
+ Row("X.", Row("Jane", "X.", "Doe")) ::
+ Row("Y.", Row("John", "Y.", "Doe")) ::
+ Row(null, Row("Janet", null, "Jones")) ::
+ Row(null, Row("Jim", null, "Jones")) ::
+ Nil)
+ }
+
+ testSchemaPruning("select a single complex field array and its parent
struct array") {
+ val query = sql("select friends.middle, friends from contacts where
p=1 order by id")
+ checkScanSchemata(query,
+
"struct<id:int,friends:array<struct<first:string,middle:string,last:string>>>")
+ checkAnswer(query,
+ Row(Array("Z."), Array(Row("Susan", "Z.", "Smith"))) ::
+ Row(Array.empty[String], Array.empty[Row]) ::
+ Nil)
+ }
+
+ testSchemaPruning("select a single complex field from a map entry and
its parent map entry") {
+ val query =
+ sql("select relatives[\"brother\"].middle, relatives[\"brother\"]
from contacts where p=1 " +
+ "order by id")
+ checkScanSchemata(query,
+
"struct<id:int,relatives:map<string,struct<first:string,middle:string,last:string>>>")
+ checkAnswer(query,
+ Row("Y.", Row("John", "Y.", "Doe")) ::
+ Row(null, null) ::
+ Nil)
+ }
+
+ testSchemaPruning("select a single complex field and the partition
column") {
+ val query = sql("select name.middle, p from contacts order by id")
+ checkScanSchemata(query, "struct<id:int,name:struct<middle:string>>")
+ checkAnswer(query, Row("X.", 1) :: Row("Y.", 1) :: Row(null, 2) ::
Row(null, 2) :: Nil)
+ }
+
+ ignore("partial schema intersection - select missing subfield") {
--- End diff --
This test is passing now. I think that I was so convinced that there was a
problem in the Spark codebaseâbased on several confounding factorsâthat I
overlooked the fact that the failing test was actually alerting me to a problem
in my patch. Lesson learned.
I am re-running the entire catalyst and sql-core test suite again locally
with the default setting for schema pruning set to `true`. If those tests pass,
I will publish a revised commit to my branch with the ignored test re-enabled
and the new default value of the schema pruning configuration. I will ask
@ajacques to rebase off of my branch at that time. If that goes well, then I
suggest we merge this PR to master.
Incidentally, before closing SPARK-4502, we should consider how to document
and track future work on this functionality in Jira. I see at least three more
PRs following this one in order to get back to the functionality of #16578.
First is to restore support for displaying the number of physical columns that
will be read in the physical query plan. That is *invaluable* in verifying that
schema pruning is working as expected. Second is to restore support for pruning
query filters ("WHERE" clauses). Following that is support for pruning in joins
and aggregations in one or two more PRs. I may be forgetting some things. That
will cover a good deal of use cases for analytic queries. There are some
additional use cases I have in my back pocket that can be worked on, too, such
as support for window queries.
So... do we create sub-issues in SPARK-4502? Or do we close SPARK-4502 and
create new issues?
Overall, after 2+ years in review, I'm excited to see us *this close* to
merging something in. Once we have this in master, I believe it will be ready
for an ORC expert to contribute support for schema pruning for that file
format. Support for other columnar file formats should theoretically be
possible, too.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
