Github user ericl commented on a diff in the pull request:
https://github.com/apache/spark/pull/14690#discussion_r77426535
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileSourceStrategy.scala
---
@@ -79,8 +79,16 @@ object FileSourceStrategy extends Strategy with Logging {
ExpressionSet(normalizedFilters.filter(_.references.subsetOf(partitionSet)))
logInfo(s"Pruning directories with:
${partitionKeyFilters.mkString(",")}")
+ val prunedFsRelation = fsRelation.location match {
--- End diff --
That's correct. Let's wait for #14750 then. My feeling here is that schema
inference over a subset of files is not acceptable since then the schema can
vary with the partition predicate.
For the broadcast join optimization, I think that is a larger issue that
should not affect the design of this pr.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
