Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21893#discussion_r206234291
--- Diff:
sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala
---
@@ -437,6 +437,8 @@ private[hive] class HiveClientImpl(
throw new AnalysisException("Hive index table is not
supported.")
},
schema = schema,
+ hasMultiFormatPartitions =
+ shim.getAllPartitions(client,
h).map(_.getInputFormatClass).distinct.size > 1,
--- End diff --
Fetching the metadata of all the partitions will slow down all the queries
when the number of partitions is huge. We will face a huge perf regression.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
