Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21111#discussion_r182946222
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/OptimizeMetadataOnlyQuery.scala
---
@@ -114,11 +119,8 @@ case class OptimizeMetadataOnlyQuery(catalog:
SessionCatalog) extends Rule[Logic
relation match {
case l @ LogicalRelation(fsRelation: HadoopFsRelation, _, _,
isStreaming) =>
val partAttrs =
getPartitionAttrs(fsRelation.partitionSchema.map(_.name), l)
- val partitionData = fsRelation.location.listFiles(relFilters,
Nil)
- // partition data may be a stream, which can cause
serialization to hit stack level too
- // deep exceptions because it is a recursive structure in
memory. converting to array
- // avoids the problem.
--- End diff --
I believe this is already fixed in
https://issues.apache.org/jira/browse/SPARK-21884
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
