parthchandra commented on a change in pull request #24865: [SPARK-27100][SQL]
dag-scheduler-event-loop" java.lang.StackOverflowError
URL: https://github.com/apache/spark/pull/24865#discussion_r294423295
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
##########
@@ -381,7 +381,7 @@ case class FileSourceScanExec(
fsRelation: HadoopFsRelation): RDD[InternalRow] = {
logInfo(s"Planning with ${bucketSpec.numBuckets} buckets")
val filesGroupedToBuckets =
- selectedPartitions.flatMap { p =>
+ selectedPartitions.toArray.flatMap { p =>
Review comment:
@dbtsai I like your suggestion but it requires that we either change the
return type of `FileIndex.listFiles` or convert the value returned by
`relation.location.listFiles` to an array, which doesn't save us anything.
Changing the return type of `FileIndex.listFiles` is a bigger change that I
would want to make at this point, given my relative inexperience with the code.
@srowen,
```
val filePartitions: Seq[FilePartition] =
Seq.tabulate(bucketSpec.numBuckets) { bucketId =>
FilePartition(bucketId,
prunedFilesGroupedToBuckets.getOrElse(bucketId, Nil).toArray)
}.toArray
```
doesn't compile
```
/spark/sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala:404:
polymorphic expression cannot be instantiated to expected type;
[error] found : [B >:
org.apache.spark.sql.execution.datasources.PartitionedFile]Array[B]
[error] required:
Seq[org.apache.spark.sql.execution.datasources.PartitionedFile]
[error] FilePartition(bucketId,
prunedFilesGroupedToBuckets.getOrElse(bucketId, Nil).toArray)
```
Scala isn't my first language so if you can help me with a way to satisfy
the compiler, I'll be happy to make the change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
