ankurdave commented on a change in pull request #34369:
URL: https://github.com/apache/spark/pull/34369#discussion_r734822511
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileScanRDD.scala
##########
@@ -85,6 +85,17 @@ class FileScanRDD(
private[this] var currentFile: PartitionedFile = null
private[this] var currentIterator: Iterator[Object] = null
+ private def resetCurrentIterator(): Unit = {
+ currentIterator match {
+ case iter: NextIterator[_] =>
+ iter.closeIfNeeded()
+ case iter: Closeable =>
+ iter.close()
+ case _ => // do nothing
Review comment:
There are currently two cases aside from null:
- OrcFileFormat produces an ordinary non-Closeable Iterator due to
unwrapOrcStructs().
- The user can create a FileScanRDD with an arbitrary readFunction that does
not return a Closeable Iterator.
It would be ideal if we could disallow these cases and require the iterator
to be Closeable, but it seems that would require changing public APIs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
