cloud-fan commented on code in PR #57388:
URL: https://github.com/apache/spark/pull/57388#discussion_r3627128276
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/SupportsArchiveFormat.scala:
##########
@@ -88,10 +90,124 @@ trait SupportsArchiveFormat extends Logging {
}
}
+ /** Which archive entries are data files of this format; others are skipped.
*/
+ protected def archiveEntryFilter(name: String): Boolean =
+ throw new UnsupportedOperationException(
+ s"${getClass.getName} does not support random-access archive reads")
+
+ /**
+ * Materializes each kept entry (those passing [[archiveEntryFilter]]) to a
temp file under
+ * `localDir`, lazily one at a time, so only one entry occupies disk at once.
Review Comment:
Please scope this guarantee to `readLocalizedEntries`, or document that
callers must delete each returned file before advancing. `localizeEntries`
creates a new temp file on every `next()` and does not remove earlier results,
so the helper itself does not ensure that only one entry occupies disk.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
