srowen commented on code in PR #38024:
URL: https://github.com/apache/spark/pull/38024#discussion_r983008731
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FilePartitionReader.scala:
##########
@@ -36,8 +36,15 @@ class FilePartitionReader[T](
private def ignoreMissingFiles = options.ignoreMissingFiles
private def ignoreCorruptFiles = options.ignoreCorruptFiles
+ private def ignoreCorruptFilesAfterRetries =
options.ignoreCorruptFilesAfterRetries
override def next(): Boolean = {
+
+ def shouldSkipCorruptFiles(): Boolean = {
Review Comment:
Well, some kind of file corruption is just one reason for an error, and
surely the config is meant to denote any inability to correctly read data.
You're really distinguishing between transient and permanent failures, but,
IOException doesn't tell you that.
It's not a correctness issue, as you've already said you're willing to
ignore data that can't be read (otherwise, don't specify this, right?). Now
that's being replaced with "data that can't be read if I try N times" which
isn't fundamentally different.
Or I could spin a different argument: if the data can't be read 2 times, and
is read a 3rd time, are you OK with that being correct?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
