rdblue commented on code in PR #4325:
URL: https://github.com/apache/iceberg/pull/4325#discussion_r1129894273
##########
data/src/main/java/org/apache/iceberg/data/TableMigrationUtil.java:
##########
@@ -107,8 +176,13 @@ public static List<DataFile> listPartition(
.filter(FileStatus::isFile)
.collect(Collectors.toList());
DataFile[] datafiles = new DataFile[fileStatus.size()];
- Tasks.Builder<Integer> task =
-
Tasks.range(fileStatus.size()).stopOnFailure().throwFailureWhenFinished();
+ Tasks.Builder<Integer> task = Tasks.range(fileStatus.size());
+
+ if (skipOnError) {
+ task.suppressFailureWhenFinished();
Review Comment:
I think it would be better to skip based on specific failures. As this is
written, this would skip files when there is a transient S3 error, like a 503
"slow down". It would be better to handle errors like
`ParquetDecodingException` to skip instead.
I don't think that would be an easy thing to implement, so I would say that
this should at least log and should include retries for certain exceptions
types.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
