Akash3121 commented on code in PR #10716:
URL: https://github.com/apache/paimon/pull/10716#discussion_r4372612272
##########
paimon-core/src/main/java/org/apache/paimon/io/DataFileRecordReader.java:
##########
@@ -256,4 +270,76 @@ private static boolean ignoreCorruptException(Throwable e,
boolean ignoreCorrupt
|| e instanceof RuntimeException
|| e instanceof InternalError);
}
+
+ /**
+ * Row-based formats (CSV, JSON, TEXT, Avro, ...) decode lazily in {@link
#next()}, so a file
+ * that is truncated or corrupt in the middle only fails after {@link
#readBatch()} returned.
+ * This iterator extends {@code scan.ignore-corrupt-files} to that point:
the rows already
+ * returned are kept and the rest of the file is skipped.
+ */
+ private class IgnoreCorruptFileRecordIterator implements
FileRecordIterator<InternalRow> {
+
+ private final FileRecordIterator<InternalRow> iterator;
+
+ private
IgnoreCorruptFileRecordIterator(FileRecordIterator<InternalRow> iterator) {
+ this.iterator = iterator;
+ }
+
+ @Nullable
+ @Override
+ public InternalRow next() throws IOException {
+ if (corrupted) {
+ return null;
+ }
+ try {
+ return iterator.next();
+ } catch (Exception e) {
+ return handleCorruptException(e, null);
Review Comment:
`ignoreCorruptException` includes `InternalError` (notably `ZipError`), but
this catch only intercepts `Exception`, so that supported corrupt-file failure
still escapes and aborts the scan. Could we catch the same throwable set here
(and in `skip()`), with a regression test using `ZipError`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
