COLDMANYU opened a new pull request, #10716:
URL: https://github.com/apache/paimon/pull/10716
### Purpose
`scan.ignore-corrupt-files` only catches failures while a file is opened and
while
`readBatch()` is called. Row-based formats (CSV, JSON, TEXT, Avro) return a
lazy iterator from
`readBatch()` and do the actual decoding in `next()`, so a file that is
corrupt in the middle
escapes the option and fails the whole query.
A common case is a truncated gzip file in a Format Table: the file opens
fine, and
`java.io.EOFException: Unexpected end of input stream` is only thrown when
decompression reaches
the cut. Spark's `spark.sql.files.ignoreCorruptFiles` does not help either,
because a Paimon
Format Table is read by Paimon's own reader rather than Spark's file scan.
This PR makes `DataFileRecordReader` also guard the iterator it returns when
`scan.ignore-corrupt-files` is enabled. When `next()`/`skip()` fails with an
exception the option
already treats as corruption, the rows returned so far are kept, a warning
is logged, and the rest
of the file is skipped. This matches the semantics of Spark's
`ignoreCorruptFiles`.
Notes:
- Nothing changes when the option is disabled (the default): the iterator is
not wrapped.
- Columnar iterators (`ColumnarRowIterator`, Parquet/ORC/...) are left
unwrapped. They are fully
materialized in `readBatch()`, which is already covered, and callers
detect vectorized batches
by their type.
- The set of exceptions treated as corruption is unchanged.
### Tests
- `FormatTableIgnoreCorruptFilesTest` (new): a Format Table with a gzip file
truncated in the
middle plus an intact file, for CSV and JSON. Without the option the read
fails with
`EOFException`; with the option the rows before the cut and all rows of
the intact file are
returned. The test fails on master and passes with this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
