sunchao commented on code in PR #54805:
URL: https://github.com/apache/spark/pull/54805#discussion_r3713986001
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSourceUtils.scala:
##########
@@ -200,7 +200,10 @@ object DataSourceUtils extends PredicateHelper {
}
def shouldIgnoreCorruptFileException(e: Throwable): Boolean = e match {
- case _: RuntimeException | _: IOException | _: InternalError => true
+ case _: RuntimeException | _: IOException | _: InternalError =>
+ val m = e.getMessage
+ m == null || !m.contains(
Review Comment:
[P2] Preserve ignoreCorruptFiles for malformed DELTA_BYTE_ARRAY pages
This message is not unique to genuine reader-capacity exhaustion. A corrupt
DELTA_BYTE_ARRAY page with prefix lengths [0, -1] and empty suffixes decodes
the first value successfully, then VectorizedDeltaByteArrayReader calls
arrayData.appendBytes(-1, previous.array(), previous.position()) for the
second. WritableColumnVector.appendBytes consequently calls reserve(-1), which
throws the same 'Cannot reserve additional contiguous bytes ... (integer
overflow)' exception excluded here. The base version skips this genuinely
corrupt file when ignoreCorruptFiles=true, but this change aborts the entire
scan. Please validate malformed encoded lengths as file corruption, or
otherwise distinguish corrupt-input overflows from real capacity exhaustion,
and add a regression test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
