krismo-oai commented on code in PR #58176:
URL: https://github.com/apache/spark/pull/58176#discussion_r3917229493
##########
sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/WritableColumnVector.java:
##########
@@ -156,7 +156,7 @@ private void throwUnsupportedException(int
requiredCapacity, Throwable cause) {
"refer to " + SQLConf.ORC_VECTORIZED_READER_BATCH_SIZE().key() +
" (default " +
SQLConf.ORC_VECTORIZED_READER_BATCH_SIZE().defaultValueString() +
") and " + SQLConf.ORC_VECTORIZED_READER_ENABLED().key() + ".";
- throw new RuntimeException(message, cause);
+ throw new VectorizedReaderCapacityOverflowException(message, cause);
Review Comment:
### [P2] Keep malformed Parquet lengths in the corrupt-file path
Following up on the malformed-length case discussed above: this is still
reproducible at `0b78648c`.
The helper changed in `WritableColumnVector.java:159` is reachable for
malformed encoded lengths, not just genuine capacity exhaustion. For example, a
`DELTA_BYTE_ARRAY` page with prefix lengths `[0, -2]`, suffix lengths `[1, 1]`,
and suffix bytes `ab` successfully decodes its first value. The second value
then calls `appendBytes(-2, ...)`, which requests `reserve(1 + -2)` before
copying any bytes. This now throws `VectorizedReaderCapacityOverflowException`,
which `DataSourceUtils.shouldIgnoreCorruptFileException` explicitly excludes.
Consequently, a genuinely corrupt file aborts the query even with
`ignoreCorruptFiles=true`.
I reproduced this with a 246-byte Parquet file alongside a healthy file
across all eight combinations of V1/V2, row/batch delivery, and heap/off-heap
vectors. The reviewed head fails; substituting the exact base versions of
`WritableColumnVector` and `DataSourceUtils` correctly skips the corrupt file
and returns the healthy row. Restoring the head reproduces the failure again.
Please validate decoded lengths—including delta-prefix bounds against the
previous value—before they reach capacity arithmetic, while preserving the
dedicated exception for genuine overflow/allocation failures. The new test
manually constructs an `IllegalArgumentException`; it does not verify that
malformed input actually takes that path. A decoder or file-scan regression
test should cover this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
