HyukjinKwon commented on code in PR #59117:
URL: https://github.com/apache/spark/pull/59117#discussion_r4131725984
##########
sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java:
##########
@@ -976,14 +984,26 @@ private int readIntLittleEndianPaddedOnBitWidth() throws
IOException {
}
/**
- * Reads the next group. Returns false if no more group available.
+ * Reads the next non-empty group, so that `currentCount` is positive on
return.
+ * <p>
+ * Callers never read more values than the page declares, so running out of
encoded data here
+ * means the page is corrupted. With a bit width of 0, `initFromPage`
already provides all the
+ * values of the page, so reaching here also means reading past the end.
*/
- private boolean readNextGroup() {
- if (in.available() <= 0) {
- currentCount = 0;
- return false;
- }
+ private void readNextGroup() {
+ do {
+ if (bitWidth == 0 || in.available() <= 0) {
Review Comment:
The "fail on truncated input" goal has one remaining gap: a run header whose
varint is truncated at end-of-page still hangs rather than throwing.
This guard rejects an empty stream, but a stream ending in a lone
continuation byte (e.g. a trailing `0x80`) passes it, and the `readGroup()`
below then loops forever in `readUnsignedVarInt()`: at EOF `in.read()` returns
`-1`, and `(-1 & 0x80) != 0` is always true, so the `do/while` never
terminates. It's reachable on a corrupt V1 level section (e.g. a declared
length of 1 whose only byte is `0x80`).
This is pre-existing — the old `readNextGroup` had the same guard and called
the same `readUnsignedVarInt` — so it's a "fold in here or follow-up?" like the
short-last-group case. The minimal fix is to treat EOF as corruption in the
varint loop:
```java
b = in.read();
if (b < 0) {
throw new ParquetDecodingException(
"Corrupted RLE data: reading past the end of the encoded values");
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
