iemejia commented on PR #55932:
URL: https://github.com/apache/spark/pull/55932#issuecomment-5509829447
Good point - updated the description. The **user-facing change** section now
covers all three affected encodings explicitly, not just
DELTA_LENGTH_BYTE_ARRAY:
1. **DELTA_LENGTH_BYTE_ARRAY `skipBinary`** on a truncated page - the
original unkillable-spin fix.
2. **PLAIN `skipBinary`** - `in.skip(len)` (return value ignored) ->
`in.skipFully(len)`: before, a truncated page skipped too few bytes and kept
decoding remaining values from the wrong position; now throws
`ParquetDecodingException`.
3. **Negative decoded lengths in PLAIN and DELTA_BYTE_ARRAY** - before, a
negative length silently rewound/no-op'd in PLAIN (moving `elementsAppended`
backwards) and failed with a different, unrelated exception in
DELTA_BYTE_ARRAY; now both throw `ParquetDecodingException("Encountered
negative length: ...")` before the length reaches a slice/skip or the column
vector.
Each case spells out the default (`ignoreCorruptFiles=false`) fail vs. the
`ignoreCorruptFiles=true` skip-rest-of-file behavior, and I also expanded "How
was this patch tested?" to list the DELTA_BYTE_ARRAY and PLAIN cases alongside
the DELTA_LENGTH ones. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
