viirya commented on PR #59087: URL: https://github.com/apache/spark/pull/59087#issuecomment-5864189936
Thanks for the thorough review, @dongjoon-hyun. All addressed: - **Oversized lengths.** PLAIN `readLength()` now also rejects a length larger than `in.available()`, so `readGeoData` fails instead of getting a short array from `readNBytes`. For DELTA_LENGTH_BYTE_ARRAY this is covered by the up-front validation below, which also bounds what `slice` can allocate on a `MultiBufferInputStream`. - **Validating DELTA_LENGTH_BYTE_ARRAY lengths in `initFromPage`.** Done as you suggested: the count must not exceed `valueCount`, every length must be non-negative, and their sum must fit in the bytes left in the page. `readBinary`, `readGeoData`, `skipBinary` and `getBytes` then only check that the rows they consume were decoded, replacing the per-use `checkLength` calls. The same count issue existed for the prefix lengths in `VectorizedDeltaByteArrayReader`, so it now also rejects more prefix lengths than values, or a prefix count that differs from the suffix count. Otherwise a row past the decoded prefixes silently read a zero prefix length. - **Tests.** Added coverage for the geo paths (negative and oversized lengths in PLAIN, and undecoded rows in DELTA_LENGTH_BYTE_ARRAY), plus the new count and row checks. The new tests fail without this change. - **Nits.** The messages now say "DELTA_LENGTH_BYTE_ARRAY data", since DELTA_BYTE_ARRAY suffixes use that encoding. `skipBooleans` uses the `skipFully` helper, the DELTA_LENGTH_BYTE_ARRAY read errors keep their cause, and the comment now says the position only moves backwards on a `SingleBufferInputStream`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
