LuciferYang opened a new issue, #9570: URL: https://github.com/apache/paimon/issues/9570
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version master, `2788fe596` (2.1-SNAPSHOT). ### Compute Engine Flink and Spark, reading a Parquet file whose byte-array column uses the DELTA_LENGTH_BYTE_ARRAY page encoding, with a predicate that lets the reader skip values. parquet-mr never writes that encoding for BINARY, so the files come from other writers: Arrow C++ and pyarrow, parquet-rs, DuckDB. ### Minimal reproduce step Take a DELTA_LENGTH_BYTE_ARRAY page and drop a few bytes off its data section, so the lengths at the front add up to more than the page holds, then skip through it: ```java byte[] truncated = Arrays.copyOf(page, page.length - 3); reader.initFromPage(3, ByteBufferInputStream.wrap(ByteBuffer.wrap(truncated))); reader.skipBinary(3); // returns normally, ~0.9s later ``` `VectorizedDeltaLengthByteArrayReader.skipBinary`: ```java int remaining = lengthsVector.getInt(currentRow + i); while (remaining > 0) { remaining -= in.skip(remaining); } ``` Once the stream is dry, `ByteBufferInputStream.skip` returns -1, not 0, so `remaining -= -1` grows by one per iteration. The loop ends only when the int wraps negative, which is about 2^31 iterations and takes roughly 0.9s per value on my machine, and then `skipBinary` returns with no error. The page stream is empty by then, so a later read on the same page reports `Failed to read N bytes` against a value that is not the corrupt one, and if the skip was the page's last operation nothing is reported at all. ### What doesn't meet your expectations? The same page read rather than skipped fails cleanly: `readBinary` and `getBytes` both call `in.slice(length)` and turn the `EOFException` into `ParquetDecodingException("Failed to read N bytes")`. Only the skip path burns that CPU first and then either blames the wrong value or says nothing. A negative declared length is a second hole in the same loop, and that one is silent: `while (remaining > 0)` never enters, so nothing is skipped while `currentRow` advances, and the values after it are read from the wrong offsets. ### Anything else? The class came from Spark's vectorized reader in #4982. Spark still has the same loop, and parquet-mr's own non-vectorized `DeltaLengthByteArrayValuesReader.skip` does not: it calls `in.skipFully(length)` and converts the failure. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
