LuciferYang opened a new issue, #9643: URL: https://github.com/apache/paimon/issues/9643
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version master, `475be566f` (2.1-SNAPSHOT). ### Compute Engine Any engine reading Parquet through the vectorized reader, for a column encoded with `DELTA_BYTE_ARRAY` (Parquet v2) where the reader skips values, which happens when a row filter gives the reader a row-index list. ### Minimal reproduce step Skip an odd number of values in a `DELTA_BYTE_ARRAY` column, then skip again, then read. The value that comes back has NUL bytes where its prefix should be: ```java String[] vals = new String[] {"aaaa", "aaab", "aaac", "aaad"}; Utils.writeData(writer, vals); reader.initFromPage(vals.length, writer.getBytes().toInputStream()); reader.skipBinary(1); reader.skipBinary(1); reader.readBinary(0); // "\0\0\0c" instead of "aaac" ``` `skipBinary` alternates two vectors and leaves `previous` pointing at the buffer of whichever one it wrote last: ```java c1.reset(); ... c1.putByteArray(0, bytes, 0, length); BytesColumnVector.Bytes b = c1.getBytes(0); previous = ByteBuffer.wrap(b.data, b.offset, b.len); ``` After an odd number of skipped values `previous` points into `tempBinaryValVector`, and the next `skipBinary` call starts by resetting that same vector. `HeapBytesVector.reset()` zeroes the data buffer: ```java // We don't reset buffer to avoid unnecessary copy. Arrays.fill(buffer, (byte) 0); ``` so the prefix copied out of `previous` right after is all zeros. No error is raised; the value is simply wrong. ### What doesn't meet your expectations? Two things. Reading a value back should not depend on how many values were skipped before it. And `reset()` should not wipe the data buffer at all: the comment one line above says it does not, every read is bounded by the `start` and `length` arrays that `reset()` does clear, and wiping costs O(buffer size) per batch on the vectorized read path. The `Arrays.fill` arrived with the reader rework in #4982, contradicting the comment it sits under. ### Anything else? What I did not establish is whether Paimon's own writer produces `DELTA_BYTE_ARRAY` (that needs the v2 writer) and whether `readNextFilteredRowGroup` is reached in a normal read. So the corruption may need an externally written v2 file plus a row filter; the encoding is supported on the read side either way. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
