LuciferYang opened a new pull request, #9614:
URL: https://github.com/apache/paimon/pull/9614
### Purpose
close #9613
`BitmapFileIndexMetaV2.BitmapIndexBlock.tryAdd` refused an entry whenever it
pushed the block past `index-block-size`, without asking whether the block
already held anything. `serialize` answers a refusal by opening one new block
and retrying once, and that block is empty too, so the same entry is refused
again and the write ends at `throw new RuntimeException("index fail")`. A
single dictionary key larger than the limit therefore fails the whole data
file, not just its index.
The threshold is `index-block-size` minus 16 bytes: a block starts at
`Integer.BYTES` for its entry count, and a STRING entry costs `2 *
Integer.BYTES` plus the `Integer.BYTES + sizeInBytes` that
`getSerializeSizeMeasure` reports. With the default 16kb that means one
16369-byte value, and a single row carrying it is enough. Fixed-width types
cannot get close to the limit, so this is a variable-length column problem.
An empty block now takes its first entry unconditionally, so an oversized
key gets a block to itself and the entries after it spill over as before. That
treats the limit as the packing target it is, which is also how the two sibling
implementations read it: `BitmapGlobalIndexFormat.write` only enforces its
dictionary block size once the block `hasEntries()`, and
`ChunkedDictionary.append` passes each chunk's first key through the chunk
constructor rather than `tryAdd`.
Nothing on the read side depends on a block fitting inside the limit. Block
offsets accumulate from each block's real `serializedBytes`, the entry count
written ahead of a block bounds it, and the block keys stay sorted, so
`findBlock`'s binary search still lands on the right block. Inputs that used to
serialize produce identical bytes, since an entry an empty block rejected could
not produce a file at all before.
One thing I left alone: now that an empty block never refuses, the `index
fail` throw in `serialize` is unreachable. Removing it means restructuring that
loop, which felt like more than this fix should carry, but I'm glad to fold it
in if you would rather see it go.
### Tests
`BitmapFileIndexTest.testV2EntryLargerThanBlockSize` builds an index over a
20000-character value, a null and two short values at the default 16kb block
size, then reads all four back through the reader. The two short keys share the
first block, so the size check runs and admits the second one; the long key is
rejected there and then accepted as the first entry of the next block. Both
sides of the new condition run within one index.
Against the unfixed writer the test errors with `RuntimeException: index
fail` out of `BitmapFileIndex$Writer.serializedBytes`.
`mvn -pl paimon-common -Dtest=BitmapFileIndexTest test` on JDK 8: 7 tests, 0
failures. `spotless:check` and `checkstyle:check` on paimon-common are clean.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
