LuciferYang opened a new issue, #9613: URL: https://github.com/apache/paimon/issues/9613
### Search before asking - [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar. ### Paimon version master, `475be566f` (2.1-SNAPSHOT). ### Compute Engine Any. The failure is in the bitmap index writer in paimon-common, so it is reachable from every engine that writes a table carrying `file-index.bitmap.columns` on a variable-length column. ### Minimal reproduce step Put a bitmap file index on a STRING column and write one row whose value serializes to more than `index-block-size` minus 16 bytes. With the default 16kb that threshold is 16368 bytes, so a single row holding a long text field or a 20KB JSON blob is enough: ```sql CREATE TABLE t (id INT, payload STRING) WITH ('file-index.bitmap.columns' = 'payload'); -- payload longer than 16368 bytes INSERT INTO t VALUES (1, '<20000 characters>'); ``` Driving `BitmapFileIndex`'s writer directly with such a value (`BitmapFileIndexTest` style, `index-block-size` left at its 16kb default) fails at serialization time: ``` java.lang.RuntimeException: java.lang.RuntimeException: index fail at org.apache.paimon.fileindex.bitmap.BitmapFileIndex$Writer.serializedBytes(BitmapFileIndex.java:177) ``` From the table side that call is `DataFileIndexWriter.close` -> `serializeMaintainers` -> `FileIndexMaintainer.serializedBytes` -> `BitmapFileIndex$Writer.serializedBytes`, reached when the data file writer closes, so the data file it belongs to is not written either. The packing loop in `BitmapFileIndexMetaV2.serialize` asks the current block to take the entry and, if the block refuses, opens one new block and retries exactly once: ```java BitmapIndexBlock last = indexBlocks.peekLast(); if (!last.tryAdd(e)) { BitmapIndexBlock next = new BitmapIndexBlock(last.offset + last.serializedBytes); indexBlocks.add(next); if (!next.tryAdd(e)) { throw new RuntimeException("index fail"); } } ``` `tryAdd` refuses on size alone, without asking whether the block already holds anything: ```java int entryBytes = 2 * Integer.BYTES + keyBytesMapper.apply(entry.key); if (serializedBytes + entryBytes > blockSizeLimit) { return false; } ``` An entry too large for an empty block is therefore too large for the fresh block as well, and the retry cannot succeed. The 16-byte margin is where the numbers come from: a block starts at `Integer.BYTES` for its entry count, and a STRING entry costs `2 * Integer.BYTES` plus the `Integer.BYTES + sizeInBytes` that `getSerializeSizeMeasure` reports. Fixed-width types cannot approach the limit, so only the variable-length ones are exposed. ### What doesn't meet your expectations? One long value should not fail the write. `index-block-size` is a packing target, not a maximum record size, and an oversized key can have a block to itself: nothing on the read side depends on a block staying inside the limit. Block offsets are accumulated from each block's real `serializedBytes`, a block's extent is implied by the entry count written ahead of it, and the block keys stay sorted, so `findBlock`'s binary search still lands on the right one. The sibling implementations in this repo already read the limit that way. `BitmapGlobalIndexFormat` only enforces its dictionary block size once the block `hasEntries()`, and `ChunkedDictionary` passes each chunk's first key through the chunk constructor instead of `tryAdd`, so an oversized key gets a chunk of its own. `BitmapFileIndexMetaV2` is the one size-limited packing loop left that still rejects an entry on an empty block. ### Anything else? `RuntimeException("index fail")` names no column, no value length and no limit, so a user sees a write failure with nothing in it to act on. Worth improving whether or not the packing is changed. ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
