dimakuz opened a new issue, #1757: URL: https://github.com/apache/iceberg-go/issues/1757
### Feature Request / Improvement `table/dv.RoaringPositionBitmap`'s only mutator is `Set(pos uint64)`, and `SerializeDV` writes containers as-is without run-length encoding. For writers whose deletion sets are range-shaped — deletes at block or row-group granularity, where a deletion vector is a handful of contiguous runs — this has two costs: 1. **Construction** is one `Set` call per deleted position: a multi-million-row dead range costs millions of calls where one range-add would do. 2. **Size**: incremental `Set` produces array/dense bitmap containers and nothing ever re-encodes them as runs, so a run-shaped DV serializes dramatically larger than the portable roaring format allows. Measured: 100,000 contiguous positions produce an 8,235-byte DV envelope; run-encoded, the same set is 49 bytes. Java Iceberg has both affordances and applies them by default: `RoaringPositionBitmap` declares `setRange(posStartInclusive, posEndExclusive)` and `runLengthEncode()`, and `BitmapPositionDeleteIndex.serialize` calls `runLengthEncode()` before serializing. The Go port has neither, so Go-written DVs are byte-inflated relative to Java's for the same positions (they remain spec-valid — run containers are flagged in the portable format's cookie header — just larger). Proposal, mirroring Java's names and layering: - `SetRange(startInclusive, endExclusive uint64)` — bucket-split across the high-32-bit keys, one roaring `AddRange` per bucket. - `RunLengthEncode()` — re-encode each bucket's containers as runs where smaller. - `SerializeDV` calls `RunLengthEncode()` before emitting, matching `BitmapPositionDeleteIndex.serialize`. The existing Java golden fixtures (including `all-container-types-position-index.bin`, which Java builds from range deletes and serializes run-encoded) continue to match byte-for-byte with this change — the all-container-types fixture only matches *because* of it. I'd like to work on this — implementation and tests are ready to submit. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
