Hi Mika, all, Thanks for tracking down the regression. I reproduced the throughput loss on all four benchmarks you listed.
We hit the other side of this in production. After a bulk reload of the versioned table in an event-time temporal join, the timer column family had no live timers but still contained uncompacted deletion markers. When the job scaled back in, the restoring task sought once for each of its 32,768 key groups. Seeks for successive empty groups could repeatedly scan the same later markers. The task spent more than ten minutes initializing, most of it in that loop, and the job produced no output during the stall. Bounding the seeks significantly reduced restore time for this workload. The four affected MapState benchmarks measure the cost on populated maps. They don’t cover the timer initialization we observed after downscaling. Is there an existing benchmark for that case? I’m happy to take FLINK-41519 and investigate reducing the iterator setup cost. I’d also like to add a timer-queue initialization benchmark with neighboring tombstones so we can track both workloads. Best, Aleksandr On Fri, 20 Nov 2026 at 15:27, Yuepeng Pan via dev <[email protected]> wrote: > > Bump this email ! > > Thanks > Roc Marshal > > Mika Naylor via dev <[email protected]> 于2026年11月16日周一 23:10写道: > > > Hi all, > > > > flink benchmarks has shown a rocksdb regression in mapstate since about > > Sep 27. The release 2-4 team needs a PMC decision on whether this > > regression is acceptable for 2.4. > > > > The regressions are (ops/ms, baseline -> current): > > - mapEntries.ROCKSDB: 294.5 -> 280.9 (-4.6%) > > - mapIsEmpty.ROCKSDB: 38.0 -> 36.3 (-4.6%) > > - mapIsEmpty.ROCKSDB_CHANGELOG: 38.3 -> 36.0 (-5.9%) > > - mapIterator.ROCKSDB_CHANGELOG: 295.0 -> 283.2 (-4.0%) > > > > The cause seems to be FLINK-40584 [0], which bounds MapState and timer > > prefix iterators with an iterate_upper_bound. > > Each isEmpty, clear and MapState iterator load now creates and frees its > > own ReadOptions copy and native Slice. > > It looks like this adds a fixed cost per call, which I was able to > > replicate by reverting and unreverting the commits and running > > the benchmarks (mapIsEmpty sees the regression, mapGet unchanged). > > > > In light of this, I've created a regression ticket to track a fix [1] > > > > For release 2.4, should we accept this and ship as-is (maybe with a > > documented delta in the release notes) or revert the two commits from 2.4 > > if a fix is not present by our target date of Nov 24? > > > > Thanks a ton, > > Mika > > > > [0] https://issues.apache.org/jira/browse/FLINK-40584 > > [1] https://issues.apache.org/jira/browse/FLINK-41519 -- Kind regards, Aleksandr
