Hi Mika, all,

Thanks for tracking down the regression. I reproduced the throughput
loss on all four benchmarks you listed.

We hit the other side of this in production. After a bulk reload of
the versioned table in an event-time temporal join, the timer column
family had no live timers but still contained uncompacted deletion
markers. When the job scaled back in, the restoring task sought once
for each of its 32,768 key groups. Seeks for successive empty groups
could repeatedly scan the same later markers. The task spent more than
ten minutes initializing, most of it in that loop, and the job
produced no output during the stall. Bounding the seeks significantly
reduced restore time for this workload.

The four affected MapState benchmarks measure the cost on populated
maps. They don’t cover the timer initialization we observed after
downscaling. Is there an existing benchmark for that case?

I’m happy to take FLINK-41519 and investigate reducing the iterator
setup cost. I’d also like to add a timer-queue initialization
benchmark with neighboring tombstones so we can track both workloads.

Best,
Aleksandr

On Fri, 20 Nov 2026 at 15:27, Yuepeng Pan via dev <[email protected]> wrote:
>
> Bump this email !
>
> Thanks
> Roc Marshal
>
> Mika Naylor via dev <[email protected]> 于2026年11月16日周一 23:10写道:
>
> > Hi all,
> >
> > flink benchmarks has shown a rocksdb regression in mapstate since about
> > Sep 27. The release 2-4 team needs a PMC decision on whether this
> > regression is acceptable for 2.4.
> >
> > The regressions are (ops/ms, baseline -> current):
> > - mapEntries.ROCKSDB: 294.5 -> 280.9 (-4.6%)
> > - mapIsEmpty.ROCKSDB: 38.0 -> 36.3 (-4.6%)
> > - mapIsEmpty.ROCKSDB_CHANGELOG: 38.3 -> 36.0 (-5.9%)
> > - mapIterator.ROCKSDB_CHANGELOG: 295.0 -> 283.2 (-4.0%)
> >
> > The cause seems to be FLINK-40584 [0], which bounds MapState and timer
> > prefix iterators with an iterate_upper_bound.
> > Each isEmpty, clear and MapState iterator load now creates and frees its
> > own ReadOptions copy and native Slice.
> > It looks like this adds a fixed cost per call, which I was able to
> > replicate by reverting and unreverting the commits and running
> > the benchmarks (mapIsEmpty sees the regression, mapGet unchanged).
> >
> > In light of this, I've created a regression ticket to track a fix [1]
> >
> > For release 2.4, should we accept this and ship as-is (maybe with a
> > documented delta in the release notes) or revert the two commits from 2.4
> > if a fix is not present by our target date of Nov 24?
> >
> > Thanks a ton,
> > Mika
> >
> > [0] https://issues.apache.org/jira/browse/FLINK-40584
> > [1] https://issues.apache.org/jira/browse/FLINK-41519



-- 
Kind regards,
Aleksandr

Reply via email to