dybyte opened a new issue, #13048: URL: https://github.com/apache/seatunnel/issues/13048
### Search before asking - [x] I had searched in the [feature](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement. ### Description ## Description The IMap load and recovery benchmarks appear to measure Hazelcast's wait loop rather than the load itself. Their scores are whole seconds plus a few tens of milliseconds, and they barely change with the amount of data loaded. | Benchmark | Parameters | Score | | --- | --- | ---: | | `IMapDagStorageBenchmark.finishedJobDagLoad` (one key) | all 6 combinations | 2.0016–2.0018 s | | `IMapJobStorageBenchmark.runningJobRecovery` | `runningJobCount=100` / `1000` | 2.03 s / 2.04–2.17 s | | `IMapWalStorageBenchmark.recoverAll` | `uniqueKeyCount=100`, any `mutationsPerKey` | 2.03–2.04 s | | `IMapWalStorageBenchmark.recoverAll` | `uniqueKeyCount=1000`, `mutationsPerKey=1` / `10` / `100` | 3.04 s / 3.05 or 4.06 s / 9.1 or 25 s | Benchmark runs: - https://github.com/apache/seatunnel/actions/runs/33723940255 (`finishedJobDagLoad`) - https://github.com/apache/seatunnel/actions/runs/39386767737 (`runningJobRecovery`) - https://github.com/apache/seatunnel/actions/runs/33771937553 (`recoverAll`) Because the scores move in 1 s steps, a change smaller than one step either does not show up or shows up as a full second. For example, in the same run `recoverAll` with `uniqueKeyCount=1000, mutationsPerKey=10` scored 4.06 s on `dev` and 3.05 s on #11494. ## Suspected cause All three benchmarks call `IMap.loadAll(..., true)`, which ends in `MapProxySupport.waitUntilLoaded()` (Hazelcast 5.1). Its `waitAllTrue()` asks every partition whether it has finished loading, asks the unfinished ones again, sleeps 1 s, and then checks the answers it got before sleeping. So a load that does not finish within a few milliseconds takes at least 2 s, and a longer one is rounded up to the next second. ## Goals 1. Confirm where the measured time goes, for example with a wall profile showing the time spent in `waitAllTrue`. 2. Change the benchmarks so the score reflects the load cost, keeping the recovery validation. 3. Establish a new baseline with the corrected benchmarks. ### Usage Scenario _No response_ ### Related issues - Parent: #12086 - #12060 covers `finishedJobDagStore` in the same benchmark class. ### Are you willing to submit a PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
