dybyte opened a new issue, #13039: URL: https://github.com/apache/seatunnel/issues/13039
### Search before asking - [x] I had searched in the [feature](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement. ### Description On a restart with `initial-mode: EAGER`, Hazelcast calls `FileMapStore.loadAllKeys()` once and then `loadAll(keys)` once for every partition that holds keys. `loadAll(keys)` ignores the keys and scans the whole map storage on every call (`mapStorage.loadAllWithExpirationTime()`), so a map is scanned `1 + partitions with keys` times, up to 272 with the default 271 partitions. The calls run in parallel, so each one builds its own copy of the whole map in memory at the same time. Measured with the real `FileMapStore` and an EAGER restart (8-core machine, warm page cache): | Keys | Storage size | Scans | Restart now | Scan once | |---|---|---|---|---| | 40 | 492 MB | 38 | 24.5 s | 5.3 s | | 1000 | 50 MB | 265 | 20.5 s | 2.4 s | ### Usage Scenario Scan once per map during the initial load and answer the other `loadAll(keys)` calls (and `loadAllKeys()`) from that result. ### Related issues #12022, #10329 ### Are you willing to submit a PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
