DanielLeens commented on issue #10856: URL: https://github.com/apache/seatunnel/issues/10856#issuecomment-4456135776
Thanks for adding this. PR #10860 looks useful as an observability step. I agree that better per-store metrics can make this discussion much more concrete, because today we are still mixing together several very different state categories. The important boundary is still the same from my side: - runtime-authoritative maps such as `runningJobStateIMap`, `runningJobStateTimestampsIMap`, `ownedSlotProfilesIMap`, and related cleanup/correctness state should stay out of the first RocksDB evaluation scope; - history / query-oriented maps such as `finishedJobStateImap`, `finishedJobMetricsImap`, and `finishedJobDAGInfoImap` are much better first candidates; - `metricsImap` is still in the middle, so it should be judged from measured pressure plus an explicit freshness / failover contract, not only from name or intuition. One reason to stay careful here is that the current file-based IMap storage itself already documents that it is not suitable for frequently updated scenarios, so better measurement is useful, but measurement alone does not automatically make a runtime-authoritative map a good RocksDB target. So I would treat PR #10860 as an enabling observability step, not as implicit approval for a backend switch. If the metrics show real pressure, the next step should still be a narrowed proposal around auxiliary/history/query stores first. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
