JeremyXin commented on issue #10856: URL: https://github.com/apache/seatunnel/issues/10856#issuecomment-4450645804
I would like to add a few observations from my mind. 1. I have submitted PR #10860 to expose some IMap store metrics. This pr exposes per-store owned entries, backup entries, and heap cost. These metrics can help us better observe which IMap stores actually cause meaningful memory or backup overhead, so RocksDB candidates can be selected based on measured pressure instead of only by map name. 2. I think different IMap stores should be classified before discussing replacement. - Runtime-authoritative stores should stay on IMap for now, like `runningJobStateIMap`, `runningJobStateTimestampsIMap`, `ownedSlotProfilesIMap`, etc. These stores are related to failover, cleanup correctness, slot ownership, or shared reference counting. - History/query-oriented stores are better candidates, such as `finishedJobStateImap`, `finishedJobMetricsImap`, and `finishedJobDAGInfoImap`. 3. From my side, RocksDB may be worth evaluating only for auxiliary/history/query stores with measurable memory or backup pressure and weaker recovery requirements. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
