DanielLeens commented on issue #10856: URL: https://github.com/apache/seatunnel/issues/10856#issuecomment-4550521137
Thanks, this is a useful refinement. I agree that `metricsImap` is not a historical TSDB in its current behavior. In the current runtime path, `SeaTunnelServer.updateMetrics(...)` groups task metrics by partition and updates a `Map<TaskLocation, SeaTunnelMetricsContext>` inside `metricsImap`, so semantically it is much closer to a latest-snapshot KV store than to append-only time-series history. That also strengthens your point that #10399 should probably be evaluated before expanding the RocksDB discussion again. The current file-based `IMapFileStorage` already documents that it is not suitable for frequently updated scenarios, so an improvement on that path may remove part of the pressure without introducing a new backend immediately. From my side, the design boundary is still the same: 1. keep runtime-authoritative maps such as `runningJobStateIMap`, `runningJobStateTimestampsIMap`, `ownedSlotProfilesIMap`, and pending cleanup state out of the first RocksDB scope; 2. treat `metricsImap` as a separate question from finished-job history maps, because its semantics are live latest-snapshot reuse, not just cold history retention; 3. if #10399 lands, re-evaluate with actual measurements which pressure remains, and only then decide whether a narrowed RocksDB experiment is still justified. So I think the most productive next step is exactly what you suggested: use the observability work plus the post-#10399 behavior to narrow the candidate set, instead of treating “RocksDB vs no RocksDB” as one broad yes/no decision. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
