dybyte commented on issue #10856: URL: https://github.com/apache/seatunnel/issues/10856#issuecomment-4548389383
> metricsImap — high-cardinality, continuously updated per-task metrics. This one feels like it might be better served by a time-series store eventually, though that's a much larger conversation in its own right. @davidzollo My understanding may still be incomplete, so please correct me if I am missing something. From what I can tell, the original RocksDB discussion came from the `metricsImap` path, where we had already seen write contention and related performance concerns in #9833. At the moment, `metricsImap` still looks more like a task-keyed latest-snapshot store than a historical time-series store. Because of that, I am not yet convinced that a TSDB-style direction is the immediate fit here. If we want to preserve the current semantics, it seems to me that a key-value backend might be a more natural direction to evaluate. That said, I still do not know how much of the real production slowdown or memory pressure is directly caused by `metricsImap` itself. From the code, it looks like a plausible pressure point to me, but I do not yet have enough operational context to judge how significant it is in real deployments. If anyone has production experience, profiling results, or more context on how important the current snapshot-reuse behavior is, I would really appreciate it. For the persistence-backend side, I wonder if it makes sense to first evaluate the effect of #10399 before pushing the RocksDB discussion further. My current impression is that #10399 already addresses some of the known weaknesses of the current file-based path (especially around repeated updates / stale historical data), so it may be better to see what pain remains after that improvement, and then revisit whether a different backend is still necessary. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
