DanielLeens commented on issue #10856: URL: https://github.com/apache/seatunnel/issues/10856#issuecomment-4414923646
Thanks for framing this so carefully. After rechecking the current Zeta state paths, I agree this is closer to a design topic than a normal feature request. Today the code is already split into very different kinds of Hazelcast state: - authoritative runtime / failover state, such as `runningJobInfoIMap`, `runningJobStateIMap`, `runningJobStateTimestampsIMap`, `ownedSlotProfilesIMap`, and the pending cleanup records; - history / query-oriented state, such as finished job state, finished job metrics, and finished job DAG info. Because of that, I do not think "replace IMap with RocksDB" is the right top-level direction. My current take is: 1. `runningJobInfoIMap`, `runningJobStateIMap`, `runningJobStateTimestampsIMap`, `ownedSlotProfilesIMap`, and pending cleanup records are poor RocksDB candidates. They are on the master failover / recovery correctness path today, so a master-local store would weaken semantics unless we introduce a much larger distributed state-store design. 2. `finishedJobStateImap`, `finishedJobMetricsImap`, and `finishedJobDAGInfoImap` are better first candidates. They are much closer to history/query payloads with TTL, so the semantic gap is smaller if we explicitly accept weaker availability during master loss. 3. `metricsImap` sits in the middle. It is more auxiliary than authoritative task state, but it is still updated live and read cluster-wide, so the recovery / freshness contract needs to be written down before changing its backend. If we want to move this forward, I think the next useful step is a small design note or STIP-style matrix for each candidate map: - who writes it - who reads it - whether it is on the correctness path or only a query/history path - what visibility is required after master failover - whether temporary loss is acceptable - cleanup / TTL behavior My recommendation would be to keep runtime-authoritative state on IMap for now, and only evaluate a very small MVP around finished-job history / heavy query-only payloads first. If there is no candidate that can tolerate the weaker recovery model, optimizing the existing IMap update/cleanup path is probably the safer direction. Happy to review a narrowed proposal if you want to continue this discussion. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
