dybyte opened a new issue, #10856: URL: https://github.com/apache/seatunnel/issues/10856
## Background There was a previous issue about replacing IMap with RocksDB: #9851. I would like to start a focused discussion about whether RocksDB is valuable for Zeta at this stage, and what scope would make sense if we ever introduce it. Currently, Hazelcast IMap is used for different kinds of engine state. Some state is part of runtime correctness and failover recovery, while some state is auxiliary and mainly used for metrics or history queries. Because of this, I think "replace IMap with RocksDB" may be too broad as a direction. ## Why RocksDB may help RocksDB may help for heavy auxiliary state because it can keep large payloads local to the active master instead of storing them as distributed IMap values. This may reduce Hazelcast memory usage, serialization cost, backup replication cost, network overhead, and cleanup cost. Possible candidates may include: - large metrics payloads - metrics snapshots - finished metrics - finished job vertex/DAG details - other history/query-only payloads ## Concern However, RocksDB itself only provides local durability. If RocksDB is used as a master-local backend, it does not automatically provide the same distributed visibility, backup replication, failover recovery, or migration semantics that Hazelcast IMap provides. It may work with an asynchronous external persistence or backup path, but that still gives a different recovery model from IMap. For example, if the active master node is lost before data is persisted externally, some RocksDB-backed data may be unavailable or lost. So I think the key question is whether local durability plus eventual external recovery is sufficient for any current IMap usage. For authoritative runtime state, RocksDB alone is not enough. Replacing that kind of IMap state would require a stronger distributed state-store design, including ownership, write ordering, replication, commit, failover, catch-up, and stale-state handling. That seems much larger than simply adding a RocksDB backend. ## My current thought My current preference is: - keep IMap for runtime authoritative state; - evaluate RocksDB only for selected heavy auxiliary state; - define the minimum durability, availability, and recovery guarantees for each candidate state; - define the backup/recovery behavior before introducing RocksDB; - avoid treating RocksDB as a direct replacement for IMap semantics. I am not proposing to introduce RocksDB immediately. My goal is to clarify whether there is any suitable scope where RocksDB provides enough benefit to justify the additional complexity. ## Questions 1. Do we have any current IMap usage where RocksDB would provide clear benefits, such as reducing memory usage, serialization cost, backup replication cost, network overhead, or cleanup cost? 2. If RocksDB is used as a master-local backend with asynchronous external persistence/backup, which IMap usages, if any, can accept its weaker recovery and availability model? 3. If no clear candidate exists, should we keep the current IMap-based approach and focus on optimizing the existing update/cleanup paths instead? Any feedback or correction would be appreciated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
