davidzollo commented on issue #10856: URL: https://github.com/apache/seatunnel/issues/10856#issuecomment-4442497854
Thanks @dybyte for getting this discussion started — it's a really thoughtful question to bring to the list, and I've been mulling it over for a few days. Sharing some thoughts below, mostly as a way to align on how we might frame the question together. I'm sure I have blind spots here, so genuinely happy to be corrected on any of it. One framing that's been helpful for me is to set aside "should we adopt RocksDB" for a moment, and instead start from where RocksDB tends to shine and where it tends to struggle as a piece of technology. Then mapping that against the IMaps we have today, the right scoping seems to fall out more naturally. ## Where RocksDB tends to be a strong fit From my reading of its design, RocksDB seems to be a good match when: - writes are high-frequency and values are small, kept local to a single node - automatic compaction is valuable (tombstone cleanup, bounded space amplification, and recovery in `O(current state size)` rather than `O(total historical writes)`) - checksums on read are useful for detecting silent corruption - sorted KV access with prefix or range scans is helpful - recovery time benefits from snapshot + WAL replay rather than full-history reconstruction These properties together make it feel well-suited for **node-local, high-write-frequency, small-value persistent state**. ## Where it tends to be less of a natural fit I think it's also worth being explicit about cases where RocksDB might not be the most natural choice — partly because I suspect some of the "replace IMap with RocksDB" framings run into one or more of these: - **Cross-node shared visibility** — RocksDB is master-local by design, so anything that needs to be readable across master transitions would likely need an additional replication or consensus layer on top. - **Large blobs (hundreds of KB or more)** — LSM compaction repeatedly rewrites large values across levels, which can cause noticeable write amplification. Shared filesystem or object storage often feels like a more natural home for large immutable blobs. - **High-cardinality time-series data** — purpose-built TSDBs (Prometheus, VictoriaMetrics, InfluxDB) are designed specifically for this workload and tend to be a better fit. - **Distributed coordination** — RocksDB provides local durability rather than distributed agreement, so it's probably not a direct substitute for something like Raft or etcd when coordination semantics are needed. ## Mapping this against the current IMaps If I try to apply this framework to each of the IMaps in Zeta, here's how I'd tentatively categorize them — and I'd genuinely love to hear if folks see these differently: - `runningJobStateIMap`, `runningJobStateTimestampsIMap`, `ownedSlotProfilesIMap`, `connectorJarRefCounters`, `pendingPipelineCleanupIMap` — small values, frequent updates, need to survive failover. These feel like they'd map fairly cleanly to what RocksDB does well (as a persistence backend behind the IMap; the IMap itself would stay for HA). - `runningJobInfoIMap` — values are full `JobImmutableInformation` blobs (~200KB each, see [[JobInfo.java:36](https://github.com/apache/seatunnel/blob/dev/seatunnel-engine/seatunnel-engine-core/src/main/java/org/apache/seatunnel/engine/core/job/JobInfo.java#L36)](https://github.com/apache/seatunnel/blob/dev/seatunnel-engine/seatunnel-engine-core/src/main/java/org/apache/seatunnel/engine/core/job/JobInfo.java#L36)), low write frequency, but need to survive master failover and be readable on the new master. This profile feels closer to shared filesystem — actually quite similar to how `CheckpointStorage` already works today. - `metricsImap` — high-cardinality, continuously updated per-task metrics. This one feels like it might be better served by a time-series store eventually, though that's a much larger conversation in its own right. - `finishedJob*` — one observation worth sharing: for streaming workloads these don't really grow much (since streaming jobs don't finish), so the often-suggested "cold-store finished history in RocksDB" direction may have less impact on dominant heap pressure than it first appears. The thing this exercise surfaces for me is that **our current IMaps actually span at least three pretty different access profiles**, and they may each want different answers. That's why I'm finding "replace IMap with RocksDB" as a single direction a bit hard to make precise — RocksDB seems to fit one of those profiles cleanly, but maybe not the others. # Three Possible Directions That Seem to Emerge If the framing above resonates, three separate questions seem to surface naturally. They feel reasonably independent to me, with different risk profiles, and I'd gently lean toward discussing them as separate threads rather than bundling them. But happy to hear if others see it differently or if there's a cleaner way to slice this. ## Direction 1 — Possibly evolving the IMap persistence backend This is where RocksDB seems like a natural candidate. I went and read through the current `IMapFileStorage` implementation, and there are a few characteristics that might make it less than ideal for the high-frequency-write IMaps it currently backs: - Synchronous `hsync` + `hflush` on every append ([[HdfsWriter.java:71-73](https://github.com/apache/seatunnel/blob/dev/seatunnel-engine/seatunnel-engine-storage/imap-storage-plugins/imap-storage-file/src/main/java/org/apache/seatunnel/engine/imap/storage/file/wal/writer/HdfsWriter.java#L71)](https://github.com/apache/seatunnel/blob/dev/seatunnel-engine/seatunnel-engine-storage/imap-storage-plugins/imap-storage-file/src/main/java/org/apache/seatunnel/engine/imap/storage/file/wal/writer/HdfsWriter.java#L71)) - No compaction; recovery reads the full historical WAL ([[WALReader.java:59-113](https://github.com/apache/seatunnel/blob/dev/seatunnel-engine/seatunnel-engine-storage/imap-storage-plugins/imap-storage-file/src/main/java/org/apache/seatunnel/engine/imap/storage/file/common/WALReader.java#L59)](https://github.com/apache/seatunnel/blob/dev/seatunnel-engine/seatunnel-engine-storage/imap-storage-plugins/imap-storage-file/src/main/java/org/apache/seatunnel/engine/imap/storage/file/common/WALReader.java#L59)) - Class names embedded as strings in the WAL format ([[IMapFileStorage.java:289-291](https://github.com/apache/seatunnel/blob/dev/seatunnel-engine/seatunnel-engine-storage/imap-storage-plugins/imap-storage-file/src/main/java/org/apache/seatunnel/engine/imap/storage/file/IMapFileStorage.java#L289)](https://github.com/apache/seatunnel/blob/dev/seatunnel-engine/seatunnel-engine-storage/imap-storage-plugins/imap-storage-file/src/main/java/org/apache/seatunnel/engine/imap/storage/file/IMapFileStorage.java#L289)) - Checksums are explicitly disabled ([[IMapFileStorage.java:157](https://github.com/apache/seatunnel/blob/dev/seatunnel-engine/seatunnel-engine-storage/imap-storage-plugins/imap-storage-file/src/main/java/org/apache/seatunnel/engine/imap/storage/file/IMapFileStorage.java#L157)](https://github.com/apache/seatunnel/blob/dev/seatunnel-engine/seatunnel-engine-storage/imap-storage-plugins/imap-storage-file/src/main/java/org/apache/seatunnel/engine/imap/storage/file/IMapFileStorage.java#L157)) - ORC is a columnar analytical format, which feels structurally less suited to KV WAL workloads than something LSM-based - Interestingly, the class Javadoc itself notes it is "not suitable for frequently updated scenarios" ([[IMapFileStorage.java:62-66](https://github.com/apache/seatunnel/blob/dev/seatunnel-engine/seatunnel-engine-storage/imap-storage-plugins/imap-storage-file/src/main/java/org/apache/seatunnel/engine/imap/storage/file/IMapFileStorage.java#L62)](https://github.com/apache/seatunnel/blob/dev/seatunnel-engine/seatunnel-engine-storage/imap-storage-plugins/imap-storage-file/src/main/java/org/apache/seatunnel/engine/imap/storage/file/IMapFileStorage.java#L62)) — and yet `runningJobStateIMap` is one of the higher-frequency write paths today None of this is meant as criticism of the original implementation — it solved a real problem at the time. But it might be a good moment to revisit, especially as workloads have evolved. The nice thing here is that the `IMapStorage` SPI already exists in the codebase, so this would be **adding a new implementation rather than introducing new abstractions**. The IMap's in-memory role and HA semantics would stay unchanged. Of the three directions, this one feels the most concrete and lowest-risk to me — though I'd love to hear other perspectives. ## Direction 2 — A possible state access abstraction layer This one is less about RocksDB specifically and more about whether the current code organization might be making future changes harder than they need to be. Today, `IMap.get()` and `IMap.put()` calls are spread across many of the coordination classes. This means any backend evolution, any state relocation, or any per-category optimization has to touch a lot of files. It also flattens the different access profiles into a single generic KV API, which I suspect is part of what's been making this whole conversation harder to scope cleanly. I'd gently lean toward something like a `StateStore` interface modeled on SeaTunnel business semantics (e.g., `lookupJobState`, `updatePipelineState`, `claimSlot`) rather than a generic `get/put` — because a generic KV interface would just move the coupling point without really solving the structural piece. But this is just one possible direction, and probably worth its own discussion if folks think the underlying observation has merit. This direction wouldn't change memory footprint or performance on its own — it's more of a structural enabler that could make the other directions safer to land. ## Direction 3 — Possibly moving large job metadata out of IMap This is where RocksDB feels like less of a fit — the access pattern seems to point more toward shared filesystem. `runningJobInfoIMap` looks like one of the larger sources of IMap heap usage under streaming workloads (rough back-of-envelope: ~400 MB for 1000 jobs at 200KB per blob × backup replica, growing linearly with job count and independent of parallelism). Its access pattern — write once at submission, read at failover or restart, immutable after that — actually looks quite similar to how `CheckpointStorage` works today. The cross-node visibility point matters here: anything that needs to survive master failover and be readable on the new master probably wants a backend that's reachable cross-node, which is part of why I'd lean toward shared filesystem (HDFS / S3 / NFS) here rather than RocksDB. That would mean shared filesystem becomes a deployment prerequisite, which has its own implications worth a separate conversation — including for single-node and Docker deployments. Of the three directions, this one has the largest operational implications, which is part of why I'd gently suggest keeping it on its own thread. --- # Possible Next Step (only if this framing resonates) If the framing above feels reasonable, one possible next step might be to: 1. See if we can align on the "where RocksDB fits / doesn't fit" mapping first 2. Then optionally split the discussion into separate threads — with Direction 1 probably being the most concrete place to start if there's interest But honestly, I'm really curious what others think — especially if anyone sees the mapping differently, thinks I've miscategorized one of the IMaps, or feels the three-directions split is artificial. There might well be a cleaner way to slice this that I'm not seeing. Thanks again @dybyte for kicking this off — really glad we're having this conversation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
