nzw921rx commented on issue #10856: URL: https://github.com/apache/seatunnel/issues/10856#issuecomment-4474676062
> I'd like to mention what dybyte is currently working on: The authoritative / auxiliary split in STIP-26 is already doing the structural work my "Direction 2" was reaching for — probably more cleanly than my framing. > > [#10812](https://github.com/davidzollo/seatunnel/issues/10812) is essentially that direction in flight, with the interface layer already well thought through > > [#10399](https://github.com/davidzollo/seatunnel/issues/10399) addresses the persistence-layer concerns I was raising via a self-rolled LSM/compaction approach — which is a perfectly reasonable answer to the same problem RocksDB would address. I should have been clearer that RocksDB is one possible backend among several, not "the" path forward. I'd like to point out that RocksDB is suitable for this task. > I agree with @davidzollo that RocksDB is not the only option for a local KV database. However, there are several technical challenges and concerns regarding long-term maintainability in developing a custom in-house LSM storage solution for SeaTunnel, which I suggest we thoroughly evaluate during our discussions. **Technical Challenges** - Durability and Consistency: A self-developed solution must strictly guarantee data durability and consistency in the event of crashes or abnormal interruptions, preventing data loss or exposure of inconsistent views. - Compaction and Amplification Trade-offs: The merge scheduling in LSM trees involves navigating long-term trade-offs among write amplification, read amplification, and space amplification. We need to assess whether the current strategy is well-suited for our primary workloads and ensure we have a clear plan for dynamic optimization and policy adjustment. - Concurrency and Snapshot Consistency: With background flushing, merging, and file switching, it is crucial to ensure that foreground reads can always obtain consistent snapshot data, minimizing race conditions and read anomalies. **Maintainability** Developing a custom LSM solution introduces an additional persistent local storage layer, which will require sustained investment in development and maintenance efforts, including manpower, security, compatibility, over the long term. In contrast, adopting a mature KV component allows us to leverage the community’s existing best practices and utilize well-established solutions for common issues. With a custom solution, however, the community would need to take on ongoing development and maintenance responsibilities internally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
