davidzollo commented on issue #10856: URL: https://github.com/apache/seatunnel/issues/10856#issuecomment-4442664658
I'd like to mention what dybyte is currently working on: The authoritative / auxiliary split in STIP-26 is already doing the structural work my "Direction 2" was reaching for — probably more cleanly than my framing. [#10812](https://github.com/davidzollo/seatunnel/issues/10812) is essentially that direction in flight, with the interface layer already well thought through [#10399](https://github.com/davidzollo/seatunnel/issues/10399) addresses the persistence-layer concerns I was raising via a self-rolled LSM/compaction approach — which is a perfectly reasonable answer to the same problem RocksDB would address. I should have been clearer that RocksDB is one possible backend among several, not "the" path forward. I'd like to point out that RocksDB is suitable for this task. The points around large-blob placement (runningJobInfoIMap on shared filesystem) might be more useful as input to the open questions already listed in STIP-26 than as a separate direction -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
