GitHub user carloea2 added a comment to the discussion: Why State flows with Data instead of being Centralized by the controller?
So my understanding now is: A centralized state manager, whether implemented in the controller, a database, or as a separate State Manager, could in principle preserve the same consistency semantics as the current ECM approach. If both tuples and state versions carry an appropriate tag, timestamp, or ID, then a downstream operator could ask the State Manager for the state version that is consistent with a particular tuple. Since state writes seem to be relatively infrequent in the current codebase, I would also expect write-lock contention in such a centralized design to be limited. One advantage of centralization is that more complex coordination logic may be easier to implement, because reads and writes can be explicitly synchronized through the State Manager. With ECM, that coordination has to be represented through explicit edges, markers, and dataflow logic. The main advantage I now see in ECM is architectural: state follows the same edges as the data. The dependency is therefore explicit in the workflow topology, and the state associated with the execution naturally travels along the same path. With a centralized State Manager, those dependencies would no longer appear as workflow edges. In that sense, we would introduce "hidden edges" between operators and the external state service. So I don't think a centralized design is fundamentally unable to provide the same semantics. Rather, ECM makes state dependencies part of the dataflow itself, while a centralized design could simplify some state-management logic at the cost of making those dependencies less explicit in the workflow graph. I discussed this with Xinyuan, and I think this is roughly where we converged. @aglinxinyuan, please correct me if your understanding differs. GitHub link: https://github.com/apache/texera/discussions/7429#discussioncomment-17985809 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
