GitHub user Xiao-zhen-Liu added a comment to the discussion: Design and merge plan: operator output port result cache (MVP)
**Design discussion of October 1: the open PRs are on hold** On October 1, @chenlica, @Yicong-Huang, @parshimers, @sshiv012 and I went over the [design update above](https://github.com/apache/texera/discussions/5880#discussioncomment-18646157). The outcome affects how the work goes on, so I am posting it here. **Under full reuse (what the open PRs build).** We discussed what a skipped operator and a skip region should do, and whether they add new region states. As I understood it, @Yicong-Huang's main requirement was that skipping should not add new region states, which the current design already meets. **Beyond full reuse.** The main question was whether the current two-layer design can be extended past full reuse in a way that is compatible with other work. In the two-layer design, the skeleton decides what runs and then `CostBasedScheduleGenerator` plans the retained part, so the reuse decision is made outside the generator. @Yicong-Huang proposed a one-layer alternative: - Treat reading a saved result as one more resource allocation choice on a materialized link, ideally handled only in the resource allocator and the cost estimator. - The resource allocator makes the reuse decision: reusing means allocating no workers. - `CostBasedScheduleGenerator` makes no reuse decision and still only chooses between pipelining and materialization. In my view, both designs explore the same space of plans. We agreed that the two-layer design cannot be extended into the one-layer form for the general setting. **Outcome.** There was no consensus that the one-layer alternative has been investigated enough to rule it out. Until it has, #8731, #8732, #8733 and #8734 are on hold, and the rest of the series waits. I will find out whether the one-layer approach works and how much complexity it adds to the code base, and whether the two-layer design can be generalized in a way that is compatible with the other work. I will discuss both with @chenlica and post the findings here. Background on how `CostBasedScheduleGenerator` forms regions: the [Pasta paper](https://dl.acm.org/doi/10.1145/3698832). GitHub link: https://github.com/apache/texera/discussions/5880#discussioncomment-18718071 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
