GitHub user Xiao-zhen-Liu added a comment to the discussion: Design and merge plan: operator output port result cache (MVP)
@chenlica, here is the plan for managing the lifecycle of cached results. **Lifecycle of a cached result** *Creation.* An entry is written when a port's result is materialized during a run, as part of saving that result. It records the cache key of the port's upstream computation, the result's storage location, and the source execution. *Reuse / invalidation.* On a later run, a port whose upstream computation is unchanged matches by cache key and reads the stored result instead of recomputing; entries persist across runs. A changed upstream (parameter, wiring, schema, operator version) yields a new cache key and a new entry; old entries are never overwritten, they just stop being matched. *Where results live.* Materialized results are stored in the shared result storage (Iceberg via the REST catalog plus object store), addressed by workflow/execution/operator/port. This storage is process-wide and independent of any computing unit, so cached results survive a computing unit being destroyed. (This supersedes the earlier assumption that results are tied to the computing unit; after the storage-from-compute separation, they are not.) *Removal.* Cleanup is driven by the workflow lifecycle, not the computing unit. The direction is to reuse `WorkflowLifecycleManager`'s idle cleanup, which already clears a workflow's resources once it has had no connected users for a timeout, and extend it to also drop the workflow's cache entries and their stored results. Two points are still open for discussion: - the retention window for cached results (TBD, configurable): the current 30s timeout (`web-server.workflow-state-cleanup-in-seconds`) is for a live session's transient results and is too short for a cache whose value is reuse across sessions; - whether cache cleanup should be tied to that idle timeout at all (with a much longer window), or decoupled from the session so entries persist until the workflow is deleted or invalidated. Either way, cache entries are also removed when the workflow is deleted (the table's foreign key to `workflow` cascades). *Out of scope (MVP).* No size-bounded or cost-based eviction; that is the cost-model work and is intentionally not in this merge. GitHub link: https://github.com/apache/texera/discussions/5880#discussioncomment-17675351 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
