akshaychitneni opened a new pull request, #2567: URL: https://github.com/apache/datafusion-ballista/pull/2567
## Which issue does this PR close? Part of #1829. First, self-contained slice; no end-to-end `cache()` behavior yet. ## Rationale for this change A cached DataFrame is shuffle output we keep and read back instead of recomputing. That mapping must **outlive** the job that produced it (unlike an `ExecutionGraph`, which is GC'd after the job). This PR adds the scheduler-side store for it — the foundation the materialize/read path, pinning, and locality layers build on. Caching *results* here is distinct from the source/scan caches in #496/#645. ## What changes are included in this PR? Scheduler-only, behind a new state backend; nothing is wired into planning or cleanup. - **`CacheRegistry`** (`state/cache_registry.rs`) — cross-job store keyed by `(session_id, cache_id)` → materialized shuffle `PartitionLocation`s. Supports hit/miss `lookup`, single-claim materialization dedup, `invalidate` / `invalidate_executor` (executor-loss policy) / `remove_session`, and `pinned_job_ids` (cleanup exclusion). - **`CacheState` trait + `InMemoryCacheState`** — a third scheduler-state backend alongside `ClusterState`/`JobState`, same shape (async, `init()` hook). In-memory today; a durable (e.g. object-store) backend drops in later with no caller changes. `BallistaCluster` now exposes `cache_state()`. - **Tests** — 8 registry unit tests + 2 async backend/cluster-wiring tests. ## Are there any user-facing changes? None. `df.cache()` still errors as before (tracked by `client/tests/context_unsupported.rs`); this only adds internal scheduler state. ## What's next (follow-ups) - Materialize-on-miss and read-back, building on the checkpoint PR (#1993). - Pin shuffle files past job cleanup. - Locality placement (coordinate with #2319). - Executor-loss invalidation, then enable the end-to-end test. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
