schenksj opened a new pull request, #4828: URL: https://github.com/apache/datafusion-comet/pull/4828
## Which issue does this PR close? Part of #4695 (Scan I/O acceleration proposal). This is **Phase 1** of that work and does not close the issue — the SSD tier and read-ahead land in follow-ups. ## Rationale for this change Today every Comet scan re-fetches data bytes from S3/HDFS/ABFS — only parsed Parquet metadata is cached, and only per-plan (Comet builds a fresh `RuntimeEnv` per plan, so nothing survives across tasks, let alone across queries). A hot table scanned by ten queries pays ten full rounds of GETs. This is one of the two big I/O wins called out in #4695 (the other being read-ahead). This PR adds a process-wide, block-aligned memory cache of the raw bytes behind the `object_store` API, plus the driver-side **cache-affinity scheduling** that makes it effective beyond a single executor: with K executors and Spark's locality-blind scheduling of S3-backed scans, each host's private cache would otherwise see only ~`1/K` of the re-reads. Default **off**. **Design document:** [`docs/design/OBJECT_STORE_CACHE_DESIGN.md`](https://github.com/schenksj/datafusion-comet/blob/docs/object-store-cache-design/docs/design/OBJECT_STORE_CACHE_DESIGN.md) (on a reference branch; the code comments cite it by section, e.g. §2.7 / §2.10). ## Why this is split into phases The full design is landing as two stacked PRs, split at the memory/SSD boundary purely to keep each one reviewable: - **Phase 1 — this PR:** the storage-neutral cache core (memory tier), the `object_store` wrapper, Comet wiring, and cache-affinity scheduling. - **Phase 2 — follow-up (not yet opened here):** the on-disk **SSD tier**, optional **unified off-heap storage-memory** accounting via Spark's `UnifiedMemoryManager`, and a real-S3 benchmark. It stacks on this PR; the memory-tier core already ships the `set_memory_budget` contract on day one so the phase-2 unified-memory work is purely additive. Keeping the on-disk region format and the `UnifiedMemoryManager` integration out of this PR means Phase 1 stays focused on the core + wiring that everything else builds on. ## What changes are included in this PR? **Two new storage-API-neutral workspace crates** - `native/block-cache` — the cache core: SIEVE memory tier, single-flight miss dedup, coalesced miss fetch, ETag/version invalidation (immutable-file workloads), `set_memory_budget` (phase-2 contract), metrics. Depends only on `tokio`/`bytes`/`futures`; a `cargo tree` test guards it against `object_store`/`opendal`/Comet deps. - `native/object-store-cache` — `CachingObjectStore` wrapping `dyn ObjectStore` + the core. Routes `get_ranges` and plain bounded `get_opts` through the cache; passes everything else through and invalidates on `put`/`delete`/`copy`. The only crate that touches the `object_store` trait. **Comet wiring** - Process-global `BlockCache` in a `OnceLock`, initialized from JNI configs on the first `createPlan`; `prepare_object_store_with_configs` wraps remote (non-`file`) stores. - `spark.comet.scan.dataCache.*` configs (default off), force-added across JNI so defaults reach native code. - `CometFileLocalityManager` — driver-side sticky `file -> host` assignment, least-loaded balancing, fair-share rebalance on scale-up — wired into `CometExecRDD.getPreferredLocations` and `CometNativeScanExec`. Deferred to later phases per the design: SSD tier, unified storage-memory integration, opendal/Iceberg coverage, readahead. ## How are these changes tested? - `block-cache`: 11 integration + 3 unit tests + a storage-neutrality guard (`cargo tree` fails the build if a forbidden dep creeps in) + doctest. - `object-store-cache`: 4 adapter tests (zero-refetch, byte-exactness vs. the unwrapped store, conditional passthrough, put invalidation). - native `data_cache` config parsing: 3 tests. - `CometFileLocalityManager`: 8 Scala unit tests (sticky reassignment, least-loaded, lost host, fair-share rebalance, prune). Native cache crates: `cargo test -p datafusion-comet-block-cache -p datafusion-comet-object-store-cache`. Scala built with JDK17 + `-Pspark-4.1`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
