XiaoHongbo-Hope opened a new pull request, #1186: URL: https://github.com/apache/paimon-rust/pull/1186
### Purpose Follow-up to #1175: reduce contention and foreground publication work in the opt-in node-shared disk cache under multi-process range reads. ### Brief change log - Stripe download coordination by the full block key instead of serializing every block of one file. Same-block requests still coordinate; whole-file/range overlap can duplicate downloads. - Read immutable cached inodes without the directory lock, checking the mutation marker and generation before and after the read. Throttle recency writes to approximately once every ten seconds. - Validate the owned encoded buffer and return a `Bytes` slice instead of copying its payload into another allocation. - Return range data before publication when a bounded background-fill slot is available. Transfer download guards into the fill task; use synchronous backpressure at saturation or for blocks larger than 1 MiB. Whole-file fills stay synchronous. - Encode/write a locked temporary inode outside the directory gate. Keep generation validation, eviction, capacity accounting and atomic rename under the gate. Recovery skips live staging files and cleans abandoned ones after process exit. - Document transient staging space and the scope of the concurrency bounds. No experimental timer module or environment-controlled retry tuning is included. ### Tests The experimental candidate passed all 81 cache tests and ten repeated complete cache-test runs. The cleaned PR branch is being rebuilt and retested with: ```sh cargo +1.94.0 test --release --locked -p paimon io::cache --lib -j4 cargo +1.94.0 fmt --all -- --check git diff --check ``` Coverage includes owned-buffer CRC/key/truncation validation, block-striped download coordination, hits during directory-lock contention, generation invalidation, bounded asynchronous publication/single-flight, live staging during recovery, process-exit orphan cleanup, and capacity/priority accounting. ### Performance evidence and limitations A complete training DataLoader experiment used one 64-vCPU/256-GiB host, a 60-CPU/200-GiB process scope, 8 ranks x 4 workers, batch 16/rank, prefetch 2, 64 warmup + 128 measured batches, per-batch synchronization, fixed input snapshots and sequences, a fresh tmpfs cache per trial, 20-GB configured capacity and 1-MB blocks. The pipeline included Steps, pose, images and JPEG decoding; image bodies used the existing Python OSS path, not Rust BLOB reading. The **staging change alone**, on top of the previous experimental lockless-hit/asynchronous-fill candidate, changed the two-run median from 225.2 to 294.7 samples/s (+30.8%). This is **not** a direct main-versus-PR benchmark. Peak memory did not improve (93.3 to 94.4 GiB), and node RX/sample increased slightly (4.66 to 4.77 MiB). A single cache-disabled reference reached 367.7 samples/s and 7.57 MiB/sample. Thus caching still traded throughput and memory for lower network reception in this workload. No claim of a disk-cache result or universal speedup is made. Diagnostic instrumentation was disabled for those headline trials. The cleaned PR branch removes that instrumentation and is not separately full-DataLoader benchmarked. Smoke value hashes matched and full-run sample sequences/image counts matched; full-run output value hashes were not enabled. Remaining directory-locked eviction and download coordination are intentionally not redesigned here. ### API and Format No public API, configuration default, or published cache-block format change. Only shared mode uses background fills. Eight fill slots and eight staging slots are per cache instance, not per node; temporary staging bytes are additional to the published capacity. Published blocks remain immutable, and storage mutation fencing remains mandatory. ### Documentation Updated the shared-cache section of `docs/src/getting-started.md` with concurrency, staging headroom, and whole-file/range overlap behavior. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
