aicam opened a new issue, #8665: URL: https://github.com/apache/texera/issues/8665
Mounted repositories are served by geesefs, which `bin/mounter/mounter.py` starts with `--memory-limit 512` and no `--cache`. The geesefs default is no cache, so nothing a mount serves is written to local disk and every read goes back to the object store through the file service and LakeFS. For a model read once per execution that is the right trade. For a reference dataset scanned on every execution it is the wrong one, and the cost scales with the dataset. Measured by reading the same 200 MiB three times through a mount: | read | throughput | | --- | --- | | first | 179 MB/s | | second | 137 MB/s | | third | 124 MB/s | There is no warming at all — geesefs holds only `--memory-limit` in RAM and the kernel does not retain the FUSE pages. The identical file on the node's local disk reads at 12.4 GB/s, roughly 80x faster. The workload that exposed this is a genetics pipeline whose mounted reference databases are ~326 GB and are rescanned on every run. Its search process is given 8 CPUs but burns only ~100-150%, so it is blocked on reads rather than computing, and a single execution spends 35-40 minutes waiting on reads a local disk would serve in well under a minute. ### Proposed solution Make the cache opt-in: when a new `mounter.cacheRoot` is set, pass `--cache <dir>` to geesefs. Key the cache by repository and commit rather than by computing unit. A commit is immutable, so every unit on a node reads one cache instead of each new unit starting cold — which matters most for a shared public unit, where the first execution pays and the rest benefit. Keep `cacheRoot` a separate value from `hostMountRoot` rather than a directory inside it: `hostMountRoot` holds only mount points and usually sits on the OS disk, while a cache is sized by the data being cached, so the two generally belong on different volumes. Off by default, so no deployment changes behaviour until an operator opts in. ### Affected area Deployment / `bin/mounter`, `bin/k8s`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
