Wilson Hong wrote: > Hi ccache, I am trying to improve ccache build performance by > improving cache hit rate. One thing I wanna try is to have multiple > folders specified in CCACHE_DIR. Where first level is local cache, > and second level in LAN NFS, third level on AWS s3. Ccache will start > looking for local ccache first, if not found, then search in NFS > folder and then S3, analogy CPU L1-L2-RAM cache architecture. I take > a look ccache man page: https://ccache.samba.org/manual.html, but > still cannot figure out how to do that. Is that supported in ccache? > If not, would be a good idea to implement that? Any advice is > welcome. Thanks!
It is not supported out-of-the-box, but it makes perfect sense. You can put your primary cache directory on NFS, this is described in the manual (but then *everything* will go out over the network). When doing support for what would become the "dev/memcached" branch, we used something called an external cache that does what you want. It designates a second (or more) directory, where ccache would also look for cache hits. The implementation of it has varied a bit... First we would introduce a new kind of hits, like a "half hit". But that was too much hassle, so we just copied the external file to the local cache and called that a "hit" too (somewhat slower). Eventually one would probably want to have separate statistics ? Now with the memcached support, it could be that one would want to make the "external" support more generic - perhaps also include S3. The newer version (for 3.2) is available from here: https://github.com/itensionanders/ccache/tree/external It allows for ne "external" directory, using the regular file layout. The older 3.1 version would allow comma-separated list of directories. This version has the new option refactored into what *could* become a storage backend, that would allow for other SQL and NoSQL variants... I'm not sure what library is best for talking ("directly") with S3, but it shouldn't be impossible to adapt - it's a simple 4 step API. The theory is that NFS and HTTP (or SQL) will "lose" compared with memcached, but it would nice to have some more solid statistics ;-) The memcached branch (for master) is available here: https://github.com/jrosdahl/ccache/tree/dev/memcached There are alternative backends for MySQL and for Couchbase (NoSQL), but those have not yet been refactored... So only the FS, for now. /Anders _______________________________________________ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
