Re: [ccache] Using ccache with memcached
Den 2015-12-07 kl. 04:48, skrev Mike Frysinger: > On 02 Dec 2015 20:16, Pierre Tardy wrote: >>> i don't think getting rid out of the fs makes sense, but having memcache >>> be available dynamically as an additional layer sounds fine.. >> >> It does make a lot of sense for me as I have a high performance network, >> which is faster than local harddrive. So I would insist on keeping an >> option for memcached only. > > that isn't what i meant. i don't care about runtime config options but > about (1) the code and (2) build time control. fs should remain in the > source and memcache should be an additional configure flag which allows > the user to select it at runtime. That is the way that it currently works. There is now a --enable-memcached flag, to avoid libmemcached being a mandatory requirement. In the code, it uses a #ifdef HAVE_LIBMEMCACHED But it still doesn't really *do* anything, unless you also set the configuration for memcached_conf (containing for instance --SERVER). Then there is a *second* boolean option, now called memcached_only, that only uses the regular cache for manifests and for stats / conf. So if that is set, it will avoid storing objects and friends in the file system cache but only store those in the memcached servers... Note that the use of binary packages (rather than using source ports) usually ends up just picking one of the options for you anyway ? So in that sense it's "better" to have it selectable at runtime, and for the feature to be there (by default). Otherwise it is "gone". Squashed everything, PR coming shortly. /Anders ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Using ccache with memcached
> The option to switch the to_cache/from_cache can be made available > separately, like it was in your PR. But it can use another config ? > Probably needs some updating and refactoring, and it would be nice > to try and keep the code duplication between them to a minimum... > > i.e. between the current filesystem code and the memcached code > > I can make an attempt to merge them, or if you want to do it... > To add a config like "memcached_only", next to "memcached_conf" ? > If you have a single server, then *neither* option makes any sense. > So it all depends on the setup, and needs to benchmarked further... Here is such an attempt, to keep *both* features available: https://github.com/itensionanders/ccache/tree/memcached-only If you set memcached_only to true, it will not use the fs cache. Currently it will store the manifests locally, as in the original. Also added a basic unit test to it. Needs some cleanup, but works OK ? /Anders ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Using ccache with memcached
On 01 Dec 2015 21:59, Anders Björklund wrote:
> Pierre Tardy made a PR (https://github.com/jrosdahl/ccache/pull/30)
> to replace the filesystem ("fs") cache with memcached altogether.
>
> We have gone with a different approach, to use memcached only as a
> secondary cache - while preserving the primary cache (on the disk).
i don't think getting rid out of the fs makes sense, but having memcache
be available dynamically as an additional layer sounds fine.
-mike
signature.asc
Description: Digital signature
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Using ccache with memcached
> > > i don't think getting rid out of the fs makes sense, but having memcache > be available dynamically as an additional layer sounds fine. > It does make a lot of sense for me as I have a high performance network, which is faster than local harddrive. So I would insist on keeping an option for memcached only. Regards, Pierre ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Using ccache with memcached
Pierre Tardy wrote: > > i don't think getting rid out of the fs makes sense, but having memcache > be available dynamically as an additional layer sounds fine. > > It does make a lot of sense for me as I have a high performance network, > which is faster than local harddrive. So I would insist on keeping an > option for memcached only. Both features could be kept. The "memccached" layer is basically the same (I extended it a bit, and changed the API a little...) and so is your memcached format. I suppose you could use our memcached with just a small disk cache, but you'd get a lot of (unnecessary) writes and cache cleanups ? IIRC, your manifests (and headers) were still using the local drive. The option to switch the to_cache/from_cache can be made available separately, like it was in your PR. But it can use another config ? Probably needs some updating and refactoring, and it would be nice to try and keep the code duplication between them to a minimum... i.e. between the current filesystem code and the memcached code I can make an attempt to merge them, or if you want to do it... To add a config like "memcached_only", next to "memcached_conf" ? If you have a single server, then *neither* option makes any sense. So it all depends on the setup, and needs to benchmarked further... I had also added a "readonly" flag, for builds with lots of misses. It's common to have a shared set of base files, and then with some local alterations to those... So you would have a "blessed" build filling up the memcached, and then individual builds based on that would reuse objects if available but not add their one-offs to it. The variant shared on NFS would just use CCACHE_READONLY for this. /Anders ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
[ccache] Using ccache with memcached
Hi all!
While the idea of using memcached with ccache is nothing new (*),
it seems to be more popular now with more memory being available...
* https://lists.samba.org/archive/ccache/2010q4/000686.html
https://lists.samba.org/archive/ccache/2013q2/001120.html
Pierre Tardy made a PR (https://github.com/jrosdahl/ccache/pull/30)
to replace the filesystem ("fs") cache with memcached altogether.
We have gone with a different approach, to use memcached only as a
secondary cache - while preserving the primary cache (on the disk).
Also added support for big files larger than memcached default (1M),
without having to modify the servers - by splitting them up if needed.
Manifests are just stored in a single entry in the memcached, while
other files are being combined into one entry per cache key (md4-len)
The idea is that hitting this secondary cache is still cheaper than
doing a compile again, but could be slower than not using the network.
The overhead of having each and every ccache invocation call memcached,
can be avoided by setting up a local memcached proxy ("moxi") server.
There is a public branch available, rebased from a 3.1 version:
https://github.com/itensionanders/ccache/tree/memcached
It has been (recently) updated to "3.2-maint", but not to "master".
Being a work in progress still, it's not ready for merging just yet.
But I would like some early feedback, and perhaps some more testing ?
More benchmarking needs to be done, and for some different scenarios.
/Anders
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache
