Re: [ccache] Shared cache (Ubuntu 14, VMWare, TeamCity, gcc)
Hi Mats, There is a memcached branch here: https://github.com/ccache/ccache/pull/58 I've been running a previous version of this for years with great success, and you can see that others also had success with it. Event for 16 agent, you will probably begin to see performance improvment compared to NFS single point of failure. At the moment, it is kept as a separate branch as we haven't figured out a way to cleanly disable it for people who do not want it while still being maximum performance for people who want it. See the PR discussion. BR Pierre Le lun. 3 oct. 2016 à 15:56, Mats Nilsson a écrit : > Hi, > > I'm a newcomer and I'm just about to start some tests with ccache and I'm > unsure of how to arrange a shared cache for my builds in TeamCity. > > We usually compile the same git commit in several ways (eg. shared or > unshared libs) and with several cross compilers on our 16 build agents. > > I wonder if the shared cache needs to be on a NFS-fileserver or if it's > better to use a shared virtual disk? > > BR, > > /Mats > > ___ > ccache mailing list > ccache@lists.samba.org > https://lists.samba.org/mailman/listinfo/ccache > ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Using ccache with memcached
> > > that isn't what i meant. i don't care about runtime config options but > about (1) the code and (2) build time control. fs should remain in the > source and memcache should be an additional configure flag which allows > the user to select it at runtime. > Ah, ok then we are on the same page :) Pierre ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Using ccache with memcached
Hi, I like it very much. I think it adds great value for ccache, and to my old memcached-only attempt. I did not realize the use for moxi also as a connection "keep-alive" mechanism, and a way to hide the syn-ack latency overhead. I think this is what you mean by "avoid some of the network overhead." .This perhaps would deserve a little bit more details in the doc. Even if it is not ready, I think it would be worth to create a pull request, and make it easier for everybody to review the current code. This is what I used to do it, but its not easy to put and track review comments. https://github.com/jrosdahl/ccache/compare/master...itensionanders:memcached-only Mike already put a bunch of coding style review comments on my own commits. I would rather not fix them myself, as I know you already have an evolved version which is more suitable for merging. Cheers Pierre Le ven. 4 déc. 2015 à 22:45, Anders Björklund a écrit : > > The option to switch the to_cache/from_cache can be made available > > separately, like it was in your PR. But it can use another config ? > > Probably needs some updating and refactoring, and it would be nice > > to try and keep the code duplication between them to a minimum... > > > > i.e. between the current filesystem code and the memcached code > > > > I can make an attempt to merge them, or if you want to do it... > > To add a config like "memcached_only", next to "memcached_conf" ? > > If you have a single server, then *neither* option makes any sense. > > So it all depends on the setup, and needs to benchmarked further... > > Here is such an attempt, to keep *both* features available: > https://github.com/itensionanders/ccache/tree/memcached-only > > If you set memcached_only to true, it will not use the fs cache. > Currently it will store the manifests locally, as in the original. > > Also added a basic unit test to it. > > Needs some cleanup, but works OK ? > > /Anders > ___ > ccache mailing list > ccache@lists.samba.org > https://lists.samba.org/mailman/listinfo/ccache > ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Using ccache with memcached
> > > i don't think getting rid out of the fs makes sense, but having memcache > be available dynamically as an additional layer sounds fine. > It does make a lot of sense for me as I have a high performance network, which is faster than local harddrive. So I would insist on keeping an option for memcached only. Regards, Pierre ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] CCACHE_NLEVELS , locking and cache size
On Thu, Aug 22, 2013 at 2:18 AM, tal zilcer wrote: > hi, > > i recently started using ccache and i have a couple of questions. > > 1)i did a little experiment with different values of CCACHE_NLEVELS [..] according to the build time results it seems it is better to use low > NLEVELS value. when should this value be high? > i dont understand the true meaning of this variable and what is he good > for. > It is rather simple: it spreads the cached object in different folders. So the overead you have is probably the time needed to create the folder. The number of level is depending on the overall number of cache files you want to store. Each level will divide by 16 the maximum number of file found in the cache directories. Depending on the filesystem where you store your cache the time to access a file can explode when the number of file in a directory is high. Reiserfs is notoriously good at storing a lot of files in the same directory. not sure of what happens if you put nfs in the whole story. > 2)when using the cache for multiple users(about 30 people) i noticed that i > get much slower compilation time because cache/NUMBER/stats.lock is locked > a lot. > is there an option to lock a more "deep" file(when using high NLEVELS > value)? for instance cache/NUMBER/NUMBER/NUMBER/stats.lock. > if not is there a solution to this problem? > Yeah NFS does not scale too much for such a high load a big ccache database can generate. You can try my memcached backend for ccache: http://permalink.gmane.org/gmane.comp.compilers.ccache/1116 memcached is a real database, so it will scale much better than the traditional ccache adhoc db + NFS Regards, Pierre ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
