On 8 December 2017 at 15:13, Scott Bennett <benn...@sdf.org> wrote: > Michael Fothergill via ccache <ccache@lists.samba.org> wrote: > > > On 8 December 2017 at 01:35, Scott Bennett via ccache < > > ccache@lists.samba.org> wrote: > > > > > Michael Fothergill via ccache <ccache@lists.samba.org> wrote: > > > > > > > I have an amd64 kaveri box with 8GB RAM and run Gentoo stable on it. > > > > > > > > I have just installed ccache with 2GB memory allocated to it. > > > > > > By that, I assume you have allocated some kind of memory-based > device > > > for the cache. Is that a correct understanding? > > > > > > > ?Thanks for your reply and comments. I am assuming that by having the > > standard command ? > > > > ?CCACHE_SIZE="4G" (I have increased the allocation) > > As ccache is installed on my system, I cannot find an environment > variable of that name documented. Are you sure it is being used? What I > do find documented are CCACHE_MAX_SIZE and the corresponding ccache.conf > parameter max_size. If you're going to build large items like libreoffice > or a LINUX world, you probably ought to have a much larger cache. I also > found it a good idea to change limit_multiple from its default value of 0.8 > to 0.95 to avoid half-hour-long cleanups in the middle of build runs. > It is worth noting that, by my understanding of ccache the last time > I dug into it a bit, ccache actually maintains 16 distinct caches, but the > max_size and limit_multiple values apply to the total size and usage > fraction of the aggregate of all 16 caches. When a cleanup occurs, ccache > chooses one of the 16 caches and begins deleting the least recently used > entries in it until the total space allocated has been reduced to the > limit_multiple fraction of the max_size. If it runs out of things to > delete in that cache while the total allocated remains above that fraction, > ccache chooses another cache from which to begin deleting entries, and so > on. > This procedure differs from the one described in the ccache man page and is > one reason I like to give more space to the cache(s) in order to prevent > recent entries from disappearing from a cache while other, far less > recently > used entries remain in the other 15 caches. By having a max_size large > enough to hold the last several iterations of frequently built items, > cleanups are more likely to satisfy the limit_multiple by deleting the > oldest > few iterations of updates while sparing the more recently used entries > in a cache. These days the disk space is cheap enough to give it 10 GB to > 30 GB without creating any problems for me, so I just do that. Another > trick > to keep the caches useful is to allocate separate ones for different > purposes. > For example, I set up one for building the OS userland and kernel, another > for building libreoffice, and a third for building everything else. Doing > this keeps the OS and libreoffice from evicting everything else or each > other > prematurely. :-) > > > then memory from the hard drive is being used by default here - I was not > > trying to use e.g. RAM memory. > > > Oh. Okay. > If your system is used heavily for compiling software, you may see > some > performance gains from putting the cache area onto an SSD. Using a system > memory-based cache is, of course, lightning fast, but the entire cache > evaporates when the device is deallocated (e.g., during a system shutdown > or > failure). I tried using software five- and six-way RAID-0 devices for the > file system containing my caches for a while, but decided their performance > was poor. At present I'm using a software CONCAT made of two two-way > software > RAID-1's, all on the same kind of hard drives as the earlier setups, and > this > setup seems to do very nicely for now. I just have to remember not to run > updates at the same time as scrubs on the six-way raidz2 that occupies the > bulk of the same drives. :-) I only scrub that pool about every three to > four > weeks, though, so it usually isn't a problem. > > > > > > > I have tried some repeat compilations to see if there would be any > speed > > > > increase. > > > > > > > > So far I have not seen much change but I am not skilled enough to > improve > > > > things yet. > > > > > > Your statistics show that slightly more than 45% of your total > > > compiler invocations (hits/(hits+misses)) were avoided. Did that not > > > make a dent in your timings? > > > > > > > > I tried compiling gcc, glibc and imagemagick but did not see much > > > > improvement. > > > > > > If you run the full build process for gcc, I would not expect > > > to see much improvement because most of it involves the use of either > > > a) a temporarily built compiler in a temporary location or b) the > > > newly built compiler being used for testing, but not yet installed > > > into the production location on your system. > > > > > > > ?Would cachecc1 perform any better with gcc?? > > > I don't know what cachecc1 is, so I have no idea. Sorry. >
cachecc1 is a gcc cache. See here http://cachecc1.sourceforge.net/ There are some other packages listed under related projects: https://ccache.samba.org/ e.g. gocache ccontrol and others Has anyone tried cachecc1 on the site here? There is no Gentoo package for it so I would need to compile it and install it by hand. Cheers Regards MF > > > > ImageMagick and GraphicsMagick both should provide useful timings > > > and ccache statistics. glibc probably would, too, though it's not > > > nearly as big. I don't know what sort of build procedures Gentoo uses, > > > but from the FreeBSD ports tree, here are some other good examples of > > > test cases: math/octave, www/webkit-gtk2, www/webkit-gtk3, > > > www/webkit2-gtk3, devel/llvm40. Be prepared to wait a long time for > > > the first compilation of each of the webkits. They are big and slow > > > to compile and, in the past, have shown instabilities in their build > > > procedures when parallel make runs were used. > > > > > > > ?I have now compiled qtwebengine, libreoffice and and dramatically > reduced > > the compile time with ccache > > now I am using the 3.3.4 version of the program ( see here > > https://forums.gentoo.org/viewtopic-t-1073298.html > > ? ). I think the earlier version of ccache was not working properly. > > Although some parts of libreoffice remain unaffected by ccache, the > build as a whole will show radically lower run times. I hope you enjoyed > seeing how much shorter it was after the first run with ccache. :-) > > > > People on the thread have suggested that maybe cachecc1 might not work so > > well with gcc in practice and I should stick > > with ccache as is. > > > > But you may take a different view. ? > > > I must plead ignorance, so no view at all. > > > > > > > YMMV on another OS. > > > One big savings for me was in running "make buildworld" and "make > > > buildkernel". buildworld, on my last machine, was taking about six > > > hours elapsed time for a first run. When running it later after > > > updating the source tree, the elapsed time was reduced by 2/3 to 3/4, > > > depending upon the number and sizes of source modules affected by the > > > updates. > > > > > > > ?These sound like the emerge --ask --update --deep @world command used in > > Gentoo (and similar versions).? > > buildworld is the make target that oversees the compilation of all of > the FreeBSD userland. buildkernel is the corresponding target for the > FreeBSD kernel and all of its loadable modules. > > > > ?It's not time to update the system again yet but I think that it could > > well speed things up as long as the cache size is big enough for a lot of > > packages (e.g. 200 plus) > > Again, I'd recommend setting up a separate cache area for it. I > currently allow 10 GB for the libreoffice cache, 16 GB for the OS build > cache, and 30 GB for everything else. All three of these and > the build temporary area for ports are physically in the same file system. > The libreoffice cache and OS build cache space allowances are probably > bigger than is really helpful, but I have the space, and space not being > used is space being wasted. FWIW, the cache for building ports averages a > 50% - 55% hit rate, which adds up to quite a bit of CPU and elapsed time > saved in the course of a month. > > > Scott Bennett, Comm. ASMELG, CFIAG > ********************************************************************** > * Internet: bennett at sdf.org *xor* bennett at freeshell.org * > *--------------------------------------------------------------------* > * "A well regulated and disciplined militia, is at all times a good * > * objection to the introduction of that bane of all free governments * > * -- a standing army." * > * -- Gov. John Hancock, New York Journal, 28 January 1790 * > ********************************************************************** > _______________________________________________ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
