On Sun, 18 Apr 2021 at 10:04, R. Diez via ccache <ccache@lists.samba.org> wrote: > First of all, I have been using ccache for years. Many thanks for this great > tool.
You're welcome. > I have come to the conclusion that these particular pauses are due to ccache > automatically pruning its cache. I haven't really observed pauses due to cleanup myself since I have used SSDs for a long time, but yes, they will occur with a slow enough disk (or loaded system) and many parallel ccache invocations. > I was surprised that completely clearing the cache with option '--clear' can > take a very long time, 12 minutes the last time around, because ccache > probably does not need to calculate anything beforehand, just delete all > files. But I haven't benchmarked the filesystem itself, so perhaps it's just > Linux being slow deleting so many small files. Yes, what you're seeing is a slow disk (or filesystem) in combination with many files in the cache directory. You will get similar performance from "rm -r". Upgrading to ccache 4.x will actually help since then only one file will be stored per cache entry instead of up to seven files (but more likely two) with ccache 3.x, so there will be fewer files to consider and remove when cleaning or clearing. > What happens if you are building in parallel, with "make -j32", and all 32 > concurrent instances decide to clean the cache at the same time? Will all of > them attempt to delete the same files at the same time? Yes, that could happen. A ccache invocation that writes the result of a compilation to one of the 16 cache buckets will start cleaning up that bucket if the cache size threshold has been reached. If another parallel compilation finishes and writes to the same bucket before the first has finished cleaning then it too will start cleaning and so on. There is no coordination between such cleanups. This is how the automatic cleanup has worked since ccache was initially created in 2002. I have ideas on how this can be improved but I don't have much time and interest to work on ccache these days so things are unfortunately slow. (I no longer use ccache personally except for building ccache...) > I wonder if there is a way to increase the number of buckets, from 16 to say > to 1024. There is not reason to have so few of them. There is a reason, and it's spelled "backward compatibility". Other minor reasons also exist. > I have also being thinking of triggering a manual cache clean-up on start-up > [...] > Is that a good strategy? Or are there better ways? I think it's a good strategy given that you have a problem with automatic cleanups. You could consider disabling automatic cleanup completely (set max_size to 0) and run "CCACHE_MAXSIZE=20G ccache -c" or so each night or more often. The limit will then only affect that ccache invocation; it won't be stored in the config file. Also, you don't have to parse the ccache configuration since the cron job has the definition of what the max cache size should be. > Is there some script able to parse the ccache config file that I could use as > an example? Not that I know of. You can query the size with "ccache -k max_size", but it will include a size suffix that you need to parse if you want to make it generic. -- Joel _______________________________________________ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
