I am looking for an efficient way to correctly combine multiple ccache from 
hundreds of build machines into a single ccache to build a super set ccache. We 
use 200+ autoscaled cloud machines in our build farm and each machine builds a 
random subsets of the source tree. ccache size on each machine is ~70GB and 
contains ~500K files. Having a superset ccache pre-built in the cloud image 
will greatly improve our build time.

I have tried the following rsync commands to combine ccache and like to get 
your expert opinions and suggestions on the correctness and efficiency of the 

rsync command 1:

"rsync -a --exclude=**.tmp.*" is very slow and takes hours to combine ccache.

rsync command 2:

"rsync -a --exclude=**.tmp.* --ignore-existing” is very fast.

I noticed the same ccache filename (*.o, *.manifest, *.d) not necessarily has 
the same content (md5sum) on different machines and wonder if rsync is the 
right tool to do this, or is it feasible at all to combine ccache.

Thanks in advance!



I am trying to avoid ccache on NFS mount due to number of machines we are 
dealing with and performance of NFS is not promising.
ccache mailing list

Reply via email to