Re: [ccache] Combining multiple ccache into one

[Jason Zhou via ccache]

Thank you Anders for your response! More questions inline:

> On Mar 18, 2018, at 5:32 AM, Anders Björklund via ccache 
>  wrote:
> 
> Jason Zhou wrote:
>> I am looking for an efficient way to correctly combine multiple
>> ccache from hundreds of build machines into a single ccache to build
>> a super set ccache. We use 200+ autoscaled cloud machines in our
>> build farm and each machine builds a random subsets of the source
>> tree. ccache size on each machine is ~70GB and contains ~500K files.
>> Having a superset ccache pre-built in the cloud image will greatly
>> improve our build time.
> 
> Including a pre-populated cache in the OS image is a novel idea,
> but I wonder if you would have to resort to that "workaround" ?
> 
> You could keep a local cache, and sync it from a "secondary cache".
> We have some code for this, but none of it is up to sync with master.

We are doing this because we need to launch new cloud instances fast to quickly 
response to build requests. Sync’ing from a secondary cache will take much 
longer.

> 
>> I noticed the same ccache filename (*.o, *.manifest, *.d) not
>> necessarily has the same content (md5sum) on different machines and
>> wonder if rsync is the right tool to do this, or is it feasible at
>> all to combine ccache.
> 
> This is normal. The created files might have different timestamps
> and such, that makes their checksum different. But they are supposed
> to be interchangable, so none of those differences should *matter*
> (if it does, then we are missing to hash something important…)

So ccache files are interchangeable as long as they have the same filenames 
(hash name) and this applies to all ccache files: *.o, *.manigest, *.d? If so 
we can use “rsync -a —ignore-existing” to combine multiple ccache much faster 
(no need to check file timestamp and size).

> 
>> I am trying to avoid ccache on NFS mount due to number of machines we
>> are dealing with and performance of NFS is not promising. 
> 
> Have you tried out the memcached version ? It was developed for
> that reason... You can have a cluster of such servers, if needed.
> 
> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ccache_ccache_tree_dev_memcached=DwICaQ=0ia8zh_eZtQM1JEjWgVLZg=ByzQxNeEuzdGEv2mSxiU8HCyQpz_6zJt6ZhQ1YL1R9w=uaT5rzgzrAj-kp2ARYq2pnY6oVlST6ntGbXXVIOWZxE=5-mcUIFb73nP0jUMbgxUBzXqFo4sWO9fAWzuFwV_yDY=

I’d love to explore memcached version. The document link above doesn’t mention 
any memcached info, do you have a link that describes how to use memcache for 
ccache?

Thanks again for your help!

Jason

> 
> To further scale out, one can keep a local memcached proxy ("moxi")
> and have the cluster be disk-backed (using couchbase) for restarts.
> 
> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.couchbase.com_memcached=DwICaQ=0ia8zh_eZtQM1JEjWgVLZg=ByzQxNeEuzdGEv2mSxiU8HCyQpz_6zJt6ZhQ1YL1R9w=uaT5rzgzrAj-kp2ARYq2pnY6oVlST6ntGbXXVIOWZxE=Hl8l6FfL0005SPe57kPTjCbvO3jTXHK4u3Irx1L5LCs=
> 
> /Anders
> ___
> ccache mailing list
> ccache@lists.samba.org
> https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.samba.org_mailman_listinfo_ccache=DwICaQ=0ia8zh_eZtQM1JEjWgVLZg=ByzQxNeEuzdGEv2mSxiU8HCyQpz_6zJt6ZhQ1YL1R9w=uaT5rzgzrAj-kp2ARYq2pnY6oVlST6ntGbXXVIOWZxE=RN0I5x9rfUNh6hpFtn0HPVXATIe0-Zx31xa9kF56aFk=


___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache

Re: [ccache] ccache test failures

[Joel Rosdahl via ccache]

On 16 March 2018 at 12:52, Forte, Greg via ccache
 wrote:
> Tried this with and without CCACHE_DIR set and exported - with it, I see the
> same output you showed, including the unset, and the test still fails: [...]

OK.

What's common between the two tests that fail for you is that you don't get a
cache hit for a second compilation in another directory. One way to understand
why is to log all data that is hashed by ccache in the two cases and see where
it starts to differ. There is a crude way of doing that already: Build ccache
with -DCCACHE_DEBUG_HASH and set CCACHE_DEBUG_HASH at runtime. Each ccache
invocation will then create a ccache-debug-hash.bin that contains the hashed
data, so you could compare the files in the two different directories. Or send
them to me (private mail) and I'll have a look.

By the way, here's a sketch of a yet unimplemented feature what would make this
kind of debugging easier: https://github.com/ccache/ccache/issues/226

-- Joel

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache