Re: [ccache] Optimizing MD4

2015-12-11 Thread Andrew Stubbs

On 11/12/15 10:16, Andrew Stubbs wrote:

[*] The implementation was included in some editions of the Sourcery
CodeBench toolchain, so you can find them in the source package if you
really want to. I think you can find it here:
https://sourcery.mentor.com/GNUToolchain/release3047 (free registration
required).


Oh, and the name changed to "cs" for reasons related to the "[it] does a 
bunch of other things besides" I mentioned before.


We had some interesting plans for sharing caches over networks, 
collecting data for support requests, and general "cloudy stuff", as 
well as the CEIP. The project is mostly now shelved.


Andrew

___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache


Re: [ccache] Optimizing MD4

2015-12-11 Thread Anders Björklund
Andrew Stubbs wrote:
>> I would be interested in your thoughts on how to speed that part up.
>
> My implementation, which does a bunch of other things besides, hence why
> it's not fit to post[*], launches a background task which creates a unix
> domain socket in the cache directory (the windows version uses plain old
> TCP).
>
> Each invocation of ccache then connects to that socket and asks the
> daemon to do the MD4 scan on its behalf. The daemon checks the mtime on
> the file and serves the MD4 from its memory cache if nothing has
> changed. The stat call could probably be optimized away if the cache is
> very fresh (<1s?)

So basically something similar to the "sloppiness" file_stat_matches ?
Compare size/mtime/ctime, rather than rehashing the content of a file.

> In theory, what you get is ccache spending less time in MD4, but more
> time in I/O wait. It does seem to be faster, over all, but that might
> depend on your hardware.
>
> However, even if the latency of each ccache invocation is the same, the
> fact that they're basically idle means you can usefully crank up the
> parallelism for all but the initial build.
>
> You could, in principle, use this communication to limit how many
> cache-miss compilations are permitted to run in parallel, and therefore
> run "make -j" for maximum parallelism without fear of melting your memory.

Something like that is what I meant with a new "prefix" wrapper for cpp.
Similar to the current wrapper for cc, which does that with e.g. distcc.

> Unfortunately, I've moved on to other projects and don't have much time
> to work on this stuff any more.

Thank you for your ideas. Will check the code out too if I get the time.
It seems that there are some opportunities left for faster manifest/cpp.

/Anders
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache