Re: [ccache] BSDiff for cache objects

2012-11-16 Thread Bogdan Harjoc
On Mon, Nov 12, 2012 at 8:09 PM, Bogdan Harjoc wrote: > Initial results from a small .ccache (3.0) dir: > The previous results were bogus (I was diffing .gz compressed objects). I tried the bsdiff approach on a new .ccache folder obtained from compiling 8 linux kernels (various v3.x versions) c

Re: [ccache] BSDiff for cache objects

2012-11-12 Thread Bogdan Harjoc
Initial results from a small .ccache (3.0) dir: - 6476 objects - 300MB - probably about 500-1000 compiles/recompiles of around 100 small to large projects The test was: 1. Find the candidates for compression, based on: objdump -t | grep " g " (defined symbols). If two objects had at least 4 symbo

Re: [ccache] BSDiff for cache objects

2012-11-12 Thread Andrew Stubbs
On 12/11/12 14:08, Bogdan Harjoc wrote: No but there is room for improvement. This could be optional, like a CCACHE_COMPRESS that saves 99% instead of 40% when I routinely recompile 20 kernel branches, for example (v2.6.x, 3.0.x, 3.4.x, -git, -next, -ubuntu, etc). I realise that the more diverg

Re: [ccache] BSDiff for cache objects

2012-11-12 Thread Bogdan Harjoc
On Mon, Nov 12, 2012 at 3:39 PM, Andrew Stubbs wrote: > On 12/11/12 11:49, Bogdan Harjoc wrote: > >> Alternatively, a "compact" operation could be run periodically, that >> compresses the cache using the same approach. >> > > Is cache size/capacity a very big issue for you? > No but there is roo

Re: [ccache] BSDiff for cache objects

2012-11-12 Thread Andrew Stubbs
On 12/11/12 11:49, Bogdan Harjoc wrote: Alternatively, a "compact" operation could be run periodically, that compresses the cache using the same approach. Is cache size/capacity a very big issue for you? Have you tried the CCACHE_COMPRESS feature? That simply gzips the binaries in the cache.

Re: [ccache] BSDiff for cache objects

2012-11-12 Thread Bogdan Harjoc
On Mon, Nov 12, 2012 at 2:30 PM, Jürgen Buchmüller wrote: > Am Montag, den 12.11.2012, 13:49 +0200 schrieb Bogdan Harjoc: > > Basically, before writing a new object file, ccache could find a similar > > object in the cache (based on object-code or source-code hashes for > > example) > > The main g

Re: [ccache] BSDiff for cache objects

2012-11-12 Thread Jürgen Buchmüller
Am Montag, den 12.11.2012, 13:49 +0200 schrieb Bogdan Harjoc: > Basically, before writing a new object file, ccache could find a similar > object in the cache (based on object-code or source-code hashes for > example) The main goal of most hashes is to give very distinct results even for even smal

[ccache] BSDiff for cache objects

2012-11-12 Thread Bogdan Harjoc
I just did a quick search, and couldn't find discussions on the idea of caching compiled objects as binary diffs from other existing objects. Basically, before writing a new object file, ccache could find a similar object in the cache (based on object-code or source-code hashes for example) and st