Re: [ccache] Implementing a Read-only HTTP CCACHE_DIR(resurrect)

2015-12-01 Thread Anders Björklund
vkr wrote:
> Hello,
> I stumbled across this thread -
> https://lists.samba.org/archive/ccache/2012q2/000879.html which is years
> old,
> Coincidentally, I did some work along similar lines already, without
> realizing there was this discussion about this topic here,
> and I appreciate some comments/suggestions on my approach so far.

This is interesting, there was some renewed interest in the memcached
patch that was proposed in the same timeabout frame (i.e. in 2013)

https://github.com/jrosdahl/ccache/pull/30
https://lists.samba.org/archive/ccache/2013q3/001124.html

> Having cache on NFS is comparatively the easy option from configuration
> point of view, however, there can be environments where
> for whatever the reasons, NFS server is a few hops away, while there are
> other machines that are closer to the build farm, in which case,
> having a HTTP CCACHE_DIR does seemed like a reasonably better option as it
> involves less configuration havoc on every machine in the build farm.

It also has lots of problems with for instance locking (workaround
is included) and overhead when updating modification timestamps etc.

> Keeping the above as use case, I've implemented HTTP CCACHE_DIR in my fork
> - https://github.com/venkrao/ccache
> This is a very crude throw-away test from a beginner C Programmer, that
> does the following. Care has been taken to ensure it does behave like
> existing ccache to the extent I know so far, and I did have successful runs
> of modified ccache with no core/crash or surprise failures.

I haven't been able to test your code, but it does sound like there
are some shortcomings in the design (e.g. like it being read-only).

The repository has some issues, in that it has been disconnected ?
It also has a bunch of generated files, being imported from tarball.

> Unfortunately, in our existing environment I could not see visible
> improvement between our NFS based cache setup and this new approach.
> I cannot attribute the lack of performance to anything right now.

It could be inherent with HTTP, just like it was in NFS before ?
Using a local filesystem cache or a shared memcached seems better...

I gave up on _my_ http version, when I found the memcached version.
Will post some more details about my own version of it separately.

/Anders
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache


[ccache] Using ccache with memcached

2015-12-01 Thread Anders Björklund
Hi all!

While the idea of using memcached with ccache is nothing new (*),
it seems to be more popular now with more memory being available...

* https://lists.samba.org/archive/ccache/2010q4/000686.html
   https://lists.samba.org/archive/ccache/2013q2/001120.html

Pierre Tardy made a PR (https://github.com/jrosdahl/ccache/pull/30)
to replace the filesystem ("fs") cache with memcached altogether.


We have gone with a different approach, to use memcached only as a
secondary cache - while preserving the primary cache (on the disk).

Also added support for big files larger than memcached default (1M),
without having to modify the servers - by splitting them up if needed.


Manifests are just stored in a single entry in the memcached, while
other files are being combined into one entry per cache key (md4-len)

The idea is that hitting this secondary cache is still cheaper than
doing a compile again, but could be slower than not using the network.

The overhead of having each and every ccache invocation call memcached,
can be avoided by setting up a local memcached proxy ("moxi") server.


There is a public branch available, rebased from a 3.1 version:
https://github.com/itensionanders/ccache/tree/memcached

It has been (recently) updated to "3.2-maint", but not to "master".
Being a work in progress still, it's not ready for merging just yet.

But I would like some early feedback, and perhaps some more testing ?
More benchmarking needs to be done, and for some different scenarios.

/Anders
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache