Re: [ccache] Implementing a Read-only HTTP CCACHE_DIR(resurrect)

2015-12-01 Thread Anders Björklund
vkr wrote:
> Hello,
> I stumbled across this thread -
> https://lists.samba.org/archive/ccache/2012q2/000879.html which is years
> old,
> Coincidentally, I did some work along similar lines already, without
> realizing there was this discussion about this topic here,
> and I appreciate some comments/suggestions on my approach so far.

This is interesting, there was some renewed interest in the memcached
patch that was proposed in the same timeabout frame (i.e. in 2013)

https://github.com/jrosdahl/ccache/pull/30
https://lists.samba.org/archive/ccache/2013q3/001124.html

> Having cache on NFS is comparatively the easy option from configuration
> point of view, however, there can be environments where
> for whatever the reasons, NFS server is a few hops away, while there are
> other machines that are closer to the build farm, in which case,
> having a HTTP CCACHE_DIR does seemed like a reasonably better option as it
> involves less configuration havoc on every machine in the build farm.

It also has lots of problems with for instance locking (workaround
is included) and overhead when updating modification timestamps etc.

> Keeping the above as use case, I've implemented HTTP CCACHE_DIR in my fork
> - https://github.com/venkrao/ccache
> This is a very crude throw-away test from a beginner C Programmer, that
> does the following. Care has been taken to ensure it does behave like
> existing ccache to the extent I know so far, and I did have successful runs
> of modified ccache with no core/crash or surprise failures.

I haven't been able to test your code, but it does sound like there
are some shortcomings in the design (e.g. like it being read-only).

The repository has some issues, in that it has been disconnected ?
It also has a bunch of generated files, being imported from tarball.

> Unfortunately, in our existing environment I could not see visible
> improvement between our NFS based cache setup and this new approach.
> I cannot attribute the lack of performance to anything right now.

It could be inherent with HTTP, just like it was in NFS before ?
Using a local filesystem cache or a shared memcached seems better...

I gave up on _my_ http version, when I found the memcached version.
Will post some more details about my own version of it separately.

/Anders
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache


[ccache] Implementing a Read-only HTTP CCACHE_DIR(resurrect)

2015-11-18 Thread vkr
Hello,
I stumbled across this thread -
https://lists.samba.org/archive/ccache/2012q2/000879.html which is years
old,
Coincidentally, I did some work along similar lines already, without
realizing there was this discussion about this topic here,
and I appreciate some comments/suggestions on my approach so far.

Having cache on NFS is comparatively the easy option from configuration
point of view, however, there can be environments where
for whatever the reasons, NFS server is a few hops away, while there are
other machines that are closer to the build farm, in which case,
having a HTTP CCACHE_DIR does seemed like a reasonably better option as it
involves less configuration havoc on every machine in the build farm.

Keeping the above as use case, I've implemented HTTP CCACHE_DIR in my fork
- https://github.com/venkrao/ccache
This is a very crude throw-away test from a beginner C Programmer, that
does the following. Care has been taken to ensure it does behave like
existing ccache to the extent I know so far, and I did have successful runs
of modified ccache with no core/crash or surprise failures.

1. Offers ONLY read-only http ccache_dir!
2. A Server process is to be started on the build machine(as of now,
independent of ccache. But plan is to integrate this into ccache)(see
server.c in the fork)
3. Server process uses a single cURL handle, and listens to all the
incoming cache download requests(which can be for .d/.o/.manifest or any
such file that ccache looks up otherwise in the CCACHE_DIR location)
4. Server process was necessary to share the cURL handle, to reuse a single
http connection as much as possible(have I got it right that this does save
some overhead on cURL that not many tcp connections are opened by cURL?)
5. ccache sends the URL, and the download location to the server, and it
gets a response code based on which ccache runs the next actions.

Unfortunately, in our existing environment I could not see visible
improvement between our NFS based cache setup and this new approach.
I cannot attribute the lack of performance to anything right now.
I do think that http ccache_dir is better than NFS based cache in our case
because, by benchmarking for data copy say so.
Example, specific amount of data copied between my webserver(used in the
new approach tests), and the build machine as compared to NFS server and
build machine shows NFS is no good for us.

In simple terms, is it a viable feature that my fork may be integrated into
ccache after careful consideration, that ccache_dir is served via http?(for
both get, and put cache)
Please consider reviewing the changes I made. Your comments and time are
greatly appreciated.

Regards,
Venkat.
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache