Re: [ccache] Implementing a Read-only HTTP CCACHE_DIR(resurrect)
vkr wrote: > Hello, > I stumbled across this thread - > https://lists.samba.org/archive/ccache/2012q2/000879.html which is years > old, > Coincidentally, I did some work along similar lines already, without > realizing there was this discussion about this topic here, > and I appreciate some comments/suggestions on my approach so far. This is interesting, there was some renewed interest in the memcached patch that was proposed in the same timeabout frame (i.e. in 2013) https://github.com/jrosdahl/ccache/pull/30 https://lists.samba.org/archive/ccache/2013q3/001124.html > Having cache on NFS is comparatively the easy option from configuration > point of view, however, there can be environments where > for whatever the reasons, NFS server is a few hops away, while there are > other machines that are closer to the build farm, in which case, > having a HTTP CCACHE_DIR does seemed like a reasonably better option as it > involves less configuration havoc on every machine in the build farm. It also has lots of problems with for instance locking (workaround is included) and overhead when updating modification timestamps etc. > Keeping the above as use case, I've implemented HTTP CCACHE_DIR in my fork > - https://github.com/venkrao/ccache > This is a very crude throw-away test from a beginner C Programmer, that > does the following. Care has been taken to ensure it does behave like > existing ccache to the extent I know so far, and I did have successful runs > of modified ccache with no core/crash or surprise failures. I haven't been able to test your code, but it does sound like there are some shortcomings in the design (e.g. like it being read-only). The repository has some issues, in that it has been disconnected ? It also has a bunch of generated files, being imported from tarball. > Unfortunately, in our existing environment I could not see visible > improvement between our NFS based cache setup and this new approach. > I cannot attribute the lack of performance to anything right now. It could be inherent with HTTP, just like it was in NFS before ? Using a local filesystem cache or a shared memcached seems better... I gave up on _my_ http version, when I found the memcached version. Will post some more details about my own version of it separately. /Anders ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
[ccache] Implementing a Read-only HTTP CCACHE_DIR(resurrect)
Hello, I stumbled across this thread - https://lists.samba.org/archive/ccache/2012q2/000879.html which is years old, Coincidentally, I did some work along similar lines already, without realizing there was this discussion about this topic here, and I appreciate some comments/suggestions on my approach so far. Having cache on NFS is comparatively the easy option from configuration point of view, however, there can be environments where for whatever the reasons, NFS server is a few hops away, while there are other machines that are closer to the build farm, in which case, having a HTTP CCACHE_DIR does seemed like a reasonably better option as it involves less configuration havoc on every machine in the build farm. Keeping the above as use case, I've implemented HTTP CCACHE_DIR in my fork - https://github.com/venkrao/ccache This is a very crude throw-away test from a beginner C Programmer, that does the following. Care has been taken to ensure it does behave like existing ccache to the extent I know so far, and I did have successful runs of modified ccache with no core/crash or surprise failures. 1. Offers ONLY read-only http ccache_dir! 2. A Server process is to be started on the build machine(as of now, independent of ccache. But plan is to integrate this into ccache)(see server.c in the fork) 3. Server process uses a single cURL handle, and listens to all the incoming cache download requests(which can be for .d/.o/.manifest or any such file that ccache looks up otherwise in the CCACHE_DIR location) 4. Server process was necessary to share the cURL handle, to reuse a single http connection as much as possible(have I got it right that this does save some overhead on cURL that not many tcp connections are opened by cURL?) 5. ccache sends the URL, and the download location to the server, and it gets a response code based on which ccache runs the next actions. Unfortunately, in our existing environment I could not see visible improvement between our NFS based cache setup and this new approach. I cannot attribute the lack of performance to anything right now. I do think that http ccache_dir is better than NFS based cache in our case because, by benchmarking for data copy say so. Example, specific amount of data copied between my webserver(used in the new approach tests), and the build machine as compared to NFS server and build machine shows NFS is no good for us. In simple terms, is it a viable feature that my fork may be integrated into ccache after careful consideration, that ccache_dir is served via http?(for both get, and put cache) Please consider reviewing the changes I made. Your comments and time are greatly appreciated. Regards, Venkat. ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Implementing a Read-only HTTP CCACHE_DIR
I can think of several network options that could work well: - zeromq - memcached - riak - redis Perhaps we need a simple plugin API to make it easier to experiment with different options here? On Mon, Apr 2, 2012 at 9:51 PM, Martin Pool wrote: > Hi, Richard, > > The general thing of network-wide caching using something other than a > network filesystem sounds good. > > There was some discussion (or patches? maybe even merged?) a while ago > about using memcached, which seems like a pretty good idea to me > rather than layering on http. > > -- > Martin > ___ > ccache mailing list > ccache@lists.samba.org > https://lists.samba.org/mailman/listinfo/ccache ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
Re: [ccache] Implementing a Read-only HTTP CCACHE_DIR
Hi, Richard, The general thing of network-wide caching using something other than a network filesystem sounds good. There was some discussion (or patches? maybe even merged?) a while ago about using memcached, which seems like a pretty good idea to me rather than layering on http. -- Martin ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
[ccache] Implementing a Read-only HTTP CCACHE_DIR
Hello, I've not written to a mailing list before so tell me if I have done anything wrong. For a project I am working on we will be rebuilding a lot of programs, so we are using ccache for that, however the idea is that a lot of people may be building the same thing, so we want to make the ccache objects publically available, especially since we could be building on machines that have better IO speeds than CPU speeds. To this end we are considering adding the ability to look up the objects on an http server. I have spent half a day looking at the source code and planning what to do, but you know more about ccache than I do, so I was wondering if anyone else had tried this kind of thing before, or if there is some fundamental problem with the approach that I haven't considered. The the plan is that after the hash has been calculated it checks whether the object is cached locally, if it hasn't it tries the http server specified by an environment variable. If the server has the object then ccache downloads the object and uses that. If it hasn't it carries on and compiles it. ___ ccache mailing list ccache@lists.samba.org https://lists.samba.org/mailman/listinfo/ccache
