Re: [ccache] Using ccache with memcached

2015-12-07 Thread Pierre Tardy
>
>
> that isn't what i meant.  i don't care about runtime config options but
> about (1) the code and (2) build time control.  fs should remain in the
> source and memcache should be an additional configure flag which allows
> the user to select it at runtime.
>
Ah, ok then we are on the same page :)

Pierre
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache


Re: [ccache] Using ccache with memcached

2015-12-06 Thread Mike Frysinger
On 02 Dec 2015 20:16, Pierre Tardy wrote:
> > i don't think getting rid out of the fs makes sense, but having memcache
> > be available dynamically as an additional layer sounds fine.
> 
> It does make a lot of sense for me as I have a high performance network,
> which is faster than local harddrive. So I would insist on keeping an
> option for memcached only.

that isn't what i meant.  i don't care about runtime config options but
about (1) the code and (2) build time control.  fs should remain in the
source and memcache should be an additional configure flag which allows
the user to select it at runtime.
-mike


signature.asc
Description: Digital signature
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache


Re: [ccache] Using ccache with memcached

2015-12-06 Thread Anders Björklund
Pierre Tardy wrote:

>> Here is such an attempt, to keep *both* features available:
>> https://github.com/itensionanders/ccache/tree/memcached-only
>
> I like it very much. I think it adds great value for ccache, and to my
> old memcached-only attempt.

Yeah, if it doesn't bloat the code base too much it makes sense to
leave the final decision up to the user with runtime configuration.

> I did not realize the use for moxi also as a connection "keep-alive"
> mechanism, and a way to hide the syn-ack latency overhead. I think this
> is what you mean by "avoid some of the network overhead." .This perhaps
> would deserve a little bit more details in the doc.

Right, that is what I meant. Think it depends a bit on the number
of servers involved, but I don't think it can hurt much either way.
Suppose another paragraph or two couldn't hurt, but the more advanced
config can remain with memcached and moxi documentation - I think ?

However, it does make a lot of sense to offer the "complete package"
and is something that we are looking into. Software and configuration.
For us that would entail ccache*, moxi, distcc, memcached and distccd.
So it spans at least three or four different open source projects.

* including zlib and libmemcached

> Even if it is not ready, I think it would be worth to create a pull
> request, and make it easier for everybody to review the current code.
>
> This is what I used to do it, but its not easy to put and track review
> comments.
> https://github.com/jrosdahl/ccache/compare/master...itensionanders:memcached-only

Yes, that works for testing. You can append a .diff or a .patch to it,
and use "diff" or "git am". But that's more read-only, and not social.

I wanted to do some more squashing and rebasing to "master" - but I 
suppose there is no reason why all that couldn't be done as a PR...

> Mike already put a bunch of coding style review comments on my own
> commits. I would rather not fix them myself, as I know you already have
> an evolved version which is more suitable for merging.

I think some of these might also have been fixed by "uncrustify" ?

/Anders
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache


Re: [ccache] Using ccache with memcached

2015-12-06 Thread Anders Björklund
Den 2015-12-07 kl. 04:48, skrev Mike Frysinger:
> On 02 Dec 2015 20:16, Pierre Tardy wrote:
>>> i don't think getting rid out of the fs makes sense, but having memcache
>>> be available dynamically as an additional layer sounds fine..
>>
>> It does make a lot of sense for me as I have a high performance network,
>> which is faster than local harddrive. So I would insist on keeping an
>> option for memcached only.
>
> that isn't what i meant.  i don't care about runtime config options but
> about (1) the code and (2) build time control.  fs should remain in the
> source and memcache should be an additional configure flag which allows
> the user to select it at runtime.

That is the way that it currently works.

There is now a --enable-memcached flag, to avoid libmemcached being a
mandatory requirement. In the code, it uses a #ifdef HAVE_LIBMEMCACHED
But it still doesn't really *do* anything, unless you also set the
configuration for memcached_conf (containing for instance --SERVER).

Then there is a *second* boolean option, now called memcached_only,
that only uses the regular cache for manifests and for stats / conf.
So if that is set, it will avoid storing objects and friends in the
file system cache but only store those in the memcached servers...


Note that the use of binary packages (rather than using source ports)
usually ends up just picking one of the options for you anyway ?
So in that sense it's "better" to have it selectable at runtime, and
for the feature to be there (by default). Otherwise it is "gone".

Squashed everything, PR coming shortly.

/Anders
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache


Re: [ccache] Using ccache with memcached

2015-12-04 Thread Anders Björklund
> The option to switch the to_cache/from_cache can be made available
> separately, like it was in your PR. But it can use another config ?
> Probably needs some updating and refactoring, and it would be nice
> to try and keep the code duplication between them to a minimum...
>
> i.e. between the current filesystem code and the memcached code
>
> I can make an attempt to merge them, or if you want to do it...
> To add a config like "memcached_only", next to "memcached_conf" ?
> If you have a single server, then *neither* option makes any sense.
> So it all depends on the setup, and needs to benchmarked further...

Here is such an attempt, to keep *both* features available:
https://github.com/itensionanders/ccache/tree/memcached-only

If you set memcached_only to true, it will not use the fs cache.
Currently it will store the manifests locally, as in the original.

Also added a basic unit test to it.

Needs some cleanup, but works OK ?

/Anders
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache


Re: [ccache] Using ccache with memcached

2015-12-02 Thread Mike Frysinger
On 01 Dec 2015 21:59, Anders Björklund wrote:
> Pierre Tardy made a PR (https://github.com/jrosdahl/ccache/pull/30)
> to replace the filesystem ("fs") cache with memcached altogether.
> 
> We have gone with a different approach, to use memcached only as a
> secondary cache - while preserving the primary cache (on the disk).

i don't think getting rid out of the fs makes sense, but having memcache
be available dynamically as an additional layer sounds fine.
-mike


signature.asc
Description: Digital signature
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache


Re: [ccache] Using ccache with memcached

2015-12-02 Thread Pierre Tardy
>
>
> i don't think getting rid out of the fs makes sense, but having memcache
> be available dynamically as an additional layer sounds fine.
>

It does make a lot of sense for me as I have a high performance network,
which is faster than local harddrive. So I would insist on keeping an
option for memcached only.

Regards,
Pierre
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache


Re: [ccache] Using ccache with memcached

2015-12-02 Thread Anders Björklund
Pierre Tardy wrote:
>
> i don't think getting rid out of the fs makes sense, but having memcache
> be available dynamically as an additional layer sounds fine.
>
> It does make a lot of sense for me as I have a high performance network,
> which is faster than local harddrive. So I would insist on keeping an
> option for memcached only.

Both features could be kept.

The "memccached" layer is basically the same (I extended it a bit,
and changed the API a little...) and so is your memcached format.
I suppose you could use our memcached with just a small disk cache,
but you'd get a lot of (unnecessary) writes and cache cleanups ?

IIRC, your manifests (and headers) were still using the local drive.

The option to switch the to_cache/from_cache can be made available
separately, like it was in your PR. But it can use another config ?
Probably needs some updating and refactoring, and it would be nice
to try and keep the code duplication between them to a minimum...

i.e. between the current filesystem code and the memcached code

I can make an attempt to merge them, or if you want to do it...
To add a config like "memcached_only", next to "memcached_conf" ?
If you have a single server, then *neither* option makes any sense.
So it all depends on the setup, and needs to benchmarked further...


I had also added a "readonly" flag, for builds with lots of misses.

It's common to have a shared set of base files, and then with some
local alterations to those... So you would have a "blessed" build
filling up the memcached, and then individual builds based on that
would reuse objects if available but not add their one-offs to it.

The variant shared on NFS would just use CCACHE_READONLY for this.

/Anders
___
ccache mailing list
ccache@lists.samba.org
https://lists.samba.org/mailman/listinfo/ccache