On Apr 9, 2008, at 12:05 PM, Peter Hoeg wrote: > > Luke, I'm not trying to pretend that I know anything concerning the > fine art of cache theory, so this is purely based on my layman's > interpretation and you should therefore take it with a truckload of > salt. > > But before we enter the discussion of what (and how) caching should > take place, wouldn't it make sense to see where the current > bottlenecks are and then seeing how those bottleneck could addressed?
Makes sense, certainly. For the record, I'm focused on caching right now purely for functional reasons, not performance. There are various parts of Puppet that actually need these caches, as explained earlier, and I'm planning on using the caches in other interesting ways. For instance, I'm going to model certificate signing with the local certificate being a cache and the certificate authority being the ultimate source. The ttl of the cached certificate will be the ttl of the cert itself, which is usually five years. If the cert expires, the client will automatically attempt to get a cert re-signed. Caching is useful for a lot more than just performance, and this problem in particular goes from being a very hard problem to model to being quite easy (I hope). > Based on my experience (admittedly running on fairly low-end > hardware), the server spends a fairly large amount of time creating > the catalog. So would it make sense to simply do the catalog in > parallel by a different thread/daemon? It's a nice idea, but this only really helps if your server doesn't have an even load, and even then it only helps your clients get their catalogs faster, which isn't necessarily a big problem. I mean, it'd be nice to have the client not sleeping for a few seconds, but it's not expensive or anything. If your server's load is relatively high, then it doesn't really matter when you compile the catalog -- it affects the load either way. > One example would be: > > a) I know the clients connect every [runinterval]) > > b) [runinterval - 5 minutes] minutes after the last connection run, > the process starts compiling the manifest for the client that will > connect in 5 minutes. > > c) Client delivered facts are then combined with the pre-compiled main > catalogue when the connection occurs and downloaded. The only way this would work is to use the server's cached facts, so you're using facts that are 25 minutes (using your example) old. Compiling absolutely requires the client facts; there's no ability to combine them after the fact. It's not necessarily a killer to use slightly old facts in most situations, though. > > I know I'm completely ignoring the issue of how to force a compilation > on demand in a development scenario and the fact that I don't know how > the time spent compiling the catalogue is spent. > > Does it make sense? I certainly understand what you're saying, but I'm not sure it's possible to solve the problems this way. Really, someone needs to spend a week or two buried in Puppet's compiler with dtrace or something similar. I made huge progress on the lexer for a client a few months ago, just because I'd never analyzed how fast the lexer was(n't). According to some work that Google has done recently, it looks like up to 70% of the time spent on the server is spent converting the catalog to YAML. The (now nearly mythical) REST support will include options to support different formats, although only yaml is likely to be provided out of the box. My plan is to make it relatively easy to support other formats, which should give us the opportunity to experiment in this area. So, if performance is the goal, really, the time is better spent optimizing existing code, rather than trying to avoid doing the work. -- Trying to determine what is going on in the world by reading newspapers is like trying to tell the time by watching the second hand of a clock. --Ben Hecht --------------------------------------------------------------------- Luke Kanies | http://reductivelabs.com | http://madstop.com --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/puppet-dev?hl=en -~----------~----~----~----~------~----~------~--~---
