On Mar 18, 2009, at 1:01 PM, Paul Nasrat wrote:
>
>> As mentioned in an email yesterday, I'm working with someone to add a
>> queueing service to Puppet so that some server-side operations are
>> queued and executed as cpu time is available - generally those that
>> aren't on the critical path but particularly the storeconfigs save
>> operation.
>
> Sounds interesting.
>
>> The goal here is that the client wouldn't have to wait for the server
>> to finish storing the catalog before it got its copy of the catalog;
>> instead, the server would just put the catalog into the queue, and a
>> queue reader would come behind and process as time was available.
>> The
>> only other operation I can think of where this makes sense right now
>> is reporting, but I expect we'll have others over time.
>
> Do we have a summary of current performance and some perfomance tests
> so we can objectively measure improvements? If so getting them visibly
> graphing from hudson builds of that branch would be awesome.
That would be great to have, but no, we have neither that nor even an
objective idea of performance.
We probably need to come up with a couple of standard configurations
with a given amount of complexity and then start tracking the times
those things take. We could stick entirely to 'notify' resources, so
that we're not hitting the disk at all.
Anyone interested in taking this up?
>
>> The big benefit here, of course, is that you get configuration
>> storage
>> out of the critical path for returning catalogs to clients, and this
>> has the benefit of making the client queries shorter and thus less
>> likely to overlap. It also means that if you get a big rush of
>> client
>> connections, we can queue up the more expensive operations, and even
>> if storeconfigs eventually becomes cheap, it would still make sense
>> to
>> queue anything that wasn't on the critical path.
>
> Do we have a set of requirements that we need both for functionality
> and for ie do we need persistence, etc)
Ethan, Steven, and I have developed largely complete requirements for
functionality, but, erm, no, we don't have a literal requirements
document or anything. Yes, persistence is required. Hopefully Ethan
can send out a summary of the requirements list he's developed.
>
>> So, I'm looking for input on what people think the right tools and
>> architecture are for this, and just generally how we should go about
>> it. I don't want to have a six week discussion on it, but I do want
>> to make sure we get as much feedback as possible during development,
>> and we'll be doing normal code review and publishing code as often as
>> possible.
>
> Without the requirements it's hard to say exactly, I'd probably look
> at starling/workling, beanstalk, and nanite amongst others.
>
> If we have good criteria we could do some time boxed evaluation to
> choose best fit for puppet at the moment, but ensuring that we
> implement it in a pluggable way (eg choose a pluggable transport or
> standard such as amqp, thrift, protocol buffers )
Yeah, the goal is to make it relatively pluggable, but it'll certainly
be a whole new category of functionality for me, so I know at least
I'll take some time to get up to speed.
--
Brand's Shortcut:
The only way to predict the future is to make sure it stays
exactly the same as the present.
---------------------------------------------------------------------
Luke Kanies | http://reductivelabs.com | http://madstop.com
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"Puppet Developers" group.
To post to this group, send email to puppet-dev@googlegroups.com
To unsubscribe from this group, send email to
puppet-dev+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/puppet-dev?hl=en
-~----------~----~----~----~------~----~------~--~---
