Hi,
Ideas regarding a potential performance boost that can be gained by
performing batch processing of package installs/operations has been
floating around in the Puppet echo system for quite some time.
There is a discussion (and a somewhat dated implementation/proposal) in
http://projects.puppetlabs.com/issues/2198 which is good background
reading for this topic.
In issue #2198 (if you skipped reading it ;-)), the idea is that Puppet
should have the feature to install a list of packages given by the user.
It seems doable to generalize this idea and let puppet automatically
optimize package installs under certain conditions. Performing
individual package installs is quite expensive and even if the
optimization opportunities may not be extensive (e.g. say that 20%
(number completely made up) of packages could at least be paired with
one other package) this is still a worth while activity.
To kick this off, we need to do some research and design. So, here is an
attempt to get this started by asking a bunch of questions.
Under what conditions can two (or more) packages operations be batched?
-----------------------------------------------------------------------
As an example, say that a class contains a series of package resources
without any explicit dependencies between them. The idea is that this
could be optimized. Are there any conditions that makes this impossible?
What if the resources are chained with explicit dependencies? (Guess is
that the dependencies were added for a reason, and should be done as
individual dependencies).
What if the list of packages are of different type? Is an chain of
implicit dependencies between packages of the same type required to make
it possible to batch them? (Does it depend on the policy for implicit
dependencies; parse-order, random, etc.)?
What if there are other implicit dependencies. Can it be deduced that an
intermixed resource has no effect on the outcome of a following package
operation? (Exec's can for sure do things).
Is it possible to optimize across class boundaries?
Is it enough to look at the queue of actions in the "planned catalog"
and simply look-ahead for packages handled by the same provider. An
unbroken chain of operations handled by the same provider is collected
and then handed off to the provider? Does this provide enough
optimization, or are we then likely to miss optimization opportunities?
How can we collect metrics for this?
What needs to be done to providers?
-----------------------------------
Clearly the capability to handle multiple requests must be implemented
for package managers that support this. What should the API look like?
What needs to be done to the Package type?
------------------------------------------
Is it all an ordering issue and handing off resources to the provider,
or do we need to do things to the Package type as well?
Are there situations were it is of value to veto batching per resource?
(depending on how much optimization than can be deduced by looking at
resource-dependencies).
Explicit group/list?
--------------------
If we want users to be able to explicitly give a group of packages to
manage - how should that work? A new resource type? An attribute on
Package? A defined type?
If we cannot optimize across classes, can we support explicit
grouping/batch operation? (Seems complex with yet another containment
hierarchy - or can this be done by introspecting a dependency chain of
custom resources/classes perhaps used specifically for this purpose?
- henrik
--
You received this message because you are subscribed to the Google Groups "Puppet
Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-dev+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-dev@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-dev.
For more options, visit https://groups.google.com/groups/opt_out.
