On 2014-12-02 17:16, Brice Figureau wrote:
On Wed, 2014-02-12 at 15:15 +0100, Henrik Lindberg wrote:
On 2014-12-02 9:43, Brice Figureau wrote:
Discussed this with zaphond42 today and I want to try an approach that
is different - since we will be visiting all directories at least once
when looking for resource type implementations in ruby, we could just as
well just cache their names up front. That way, subsequent searches are
simply a hash lookup, and we can continue to lookup ruby plugin resource
types before user defined.
This works when we change the semantics to perform a stale check only
before each request is processed. I am counting on that there are far
fewer plugin types than there are definitions.
Since the plan is to not load all of them upfront, but only record their
existence the entire operation should be fast enough even if it will map
unused types.
That sounds great, if it solves the agent case too. IMHO we are focusing
a bit too much on the master performance :)
I think it will.
Lookup of user defined definitions could also benefit from similar
caching of directory entries - with the goal of only visiting each
directory once, but it is a bit more complicated to implement.
The current way of loading definition works fine, because it is done
during parsing, so the resource lookup is just a hash lookup, it doesn't
involve reparsing everything. In the agent, the known_resource_types is
empty and that means we constantly hit the type loader. If the type
loading scheme you propose also works in the agent, then we'll be mostly
fine except we won't be able to cache the definitions resources types
(and thus will hit the type loader every time). That was the root of my
ideas of marking resources, because once we know what kind a resource
is, everything becomes simpler, as you don't have to guess what kind of
type your resource is, and it can be looked up more easily.
Not sure what you are saying about known_resource_type - are you saying
it is always empty? What I am proposing will result in 1 scan of all
plugin resource types - then when a lookup of a type name is performed
it first checks if this could be a plugin type (one hash lookup), if not
then it must be a user defined type. Is there something in that you
suspect will not work on the agent side?
I meant that on the agent known_resource_type is always empty. That's
the root of the issue I discovered in PUP-1592: we're constantly calling
Puppet::Type.type() because we can't cache the resource_type result.
The reason is Puppet::Type.type() returns nil (and scans all the paths
each time) because this is a definition, and the last resort check
(known_resource_type.definition) also returns nil because this hash is
valid only on the master, so ultimately we return nil (which is not an
issue in the agent) which by definition renders the cache unuseful.
So yes, your idea will make sure we don't scan the disk (except once),
but we'll still not cache the resource type and call Puppet::Type.type
everytime. If your solution makes it only a negative cache lookup, I
think this is quite safe (on a performance standpoint).
I am now looking at implementing the new cache in metatype/manager since
it seems to be the manager of the autoloader that is used. Either that,
or in the autoloader itself.
That is where Puppet::Type.type(name) ends up (i.e. metatype/manager).
- henrik
--
You received this message because you are subscribed to the Google Groups "Puppet
Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-dev/ldg9i9%24tje%241%40ger.gmane.org.
For more options, visit https://groups.google.com/groups/opt_out.
