I'd certainly be interested in working this through more at PuppetConf if
there's time.

I also don't see why this needs to wait until 5.0. It should be 100%
backward compatible with the 4.x series in theory (it would be nice anyway).

Thanks!

Trevor


On Thu, Jun 19, 2014 at 6:41 PM, Henrik Lindberg <
[email protected]> wrote:

> On 2014-19-06 23:25, John Bollinger wrote:
>
>> On Wednesday, June 18, 2014 6:07:02 PM UTC-5, henrik lindberg wrote:
>>
>>     On 2014-18-06 22:44, John Bollinger wrote:
>>      >
>>      > Puppet already has a tool for separating the details of a
>>     resource from
>>      > the decision about whether to include it in the catalog: virtual
>>      > resources.  Those decisions can be tied to whether a given class is
>>      > declared by having that class realize or (especially) collect the
>>      > resources.  In many cases that can be done via a wrapper class or
>>      > definition if the target class itself must not be modified.
>>      >
>>      > Here's an idea: Collectors are already processed very late, so
>> maybe
>>      > they are late enough -- or could be /made/ late enough -- to
>>     provide the
>>      > kind of context needed for safe reflection.  One might then write
>>      > something along these lines:
>>      >
>>      > Firewall::Rule<| title == 'whatever' and defined == 'firewall' |>
>>      > @firewall::rule { "whatever":
>>      >    # ...
>>      > }
>>      >
>>      > Of course, the collector wouldn't need actually to be close to the
>>      > resource declaration, and it might be more general than just one
>>      > resource, but it could be right there with the resource
>>     declaration to
>>      > provide something much like an if defined() effect.
>>      >
>>     The only thing this does is to move the problem to a new "phase".
>>     Since the purpose is to add resources, naturally you can add defined
>>     types, which in turn creates many resources. The set of rules then
>>     operate on a moving target (constraint solving).
>>
>>
>>
>> I'm thinking about something more restricted than it may seem I've
>> suggested: an ability only to reflect which *classes* are in the
>>
>> catalog, in a context and time where no new classes can be added (but
>> resources can).  That characterization draws on my position that classes
>> should not be considered resources, at least at DSL level, but you can
>> say instead "(but resources other than classes can [be added])" if you
>> prefer.
>>
>>      > For that to be safe though, it would be necessary to prevent the
>>      > collected resources from adding classes to the catalog, probably by
>>      > failing the catalog if any of them try to do so.  That
>>     restriction might
>>      > be reserved for when the 'defined' key is used in the collector's
>>     select
>>      > expression, provided that such collectors were processed after
>>     all others.
>>      >
>>     I don't think we absolutely have to protect classes from being
>>     included..
>>
>>     In a way they are just resources only singletons.
>>
>>
>>
>> The idea was that this limited form of reflection escapes
>> evaluation-order dependencies because it draws on information that will
>> not subsequently change -- to wit, which classes are in the catalog.
>> That's the reason for preventing new classes from being added.
>>
>>  A new mechanism is then needed, since it is not possible to statically
> derive the set of classes that will actually be included in the catalog and
> what parameters they will have. It is possible (given a few caveats) to
> know the set of classes that can potentially be included, but that is about
> it.
>
>
>  Now that I think about it, though, I'm no longer sure that would be
>> sufficient.  I think this would introduce a new form of evaluation-order
>> dependency involving the effects of collectors, since the full set of
>> resources selected by any given collector can be determined reliably
>> only after all resources that will be added to the catalog have been
>> added, at least virtually.  On the other hand, maybe this problem isn't
>> new -- it seems like collectors can already suffer from this problem on
>> their own.
>>
>> And I think any solution to that problem creates its own problems.
>> Consider this artificial example:
>>
>> define loop::a ($index = 0) {
>>    @loop::a { "a$index": index => $index + 1 }
>> }
>>
>> @loop::a { 'start': }
>>
>> Loop::A<| |>
>>
>> Obviously, you shouldn't write infinite loops, but my point is that on
>> the one hand, whether that is in fact an infinite loop depends on the
>> evaluator implementation, but on the other hand, an evaluator for which
>> it is NOT an infinite loop does not reliably collect all resources that
>> should be collected.  (And on the third hand, it might not even be
>> possible to collect all the resources that should be collected.)
>>
>>
> You can do loops now with resources. If you can also evaluate code as a
> side effect of collection (or when dependencies are formed), one more such
> looping possibility is added. IMO this makes things worse.
>
>
>      I am thinking that a module can specify constraints as a predicate
>>     (either on other modules being present, other classes, or resources,
>> or
>>     indeed on some abstract "capability"), and if the predicate becomes
>>     true
>>     at the end of the catalog production, then it includes one class from
>>     the module.
>>
>>
>>
>> But if a class is added as a result of that mechanism, then it turns out
>> that the predicate wasn't evaluated at the end of catalog production
>> after all.  That's not an issue for just one such constraint, but as
>> soon as you have two or more you are right back to having an
>> evaluation-order problem.  And consider also the issue with collectors
>> that I raised above -- I think that probably applies here, too.
>>
>>
>>     This adds a much asked for feature; autoinclude. Inclusion in the
>>     catalog by just adding a module to the modulepath.
>>
>>
>>
>> As opposed to adding "include 'mymodule'" to the top level of the
>> starting-point manifest?  Or adding a file containing that to the
>> starting-manifest directory?  What's hard about that?  The latter could
>> even be arranged as part of module installation.
>>
>> Or why not have a separate module path for autoinclude modules?  Users
>> then select autoinclude behavior by where they put the module -- or even
>> by where they put a symlink to the module.
>>
>> I think simple autoinclude is a relatively tractable problem, but I
>> guess you're looking at conditional autoinclusion.  Honestly, I'm not
>> very comfortable with that. Moreover, I'm inclined to think that
>> whatever framework might be devised for evaluating which of the
>> available modules to "auto"include would be no simpler, better, or
>> easier than what we already have.
>>
>>     (This is the reverse
>>     dependency I was talking about earlier).
>>
>>     In all of these cases, some kind of constraint solver ability is
>> needed
>>     across the new feature, queries and the forming of dependencies (since
>>     you most certainly need to also have control over the order in which
>>     the
>>     "injected" classes/resources are later applied.
>>
>>
>>
>> I think you're saying there needs to be a way to set up dependencies
>> between classes and resources added to the catalog during the "normal"
>> evaluation phase and those added as a result of this hypothetical coda.
>> That supposes, I guess, that those cannot be declared either during the
>> normal evaluation pass (e.g. via a collector) or during the additional
>> one.  I'm uncertain whether that's true.
>>
>>  Dependencies are evaluated after all classes and resource have been
> evaluated. An operand of a dependency may be a query (a.k.a collection).
> Therefore, dependencies are added last (I think)). Until that point, all
> references are unchecked "future references".
>
> The issues with dependencies and conditional inclusion is that neither
> side (the one that causes the inclusion as a side effect, or the side that
> is being included) knows enough. (Maybe this is an imaginary problem and
> dependencies are more coarse grained).
>
>
>      The other approach I can think of (I wrote about it earlier) is that
>>     you
>>     make explicit statements about things that must have been evaluated
>>     before "you" can be evaluated (i.e. "I need these Promises fulfilled /
>>     defined before I can say what I want", and modules are given the
>>     ability
>>     to define such promises. This (as John also points out) requires a
>>     context in which a resolution of available Promises can take place
>>     reliably). Maybe a Promise is nothing else but a class, but that you
>>     can
>>     tie your evaluation to the outcome to the evaluation of one or many
>>     other classes. (This is an area I hope to be able to work on ideas
>>     after
>>     we have released puppet 4).
>>
>>
>>
>> Basically that's a way to constrain evaluation order, right?  Such a
>> mechanism would indeed address the core problem, and conditional
>> evaluation can falls out as a special case.  It might even fit in easily
>> with the current (future) evaluator, as it seems like it has some
>> parallels with 'include'.  So, yes, this seems like a promising
>> direction to explore.
>>
>>  Yes it is, and in a safe way. (I think this is the most promising, the
> other ways to solve it that have come up all seem to result in a game of
> Whack-a-Mole).
>
> And, to expand on this idea a little. I think the exact same mechanism is
> applicable when the resources are applied on an agent. i.e. it is an
> evaluation plan, workflow, or orchestration if you like.
>
> I doubt I will have time to work on these ideas before Puppet Conf - I
> don't have this in presentable form yet, maybe something that can be shared
> by then. Some of these ideas have implications that we are thinking about
> and working into Puppet 4.0. We also considered these ideas for the design
> of the next phase; the catalog builder (replacement for the current
> compiler) - that works will take place during the 4x series with the intent
> of releasing it as Puppet 5.0.
>
>
> - henrik
>
> --
>
> Visit my Blog "Puppet on the Edge"
> http://puppet-on-the-edge.blogspot.se/
>
> --
> You received this message because you are subscribed to the Google Groups
> "Puppet Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/puppet-dev/lnvots%24eod%241%40ger.gmane.org.
>
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Trevor Vaughan
Vice President, Onyx Point, Inc
(410) 541-6699
[email protected]

-- This account not approved for unencrypted proprietary information --

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to [email protected].
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoX_U-T0i5ZEuBQ1rH0LxT11yF5qDrxhs4gctf%3DYky4Jog%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Reply via email to