[Puppet-dev] Re: Improving the Resources Resource Type
On 2014-07-11 18:28, Erik Dalén wrote:
On Fri Nov 07 2014 at 5:29:15 PM John Bollinger
mailto:john.bollin...@stjude.org>> wrote:
Having said all that, I can completely get behind Luke's idea to
provide for client-side queries. I am confused, however, as to why
the ideas for that seem to be running in directions such as
overloading resource titles and introducing new collector-like
forms, when the entire purpose of the Resources type is to serve
this kind of function. Why not just add a parameter to Resources to
specify the a query predicate (the default being something like
"!managed", and another to specify a hash of resource properties to
set, e.g. { ensure => absent }. Deprecate the existing 'purge' and
'unless_system_user' parameters. This would look similar to some of
the forms being proposed, but it would rely mostly on existing
facilities for everything other than actually performing queries.
Example:
resources {'remove-unmanaged-users':
name =>'user',
match =>"uid > 100 && !managed",
set=>{ensure=>'absent'}
}
I admit that I'm a bit leery of the "!managed" bit of the predicate,
as its presence suggests that maybe it can be omitted to allow
whatever is implemented for this to modify the properties of
/managed/ resources, too. This could be especially nasty if it were
possible to stack these things so that more than one applies to the
same resource. *I urge that that possibility be foreclosed*, at
least in the initial implementation. It can be added later if there
turns out to be sufficient justification, but it cannot easily be
taken away once granted.
I think this is sensible, likely the parser for this query should be
implemented in the munging method of the type though, so it can happen
server side and turn it into some sort of AST like format (which is what
I meant with something looking a bit like the PuppetDB query API).
Possibly it should allow users to submit that AST format directly, so
then you could have other parser functions to implement different
syntaxes for the query language.
as an example:
instead of match => "title='foo' and version='2.1.2'"
allow something like ["and", ["=", "title", "foo"], ["=", "version",
"2.1.2"]]
And then you could have a function that could return something like that
instead of having to turn it into a string that then gets parsed again.
That is what a query would produce as serialization if allowed as a
value expression. e.g.
match => <| title == foo and version = '2.1.2' |>
the big benefit is that the parser can validate the query, all querying
is expressed the same way, etc. Just as if this was done with a string,
any interpolation is done server side.
match => <| title == foo and version = $ver |>
i.e. $ver is evaluated and the result is stored in the resulting "baked"
query that is sent to the agent.
As part of getting rid of the old 3x AST, we just finished the rewrite
of the Collector. It is now much easier to do the collector expression
encoding transformation to "array form" or "ruby proc predicate" (used
internally for catalog collection and exported collection), and we can
now easily add an "array form" to "ruby proc predicate" transformation
that makes it really easy to use as a filter in a type.
- henrik
--
Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/
--
You received this message because you are subscribed to the Google Groups "Puppet
Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-dev/m3jrev%24mfb%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.
[Puppet-dev] Re: Improving the Resources Resource Type
On 2014-07-11 17:29, John Bollinger wrote: Having said all that, I can completely get behind Luke's idea to provide for client-side queries. I am +1 on that too. Much better idea. You can forget my earlier proposals. - henrik -- Visit my Blog "Puppet on the Edge" http://puppet-on-the-edge.blogspot.se/ -- You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/m3jqeb%2480i%241%40ger.gmane.org. For more options, visit https://groups.google.com/d/optout.
[Puppet-dev] puppet-dev status week ending 2014-11-06
** Next PR Triage Wednesday, November 12th @ 10:00 am Pacific. ** Priorities 1. Puppet 3.7.4 2. Native Facter 0.3.0 3. Puppet-server 0.4.0 4. Code removal for puppet 4 5. New puppet doc implementation Commentary This email is being put together a little differently this week. Normally I write this up by putting down what I’ve seen going on around me and what I can divine from various statuses in Jira. We’ll, this time it was a collaboration between me, Kylo, and Chris, which means that it has a bit more information about things that *aren’t* the language :) Puppet 3.7.3 was shipped. Now taking a look at 3.7.4. In 3.7.4 we are going to fix the “bug” in the language where strings that *look* like numbers are converted to numbers for comparison. This change is only happening in the future parser (and so will be default in puppet 4). Native Facter: nearing a 0.3.0 release (Solaris, Windows, ec2/gce facts). Also, we should have packages for the RHELs and Wheezy real soon now! Code removals are plowing ahead. So many that I’m not going to list them all. The big one that I’ve been watching is PUP-2906, which is up as a PR now and being reviewed. We’re beginning to scope some work related to changing the URL structure for the master/agent HTTP API just a bit. Specifically, we’d like to move the environment name away from being the first segment of the URL, so that the URL space is more well-contained and predictable. Puppet Server 0.4.0 is packaged and we’re targeting ship for Monday, Nov. 10. This release contains some minor bugfixes related to certificate generation, as well as some packaging improvements for a few platforms. The next release of Puppet Server after 0.4.0 will likely be our official 1.0 release, and will contain a few new features such as an API for flushing the environment caches. Puppet-dev conversations of note: - Improving the Resources Resource Type - We had a PR for the resources type...it scared us - The thread covers how the resources type *should* be handled. - Setting log levels in puppet.conf - The command line overriding the config file is great in a lot of cases, except for log level. - Consensus seems to be that making the command line --verbose and --debug flags mean “at least” rather than “exactly” those levels is the most sensible approach. - Leaving Puppet Labs - Yep, I’m leaving. :( - The string to number torture never stops - This is the conversation that cemented that we would remove the automatic conversion for puppet 4. It was unanimous that it just causes problems. Data Let’s spend some time admiring these amazing charts from Chris’s PuppetConf talk about the Puppet Server. -- Andrew Parker a...@puppetlabs.com Freenode: zaphod42 Twitter: @aparker42 Software Developer *Join us at **PuppetConf 2015, October 5-9 in Portland, OR - * http://2015.puppetconf.com *Register early to save 40%!* -- You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CANhgQXswfhXB%2BTus--b3o-_7MPSVwE9RUUSd8mtKKupKWhJ9kw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: [Puppet-dev] Re: Improving the Resources Resource Type
On Fri Nov 07 2014 at 5:29:15 PM John Bollinger wrote: > > > On Wednesday, November 5, 2014 8:37:08 PM UTC-6, henrik lindberg wrote: >> >> Hi, >> I am bringing up this topic because of a recent discussion and PR for >> the ticket https://tickets.puppetlabs.com/browse/PUP-1486 >> >> Here is a recap. >> >> The Resources resource is used to "manage unmanaged" resources of a >> particular given type - e.g. to purge users, groups etc. that are not >> otherwise managed. >> >> > > Purging unmanaged resources is what Resources can do now, but I've always > thought the original idea was much grander. The type's own documentation > is couched partially in terms of generating resource instances, which fits > with "managing [otherwise] unmanaged resources", but it also described as > "managing other resource types", which sounds like something different. I > suspect that some of the features that might otherwise have been > implemented via the Resources type went other places instead (especially > into collectors). > > >> The issues are that "managing the unmanaged" may require additional >> attributes that are specific to the type that is referenced. The base >> type only has the attributes "unless_system_user" and "unless_uid" (se >> https://docs.puppetlabs.com/references/latest/type.html#resources for >> the documentation). >> > > > Yes, I think this essential problem stunted the development of the > Resources type. > > > Instead, I would like to see something where the concerns are separated >> (to avoid the problem of "the base class knows about all the things"). >> One good pattern is "the strategy pattern" / "adapter pattern" which is >> used internally in the future parser. > > > > Most everyone here should be deeply familiar with this pattern, though > they may not know it by name. Puppet has relied on it forever in the form > of resource providers. > > I'm not sure it quite fits the situation, though, because fundamental to > the Strategy pattern is a common view of the inputs directing the work to > be performed. For strategies to be applicable to a very general problem, > such as "managing unmanaged resources", the form of the inputs must be very > general. If you went down this path you would soon have to consider what > you are gaining by using the Resources type as a front end. > > > >> One very simple approach is to add the basic ability to handle such >> "adapters", or "extra data" if you like, in the Resources type. We >> cannot use the actual adapter pattern since when it comes to resources, >> they are serialized and processed in ways that would drop such adapters >> (since those are really a runtime concept). >> >> > > I'm not buying that. The adapter pattern works fine for resources in > general, including specifying non-default strategies (providers) for > particular resources. This is a problem that could be solved for Resources > -- for instance, by using the existing provider system, as Felix suggested. > > > >> Proposal: >> >> * Add the attribute options (with the type Array[Resource]) >> * Add the ability in a resource type to specify a list of type names >> that it allows instances of as options (typically one) >> * Add the ability in type to get the options. >> >> > > So the idea here is what? Resource instances for unmanaged physical > resources of the specified type(s) are created, with the specified > parameters, to manage those physical resources? > > Or are the options supposed to effectively be resource-type-specific > parameters to some sort of "things_Resources_can_do_with_this_type()" > function that all resources will provide? If the latter, then what are the > implications for providers? > > > >> How the rest of the protocol between Resources and the types that >> support options should work is left to be designed, but basically; the >> Resources implementation should present the options to the type either >> for just asking if a particular resource should be purged or not (as it >> does now but done as a delegation to the particular type). >> > > > Are you supposing that this would be limited to purging unmanaged > resources, or is that just an example? I know there are issues with > purging resources of types that don't prefetch or that aren't Ensurable, > but if that's all you're going after then this sounds a bit out of > proportion. > > > >> >> Alternative - using the type system in Puppet 4.0 >> >> Another alternative is to simply use the new type system in Puppet 4.0. >> A type that supports options to Resources returns a Struct type that >> describes the options. (A Struct is a detailed type specification of a >> hash). (Read more about the Struct type here: >> http://puppet-on-the-edge.blogspot.se/2014/02/adding- >> struct-and-tuple-to-puppet-type.html) >> >> In the catalog a Struct type is simply encoded as a String, and it is >> easily converted back to an instance of the type which can be used for >> type checking the options received from the Resource
Re: [Puppet-dev] How to run tests on custom report parser
On Nov 4, 2014 6:32 AM, "tracyde" wrote: > > I am new to ruby development and want to integrate my report parser with TravisCI. I have read articles and blog posts on how to integrate Puppet modules with rspec, puppet-lint, and travisci but those are not directly applicable to modules that do nothing but provide lib instances (facts, report parsers, etc.) > Alas, I have had my eye out for documentation or utilities for testing report processors for a while and have seen nothing. For that matter, even figuring out what the incoming data looked like room a bit of digging -- I just read the YAML dump of a 'store' report and assumed I knew what values meant. My guess is that testing would involve one or more store report YAML files and just enough framework to deserialize that and pass the resulting data to your report processor. Then you'd need to stub the objects & methods your processor calls for output. I'm not sure you'd actually need rspec-puppet -- I don't think it provides any facilities that would be useful in this case. Wil Wil -- You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/CAMmm3r5tbTuu6b3WUhGTasXsoCai3NiAaR%3D97%3DpPEKJcA%2BPgnA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
[Puppet-dev] Re: Improving the Resources Resource Type
On Wednesday, November 5, 2014 8:37:08 PM UTC-6, henrik lindberg wrote: > > Hi, > I am bringing up this topic because of a recent discussion and PR for > the ticket https://tickets.puppetlabs.com/browse/PUP-1486 > > Here is a recap. > > The Resources resource is used to "manage unmanaged" resources of a > particular given type - e.g. to purge users, groups etc. that are not > otherwise managed. > > Purging unmanaged resources is what Resources can do now, but I've always thought the original idea was much grander. The type's own documentation is couched partially in terms of generating resource instances, which fits with "managing [otherwise] unmanaged resources", but it also described as "managing other resource types", which sounds like something different. I suspect that some of the features that might otherwise have been implemented via the Resources type went other places instead (especially into collectors). > The issues are that "managing the unmanaged" may require additional > attributes that are specific to the type that is referenced. The base > type only has the attributes "unless_system_user" and "unless_uid" (se > https://docs.puppetlabs.com/references/latest/type.html#resources for > the documentation). > Yes, I think this essential problem stunted the development of the Resources type. Instead, I would like to see something where the concerns are separated > (to avoid the problem of "the base class knows about all the things"). > One good pattern is "the strategy pattern" / "adapter pattern" which is > used internally in the future parser. Most everyone here should be deeply familiar with this pattern, though they may not know it by name. Puppet has relied on it forever in the form of resource providers. I'm not sure it quite fits the situation, though, because fundamental to the Strategy pattern is a common view of the inputs directing the work to be performed. For strategies to be applicable to a very general problem, such as "managing unmanaged resources", the form of the inputs must be very general. If you went down this path you would soon have to consider what you are gaining by using the Resources type as a front end. > One very simple approach is to add the basic ability to handle such > "adapters", or "extra data" if you like, in the Resources type. We > cannot use the actual adapter pattern since when it comes to resources, > they are serialized and processed in ways that would drop such adapters > (since those are really a runtime concept). > > I'm not buying that. The adapter pattern works fine for resources in general, including specifying non-default strategies (providers) for particular resources. This is a problem that could be solved for Resources -- for instance, by using the existing provider system, as Felix suggested. > Proposal: > > * Add the attribute options (with the type Array[Resource]) > * Add the ability in a resource type to specify a list of type names > that it allows instances of as options (typically one) > * Add the ability in type to get the options. > > So the idea here is what? Resource instances for unmanaged physical resources of the specified type(s) are created, with the specified parameters, to manage those physical resources? Or are the options supposed to effectively be resource-type-specific parameters to some sort of "things_Resources_can_do_with_this_type()" function that all resources will provide? If the latter, then what are the implications for providers? > How the rest of the protocol between Resources and the types that > support options should work is left to be designed, but basically; the > Resources implementation should present the options to the type either > for just asking if a particular resource should be purged or not (as it > does now but done as a delegation to the particular type). > Are you supposing that this would be limited to purging unmanaged resources, or is that just an example? I know there are issues with purging resources of types that don't prefetch or that aren't Ensurable, but if that's all you're going after then this sounds a bit out of proportion. > > Alternative - using the type system in Puppet 4.0 > > Another alternative is to simply use the new type system in Puppet 4.0. > A type that supports options to Resources returns a Struct type that > describes the options. (A Struct is a detailed type specification of a > hash). (Read more about the Struct type here: > > http://puppet-on-the-edge.blogspot.se/2014/02/adding-struct-and-tuple-to-puppet-type.html) > > > > In the catalog a Struct type is simply encoded as a String, and it is > easily converted back to an instance of the type which can be used for > type checking the options received from the Resources type. > > I know that the type system is a shiny new toy, and it has lots of promise, but it seems a lot lower level than the Resources
Re: [Puppet-dev] Setting log levels in puppet.conf
On 11/04/2014 10:46 PM, Josh Cooper wrote: > 2. Change --test to imply --debug instead of --verbose I don't think this is an option. Many users are likely in the habit of using --test whenever actually testing or even manually running the catalog from the master. The percentage of those that would want to get the debug details all over their terminals each and every time is most likely insignificant. -- You received this message because you are subscribed to the Google Groups "Puppet Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/545CD4DA.1090507%40alumni.tu-berlin.de. For more options, visit https://groups.google.com/d/optout.
Re: [Puppet-dev] Improving the Resources Resource Type
On 11/07/2014 05:59 AM, Luke Kanies wrote:
>> To get a behavior that matches today's resources type, we'd need to
>> > explicitly query unmanaged resources.
>> >
>> > User<~| managed == false |~> { ensure => absent }
>> >
>> > This has the advantage of being more explicit about the unmanaged part than
>> >
>> > resources { 'user': purge => true }
> Agreed.
>
> We do need some way to make it clear that this query happens on the client,
> rather than on the server, and then the catalog needs some way to store that
> query.
>
> The only other way to do it would be to have the server be able to do
> real-time queries against the client, but I think that would have too many
> crazy consequences.
>
> It’s been a long time, but I seem to remember trying to treat the resource as
> a query; something like:
>
> user { *:
> uid >= 100,
> purge => true
> }
>
> But I think this is not the right answer.
>
> Maybe something like:
>
> user { <| uid > 100 |>:
> purge => true
> }
>
> Then the title is the query, and the parameters get applied to all resources
> that match that title.
>
> Thoughts?
>
I think I would prefer a mix of both approaches:
user { 'arbitrary-purger-title':
match => <| uid > 100 && !managed |>,
ignore => <| title == sysmaint |>,
ensure => absent,
}
This would have the benefit that the user can
exec { 'tidy-all-home-dirs: require => User['arbitrary-purger-title'] }
at the expense of more metaparameters, putting restraints on type
authors wrt. attribute names that they can use for their own types.
Hell, we have at least one core type that uses "ignore" even today.
So perhaps a more dedicated syntax like
user { 'arbitrary-purger-title':
<| uid > 100 && !managed |>,
...
}
has more merit.
On 11/07/2014 05:39 AM, Luke Kanies wrote:> On Nov 7, 2014, at 2:02 PM,
Joshua Hoblitt wrote:
>> How would make use of that information on the agent if expressions are
>> > evaluated on the master? As an example, I'd like to be able to 'query'
>> > the home dir of a role account (without having to install a fact) and
>> > interpolate it into a string to form a path. I've always found this
>> > scenario a bit frustrating as the type has the data I want; it's just
>> > inaccessible while the catalog is built.
> As others have mentioned, the only way this is possible is if we find
a way to send the query down to the client. We need different queries
for different locations, but hopefully similar enough that it doesn’t
like a completely different system, like it is today.
There be dragons.
System entities are queried at the start of the agent transaction at the
moment. The transaction starts after the catalog has been retrieved. We
would have to intertwine these steps, and this would raise a plethora of
problems, I imagine.
For example - you cannot be sure what provider the catalog will end up
assigning to the resource that you are querying before everything is
properly resolved. It would be awkward if a query is performed using the
default provider, but the catalog ends up specifying that the user in
question be managed through LDAP, for example.
The compiler calling back to the agent during its catalog request would
be a major game changer. I'm thinking "whole new paradigm".
Cheers,
Felix
--
You received this message because you are subscribed to the Google Groups
"Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-dev/545C88B5.7020707%40alumni.tu-berlin.de.
For more options, visit https://groups.google.com/d/optout.
