[Puppet-dev] Re: Custom Hiera Interpolation Functions

[Henrik Lindberg]

On 18/04/18 03:56, adam.gard...@magicmemories.com wrote:
So, I'm aiming to have a method of interpolating Facter facts in Hiera 
data files that works more or less exactly like the `scope` lookup 
function, except that if the Facter fact is not defined, this 
interpolation should result in Hiera moving on to the next hierarchy 
level as if the key were not found, rather than simply interpolating an 
empty string.


I've managed to make a rough hack 
of https://github.com/puppetlabs/puppet/blob/master/lib/puppet/pops/lookup/interpolation.rb 
that seems to accomplish what I'm looking for, but I'm not really sure 
how to work with Puppet's test infrastructure, nor whether this is 
really an idea the Puppet team would actually be interested in.


My actual goal is to allow Jenkins to run the Puppet Agent on a CI node 
while setting an environment variable such as `FACTER_myapp_version`, 
and have a node-specific YAML file that reads `profile::myapp::version: 
"%{maybe('facts.myapp_version')}"`, and thus allow Jenkins to control 
the version of the app that gets deployed; at the same time, during a 
run which doesn't set that particular environment variable I'd want 
Hiera to continue looking in other hierarchy levels, not just say that 
profile::myapp::version is an empty string. And, I do want to control 
this in Hiera if at all possible, not using 
`pick($facts['myapp_version'],lookup('profile::myapp::version'))` in 
puppet code, or anything similar.


So, I have three questions:

 1. Is there an easier way to accomplish this that I'm not seeing, with
a custom Hiera backend or something similar?


With a custom hiera 5 backend function you can do interpolation any way 
you like. The easiest is probably do do one that does exactly what you 
want rather than a configurable one. For your case you can probably 
write it in the puppet language in just a couple of lines of code.



 2. If not, is there a system for implementing custom interpolation
functions as anything other than a patch to the
`interpolation_method` method in
lib/puppet/pops/lookup/interpolation.rb? (I couldn't find one, but
that doesn't mean one doesn't exist)


There is no such extension mechanism for hiera interpolation functions.


 3. If not, would anyone actually be interested in a pull request for this?



Rather than adding/inventing new hiera interpolation functions (there 
will never be an end to the requests people will have). I would rather 
want to support a format where interpolation is done using puppet 
notation. You can then call whatever functions you want.


That is, the function would be a lookup_key kind of function.
It would read a yaml file and cache it, then when returning a value it 
would use the Puppet::Pops::EvaluatingParser class and call 
evaluate_string. The evaluated string is returned.


Another alternative is to write a backend that supports a .pp file as 
data with the expected result of a hash. Then the function is more or 
less a wrapper for EvaluatingParser's method 'evaluate_file'.


If you want to do one of those, and you want to work on a generic 
solution, (a .pp backend, either yaml + PP interpolation, or just .pp),

then I am happy to help out with the details, review etc.

- henrik

--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups "Puppet 
Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/pb7s92%2498p%241%40blaine.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: Ephemeral resources

[Henrik Lindberg]

On 16/04/18 23:13, Trevor Vaughan wrote:
I thought there were dangers with Virtual Resource being accidentally 
realized sometimes?




Not without users doing something bad (like realizing all) - puppet 
itself does not realize virtual resources willy-nilly.


We found a super-hacky way to call functions at the end of a compile 
which we use in 
https://github.com/simp/pupmod-simp-compliance_markup/blob/master/manifests/map.pp. 
It would be *really nice* if there were an actual 'hook in here, no 
resources will be processed after this point' hook.




What if the logic called "after all resources processed" creates new or 
modifies existing resources? Also, should it run after 
collection/overrides or before?


- henrik



Trevor

On Mon, Apr 16, 2018 at 5:00 PM, Henrik Lindberg 
<henrik.lindb...@puppet.com <mailto:henrik.lindb...@puppet.com>> wrote:


On 16/04/18 17:38, Trevor Vaughan wrote:

How difficult would it be to create a third type of resource
which is an 'ephemeral resource' whose only purpose is data
collection on a host to be used by some other collector?

These items would not be part of the catalog or added to the
graph but would instead just hang around for reference during
compilation.

This would fix the catalog explosion issue when you start doing
exported resources based on large numbers of things and/or
things like firewall rules and copious file_line resources.

Basically, a 'data' -> 'collector' pattern where you can
optimize...well...everything into a MUCH smaller catalog that is
sent to the client for processing.


Sounds a bit like the existing virtual resources, but with a better
collection mechanism. Would not be too difficult to write a function
that takes a data type predicate to match against virtual resources
data type predicate) and then calling a lambda with each.

Virtual resources do not end up in the catalog unless they are realized.
With the function I imagined, you would select virtual resources and
then do whatever you want in the lambda.

The function should probably return an Iterator over the resources.
That can then be iterated with each, map, or reduce.

The issue then is when to call that function - you want it at the
very end which we do not have a mechanism for.

The virtual collector could be modified to accept a lambda - since
collection runs late it would be at the right time. This is a much
bigger change naturally as it changes the language.

- henrik


-- 


Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/
<http://puppet-on-the-edge.blogspot.se/>

-- 
You received this message because you are subscribed to the Google

Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to puppet-dev+unsubscr...@googlegroups.com
<mailto:puppet-dev%2bunsubscr...@googlegroups.com>.
To view this discussion on the web visit

https://groups.google.com/d/msgid/puppet-dev/pb32pa%24d3q%241%40blaine.gmane.org

<https://groups.google.com/d/msgid/puppet-dev/pb32pa%24d3q%241%40blaine.gmane.org>.

For more options, visit https://groups.google.com/d/optout
<https://groups.google.com/d/optout>.




--
Trevor Vaughan
Vice President, Onyx Point, Inc
(410) 541-6699 x788

-- This account not approved for unencrypted proprietary information --

--
You received this message because you are subscribed to the Google 
Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send 
an email to puppet-dev+unsubscr...@googlegroups.com 
<mailto:puppet-dev+unsubscr...@googlegroups.com>.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoUUaCqmHhKjUWiFRyqFF3%2Bc5BpgLEHe-faYZxcyUutcGA%40mail.gmail.com 
<https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoUUaCqmHhKjUWiFRyqFF3%2Bc5BpgLEHe-faYZxcyUutcGA%40mail.gmail.com?utm_medium=email_source=footer>.

For more options, visit https://groups.google.com/d/optout.



--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups "Puppet 
Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/pb42so%24oqc%241%40blaine.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: Ephemeral resources

[Henrik Lindberg]

On 16/04/18 17:38, Trevor Vaughan wrote:
How difficult would it be to create a third type of resource which is an 
'ephemeral resource' whose only purpose is data collection on a host to 
be used by some other collector?


These items would not be part of the catalog or added to the graph but 
would instead just hang around for reference during compilation.


This would fix the catalog explosion issue when you start doing exported 
resources based on large numbers of things and/or things like firewall 
rules and copious file_line resources.


Basically, a 'data' -> 'collector' pattern where you can 
optimize...well...everything into a MUCH smaller catalog that is sent to 
the client for processing.




Sounds a bit like the existing virtual resources, but with a better 
collection mechanism. Would not be too difficult to write a function 
that takes a data type predicate to match against virtual resources

data type predicate) and then calling a lambda with each.

Virtual resources do not end up in the catalog unless they are realized.
With the function I imagined, you would select virtual resources and 
then do whatever you want in the lambda.


The function should probably return an Iterator over the resources. That 
can then be iterated with each, map, or reduce.


The issue then is when to call that function - you want it at the very 
end which we do not have a mechanism for.


The virtual collector could be modified to accept a lambda - since 
collection runs late it would be at the right time. This is a much 
bigger change naturally as it changes the language.


- henrik


--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups "Puppet 
Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/pb32pa%24d3q%241%40blaine.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: Feature Flags location

[Henrik Lindberg]

On 16/02/18 04:56, Juan Hoyos wrote:

Hi there!

In the COMMITTERS.md [0] file it is clearly stated that Puppet uses 
Feature Flags as the primary opt-in behavior, but cannot find a clear 
location where these are being used. In defaults.rb [1] I see 
configuration settings without no explicit distinction between stable or 
experimental features.




An experimental status is mentioned in the documentation for such 
settings/features.



Where can I find the used Feature Flags with their defaults values?
How is decided that a new feature in Puppet should be behind a Feature Flag?



In Puppet a "feature" is technically something else - it is a mechanism 
to detect if being on 'windows' or if some optional thing has been 
installed or not.


When we talk about a feature flag we usually mean a setting that affects 
the behavior of puppet. --strict_variables in one such setting that is a 
"feature flag".


We had one major "feature flag" during the 3.x series for 
--compiler=future (or --compiler=current). Which enabled us to introduce 
the 4.x language compiler gradually. This feature flag was dropped in 
4.x as the future was then the present.


We do not have a standard way of handling "feature flags", it is done on 
an ad-hoc basis. Currently we are testing having a single 
"all-future-things" flag (which right now means, use with current 
version of features to be released in next major). Not sure how many 
such actual behaviors we have atm.


I'm currently undergoing an academic research regarding Technical Debt 
and Feature Toggles and I'm trying to determine if Puppet is a good 
candidate to be analyzed.




Hope my answers above are of some help.

Best,
- henrik


Thank you in advance,

Juan

References:

[0] - https://github.com/puppetlabs/puppet/blob/master/COMMITTERS.md
[1] - 
https://github.com/puppetlabs/puppet/blob/master/lib/puppet/defaults.rb


--
You received this message because you are subscribed to the Google 
Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send 
an email to puppet-dev+unsubscr...@googlegroups.com 
.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/a96375a0-2676-49e2-a3ee-b01ce90bb732%40googlegroups.com 
.

For more options, visit https://groups.google.com/d/optout.



--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups "Puppet 
Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/p66gsj%246tf%241%40blaine.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: Puppetserver, never-ending requests and watchdogs

[Henrik Lindberg]

On 21/08/17 09:22, Nacho Barrientos wrote:

Hi,

I'd like to bring up a point that was raised during the resolution of a 
ticket.
The idea is to hopefully trigger a discussion and derive actions from 
it, if

necessary.

Bugs like the one described in PUP-7848 [0] (for which there's a fix 
already,
thanks!) are quite dangerous from the operations' point of view as they 
could

quickly reduce the performance of a production Puppet infrastructure.

Is there any kind of watchdog that can be configured at Puppetserver 
level to
automatically destroy instances that are misbehaving like these ones 
(perhaps

based on the CPU wall time, age...)? We're already using over here
max-requests-per-instance but for obvious reasons it's not useful in 
this case

:)

The more agents exercising the bad code and triggering the issue, the 
faster

the load goes up and therefore the slower the infrastructure becomes. There
should be a way to tell Puppetserver how to protect itself. Perhaps there's
already but we could not find it [1]. In the meantime, what we're doing 
is to
put some extra (and very specific) monitoring in place on our side to 
try to
detect this situation and alarm it but perhaps there's something that 
could be

done directly at Puppetserver level to act earlier.

In case it helped we're running 2.7.2 over here.

What do you think?

Thanks!



I think it is a very good idea for a Puppet Server feature. I suppose it 
would need to look at the compilation time and compare that against a 
set timeout as it would otherwise be very difficult to figure out if a 
legit load is rogue or not - a regular compilation could consume a lot 
of CPU, as all depends on what is in the manifests being compiled.


I think you can log a feature request for Puppet Server with your idea
as that will more readily put this in front of those that prioritize 
between features to add in upcoming releases.


Best,
- henrik


[0] https://tickets.puppetlabs.com/browse/PUP-7848
[1] 
https://docs.puppet.com/puppetserver/latest/config_file_puppetserver.html





--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups "Puppet 
Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/op91o5%24kkd%241%40blaine.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: lookup from external script

[Henrik Lindberg]

On 21/08/17 09:26, Peter Meier wrote:

Hi all,

I have a bunch of ruby script that are munging data around, part of that
data is based out of hiera.

Until now, with hiera 3, I just required hiera in my ruby script, called
lookup with a certain scope and I got my data out of hiera.

Now looking forward to puppet not anymore shipping hiera as a gem (as
everything moved to hiera 5 and hence lookup), how would I be able to
use the same functionality with lookup?

What I did (simplified):

require 'hiera'
def hiera
   @hiera ||= Hiera.new(:config => '/etc/hiera.yaml')
end

def hiera_def_scope
   @hiera_def_scope ||= { 'environment' => 'production' }
end

def hiera_lookup(key,default={})
   hiera.lookup(key,default,hiera_def_scope)
end

mydata = hiera_lookup('mydata','a default')

While I see that with lookup this might get quickly very complex with
different environments, module data and so on. BUT I could imagine, that
given the data in hiera is central to your infra, lots of people might
also like to use that data for something else. So how to get that data out?

What would you recommend to do, how would you approach the problem with
getting data out of hiera in something else than puppet? Just stick
around with Hiera 3 and an old non-hiera-5 config file? Which would
likely be feasible if hiera stays somewhat supported, which it doesn't
from my understanding?! Write your own self-supported hiera?

best



Over time the amount of features have grown for hiera and the move into 
puppet was because of the increasing difficulties of making an external 
hiera do all the required things as the external hiera would have needed 
to know about environments and module paths and finding information in 
modules, ability to reference to puppet variables etc. etc.


Thus, to use heira 5 as a library, it is basically the same as a 
compilation + calls to the lookup function. To do that you would need to 
do what the lookup CLI is doing in terms of setup. That would then give 
you the full feature set.


There are however several use cases where people want to be able to 
instantiate "a hiera" with a hierarchy of their own to do lookups. There 
is the beginnings of such an API in puppet, but it is not well 
documented and needs a bit more work.


In your case, do you want exactly the same as what a puppet compilation 
would see (and support env, module data etc.) or do you need the 
functionality of having your own hierarchy somewhere that is not mixed 
into puppet?


Hiera 3, the freestanding gem will at some point no longer be supported.
It may be too early to drop it for Puppet 6, but it also depends on when 
it will be released. We are currently just fixing critical problems with 
the hiera 3 separate gem.


Best,
- henrik
--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups "Puppet 
Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/onelpj%24u9d%241%40blaine.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: Hiera Merge

[Henrik Lindberg]

On 10/08/17 20:48, ggun wrote:
Thanks, but the main point to merge the hash is that the sub hash header 
value is not fixed in the example I have mentioned as 
esa-user-profile-service or esa-group-service.  It can be anything. I 
need to make the code free from using direct Key value .




So 'was_dmgr_data' is a hash of unknown hashes. Did we understand 
correctly that the "common part" is some kind of default?


Can you change the original structure, or are you stuck with this design?

In puppet 5.0.0 there is a function named 'tree_each' that can be used 
together with 'map()' and the result of that fed into a Hash.new with an 
option of 'tree', or 'tree_hash'. This is a very powerful function but 
requires a little bit of setup.


I also included a simpler variant using the reduce() function.
That works well here since the task is to map one level of data in a 
known way, so using tree_each is a bit of an overkill.


Using tree_each
===

-example.pp

# Sample data for illustration
$hash = {
  common => { x => 10, y => 20},
  was_data => {
thing => {a => 'in thing'},
other_thing => {a => 'in other thing', x => 1000 }
  }
}
# The common to part to merge as defaults
#
$common = $hash[common]

# Map the tree to a flat structure. Skip the root
# (we want data one level down),
# and skip all values as they are included in the hash containers (and
# we do not need to transform them)
#
$flat_tree = $hash[was_data]
 .tree_each({include_root => false, include_values => false} )
 .map |$entry| {
   # assign the path and value in $entry to separate variables
   [$path, $val] = $entry

   # only map top level keys (the path has length == 1) to a new value
   # and make all other entries undef.
   # The mapped entries is the common part with the specific data
   # merged ($common + $val)
   #
   if $path =~ Array[1,1] {
 [ $path, $common + $val ]
   }
}.filter |$x| { $x != undef } # filter out the undef entries

# create a new hash and notice it
notice Hash($flat_tree, 'hash_tree')
--

When running this the output would be:
{thing   => {x => 10, y => 20, a => in thing},
 other_thing => {x => 1000, y => 20, a => in other thing}}

As you can see the result is a hash with the keys under 'was_data'
with the expected merged data.

Using reduce

If  you are on a version before Puppet 5.0.0 it can be done with the 
reduce function.


-example.pp
# Sample data for illustration
$hash = {
  common => { x => 10, y => 20},
  was_data => {
thing => {a => 'in thing'},
other_thing => {a => 'in other thing', x => 1000 }
  }
}
$common = $hash[common]

$mapped = $hash[was_data].reduce({}) |$memo, $entry| {
  [$k, $v] = $entry
  $memo + { $k => $common + $v}
}

notice $mapped

-

I hope these two examples help.
Regards
- henrik


On Wednesday, August 9, 2017 at 2:24:34 PM UTC-4, Reid Vandewiele wrote:

If you're just trying to transform the data in Puppet code and
assuming (as Henrik was) that you can't change how the data is
stored, something like this might work.

# Assuming $was_data is the hash of data from Hiera
$common_data = $was_data.filter |$pair| { $pair[0] != 'was_dmgr_data' }
$hash1 = {'esa-group-service'=> $common_data +
$was_data['was_dmgr_data']['esa-group-service'] }
$hash2 = {'esa-user-profile-service' => $common_data +
$was_data['was_dmgr_data']['esa-user-profile-service'] }

~Reid

On Wednesday, August 9, 2017 at 6:15:41 AM UTC-7, ggun wrote:

Thanks

On Tuesday, August 8, 2017 at 7:10:13 PM UTC-4, ggun wrote:

Hi Experts,

I have a requirement as below.
I need to create a Hash from below hiera data.

was_data:
   hs3sourcepath: 'glic.binaries/websphere'
   hdaresponse_file: /opt/software/WAS8.5.5.10_Install.xml
   hibmagentpath:

/opt/software/agent.installer.linux.gtk.x86_64_1.8.2000.20150303_1526.zip
   hbase_dir: '/opt/was/was855'
   hinstance_name: WebSphere
   was_dmgr_data:
 esa-group-service:
   hgroup: websph
   hdmgr_profile: TST
   hdmgr_cell: CELL
   hcluster_name: CLUSTER
   hpptdmgrsrvport: 8080
 esa-user-profile-service:
   hdmgr_profile: ABC
   hdmgr_cell: PQS
   hcluster_name: IOP
   hpptdmgrsrvport: 


I need a hash of above data as
Hash 1 :
 esa-group-service:
   hgroup: websph
   hdmgr_profile: TST
   hdmgr_cell: CELL
   hcluster_name: CLUSTER
   hpptdmgrsrvport: 8080
   hs3sourcepath: 'glic.binaries/websphere'
   hdaresponse_file: /opt/software/WAS8.5.5.10_Install.xml
 

[Puppet-dev] Re: Hiera Merge

[Henrik Lindberg]

On 09/08/17 01:10, ggun wrote:

Hi Experts,

I have a requirement as below.
I need to create a Hash from below hiera data.

was_data:
   hs3sourcepath: 'glic.binaries/websphere'
   hdaresponse_file: /opt/software/WAS8.5.5.10_Install.xml
   hibmagentpath: 
/opt/software/agent.installer.linux.gtk.x86_64_1.8.2000.20150303_1526.zip

   hbase_dir: '/opt/was/was855'
   hinstance_name: WebSphere
   was_dmgr_data:
 esa-group-service:
   hgroup: websph
   hdmgr_profile: TST
   hdmgr_cell: CELL
   hcluster_name: CLUSTER
   hpptdmgrsrvport: 8080
 esa-user-profile-service:
   hdmgr_profile: ABC
   hdmgr_cell: PQS
   hcluster_name: IOP
   hpptdmgrsrvport: 


I need a hash of above data as
Hash 1 :
 esa-group-service:
   hgroup: websph
   hdmgr_profile: TST
   hdmgr_cell: CELL
   hcluster_name: CLUSTER
   hpptdmgrsrvport: 8080
   hs3sourcepath: 'glic.binaries/websphere'
   hdaresponse_file: /opt/software/WAS8.5.5.10_Install.xml
   hibmagentpath: 
/opt/software/agent.installer.linux.gtk.x86_64_1.8.2000.20150303_1526.zip

   hbase_dir: '/opt/was/was855'
   hinstance_name: WebSphere

Hash 2
   esa-user-profile-service:
   hdmgr_profile: ABC
   hdmgr_cell: PQS
   hcluster_name: IOP
   hpptdmgrsrvport: 
   hs3sourcepath: 'glic.binaries/websphere'
   hdaresponse_file: /opt/software/WAS8.5.5.10_Install.xml
   hibmagentpath: 
/opt/software/agent.installer.linux.gtk.x86_64_1.8.2000.20150303_1526.zip

   hbase_dir: '/opt/was/was855'
   hinstance_name: WebSphere

So I trying to merge the has of esa-group-service to was_data and 
esa-user-profile-service to was_data


Please let me know if there is a way



"There is always a way..." - question is if there is a good way ;-)
I take it that it is not feasible to rearrange how the data is organized 
in hiera in the first place - that would otherwise be the best.


The issue here is that you cannot simply both merge and split up the 
hash at the same time - you have one part that looks like a common part;

defaults for the other two parts if you like.

With hiera 5, if you change the data structure to consist of a defaults 
part - under one key, say "esa_default", and then store the other two 
under their respective keys 'esa_group_part', 'esa_user_part', then you 
have several options:


* write a hiera 5 backend function that returns the merged result for 
the two full keys for the group and user structures
* write a class where you use APL to get the three parts, and then merge 
in that class, binding to a variable that you then use to get the merged 
result.


I would do the hiera backend function.

If you also need the original data structure (as shown in your example) 
then also merge a result that looks like that in your backend function.


The function would be simple - basically do something like:

lookup('esa_default') + lookup('esa_group_part')
or
lookup('esa_default') + lookup('esa_user_part')

to override defaults with more specific values (if that is what you want 
to do - or some variation on that if it is not defaults in the "common" 
part).


I choose the names "group_part" and "user_part" simply for illustration.
Hope this helps you.

You can read more about how to create a backend function here: 
https://docs.puppet.com/puppet/5.0/hiera_custom_lookup_key.html


- henrik

--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups "Puppet 
Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/omejnq%24j0t%241%40blaine.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: So how exactly the immutable hashes are supposed to be used?

[Henrik Lindberg]

On 13/07/17 19:00, Mariusz Gronczewski wrote:

Hi,

I've been slowly converting old 3.x codebase (which has seen days of 
0.24...) to 4.x and there is a lot of following pattern used for hashes:'


 $hash = {
   "some" => "defaults"
 }
 if $thing_one == "is_true" {
 $hash["option1"] = $thing_one,
 }
 if $thing_two == "is_something" {
 $hash["option2"] = "something"
 }
 else {
 $hash["option2"] = "something else"
 }

etc. Now with immutable hashes I'm forced either to do:'

 $hash1 = {
   "some" => "defaults"
 }
 if $thing_one == "is_true" {
 $hash2 = {
 "option1" => $thing_one,
 }
 }
 else { $hash2 = {} }

 if $thing_two == "is_something" {
 $hash3 = {
 "option2" => "something"
 }
 }
 else {
 $hash3 = {
 "option2" => "something else"
 }
 $hash4 = merge($hash1, $hash2, $hash3, $hash3)

or go the hacky route and go to erb and back to use ruby language to do it.

Am I missing something here ? is there a better way to do it? We have a 
lot of "get a hash, munge it, feed it to either puppet resource or to 
external config" pattern used in the code and immutability so far 
doesn't seem like a very useful property




You can do like what John Bollinger showed - or something like this 
where you merge hash snippets using + operator.


$hash = { "some" => "defaults" }
+ if $thing_one == "is_true" {
{"option1" => $thing_one}
  }
  else { {} }
+ if $thing_two == "is_something" {
{"option2" => "something"}
  }
  else {
{"option2" => "something else"}
  }

- henrik

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups "Puppet 
Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/ol7f3r%24vg5%241%40blaine.gmane.org.
For more options, visit https://groups.google.com/d/optout.

Re: [Puppet-dev] calling functions from (ruby-)functions and puppet data types

[Henrik Lindberg]

> On 24 Jul 2017, at 22:29, Peter Meier  wrote:
> 
>> Now, the reason your call on line 74
>> 
>>call_function('lookup', key, 'Hash', 'hash',{})
>> 
>> fails is that you have to pass in a ruby object for the type, not a
>> string representation of the type. There's probably a better way that
>> would generate an appropriate object based on the string, but this
>> works for this case:
>> 
>>call_function('lookup', key,
>> Puppet::Pops::Types::PHashType::DEFAULT, 'hash',{})
>> 
>> That invocation gives the expected error message:
>> 
>>Evaluation Error: Error while evaluating a Function Call, Found
>> value has wrong type, expects a Hash value, got String
> 
> Awesome, that works! :) I was navigating within the pops subsystem code
> once into this direction, but it felt a bit too clumsy and I didn't
> bother to try.
> 
> Thanks for the help!
> 
> I agree that there should be a better (and documented! ;) ) way that
> would generate an appropriate object based on the string and if there
> isn't there should probably be a user story about it.
> 

Puppet::Pops::Types::TypeParser#parse can be used to get a data type object 
given a
type expressed as in the puppet language. As an example - this:

tparser = Puppet::Pops::Types::TypeParser.singleton
t = tparser.parse(‘Array[String]’, scope)

would assign a data type of Array type accepting String values to the
variable t.

An alternative is to use the Puppet::Pops::Types::TypeFactory to create
the same Array[String] datatype:

tf = Puppet::Pops::Types::TypeFactory
t = tf.array_of(tf.string)

Hope those help in the future
Best,
- henrik


>> I've filed DOCUMENT-703 to correct the docs.
>> https://tickets.puppetlabs.com/browse/DOCUMENT-703
> 
> Cool, thanks!
> 
> best
> 
> ~pete
> 

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/FDFB80E4-1A5C-44DD-98C1-0D247537E143%40puppet.com.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: Puppet 4 function dispatch

[Henrik Lindberg]

On 23/05/17 20:21, Dean Wilson wrote:

Hi,
  I'm currently trying to write a Puppet 4, EPP, version of
https://forge.puppet.com/deanwilson/multitemplate but I'm having an
issue with the dispatch rules I'd like to use.

In order to support multiple template file names, and provide values
for the templates, I think I want something like this (but with a
better Pattern):

  dispatch :template_data do
scope_param
repeated_param 'String', :templates
param 'Hash[Pattern[/^\w+$/], Any]', :parameters
  end

But that's not allowed as repeated patterns have to be the last thing
(excluding blocks.) I know in modern rubies you can have a 'splat' in
the middle and unpack the first and last params so I was wondering if
it'd be considered for puppet function dispatch too.



We did consider adding that but rejected the idea. Mostly because the 
varargs concept exists in most languages (as last param), but not 
somewhere in the middle.


Why not simply switch the arguments around so you have the list of files 
at the end of the list?


- henrik


I can currently get around this with a call like
multi_epp_template(['file', 'file2'], { }) but that extra bit of array
syntax feels unwieldy.

In general the puppet 4 function dispatch stuff has been quite nice to
work with, so congrats on that.


Glad you like it.

- henrik


  Dean




--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups "Puppet 
Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/og23lu%24h1p%241%40blaine.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: listing datatypes

[Henrik Lindberg]

On 31/03/17 22:49, Thomas Hallgren wrote:

The problem is that there is no known list. Loaders builds their set of
type (the list) in response to the types being referenced during
evaluation. Different loaders will have different set of types. So what
list is it you want to present to the user? At what time?

I think that would be "the name of every type in every module and in the 
environment".


- henrik


- thomas

On Fri, Mar 31, 2017 at 8:06 PM, Corey Osman > wrote:

Thanks for the reply.

The debugger doesn't need the types loaded in order to use them
since it passes that work down to puppet when evaluating code.   The
only thing I want to do here is just present a list so the user
knows what types and dataypes are available to them.


/I am going to play around with your suggestions to see what I can
come up with./



On Thursday, March 30, 2017 at 10:21:12 PM UTC-7, Thomas Hallgren wrote:

Hi Corey,

Yes, your solution is indeed hackish as it performs surgery on
undocumented implementation internals that might change without
notice :-). Please don't do that.

We currently have no way of presenting a full list. In order to
do that, we must implement new functionality on the
Puppet::Pops::Loaders::Loader. The loaders form a directed
acyclic graph where the leafs are the private loaders of each
module. Different loaders see different things. Code loaded from
one module will not automatically see all things loaded from
other modules. It will just see modules that it is immediately
dependent on. This is probably something that the debugger needs
to be aware of.

I would envision that the debugger would obtain the loader by
doing Puppet::Pops::Loaders.find_loader() or by
asking what loader that is attached to the current AST using the
Puppet::Pops::Adapters::LoaderAdapter.loader_for_model_object
method. It would then obtain the list from there. The list will
vary depending on loader.

A complicating factor might be that most types are loaded on
demand. I.e. the loader doesn't know about them until the type
is referenced loaded (or missing) as a response to that
reference. Perhaps we'll need to design something that loads
everything eagerly.

In any case, please enter a ticket (or tickets) where you
explain what the debugger would need. I think we'll need to work
together on this to create a good solution.

Short term, for the statically loaded things, take a look at
Puppet::Pops::Types::TypeFactory.type_map. It's "@api private"
but it's not likely to change any time soon.

- thomas


On Fri, Mar 31, 2017 at 3:19 AM, Corey Osman
 wrote:

Hi,

I have found a way to list datatypes for the puppet-debugger
(https://github.com/nwops/puppet-debugger
) but my approach
seems hackish.  Additionally my approach only seems to pick
up the core datatypes and not 3rd party types loaded from
stdlib puppet module and such.


Is there a better way than this?

https://gist.github.com/logicminds/2a7bf4124f3bdb01f1a77d0bb521dffc




Corey

--
You received this message because you are subscribed to the
Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails
from it, send an email to puppet-dev+...@googlegroups.com.
To view this discussion on the web visit

https://groups.google.com/d/msgid/puppet-dev/D35D28B2-7D65-47C9-8A1A-D1E1C2278AD1%40nwops.io

.
For more options, visit https://groups.google.com/d/optout
.


--
You received this message because you are subscribed to the Google
Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to puppet-dev+unsubscr...@googlegroups.com
.
To view this discussion on the web visit

https://groups.google.com/d/msgid/puppet-dev/9a02b21a-6e0a-478a-adca-9117df9ae129%40googlegroups.com

.

For more options, visit https://groups.google.com/d/optout
.


--
You 

[Puppet-dev] Re: Extend data_provider lookups to eyaml in the 4.7.x Puppet implementation

[Henrik Lindberg]

On 12/12/16 22:17, Stan Chan wrote:

Hi Puppet Devs,

I'm trying to extend the data_provider lookup types to support eyaml for
the v4 implementation of Hiera. Has anyone started down this path and/or
can provide some suggestions as to how best to extend the 4.7.x codebase
to support this?

Thanks
Stan



Stan, we are doing quite a lot with lookup and "data providers" in 
puppet 4.9.0 to make it really simple to write new "backends/data 
providers". The work named "hiera 5" is based on functions.

I am in the process of writing up documentation for that now.

Love to help out in getting this work done for EYAML and I am happy to 
help out. Are you by any chance in Slack and/or participating in today's 
Hack day? I think it would be best to work out where to start in an 
interactive fashion rather than trying to convey everything there is to 
know in a post like this.


- henrik
(helindbe in one form or another on Slack, IRC, etc).


--
You received this message because you are subscribed to the Google
Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to puppet-dev+unsubscr...@googlegroups.com
.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-dev/422dc9c5-9717-41dc-bb18-35e8ee03b529%40googlegroups.com
.
For more options, visit https://groups.google.com/d/optout.



--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups "Puppet 
Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/a22ea83f-f59d-3154-3459-66909536383b%40puppet.com.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: list all available functions by input data type

[Henrik Lindberg]

For 3.x functions this is impossible as they do not have type information. 
(You would have to call each function with each data type and see if it 
fails. Or, you would have to parse the ruby code and then via static 
analysis of the ruby code try to determine if a data type is acceptable or 
not) = impossible. 

For 4.x this is doable. You would first need to load all functions, 
then look them up. For the 4.x functions it is possible to get the 
signatures defined by the dispatchers, and you can thus calculate the 
acceptable data type(s). You can then use the TypeCalculator to compute the 
type of the argument (-100 in your example), and then check if the 
acceptable data type is assignable from the inferred data type (in the 
example an Integer[-100, -100]). When inferring the value you should use 
TypeCalculator.infer_set to get the richest possible inference. 

With this approach you may get false positives as functions may have a 
wider type than what is actually allowed (checked at runtime), or where 
there are additional constraints posed by invariant combinations of 
parameters. That can probably be ignored for this use case, you would 
simply see some methods that may not be applicable. 

For 3.x functions, if you were to include them in the list, you would have 
to include all of them as you cannot determine if they will work or not. 
That in turn is a problem since if you have stdlib included you would get 
all of those functions in every list, which would make the feature less 
valuable (lots of false entries). 

Best, 
- henrik 



-- 

Visit my Blog "Puppet on the Edge" 
http://puppet-on-the-edge.blogspot.se/

On Friday, July 29, 2016 at 9:49:43 PM UTC+2, Corey Osman wrote:
>
> Hi, 
>
> I am curious what the best approach would be to get a list of puppet 
> functions that respond to a particular data type. 
>
> I am looking to add a feature to the puppet-repl where tab complete would 
> work off of the . operator. 
>
> Example: 
>
>‘-100’. 
>abs 
>max 
>min 
>split 
> 
>
> There is similar magic that occurs with the Ruby based PRY REPL except 
> Ruby already knows its methods due to object ancestors. 
>
>
> Corey 
>

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/48c196a7-4c14-45bf-bf8f-4ed1e0347330%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: Facter config file

[Henrik Lindberg]

On 23/05/16 22:06, Eric Sorenson wrote:

On Wed, 2 Mar 2016, Henrik Lindberg wrote:


I would very much like to be able to type facts using the puppet type
system. While this could be done in other places, a Facter config
would be very useful to have server side when compiling as the
compiler can then validate against such a configuration without having
to do the full compilation with a full set of values.

This also opens up for the possibility to do real type inference. This
is not possible when facts are in essence untyped.


This seems orthogonal to an agent-side facter config; you're talking
about a Type "signature" of what the returned structure of a fact should
look like, right?



Yes like in "the fact 'foo' is an Integer" - preferably expressed in 
terms of the Puppet Type System.


- henrik


Eric Sorenson - eric.soren...@puppetlabs.com - freenode #puppet: eric0
puppet platform // coffee // techno // bicycles




--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups "Puppet 
Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/387ced45-91af-e358-9bd3-1be8a4ed4cc7%40puppet.com.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: server facts

[Henrik Lindberg]

On 17/05/16 02:37, Corey Osman wrote:

During a puppet-repl session any code that contains references to
$server_facts would fail because those special facts did not carry across
node indirector calls.  I ended up reimplementing the assignment of the
facts as suggested in order to provide these facts in the repl session.

Working example here: https://www.puppet-repl.com/play?content=vars


Glad that worked for you. I think that was the right decision.


Another annoyance with the node indirector is that it does not populate
the facts variable as it just merges all facts and variables into the
parameters variable.   Any reason why this is done this way?



Poor judgement?

Jokes aside - it is pretty much Bouncy Castle all the way from a request 
comes in until it has passed the initial set up and can get down to just 
loading manifests and evaluating the catalog. Then the fun begins again 
on the path out to the agent.


Specifically, you can ask for a node for different purposes and in 
contexts where there are no scopes or variables. You don't know until 
you start compiling that you have those things for sure.


Once there were only global variables; no $facts - so all node 
parameters were simply set as variables. Once set there were no 
difference between those variables that came from the node's facts, and 
regular variables set in a manifest.


There is a similar situation with the nodes parameters - IIRC (and as 
you say) they are just mixed in to the node's parameters, and then 
become both global variables (for backwards compatibility reasons), and 
in $facts.


Hope that explains a bit and that it helps you.
Best,
- henrik


Corey

On Sunday, May 15, 2016 at 3:58:15 PM UTC-7, Henrik Lindberg wrote:

On 15/05/16 23:53, Corey Osman wrote:
> Hi,
>
> I want to retrieve server_facts as documented below but the puppet
node
> face doesn’t seem to create a variable called $server_facts in the
facts
> hash.
>
>

https://docs.puppet.com/puppet/latest/reference/lang_facts_and_builtin_vars.html#serverfacts-variable

<https://docs.puppet.com/puppet/latest/reference/lang_facts_and_builtin_vars.html#serverfacts-variable>

>

It is the compiler indirection that sets those by doing
add_server_facts
on the node. It first computes them by calling the private method
set_server_facts, which just remembers them in the compiler.

Then, when the compiler (the real one) starts compiling it ensure that
the variables are set up in top scope.

To get them (without copy/paste of the code in question and doing it
yourself, it is around 10-20 lines of code) you would probably need to
do a dummy compilation via a find catalog for a dummy node. Then get
the
data from the node.

Not pretty.


> Is there a way to get this data via the node face?
>
> ie. puppet node find puppetdev.localdomain —terminus rest
—render-as yaml
>
>

Does not seem so. And why would it? The server facts are not part of
the
node (even if technically they are delivered that way from one place to
another inside puppet - but that is mainly because it is hard to pass
information around when the indirector is involved).

What is it you are trying to do?

- henrik

--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/
<http://puppet-on-the-edge.blogspot.se/>

--
You received this message because you are subscribed to the Google
Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to puppet-dev+unsubscr...@googlegroups.com
<mailto:puppet-dev+unsubscr...@googlegroups.com>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-dev/5c68e1f8-e0be-4e84-b5a8-8b6a6aa02b58%40googlegroups.com
<https://groups.google.com/d/msgid/puppet-dev/5c68e1f8-e0be-4e84-b5a8-8b6a6aa02b58%40googlegroups.com?utm_medium=email_source=footer>.
For more options, visit https://groups.google.com/d/optout.



--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups "Puppet 
Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/6299276b-ffe3-c659-f8bd-c7c44cfe61c8%40puppet.com.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: server facts

[Henrik Lindberg]

On 15/05/16 23:53, Corey Osman wrote:

Hi,

I want to retrieve server_facts as documented below but the puppet node
face doesn’t seem to create a variable called $server_facts in the facts
hash.

https://docs.puppet.com/puppet/latest/reference/lang_facts_and_builtin_vars.html#serverfacts-variable



It is the compiler indirection that sets those by doing add_server_facts 
on the node. It first computes them by calling the private method 
set_server_facts, which just remembers them in the compiler.


Then, when the compiler (the real one) starts compiling it ensure that 
the variables are set up in top scope.


To get them (without copy/paste of the code in question and doing it 
yourself, it is around 10-20 lines of code) you would probably need to 
do a dummy compilation via a find catalog for a dummy node. Then get the 
data from the node.


Not pretty.



Is there a way to get this data via the node face?

ie. puppet node find puppetdev.localdomain —terminus rest —render-as yaml




Does not seem so. And why would it? The server facts are not part of the 
node (even if technically they are delivered that way from one place to 
another inside puppet - but that is mainly because it is hard to pass 
information around when the indirector is involved).


What is it you are trying to do?

- henrik

--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups "Puppet 
Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/005974d9-0f9b-882e-83b8-edd651bd2bb5%40puppetlabs.com.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: Puppet RFC 23 - XPP Files

[Henrik Lindberg]

On 30/03/16 18:24, Eric Sorenson wrote:

Hi, I've just posted a new Puppet RFC that describes pre-parsed and
pre-validated Puppet files, akin to '.pyc' files for Python. It's called
XPP and the doc is open for comments here:

https://docs.google.com/document/d/17SFn_2PJYcO5HjgA4R65a5ynR6_bng_Ak5W53KjM4F8/edit?usp=sharing



Just a quick note that I am now editing that document to make a revision 
where received comments have been addressed.


Will announce when there is a second draft.

- henrik

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups "Puppet 
Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/ne0lap%24sd6%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: Puppet RFC 23 - XPP Files

[Henrik Lindberg]

On 04/04/16 19:42, Thomas Gelf wrote:

Probably not evaluated, because that's where from my understanding the
"it's no longer data" starts. If I'm wrong on that: nice. If not, just
out of curiosity: is evaluation in Ruby expensive?


I forgot - yes evaluation in Ruby is also slow.

- henrik

--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups "Puppet 
Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/5702B6CD.8050508%40puppetlabs.com.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: Puppet RFC 23 - XPP Files

[Henrik Lindberg]

On 02/04/16 01:18, Thomas Gelf wrote:

Hi Henrik,

thanks a lot for your response!

Am 01.04.2016 um 20:21 schrieb Henrik Lindberg:

The C++ implementation is several orders of magnitudes faster than the
ruby implementation. i.e. something silly like tens of thousands of
times faster.


No doubt on this, I believe you without any benchmark.



Although, I got an order or two too carried away with superlatives here.
We will probably end up with 100s' to 1000's times faster.


The Ruby lexing/parsing and validation alone can take minutes on a
complex set up. We have shown earlier though benchmarks that lexing
alone is a bottleneck in any catalog compilation - every optimization
there contributes greatly to the bottom line.


Could you share some details on this? What kind of catalogs are you
talking about? How many resources, parameters, how large are they - and
what makes them so large and slow? Still, no doubt that C++ will be able
to lex the same catalogs in a fraction of the time.



It is pretty much linear to the amount of source text. Though, depending 
on which kind of performance we are talking about here, it does 
naturally have 0 impact when an environment is cached since everything 
is parsed just once.



We have already measured the approach. The benefit on the ruby side is
that the lexing is delegated to a native implementation that reads
binary. A spike was performed with Ruby Marshal, which also compared to
a native MsgPack.


Ok, so basically a linked c-based lexer could give the same performance
boost? Yes, I know, JRuby. But still, could this be true?



A c++ based lexer would indeed be beneficial. The parser is already 
using a c++ driver, but it makes judicious call-outs to Ruby. The 
construction of the AST is the second bottleneck - on par with lexing.


Thirdly, the decoupling of the lexing/parsing from the runtime makes it 
possible to tabulate and compact the AST (as data is serialized). That 
is less meaningful if done in the same process since it takes time 
although it would reduce the number of objects managed by the ruby 
runtime (less memory, lighter job for the GC).


We felt that although doable (we have this as an alternative; link the 
lexer/parser/validator into the runtime (MRI or into Puppet 
Server/JRuby) - that would give us a number of interesting technical 
challenges to solve that we are not sure the right problems to solve.
It may be - we are not sure, but initially we rather spend time making 
the c++ based lexer/parser/validator as good as possible (rather than 
solving the technical packaging the most complicated way).



The main point here is that we are transitioning to a full
implementation of the puppet catalog compiler to C++ ... The use of XPP
makes this possible.


This is where I started to feel no longer comfortable while reading the
proposal. No caching mechanism that helped my in Puppet comes to my
mind, but I could immediately tell a lot of anecdotes involving severe
Puppet issues breaking whole environments just because of caching issues.


We are happy if we initially only get 5-10% out of this...


And this is where I currently disagree. Very often I invest lots of time
for just 1%. But being able to run without a fragile caching layer could
be worth even 50% as long as I'm able to scale. When someone has to stop
a deployment chain because he needs to troubleshoot a caching layer,
lot's of people are sitting around and cannot work. Ask them whether
they would have preferred to buy more hardwar.



We have to start somewhere and when doing so we want to apply the KISS 
principle. The intent is for puppet server to automatically keep the XPP 
files in sync. There may be no need for "caching" - it is simply done as 
a step in atomic deploy of modified puppet code.



We are hoping for more though.


I hope you're pretty confident on this ;)

yes, but unsure how quickly we get there and if facing a tradeoff of 
doing more work on the Ruby side vs. spending the time on the c++ side 
to make more parts of compilation work (i.e. integrating "ruby legacy") 
- what we actually will decide to do. There is some fairly low hanging 
fruit on the Ruby side we could speed up like the AST model itself.



That is not the intent (drop all Ruby things) - our plan is to make a
smooth transition. All "boil the ocean" strategies tends to fail, so
backwards compatibility and gradual change is important to us.


Eeh... Sorry, this is slightly OT, but by the end it isn't. This
"transition" is the root cause for a proposal with the potential for a
lot of additional trouble. You do not want to "drop all the Ruby
things", but you want to have a smooth transition. Without knowing where
this transition should lead this sounds like a contradiction to me.

So, where will this transit phase lead to? That's IMO the question that
many would love to see answered. Will ruby still be there? So where is
the transition? If it 

[Puppet-dev] Re: Puppet RFC 23 - XPP Files

[Henrik Lindberg]

On 01/04/16 04:02, Thomas Gelf wrote:

Hi Eric,

your dedication in getting Puppet faster is really appreciated. My post
is absolutely not in favor of XPP, but please don't get me wrong: it is
meant to be a constructive contribution to the current design process.

In my personal opinion we have a sad history of optimizations focusing a
lot on blaming different languages and tools. Puppet often created fancy
new tools with new languages and components, but we rarely tackled the
root causes of our problems. This would be off topic, but I guess I'll
add a few examples by the end of this mail to let you understand what I
mean.


So let me start with the stated "Problems":

* Performance: I didn't do any measurements, but I guess the compiler
spends more time in resolving dependencies and traversing graphs than it
does in parsing and validating .pp files. Not to mention a lot of compat
hacks, alias-handling voodoo, insane Hiera lookups, type validation for
those lookups and legacy support hacks. So do you have any related
numbers? Where is most of the time spent when building and shipping
(real-world) catalogs? Are you really sure an AST-cache (per manifest?!)
would be worth the effort and solve the "performance problem"? I guess
the C++ parser itself is not so slow that it already needs an AST cache,
because then there would be something wrong with it.



The C++ implementation is several orders of magnitudes faster than the 
ruby implementation. i.e. something silly like tens of thousands of 
times faster.


The Ruby lexing/parsing and validation alone can take minutes on a 
complex set up. We have shown earlier though benchmarks that lexing 
alone is a bottleneck in any catalog compilation - every optimization 
there contributes greatly to the bottom line.



* Cross-Language support: You wrote that the C++ parser needs to provide
the compiled AST to the Ruby runtime. Makes sense to me. But parsing .pp
files with C++, serializing them to a custom not yet designed format,
parsing that custom format with Ruby again and then re-resolve all
(most, some?) dependency graphs across the whole catalog with Ruby...
this doesn't sound like something that could help with getting things
faster. Sure, it would help the C++ parser to hand over it's AST. Or
store it to disk. But would this speed up the whole process? I have some
serious doubts in that relation.



We have already measured the approach. The benefit on the ruby side is 
that the lexing is delegated to a native implementation that reads 
binary. A spike was performed with Ruby Marshal, which also compared to 
a native MsgPack.


The main point here is that we are transitioning to a full 
implementation of the puppet catalog compiler to C++. It will take some 
time to get there and we want to give user the benefit of the 
performance improvements and increased quality (we can check more, 
record more information as it is cheaper) sooner rather than later.


The use of XPP makes this possible. We are happy if we initially only 
get 5-10% out of this - for some that will be enough as it can translate 
to supporting hundreds of additional agents on the same master. We are 
hoping for more though. If we get a yield that is too disappointing we 
will naturally not make the XPP feature a regular one, and instead try 
something else.



IMHO this wouldn't help much, at least not unless "drop all Ruby
interfaces in the long run" is the final goal on your agenda. In that
case please let us know. Those who want to support that goal could unite
their forces to get it accomplished as fast as possible, the others
would at least know what to expect.



That is not the intent (drop all Ruby things) - our plan is to make a 
smooth transition. All "boil the ocean" strategies tends to fail, so 
backwards compatibility and gradual change is important to us.



In a current Puppet ecosystem a C++ parser able to generate an AST from
a .pp file to me still seems far from anything that could completely
replace the current Ruby-based parser in a helpful way very soon. At
least not in a real-world environment with lot's of modules, custom
functions and external data sources, often provided by custom lookup
functions. At least not in a way that would bring any benefit to the
average Puppet user.



The goal is to do this transparently.


So, to me the former one remains a key question to the performance
benefit we could get from all this. As long as the Ruby runtime is
supported, I do not really see how this could work out. But this is just
a blind guess, please prove me wrong on this. Obviously the C++ Puppet
will be faster as soon as you drop the Ruby runtime. But then we should
add something else to the big picture: how should we build custom
extensions and interfaces to custom data in the future? Forking plugins?
Talking with web services? Because adding a C++ compiler to a (dev)ops
deployment pipeline will not convince many people I guess.



That topic is indeed a big topic, and 

[Puppet-dev] Re: Puppet RFC 23 - XPP Files

[Henrik Lindberg]

On 30/03/16 23:12, Jeremiah Powell wrote:

ASTs don't need to be built on a node-by-node basis (unless you
meant manifest-by-manifest basis


Well, manifest-by-manifest where the manifest will vary depending on the
node the compile job is depending upon.  Reviewing the code I get the
impression I just don't understand the existing parser[1] enough today
to hold a valid opinion on this.  But honestly, I'm not trying to troll.



In puppet it is the "compilation of a catalog" that is specific to a 
node, not the parsing of the individual .pp files. The word "parsing" 
does not imply "evaluation" or "catalog compilation".


The puppet parser does the following:

* reads the source text
* builds a representation of this in memory (the Abstract Syntax Tree (AST)
* Performs static validation of the produced AST (to report semantic 
problems that are not covered by syntax alone.


The compiler does this:

* given a node, with its facts, settings from a node classifier etc. it 
is determined where it should start evaluating puppet code


* when it needs to evaluate something - e.g. "site.pp", it first needs 
to parse this file into AST (the steps above).


* When it has the AST, it starts evaluating the AST (expressions like 1 
+ 1, function calls, resource declarations etc).


* The result of the evaluation is that it has built up a catalog of 
resources.


* The catalog is typically sent to an agent for application, but can be 
written to disk etc.


The XPP PRFC is only for the three parsing steps. There will be no 
difference what so ever in what happens once the AST is loaded into 
memory. The only difference between the "Ruby parser parsing .pp file 
into AST" and the "reading an XPP containing an AST" is that we do not 
have to use the very slow Ruby runtime to do all the processing.




The goal of this particular initiative is to enable the C++ parser
(i.e. the frontend) to interop with the Ruby evaluation
implementation (i.e. the backend).  The Puppet code is not being
pre-compiled, but pre-parsed/pre-validated; the C++ implementation
will not (yet) evaluate any Puppet code or load custom types or
functions defined in Ruby.


How will this work with create_resources[2]?


They are completely unrelated. The call to the "create_resources" will 
take place in exactly the same way. The AST that is evaluated looks 
exactly the same if it came from an XPP file or if it was read in source 
form from a .pp file and parsed by the ruby parser.




In compiler terminology, there's a "frontend" and a "backend".


In compiler terminology the frontend is a scanner composed of a parser
and a lexer.  The front-end validates the parse of the code as a
side-effect.  This is beyond the scope of the discussion of the PRFC and
into a sizing competition about who's read Aho, Lam, Sethi and Ullman.

The only point form this is that this is not compiling but a partial
parsing. Some of my concerns cannot be raised until there is actual
output to examine.



Puppet does not have a compiler in the typical computer science sense. 
The puppet term "compiler" uses the word in an English/generic sense 
since what it is doing is "compiling a catalog" (putting a catalog 
together out of the pieces that are supposed to be in it).


Puppet is an interpreter that interprets the AST that is produced by the 
puppet parser. The Puppet compiler uses that interpreter when it is 
compiling the catalog.



The above seems to be confusing, understandably so, pre-compiling a
resource catalog with pre-parsing a manifest.  In terms of a
language like Python, the "pre-compiled" pyc files are simply a
representation of the source that is more efficient to load and
execute than having to parse the Python source again


That is because the Java code and CPython code is completely compiled
and ready to link in at runtime.In this the XPP proposal does not
appear similar to .pyc files or Java bytecode.



Java is compiled to byte code. This byte code is then either interpreted 
or "just in time" compiled into machine code.


The Puppet AST is to Puppet what the Java byte code is to a JVM.
(Although puppet it is not byte code based; we simply use the AST).

Byte code is typically for a virtual machine that is stack based.
As a simple example if you have the source:

  a = 2 + 3

Byte code may be something like

 Push Literal 2
 Push Literal 3
 Add
 Store a

Whereas the AST is a tree of nodes (which is hard to draw, so here using
a list notation. The same in puppet would be:

(AssignmentExpression a
  (ArithmeticExpression +
(Literal 2)
(Literal 3)))



It does appear to me too be very similar to the Ecore technology[3] from
Eclipse, and thus Geppetto as mentioned in the references on RGen in the
prior art section.   It does appears to be similar in how you write a
Coffeescript parser for grammars in Atom or languages in Sublime Text.
It is just that you plan to serialize the result to disk 

[Puppet-dev] Re: RFC: deprecating the 'resource_types' endpoint in Puppet Server

[Henrik Lindberg]

On 11/03/16 15:06, Chris Price wrote:



On Fri, Mar 11, 2016 at 3:54 AM, Erik Dalén > wrote:

The API looks good to me.

Will this be available as a command line tool (standalone one, not
contacting a master)? That would be useful for scripts and local
development where you might not want to upload the code to a
puppetmaster in order to find the classes.


I don't think we have immediate plans for that, but I agree that it
sounds like it could be useful, and probably pretty easy.  Might be
worth creating a Jira ticket for.

Henrik may have thought about this more than I have.



No, I have not thought about it. But have now :-)

We have several kinds of definitions that are of interest (Puppet 4.5.0):

- Resource Types
- Functions (ruby and puppet)
- Data Types (ruby and puppet)
- Aliased Data Types (typically puppet)

To be able to do something useful, somone needs to figure out the 
requirements, what they want to get out of this - as opposed to say 
generating documentation for the above and reading the documentation.


Is this a "quick list" of what is available?


If it is using the indirector it should be pretty trivial to create
a puppet face for it.

Also, will you create a similar one for defined types? Functions?



Functions and data types have no indirector related functionality.

- henrik



For defined types, we are trying to determine whether there were use
cases that people needed that were being satisfied by the resource_types
endpoint.  If so, we'll need to do that before we get rid of
'resource_types'.  If not, and for functions, I would imagine that we
will eventually end up creating them, but perhaps not until after we get
rid of resource_types.  If you think those would be particularly useful,
then we should capture them in Jira as well.  I hadn't thought of the
idea about functions before, that's pretty interesting!

Because removing resource_types is something that would need to happen
on a major version boundary, right now the main goal is just to figure
out what we need to do to be ready for that whenever we start lining up
dates for future major version bumps.


On Wed, 9 Mar 2016 at 21:45 Chris Price > wrote:

Hello,

We're looking into the possibility of deprecating and removing the `
resource_types` endpoint from Puppet Server.

The current implementation can return information about a lot of
different
things in Puppet, but it's very expensive (in terms of CPU and
memory
usage), and has some unexpected side effects that can cause
subsequent
requests to other endpoints (including catalog compilation! see
https://tickets.puppetlabs.com/browse/SERVER-1200 ) to misbehave.

The main use case that we're aware of for the endpoint is to get
a list of
all of the classes (and their associated parameters) for an
environment.
We've just finished building a new HTTP endpoint called
'environment_classes' (which will ship in the upcoming releases
of OSS and
PE Puppet Server) that will provide that data in a cleaner, less
expensive
fashion.  You can see a sneak preview of the documentation for
this new
endpoint, including its wire formats, here:


https://github.com/puppetlabs/puppet-server/blob/dc58bdd94246e5b68bf1adff2d38bf574ca22662/documentation/puppet-api/v3/environment_classes.md

Once that endpoint is available we'd like to deprecate and
eventually
remove the resource_types endpoint.  However, we'd first like to
make sure
that there aren't other important use cases that users are
relying on it
for.  (If there are, we'll probably try to address those by adding
additional new HTTP endpoints until we're at feature parity for
the things
that people rely on resource_types for.)

So... this is a solicitation for input.  Do you currently rely
on the
resource_types endpoint for critical parts of your workflow?
What kinds of
things are you using it for?  What would you be missing if it
were to be
removed?

We've got a placeholder ticket in our issue tracking system for this
deprecation / removal:

https://tickets.puppetlabs.com/browse/SERVER-1120

It doesn't have a ton of detail yet, but if you're interested in
tracking
this, you can watch that ticket as that is where we'll end up
tracking
things as this progresses.

Thanks in advance for your feedback!

--
You received this message because you are subscribed to the
Google Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails 

[Puppet-dev] Re: Facter config file

[Henrik Lindberg]

On 01/03/16 19:52, Eric Sorenson wrote:

I've been thinking about a config file for Facter, which has
historically not been run-time configurable.

The two problems in front of me that seem applicable are:

* Sometimes, certain facts are just plain bad to collect and users would
like to prevent them from even being resolved (see FACT-718, FACT-449, ).
* Some facts are not inherently bad but _are_ expensive and/or change
infrequently, so preventing them from being resolved every time would be
beneficial (FACT-348)

Are there other problems you're running into in this area that you'd
like to see addressed with a "facter.conf"? I'd like to gather all the
requirements and start up a little Puppet RFC based on them.

--eric0



I would very much like to be able to type facts using the puppet type 
system. While this could be done in other places, a Facter config would 
be very useful to have server side when compiling as the compiler can 
then validate against such a configuration without having to do the full 
compilation with a full set of values.


This also opens up for the possibility to do real type inference. This 
is not possible when facts are in essence untyped.


- henrik

--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups "Puppet 
Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/56D66026.9010501%40puppetlabs.com.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: How strict do you want puppet to be?

[Henrik Lindberg]

On 25/02/16 23:50, Trevor Vaughan wrote:

I just want to make sure I'm reading this right.

For the first scenario, noop means don't order anything, just do things
in whatever order they happen.

And the second scenario is: don't fail B if A fails (don't create a
relationship), but do A before B

Is this correct?



I have problems understanding what you mean by noop and fail here.

The fundamental question is, does this expression define the order of A 
and B?


A -> [ ] -> B

You can view it either as it defines the order [A, B], or that it does 
not and their partial order is unaffected.


If you like the answer to be [A, B], does it bother you that the 
following is not the same thing:


A -> [ ]
[ ] -> B

As that would not order them - because [ ] here can be seen as either 
first/last or "in the partial order that is defined elsewhere".


Trying to make that define the order [A, B] is impossible since every 
resource starts out with the partial order [[], R, []] (nothing before 
R, nothing after R).


- henrik



If it is, I favor the second scenario as the default as it is more
intuitive to the 'ordered compile' of Puppet 3+.

Thanks,

Trevor

On Thu, Feb 25, 2016 at 5:31 PM, Henrik Lindberg
<henrik.lindb...@puppetlabs.com <mailto:henrik.lindb...@puppetlabs.com>>
wrote:

On 25/02/16 22:08, John Bollinger wrote:



On Thursday, February 25, 2016 at 8:55:01 AM UTC-6, Trevor
Vaughan wrote:

 Hmm.

 I think, as long as it is documented, then whatever behavior is
 deterministic is fine.
 '
 I think that there is value in the following resolutions:

 Notify['left'] -> [] -> Notify['right']
* Noop since there is nothing in []

 Notify['left'] -> [] -> Notify['right']
 Notify['left'] -> Notify['right']
* First == Noop
* Second == Expected ordering



I flip-flopped a bit on this.  I started from the position that
it would
be inappropriate for

|
Notify['left']->$stuff ->Notify['right']
|

or

|
Notify['left']->Stuff<||>->Notify['right']
|

ever to fail to cause Notify['left'] to be applied before
Notify['right'], as indeed it now does fail to do in the event
that the
stuff in the middle represents zero resources.  My basis there
was that
it is counterintuitive for such expressions to not establish
relative
ordering of the two Notifys.

Ultimately, however, that basis is completely subjective.
Others might
reasonably intuit that such an expression would have exactly the
semantics it actually does have, which are roughly equivalent to
those of

|
Notify['left']->$stuff
$stuff ->Notify['right']
|

.  Upon reflection, this alternative is less magic.  It falls out
naturally from understanding that chain operators associate from
left to
right, and that each binary chain expression /evaluates to/ the
value of
its right-hand operand.  That some people might find that behavior
surprising is a consideration, but not a primary one for me.


It depends on the operator if it is left or right associative and if
it is cumulative. And also what the result of '(nothing) op
(something)' results in.

if
   (nothing) -> X
   X -> (nothing)

both evaluate to X

as in

   0 + 1
   1 + 0

Then empty man in the middle would be a noop.

If you consider the arrows to abstractly order a set of resources in
an array which at the end of the chain defines the partial order of
the resources it would be the same as doing array concatenation.

i.e.
[A] + [] + [B] = [A, B]

So - it is all down to definition of the arrow operators; i.e.
taste/most practical/least surprise.

- henrik


--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google
Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to puppet-dev+unsubscr...@googlegroups.com
<mailto:puppet-dev%2bunsubscr...@googlegroups.com>.
To view this discussion on the web visit

https://groups.google.com/d/msgid/puppet-dev/56CF80DD.7010200%40puppetlabs.com.

For more options, visit https://groups.google.com/d/optout.




--
Trevor Vaughan
Vice President, Onyx Point, Inc
(410) 541-6699

-- This account not approved for unencrypted proprietary information --

--
You received this message because you are subscribed to the Google
Groups "Puppet Developers" group.
To unsu

[Puppet-dev] Re: How strict do you want puppet to be?

[Henrik Lindberg]

On 25/02/16 22:08, John Bollinger wrote:



On Thursday, February 25, 2016 at 8:55:01 AM UTC-6, Trevor Vaughan wrote:

Hmm.

I think, as long as it is documented, then whatever behavior is
deterministic is fine.
'
I think that there is value in the following resolutions:

Notify['left'] -> [] -> Notify['right']
   * Noop since there is nothing in []

Notify['left'] -> [] -> Notify['right']
Notify['left'] -> Notify['right']
   * First == Noop
   * Second == Expected ordering



I flip-flopped a bit on this.  I started from the position that it would
be inappropriate for

|
Notify['left']->$stuff ->Notify['right']
|

or

|
Notify['left']->Stuff<||>->Notify['right']
|

ever to fail to cause Notify['left'] to be applied before
Notify['right'], as indeed it now does fail to do in the event that the
stuff in the middle represents zero resources.  My basis there was that
it is counterintuitive for such expressions to not establish relative
ordering of the two Notifys.

Ultimately, however, that basis is completely subjective.  Others might
reasonably intuit that such an expression would have exactly the
semantics it actually does have, which are roughly equivalent to those of

|
Notify['left']->$stuff
$stuff ->Notify['right']
|

.  Upon reflection, this alternative is less magic.  It falls out
naturally from understanding that chain operators associate from left to
right, and that each binary chain expression /evaluates to/ the value of
its right-hand operand.  That some people might find that behavior
surprising is a consideration, but not a primary one for me.



It depends on the operator if it is left or right associative and if it 
is cumulative. And also what the result of '(nothing) op (something)' 
results in.


if
  (nothing) -> X
  X -> (nothing)

both evaluate to X

as in

  0 + 1
  1 + 0

Then empty man in the middle would be a noop.

If you consider the arrows to abstractly order a set of resources in an 
array which at the end of the chain defines the partial order of the 
resources it would be the same as doing array concatenation.


i.e.
[A] + [] + [B] = [A, B]

So - it is all down to definition of the arrow operators; i.e. 
taste/most practical/least surprise.


- henrik


--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups "Puppet 
Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/56CF80DD.7010200%40puppetlabs.com.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: How strict do you want puppet to be?

[Henrik Lindberg]

On 25/02/16 15:08, Gary Larizza wrote:



On Wednesday, February 24, 2016, Gary Larizza <g...@puppetlabs.com
<mailto:g...@puppetlabs.com>> wrote:



On Wed, Feb 24, 2016 at 10:47 PM, Henrik Lindberg
<henrik.lindb...@puppetlabs.com
<javascript:_e(%7B%7D,'cvml','henrik.lindb...@puppetlabs.com');>> wrote:

    On 23/02/16 01:47, Henrik Lindberg wrote:

Hi, I am thinking ahead a bit regarding puppet 5 and how we
should deal
with all the requests for features that require
deprecations. (There are
some related things like requests for additional validation
and warnings
that are different from deprecations).

In the past we merrily started issuing deprecation warnings,
but the
community pretty unanimously said "stop doing that" we
cannot deal with
all of those warnings. Since then we then pretty much
stopped doing
deprecation warnings.

There has also been a long standing wish for a "strict mode"
in puppet,
that like a harsh teacher would point out every itty-bitty
problem.

So - what should we do?

In PUP-5889 I have described an idea. This is the text from
that ticket
as it stands right now.

PUP-5889

-
Add a flag --strict to puppet settings. When in effect this
will turn on
--strict_variables, and will also enable other "helpful" but
undesirable
behavior. (Each such behavior to be defined in a separate
ticket).

The semantics of this flag should be:

* '--strict=ignore'; no strictness checks are to be
performed, nothing
is reported.

* '--strict=warn'; strictness checks are performed, they are
reported as
warnings, individually configurable warnings follow their
own setting
(i.e. if they are added to disabled_warnings).

* '--strict=errors'; strictness checks are performed, they
are reported
as errors and stop the execution. Further configuration to error
individually is not supported.

When we add this we promise to not change the set of things
that lead to
warning/error in .z releases, but we reserve the right to do
so for .x
releases. The idea being is that you can safely accept
updates for .z
without having to do anything. For .x releases you may need
to step back
to '--strict=warning' and then correct the problem before
going back to
'--strict=error'.

This scheme should cater to those that are pedantic about
following best
practices and not using deprecated features while those that
only care
at major version boundaries can do so in peace without being
bothered
with lots of deprecation warnings.

--

What do you think about this idea? Control all strictness and
deprecation warnings/errors with one flag, and handle
individual ones
(where applicable) by disabling those checks.

The benefit for us developing puppet is that we can
introduce the new
behavior much sooner and we do not need to add flags for
each and every
kind of validation/deprecation. This means we are more
likely to improve
things as we are not holding off until the very last release
in a major
series (and where inevitably some tickets will not make it).

Ironically, if this feature is liked it will make it into
4.5.0 which
may be the last in the 4.x series, but no decision has been
made yet.

- henrik


I am following up with a runtime type strictness thing.

If you have a construct like this in your manifests:

Notify['left'] -> $stuff -> Notify['right']

and at runtime $stuff happens to be an empty array, puppet
currently silently skips the middle part, and thus 'left' and
'right' are not ordered via the dependency in the middle.

Should it warn? Is it an error? (I understand there will be a
difference of opinion here if it should always be one or the
other, or if it should be controlled by the --strict option). I
just wanted to include it as an example of something that is not
caught by static 

[Puppet-dev] Re: How strict do you want puppet to be?

[Henrik Lindberg]

On 23/02/16 01:47, Henrik Lindberg wrote:

Hi, I am thinking ahead a bit regarding puppet 5 and how we should deal
with all the requests for features that require deprecations. (There are
some related things like requests for additional validation and warnings
that are different from deprecations).

In the past we merrily started issuing deprecation warnings, but the
community pretty unanimously said "stop doing that" we cannot deal with
all of those warnings. Since then we then pretty much stopped doing
deprecation warnings.

There has also been a long standing wish for a "strict mode" in puppet,
that like a harsh teacher would point out every itty-bitty problem.

So - what should we do?

In PUP-5889 I have described an idea. This is the text from that ticket
as it stands right now.

PUP-5889
-
Add a flag --strict to puppet settings. When in effect this will turn on
--strict_variables, and will also enable other "helpful" but undesirable
behavior. (Each such behavior to be defined in a separate ticket).

The semantics of this flag should be:

* '--strict=ignore'; no strictness checks are to be performed, nothing
is reported.

* '--strict=warn'; strictness checks are performed, they are reported as
warnings, individually configurable warnings follow their own setting
(i.e. if they are added to disabled_warnings).

* '--strict=errors'; strictness checks are performed, they are reported
as errors and stop the execution. Further configuration to error
individually is not supported.

When we add this we promise to not change the set of things that lead to
warning/error in .z releases, but we reserve the right to do so for .x
releases. The idea being is that you can safely accept updates for .z
without having to do anything. For .x releases you may need to step back
to '--strict=warning' and then correct the problem before going back to
'--strict=error'.

This scheme should cater to those that are pedantic about following best
practices and not using deprecated features while those that only care
at major version boundaries can do so in peace without being bothered
with lots of deprecation warnings.
--

What do you think about this idea? Control all strictness and
deprecation warnings/errors with one flag, and handle individual ones
(where applicable) by disabling those checks.

The benefit for us developing puppet is that we can introduce the new
behavior much sooner and we do not need to add flags for each and every
kind of validation/deprecation. This means we are more likely to improve
things as we are not holding off until the very last release in a major
series (and where inevitably some tickets will not make it).

Ironically, if this feature is liked it will make it into 4.5.0 which
may be the last in the 4.x series, but no decision has been made yet.

- henrik



I am following up with a runtime type strictness thing.

If you have a construct like this in your manifests:

Notify['left'] -> $stuff -> Notify['right']

and at runtime $stuff happens to be an empty array, puppet currently 
silently skips the middle part, and thus 'left' and 'right' are not 
ordered via the dependency in the middle.


Should it warn? Is it an error? (I understand there will be a difference 
of opinion here if it should always be one or the other, or if it should 
be controlled by the --strict option). I just wanted to include it as an 
example of something that is not caught by static checking at 
parse/validation time.


Statically we could possibly check if the outcome is statically known
or if there is a potential problem (i.e. where evaluation can lead to 
"empty thing in the middle"). But I am not sure what the quality of that 
would be and how expensive it would be to implement.


There are number of such cases where puppet is trying to be helpful and 
ends up with a messy situation that it silently ignores or goofs up on.


- henrik



--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups "Puppet 
Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/56CE8764.2000703%40puppetlabs.com.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: How strict do you want puppet to be?

[Henrik Lindberg]

On 24/02/16 20:24, Walter Heck wrote:

On Wednesday, February 24, 2016 at 11:49:17 AM UTC+1, Trevor Vaughan wrote:

I'm also a fan of per module which can override a global setting.

If it could be part of the metadata.json, that would be ideal and
would allow for attestation on the Forge if appropriate.

I gave this some thought and I would be heavily against this: a module
doesn't get to decide for it's consumers wether they will want to run in
strict mode. The person running the puppetmaster should decide this,
preferrably on a per environment base.

What I do think would be good to have in metadata.json is a flag that
says which modes the module supports for which puppet versions. That way
it can be displayed on the forge for instance and I can choose in
advance to only download modules supporting strict mode.



Good idea. Ideally the forge should validate the module by parsing 
everything in it and automatically assert if it passes strict without 
warnings for the version the module states it supports.


This would be a reasonable measure of quality.

Forge can however not validate the runtime strictness (it would need to 
evaluate / compile and that is not guaranteed to find all such problems.


We naturally want to make as much as possible be found when parsing and 
validating statically.


If the forge ere to check strictness for statically discoverable things, 
would you need to manually state strictness in metadata.json?


- henrik


cheers,

--
You received this message because you are subscribed to the Google
Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to puppet-dev+unsubscr...@googlegroups.com
.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-dev/2a0bec94-ff66-4e0f-9930-a76a193a5f88%40googlegroups.com
.
For more options, visit https://groups.google.com/d/optout.



--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups "Puppet 
Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/56CE8499.1020904%40puppetlabs.com.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: How strict do you want puppet to be?

[Henrik Lindberg]

On 24/02/16 11:49, Trevor Vaughan wrote:

I'm also a fan of per module which can override a global setting.

If it could be part of the metadata.json, that would be ideal and would
allow for attestation on the Forge if appropriate.

The global --strict=off should override any module-level setting.




I have been thinking about this for a bit now. Unfortunately it is going 
to be very difficult to achieve any sort of "per module" handling of 
strictness. While it can be applied statically to parsing, most of the 
deprecations are about semantic problems that occur at evaluation time 
and in places where there is absolutely no clue about the origin of the 
logic - or for that matter a mix of modules and data sources that 
together resulted in a state that in the future will be an error.


I think we will start with just a global --strict flag that can be used 
in combination with tools like puppet parser validate at authoring time 
to catch problems. In production I think you always set --strict=off 
since you must already have reviewed and tested the logic that is in 
production, being reminded about warnings for all nodes every 30 minutes 
is not really what you want.


It makes a lot of sense to set it per environment. We do have issues 
with that though; when you want to treat things differently in 
development and production as you must modify a file that is checked in 
and you must remember to not check in the settings that are only 
applicable when an environment is in test or development state. Still 
better than just a global flag.


- henrik

--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups "Puppet 
Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/56CDB6FB.7040105%40puppetlabs.com.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: How strict do you want puppet to be?

[Henrik Lindberg]

On 24/02/16 00:27, Ryan Whitehurst wrote:

On Tue, Feb 23, 2016 at 3:22 PM, Walter Heck  wrote:

On Tuesday, February 23, 2016 at 11:31:18 PM UTC+1, Ben Ford wrote:


Would it be possible in this scheme to mark strict mode per class? I could
mark my own code as being strict and therefore get compile time failures
when I make a typo myself, but wouldn't have to enforce that on all third
party code.



Instead of per class I'd like to be able to set it per (module)path. We use
r10k and split the role and profile modules to their own modulepath
(typically called 'site'), all 3rd party modules live in the standard
modulepath.

I could also imagine people wanting to set this per environment.

Lastly, I think --strict=ignore should be --strict=off, but that's personal
preference.


I agree with everything Walter is saying here. Per-class would be
difficult, but if there was a way to set it per-modulepath, you could
accomplish what Ben is asking for. It should also definitely be an
available setting in environment.conf.



By per modulepath, do you mean per directory containing modules?
the module path consists of multiple directories and the environment has 
one.


Would adding strictness to the module metadata.json work for you?
Then each module author decides how clean their code is.


--strict=off seems a bit nicer to me as well than does
--strict=ignore, but I don't care much.

I also bought the --strict='off' since in many cases it also actually 
turns of the checking (but not always, and it may just be a 
"ignore"/suppression of the logging.



My first thought is that --strict=warn would make a sensible default
in this scenario, as it's most similar to puppet's traditional
behavior of showing deprecation warnings every time.



In the PR that is up, --strict=warning is the default.


--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups "Puppet 
Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/56CD0F8D.1010201%40puppetlabs.com.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: How strict do you want puppet to be?

[Henrik Lindberg]

On 24/02/16 00:22, Walter Heck wrote:

On Tuesday, February 23, 2016 at 11:31:18 PM UTC+1, Ben Ford wrote:

Would it be possible in this scheme to mark strict mode per class? I
could mark my own code as being strict and therefore get compile
time failures when I make a typo myself, but wouldn't have to
enforce that on all third party code.


Instead of per class I'd like to be able to set it per (module)path. We
use r10k and split the role and profile modules to their own modulepath
(typically called 'site'), all 3rd party modules live in the standard
modulepath.

I could also imagine people wanting to set this per environment.



Need to think about that for a bit. It makes it a lot more complicated.
Multiple paths etc. and where to specify them etc.


Lastly, I think --strict=ignore should be --strict=off, but that's
personal preference.



I think that is a good point. Will use 'off' instead of 'ignore'.


great idea, hope to see it come to life sooner rather then later :)



There is a PR up already :-)

- henrik


--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups "Puppet 
Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/56CD0E58.4020708%40puppetlabs.com.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: How strict do you want puppet to be?

[Henrik Lindberg]

On 23/02/16 23:31, Ben Ford wrote:

Would it be possible in this scheme to mark strict mode per class? I
could mark my own code as being strict and therefore get compile time
failures when I make a typo myself, but wouldn't have to enforce that on
all third party code.



Good idea, you probably also need to be able to override because you may 
want to assert the quality of things you are using from 3d parties, 
either report the issues or fix and contribute back.


- henrik


--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups "Puppet 
Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/56CD0DD3.2050806%40puppetlabs.com.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] How strict do you want puppet to be?

[Henrik Lindberg]

Hi, I am thinking ahead a bit regarding puppet 5 and how we should deal 
with all the requests for features that require deprecations. (There are 
some related things like requests for additional validation and warnings 
that are different from deprecations).


In the past we merrily started issuing deprecation warnings, but the 
community pretty unanimously said "stop doing that" we cannot deal with 
all of those warnings. Since then we then pretty much stopped doing 
deprecation warnings.


There has also been a long standing wish for a "strict mode" in puppet, 
that like a harsh teacher would point out every itty-bitty problem.


So - what should we do?

In PUP-5889 I have described an idea. This is the text from that ticket
as it stands right now.

PUP-5889
-
Add a flag --strict to puppet settings. When in effect this will turn on 
--strict_variables, and will also enable other "helpful" but undesirable 
behavior. (Each such behavior to be defined in a separate ticket).


The semantics of this flag should be:

* '--strict=ignore'; no strictness checks are to be performed, nothing 
is reported.


* '--strict=warn'; strictness checks are performed, they are reported as 
warnings, individually configurable warnings follow their own setting 
(i.e. if they are added to disabled_warnings).


* '--strict=errors'; strictness checks are performed, they are reported 
as errors and stop the execution. Further configuration to error 
individually is not supported.


When we add this we promise to not change the set of things that lead to 
warning/error in .z releases, but we reserve the right to do so for .x 
releases. The idea being is that you can safely accept updates for .z 
without having to do anything. For .x releases you may need to step back 
to '--strict=warning' and then correct the problem before going back to 
'--strict=error'.


This scheme should cater to those that are pedantic about following best 
practices and not using deprecated features while those that only care 
at major version boundaries can do so in peace without being bothered 
with lots of deprecation warnings.

--

What do you think about this idea? Control all strictness and 
deprecation warnings/errors with one flag, and handle individual ones 
(where applicable) by disabling those checks.


The benefit for us developing puppet is that we can introduce the new 
behavior much sooner and we do not need to add flags for each and every 
kind of validation/deprecation. This means we are more likely to improve 
things as we are not holding off until the very last release in a major 
series (and where inevitably some tickets will not make it).


Ironically, if this feature is liked it will make it into 4.5.0 which 
may be the last in the 4.x series, but no decision has been made yet.


- henrik

--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups "Puppet 
Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/naga6d%245r3%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: Puppet 4: defined resource types and epp template

[Henrik Lindberg]

On 2016-04-02 11:46, Martin Alfke wrote:


On 01 Feb 2016, at 19:15, Henrik Lindberg <henrik.lindb...@cloudsmith.com> 
wrote:



That is exactly what you should do. An external (file based epp) when called, 
does not get to see variables in the scope from which it was called/used. This 
design is deliberate. Think of the template as a function you are calling, and 
you have to give it its arguments.


In this case the documentation on the website is wrong:
https://docs.puppetlabs.com/puppet/latest/reference/lang_template_epp.html#example-template

The long example is wrong in that it does not show that arguments must 
be given in a hash.


The documetation for epp has unfortunately been boiled down to just the 
bare minimum and the important distinction that variables must be given 
has been lost in the series of rewrites that were made.


The implementation clearly restricts epp() to only see global scope.

The specification is also clear about scoping: 
https://github.com/puppetlabs/puppet-specifications/blob/master/language/templates.md#visibility-of-scoped-variables


I filed a documentation ticket.


The example just switches from template() to epp() function.
According to your writing you must pass all parameters to the epp() function.



yes, epp() requires that arguments are given in the call. epp() 
templates only have access to global variables (i.e. $::osfamily and such).





Contrast this with the inline_epp, which you can think of as a 
lambda/code-block. Here the code block gets to see the variables in scope, 
since it is itself in that scope (part of the same piece of code).


Got it.
To be honest: this is ugly.
I always thought of epp() being a full replacement for template() without the 
scope issue.


The ERB based template() function is for templates in files. The epp() 
function is also for files. The big difference between them is that all 
variables used by the template processed by epp() must be given to the 
function.


The inline_epp() may in the special case, when it does not declare 
parameters have access to all variables in scope. This was deemed to be 
harmless as all of the logic is in one place, and there is never a 
question of reuse. It is also seen as an easy way to transition to 
inline_epp() from inline_template().


With that, I am not sure I understand your comments. What is it that you 
think is ugly?



This especially feels bad, when we will have performance improvement for epp in 
the future.



I have not seen tickets with reports of performance problems. How bad is it?

Best
- henrik

--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups "Puppet 
Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/n8vjpn%244u0%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: Puppet 4: defined resource types and epp template

[Henrik Lindberg]

On 2016-02-02 21:15, John Bollinger wrote:


On Monday, February 1, 2016 at 12:15:12 PM UTC-6, henrik lindberg wrote:

Contrast this with the inline_epp, which you can think of as a
lambda/code-block. Here the code block gets to see the variables in
scope, since it is itself in that scope (part of the same piece of
code).



Hmmm. The docs


The docs are wrong. I have filed a ticket.

The behavior is specified here 
https://github.com/puppetlabs/puppet-specifications/blob/master/language/templates.md#visibility-of-scoped-variables


The implementation works as specified.

Best
- henrik


<https://docs.puppetlabs.com/puppet/latest/reference/lang_template_epp.html#special-scope-rule-for-inlineepp>
cast that behavior of inline_epp() as a special exception, and claim
it's applicable only when the template declares no parameters and you
don't pass any (and only with inline_epp()).  Are the docs wrong?  If
they are correct, then surely it is best to view that exception *as* an
exception, not as some sort of natural consequence of the scope in which
the template or the inline_epp() call appears.

In fact, inasmuch as the template might be presented in the form of a
variable defined in a different scope, drawing its actual value from
who-knows-where, I really think it's confusing to assert that such an
exception is a natural extension of Puppet's scoping rules.  Call it
what it is: an ease-of-use aid applicable only to cases where the
stricter ordinary scoping rules of EPP templates are of little advantage.


John

--
You received this message because you are subscribed to the Google
Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to puppet-dev+unsubscr...@googlegroups.com
<mailto:puppet-dev+unsubscr...@googlegroups.com>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-dev/02150181-1b5d-47a6-b2a4-4ebb3f8a6f98%40googlegroups.com
<https://groups.google.com/d/msgid/puppet-dev/02150181-1b5d-47a6-b2a4-4ebb3f8a6f98%40googlegroups.com?utm_medium=email_source=footer>.
For more options, visit https://groups.google.com/d/optout.



--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups "Puppet 
Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/n8vjvv%248b0%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: Puppet 4: defined resource types and epp template

[Henrik Lindberg]

On 2016-04-02 14:40, Martin Alfke wrote:

On 04 Feb 2016, at 14:30, Henrik Lindberg <henrik.lindb...@cloudsmith.com> 
wrote:

On 2016-04-02 11:46, Martin Alfke wrote:

This especially feels bad, when we will have performance improvement for epp in 
the future.


I have not seen tickets with reports of performance problems. How bad is it?


You mentioned it somewhere at PuppetConf that in the future the epp() function 
will have a performance benefit over template() function due to template() 
needing the ruby stack whereas epp() is running in the java based parser (as 
far as I have understood this).



It has a performance benefit already! What I was referring to is that in 
order to construct the ERB context we need to iterate over all variables 
in all visible scopes and create a temporary class/context where each of 
those variables are added as instance variables. This may mean hundreds 
or possible thousands of variables that must be copied over. This is 
repeated for each an every call to template() and inline_template().


The epp() and inline_epp() does not have to do this. It simply uses the 
scope (global or local depending on epp/inline_epp), then variable 
access goes directly against the scope.


I noticed a performance improvement when manually playing around with 
epp. But this was for small templates, and I did not write any 
benchmarks for epp. However, .pp logic is slower than ruby in general, 
so it depends on what you are doing.


We have ideas for further improvements.

A big improvement will come in the future when we have a native 
compiler. The EPP based templates will then run much faster, and most 
likely faster than ERB since ERB means calling Ruby in an external 
process and serializing the 100-1000 variables for each and every 
invocation.


So, unless you are using very large templates or have complex logic, 
iterate over massive data structures, or having the need to use external 
ruby code, I would absolutely recommend using EPP as soon as possible.


If anyone have real performance problems with EPP I would like to hear 
about it so we can optimize.


Best,
- henrik

--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups "Puppet 
Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/n902kb%24d6b%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: RFC - A specification for module schemas

[Henrik Lindberg]

There were many great replies to this, I am following up on this
and the comments made elsewhere in one go here.

On 2016-30-01 5:45, Corey Osman wrote:

I wanted to bring up a conversation in hopes that we as a community can
create a specification for something I am calling module schemas.
  Before I get into that I want to provide a little background info.

This all started a few years ago when hiera first came out. Data
seperation in the form of parameters and auto hiera lookups quickly
became the norm and reusable modules exploded into what the forge is
today .  Because of the popularity of hiera, data validation is now a
major problem though.  Without good data, excellent modules become useless.

Puppet 4 and stdlib brought many new functions and ways to validate
incoming data, and I consider puppet 4 to now be a loosely typed
language now.   Hell, there was even this a long time ago:
https://github.com/puppetlabs/puppetlabs-kwalify  But puppet only does
so much, and while having validation reside in code might make
troubleshooting a snap, there is still a delay in the feedback loop when
the code is tightly coupled with an external “database” of data.  Data
that is inserted by non puppet developers who don’t know YAML or data
structures.

So with that said I want to introduce something new to puppet module
development, called module schemas.  A module schema is a specification
that details the inner workings of a module.   For right now this means
a detailed specification of all the parameters for classes and
definitions used inside a module who’s goal is to make it impossible to
insert a bad data structure.  But ideally, we can specify so much more
(functions, types, providers, templates) even hiera calls in weird
places like templates and functions, which are usually things that do
not get documented and are hard to reference and usually requires
looking at source code.

What does such a schema look like?

Here is a example schema for the apache module which contains 446
parameters!.
https://github.com/logicminds/puppet_module_schemas/blob/master/apache_schema.yaml

The most immediate use case for such a schema is hiera validation as I
have outlined here:
http://logicminds.github.io/blog/2016/01/16/testing-hiera-data.  Which
works AWESOME!.  We are validating hiera data and not YAML and doing it
under 500 ms for every commit on every single file.

As a community we need a solution for validating hiera data.  Its my
belief that schemas are the way to go.   After all hiera data is now in
modules with no way to easily validate.

Other use cases that come to mind:

   - generating documentation (Many modules on the forge usually contain
a static map of parameters used inside the module).   If a schema was
present, we could just generate that same map automatically.
   - useful for other 3rd party tools like puppet strings
   Parameter specification lookup
   - Imagine a  face that shows internal puppet module specifications.
  I am not talking about puppet-strings, this would detail the
parameters given a class, or an example parameter value given a
parameter name.
 Scenario:
   - puppet module puppetlabs/apache   (outputs all the parameters,
classes for that module) in a specified format (json or yaml)
   - puppet module puppetlabs-apache::class_name (outputs all the
parameters for the class in a specified format (json or yaml)
   - puppet module puppetlabs-apache::class_name::param1  (outputs
an example value for that parameter, as well as the default value) in a
specified format (json or yaml)

Foreman and Puppet Console need this level of detail as well.
  Currently, both of these solutions spend quite a bit of time parsing
code to show parameters for UI display.   It would be much easier if a
schema was available that detailed this level of data..  Think of the
speed improvements that could be had if this information was “cached” in
a file.   These solutions currently load or intelligently scan all the
puppet code for every puppet environment to get the parameters and
defaults.

Here is how we can create a schema
http://logicminds.github.io/blog/2016/01/15/how-to-build-a-module-schema/
   (which I even automated with retrospect-puppet
(https://github.com/nwops/puppet-retrospec.git)

However,  we all need to agree on something before schemas can ever be a
“thing”.  We need a schema for module schemas.  This is important
because as soon as 3rd party tools or scripts start to use schemas and
later we decide the schema needs changing, everything breaks.  Tools
need a specification to work from.

So with this in mind and an example schema here:
https://github.com/logicminds/puppet_module_schemas/blob/master/apache_schema.yaml.
  How can this be improved?  What should we add?

About the only change I was pondering was adding another object for the
types themselves.
https://github.com/logicminds/puppet_module_schemas/blob/master/specification_with_types.yaml

What are your thoughts?  

[Puppet-dev] Re: Puppet 4: defined resource types and epp template

[Henrik Lindberg]

On 2016-01-02 24:23, Martin Alfke wrote:

Hi,

I recently had an issue with epp template within a defined resource type.
Let’s assume the following code snippets:

# modules/test/manifests/init.pp
class test {
   ::test::files { 'test':
 param1 => 'value',
   }
}

# modules/test/manifests/files.pp
define test::files (
   $param1 = '',
){
   file { "/tmp/${title}":
 ensure  => file,
 content => epp('test/files.epp'),
   }
}

# modules/test/templates/files.epp
<%= $param1 %>

The parameter Param1 will not get the data provided within define declaration.

Is this desired behaviour?
Should I open a bug?
Is this a known bug?

When passing the param1 data via hash to epp function, data will get added to 
the template.


That is exactly what you should do. An external (file based epp) when 
called, does not get to see variables in the scope from which it was 
called/used. This design is deliberate. Think of the template as a 
function you are calling, and you have to give it its arguments.


Contrast this with the inline_epp, which you can think of as a 
lambda/code-block. Here the code block gets to see the variables in 
scope, since it is itself in that scope (part of the same piece of code).


This design makes the code more maintainable and templates more 
reusable. It is also easier to test the templates (there is a command 
line utility (puppet epp IIRC) that allows you to feed values into a 
templates and render the result).


Hope that helps
- henrik



Best,
Martin




--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups "Puppet 
Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/n8o7b5%24k5j%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: Is is possible to detect what file and line from which a function was called?

[Henrik Lindberg]

On 2015-12-11 6:20, Trevor Vaughan wrote:

All,

Please disregard, the answer is source.file and source.line and works
MUCH better if you're not testing a function outside of a manifest set ;-).

Thanks,

Trevor

That is not the same as where a function is called from, it is container 
from which the evaluated code makes a call from.


For 4.x we have a ticket to make calling location available.

Question is: for what purpose is this needed? For errors? or for other 
purposes?


- henrik


On Wed, Nov 11, 2015 at 1:32 PM, Trevor Vaughan > wrote:

Hoping to use this for logging purposes.

I can get it from a custom type, but I don't seem to be able to snag
this information from a function.

Thanks,

Trevor

--
Trevor Vaughan
Vice President, Onyx Point, Inc
(410) 541-6699 

-- This account not approved for unencrypted proprietary information --




--
Trevor Vaughan
Vice President, Onyx Point, Inc
(410) 541-6699

-- This account not approved for unencrypted proprietary information --

--
You received this message because you are subscribed to the Google
Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to puppet-dev+unsubscr...@googlegroups.com
.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoUh-R0sh%3DsCSaNx8hPQrz%2BxHepu%3DDCuQqp4ggtoYzLQKQ%40mail.gmail.com
.
For more options, visit https://groups.google.com/d/optout.



--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups "Puppet 
Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/n2dhe9%24poe%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: Puppet Strings Module Release

[Henrik Lindberg]

On 2015-23-09 22:33, Ian Kronquist wrote:

We’re excited to announce the latest release of the Strings module!
Strings generates HTML documentation from inline comments and is
designed to replace /puppet doc/. Strings 0.3.1 is a backward-compatible
minor release. It includes support for generating documentation from
Types and Providers, extracting and checking type signatures from Puppet
4.0 functions and classes, and checking that documented parameter names
and types match what's actually in the code. Inline comments are the
future for documenting Puppet modules, so if you maintain a module go
ahead and take Strings for a spin and let us know what you think! This
release was driven by Ian Kronquist’s Summer intern project.

*  The CHANGELOG can be found here:
https://github.com/puppetlabs/puppetlabs-strings/blob/master/CHANGELOG.md

*  The module can be found on the Forge here:
https://forge.puppetlabs.com/puppetlabs/strings

*  Installation and usage instructions can be found here:
https://github.com/puppetlabs/puppetlabs-strings/blob/master/README.md

To track issues related to this release or file new issues, head over to
the PDOC JIRA Project:

* JIRA Project: https://tickets.puppetlabs.com/browse/PDOC

* Fixes for 0.3.1: https://tickets.puppetlabs.com/issues/?filter=15532


Ian Kronquist
iankronqu...@gmail.com 
Software Developer, OSU Open Source Lab


Congratulations Ian, and thanks for all the great work on Puppet Strings!

- henrik



--
You received this message because you are subscribed to the Google
Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to puppet-dev+unsubscr...@googlegroups.com
.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-dev/CAP9O3dcXYDgBH%3DaUZJKryZ3B%3DyeTW1B634-PyK%2BhwRnGxgr%2B-w%40mail.gmail.com
.
For more options, visit https://groups.google.com/d/optout.



--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups "Puppet 
Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/mtvcer%24fea%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: EPP templates and ruby code

[Henrik Lindberg]

On 2015-11-09 11:31, Erik Dalén wrote:

You use the range function from stdlib to create an array. For example:
range(1,5).each |$x| {notice($x)}



The same is achieved with an Integer type:

Integer[1,5].each |$x| { notice $x }

It is more efficient than the range function since it does not create an 
array of all values in the range.


- henrik


On Fri, 11 Sep 2015 at 10:41 Martin Alfke > wrote:

Hi Michael,

On 10 Sep 2015, at 17:29, Michael Smith
>
wrote:

 > Looking at the examples for the 'each' function
(https://docs.puppetlabs.com/references/latest/function.html#each)
you can replace 'times' with 'each' and I think it'll work.

each can be used on arrays. In my case the variable is an integer.
I want to print stuff $i times into the final file.

 >
 > On Sep 10, 2015, at 4:34 AM, Martin Alfke > wrote:
 >
 >> Hi,
 >>
 >> is there a way to iterate with the ruby .times function inside
an EPP template?
 >>
 >> e.g.
 >> <% $i.times |$x| { -%>
 >> i = <%= $x +1 %>
 >> <% } -%>
 >> The code above seems to need a puppet4 function with name “times”.
 >> Result:
 >> Error: Evaluation Error: Unknown function: 'times'.
 >>
 >> When changing to ruby code style another error occurs:
 >> <% $i.times do |$x| -%>
 >> i = <%= $x +1 %>
 >> <% end -%>
 >> Error: Evaluation Error: Error while evaluating a Function Call,
epp(): Invalid EPP: Syntax error at ‘|'
 >>
 >> thanks,
 >> Martin
 >>
 >> --
 >> You received this message because you are subscribed to the
Google Groups "Puppet Developers" group.
 >> To unsubscribe from this group and stop receiving emails from
it, send an email to puppet-dev+unsubscr...@googlegroups.com
.
 >> To view this discussion on the web visit

https://groups.google.com/d/msgid/puppet-dev/4F22DF3E-4CD3-410E-B602-BD5B94CE089A%40gmail.com.
 >> For more options, visit https://groups.google.com/d/optout.
 >
 > --
 > You received this message because you are subscribed to the
Google Groups "Puppet Developers" group.
 > To unsubscribe from this group and stop receiving emails from it,
send an email to puppet-dev+unsubscr...@googlegroups.com
.
 > To view this discussion on the web visit

https://groups.google.com/d/msgid/puppet-dev/3B71678C-05FD-4E4B-B608-EC4A769A2CDB%40puppetlabs.com.
 > For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to puppet-dev+unsubscr...@googlegroups.com
.
To view this discussion on the web visit

https://groups.google.com/d/msgid/puppet-dev/238A23C7-A1D5-4E09-830E-A4F49142E43F%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to puppet-dev+unsubscr...@googlegroups.com
.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-dev/CAAAzDLc5utqECijD-Zd%2BAUVTbbHC61hxuO%2BxNNT2ovmbzHApuw%40mail.gmail.com
.
For more options, visit https://groups.google.com/d/optout.



--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups "Puppet 
Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/mt1d3l%242nq%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: Puppetconf 2015: pre-summit tea party

[Henrik Lindberg]

On 2015-11-09 19:20, Daniele Sluijters wrote:

Hi everyone,

As is tradition (since last year)


It was a nice event last year.


Want to join? Throw a +1 in the list so I get a rough idea of how people
are going to show.



+1

- henrik

--

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups "Puppet 
Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/mt1d9e%245k0%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: Using data types on hashes

[Henrik Lindberg]

On 2015-27-08 9:01, Martin Alfke wrote:


On 27 Aug 2015, at 02:08, Henrik Lindberg henrik.lindb...@cloudsmith.com 
wrote:


On 2015-26-08 17:56, Martin Alfke wrote:

Hi Henrik,
On 26 Aug 2015, at 17:49, Henrik Lindberg henrik.lindb...@cloudsmith.com 
wrote:




Hash[String, Struct[{ uid = Integer, home = Pattern[/^\/.*/]}]] $hash

Am I missing something?
Within the Struct I provide the key name (uid) and Type (Integer) and the 
second key (home) with Type (Pattern).


Yes, you are indeed. Your regular expression is however not correct for the 
home directories that you have given. The regexp /^\/.*/ requires that the 
string starts with a slash. If you change your input to say

home = '/home/jones'

it should work.


Ah. It does work providing correct data.
The error message if I do provide a string with missing / at start pointed me 
into wrong direction.

Many thanks for your help.
I know learned that using Struct inside has will not give a detailed output on 
a specific sub hash to be wrong.
It complains about the whole hash being wrong without details.



We have improvements for how type mismatches are reported. Hope to get 
that into 4.3.0. It has a couple of heuristics regarding what is most 
useful to know, and in your case it will point to the path where the 
first mismatch was found.


- henrik


--

Visit my Blog Puppet on the Edge
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups Puppet 
Developers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/mrs0vn%24oqb%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: Using data types on hashes

[Henrik Lindberg]

On 2015-24-08 14:38, Martin Alfke wrote:

Hi,

I am playing around with the Puppet 4 Type system and nested hashes.
Is it possible to use the Type system on a hash and check sub hashes as Struct?


Yes, certainly.


# nested hashes
$hash = {
   ‘ben’ = {
 uid = 2204,
 home = ‘/home/ben’,
   },
   ‘jones’ = {
 uid = 2205,
 home = ‘home/jones’,
   }
}

# class definition
class users (
   Hash[String, Struct[{ uid = Integer, home = Pattern[/^\/.*/]}]] $hash,
){
…
}

# class declaration
class { ‘users’:
   hash = $hash,
}

In this case I receive the following error:
Error: Expected parameter 'hash' of 'Class[Users]' to have type Hash[String, Struct[{'uid'=Integer, 
'home'=Pattern[/^\/.*/]}]], got Struct[{'ben'=Struct[{'uid'=Integer, 'home'=String}], 
'jones'=Struct[{'uid'=Integer, 'home'=String}]}] at /root/hash.pp:16 on node 
puppetmaster.example.net

That is because you cannot match integer values (such as 2204) with a 
Pattern (it only matches strings).



When using the Struct one needs to know the keys. But the keys may be any 
arbitrary value.

If you want to support any arbitrary value of any type (including 
Undef), use the Any type. If you do not want Undef, you can use 
NotUndef[Any]. If you want to be explicit about values of integers and 
strings (not being empty strings say), you can use a Variant type - e.g. 
Variant[Integer, String] (any string or integer value), 
Variant[Integer[0, default], String[1]] (any 0 or positive integer, or 
any non empty string). etc.


You can also use the Scalar type if you want to accept non undef single 
values (matches any Integer, Float, String, and Regular expression).


Plenty of options depending on how stringent you want the check to be.


Within class users I could use the keys function and a define and check for the 
data types within the define.
But I thought that the mentioned approach is valid (at least I haven’t found a 
hint that my approach is not valid).



Your approach is valid, you just got tripped up by Integer values not 
matching with a Pattern.


Regards
- henrik
--

Visit my Blog Puppet on the Edge
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups Puppet 
Developers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/mrkn62%246jp%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: Using data types on hashes

[Henrik Lindberg]

On 2015-26-08 17:56, Martin Alfke wrote:

Hi Henrik,
On 26 Aug 2015, at 17:49, Henrik Lindberg henrik.lindb...@cloudsmith.com 
wrote:


On 2015-24-08 14:38, Martin Alfke wrote:

Hi,

I am playing around with the Puppet 4 Type system and nested hashes.
Is it possible to use the Type system on a hash and check sub hashes as Struct?


Yes, certainly.


# nested hashes
$hash = {
   ‘ben’ = {
 uid = 2204,
 home = ‘/home/ben’,
   },
   ‘jones’ = {
 uid = 2205,
 home = ‘home/jones’,
   }
}

# class definition
class users (
   Hash[String, Struct[{ uid = Integer, home = Pattern[/^\/.*/]}]] $hash,
){
…
}

…



That is because you cannot match integer values (such as 2204) with a Pattern 
(it only matches strings).


??


Doh, I misread.


Hash[String, Struct[{ uid = Integer, home = Pattern[/^\/.*/]}]] $hash

Am I missing something?
Within the Struct I provide the key name (uid) and Type (Integer) and the 
second key (home) with Type (Pattern).

Yes, you are indeed. Your regular expression is however not correct for 
the home directories that you have given. The regexp /^\/.*/ requires 
that the string starts with a slash. If you change your input to say


home = '/home/jones'

it should work.

Or change the regular expression. The ^ means anchor at start of string, 
and \/ means a slash is required, then 0 or more characters. If you 
accept any String, simply use String. If you want to ensure that it is a 
valid path you need a way more complex regular expression.


Were you attempting to specify not allowed to start with a slash?
If so, try Pattern[/^[^\/]+.*/]




Within class users I could use the keys function and a define and check for the 
data types within the define.
But I thought that the mentioned approach is valid (at least I haven’t found a 
hint that my approach is not valid).



Your approach is valid, you just got tripped up by Integer values not matching 
with a Pattern.


Good to know…


Sorry that I went of on a tangent after misreading the error message.
Hope the suggestions above will work out better for you.

Regards
Henrik

--

Visit my Blog Puppet on the Edge
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups Puppet 
Developers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/mrlkee%244st%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: Accessing resource attributes

[Henrik Lindberg]

On 2015-31-07 12:35, R.I.Pienaar wrote:

I just noticed last night we can now access resource and class attributes:

define x($y) {
}

x{foo: y = bar}

notice(X[foo][y])

this works, yay with the obvious parsing order caveats.

This works too:

define x($y) {
 notify{$name: message =  $y }
}

class one {
  x{one: y = hello from one}
}

include one

notice(X[one][y])

but this does not:

 define x($y) {
   notify{$name: message =  $y }
 }

 x{hello: y = hello world}

 notify{outside: message = hello from outside}

 notify{X[hello][y]: }
 notify{Notify[outside][message]: } # works
 notify{Notify[hello][message]: } # fails

It can't find the notify inside the defined type X:

Error: Evaluation Error: Resource not found: Notify['hello']

commenting out the problem notify I do see:

Notice: /Stage[main]/Main/X[hello]/Notify[hello]/message: defined 'message' as 
'hello world'

which suggests I didn't mess up the resource name

Is this a bug or just not supposed to work?



You said it yourself the obvious parsing order caveats. Afaict, the 
problem is that the resource has not yet been evaluated in the cases 
where it did not work for you.


if you do something like

  define foo { something { title: x = 1} }
  foo { bar: }
  notice Something['title']['x']

That fails because the evaluation of foo is lazy. When it is evaluated 
it enqueues the evaluation of the 'something' and returns. Then an 
attempt is made to access 'Something[title]' which will fail as it is 
not yet evaluated. If you stick the second in a define, it will work as 
it will be enqueued after what was enqueued in 'foo'.


e.g.

  define foo { something { title: x = 1} }
  define helper { notice Something['title']['x'] }
  foo {bar: }
  helper { yay: }

More about the order of evaluation can be read here:

http://puppet-on-the-edge.blogspot.se/2014/04/getting-your-puppet-ducks-in-row.html

- henrik


---
R.I.Pienaar




--

Visit my Blog Puppet on the Edge
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups Puppet 
Developers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/55BB73AD.60700%40cloudsmith.com.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: Getting module_name from Puppet 4 functions

[Henrik Lindberg]

On 2015-20-07 17:59, Alessandro Franceschi wrote:

Thanks Henrik,
useful and detailed info as usual.
In this case I've to manage a puppet4 version of a function,
params_lookup which is widely used in all my old modules, so it's
impossible to change the params passed to it without changing all those
modules.


Note that there is nothing forcing you to use the 4.x function API (at 
least not yet) ;-). Also well aware of the special functions people have 
to do in order to move older code along. Hence, sharing the information 
about how to achieve this (but have to include the reasons why system 
functions with special powers should be used sparingly).




I fear I'll need to make a system function, even if not recommended.
Do you mind to give a look to it, once I complete it, in order to avoid
possible bad side effects?


Ping me when you have something for review.
- henrik



Thanks as usual
al

On Monday, July 20, 2015 at 4:10:20 PM UTC+2, henrik lindberg wrote:

On 2015-19-07 18:14, Alessandro Franceschi wrote:
  This is a silly question that will take to guys like Henrik less
than a
  nanosecond to reply, I suppose, but I'm struggling hard with this
and
  haven't found a way out.
 
  I need to get the value of the compiler variable module_name from
within
  a Puppet 4 function.
  I've tried various approach, the one most resembling the approach in
  Puppet 3 function is:
  mod =  closure_scope['module_name']
  but it doesn't work.
 
  Any hint?
  Your nanosecond is worth at least a sunday afternoon of mine.
 

The 'closure scope' is the scope the function is defined in.
You can also get access to 'calling scope' by creating a 'system
function' instead of a regular function.

A 'system function' is special as it has more power, but it also comes
with more responsibility.

Ideally functions should not ever need anything but the arguments given
to them. It is good practice to write such pure functions. (i.e.
consider requiring that $module_name is given to the function).

A 'system function' is written like this:

Puppet::Functions.create_function(:inline_epp,
Puppet::Functions::InternalFunction) do

dispatch :inline_epp do
  scope_param()
  param 'String', :template
  optional_param 'Hash[Pattern[/^\w+$/], Any]', :parameters
end

def inline_epp(scope, template, parameters = nil)
  Puppet::Pops::Evaluator::EppEvaluator.inline_epp(scope, template,
parameters)
end
end

If you cannot spot it immediately, there are two things you need to do
in your function to make it a 'system function':
* Use create_function(:name, Puppet::Functions::InternalFunction)
instead of just create_functon(:name).
* Make the dispatcher inject the calling scope by calling
'scope_param()' in the dispatchers definition of parameters.

Then, use the calling scope to get the calling scope's variables.

The reason it is bad to directly get variables from a calling
scope is
that it makes it hard to write general functions. What if you in a
module want to provide a function that wraps calls to a function that
picks up $module_name, or $calling_module from scope?

Thus, use 'InternalFunction' and 'scope_param()' sparingly, if at all.

- henrik

--

Visit my Blog Puppet on the Edge
http://puppet-on-the-edge.blogspot..se/
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google
Groups Puppet Developers group.
To unsubscribe from this group and stop receiving emails from it, send
an email to puppet-dev+unsubscr...@googlegroups.com
mailto:puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-dev/2534c29c-72d7-4bbe-84c8-debe3b59f1c8%40googlegroups.com
https://groups.google.com/d/msgid/puppet-dev/2534c29c-72d7-4bbe-84c8-debe3b59f1c8%40googlegroups.com?utm_medium=emailutm_source=footer.
For more options, visit https://groups.google.com/d/optout.



--

Visit my Blog Puppet on the Edge
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups Puppet 
Developers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/mp1aqt%24e03%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: Getting module_name from Puppet 4 functions

[Henrik Lindberg]

On 2015-19-07 18:14, Alessandro Franceschi wrote:

This is a silly question that will take to guys like Henrik less than a
nanosecond to reply, I suppose, but I'm struggling hard with this and
haven't found a way out.

I need to get the value of the compiler variable module_name from within
a Puppet 4 function.
I've tried various approach, the one most resembling the approach in
Puppet 3 function is:
mod =  closure_scope['module_name']
but it doesn't work.

Any hint?
Your nanosecond is worth at least a sunday afternoon of mine.



The 'closure scope' is the scope the function is defined in.
You can also get access to 'calling scope' by creating a 'system 
function' instead of a regular function.


A 'system function' is special as it has more power, but it also comes 
with more responsibility.


Ideally functions should not ever need anything but the arguments given 
to them. It is good practice to write such pure functions. (i.e. 
consider requiring that $module_name is given to the function).


A 'system function' is written like this:

Puppet::Functions.create_function(:inline_epp, 
Puppet::Functions::InternalFunction) do


  dispatch :inline_epp do
scope_param()
param 'String', :template
optional_param 'Hash[Pattern[/^\w+$/], Any]', :parameters
  end

  def inline_epp(scope, template, parameters = nil)
Puppet::Pops::Evaluator::EppEvaluator.inline_epp(scope, template, 
parameters)

  end
end

If you cannot spot it immediately, there are two things you need to do 
in your function to make it a 'system function':
* Use create_function(:name, Puppet::Functions::InternalFunction) 
instead of just create_functon(:name).
* Make the dispatcher inject the calling scope by calling 
'scope_param()' in the dispatchers definition of parameters.


Then, use the calling scope to get the calling scope's variables.

The reason it is bad to directly get variables from a calling scope is 
that it makes it hard to write general functions. What if you in a 
module want to provide a function that wraps calls to a function that 
picks up $module_name, or $calling_module from scope?


Thus, use 'InternalFunction' and 'scope_param()' sparingly, if at all.

- henrik

--

Visit my Blog Puppet on the Edge
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups Puppet 
Developers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/moivg1%24dgk%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: How do you know what variables are frozen?

[Henrik Lindberg]

On 2015-15-07 12:22, Peter Huene wrote:

On Wed, Jul 15, 2015 at 12:18 PM, Trevor Vaughan tvaug...@onyxpoint.com
mailto:tvaug...@onyxpoint.com wrote:

Indeed, I did change it to that but it would still be nice to know
what's going on so that I can stuff it into rspec and check for it.


Puppet's compiler intentionally freezes $trusted and its values.  If you
intend to support operating on arguments like this, you should be
creating a copy and not modifying the argument (probably a good rule to
follow anyway, especially for a function named parse).

Peter is right. Rule of thumb, never ever mutate arguments that are 
given to you or that you lookup. Unfortunately deep freezing all values 
has a high cost or we would do that extensively when the puppet runtime 
returns values, or calls into user supplied logic.


Regards
- henrik


Thanks,

Trevor

On Wed, Jul 15, 2015 at 2:59 PM, R.I.Pienaar r...@devco.net
mailto:r...@devco.net wrote:



- Original Message -
 From: Trevor Vaughan tvaug...@onyxpoint.com 
mailto:tvaug...@onyxpoint.com
 To: puppet-dev puppet-dev@googlegroups.com 
mailto:puppet-dev@googlegroups.com
 Sent: Wednesday, July 15, 2015 7:50:36 PM
 Subject: Re: [Puppet-dev] How do you know what variables are frozen?

 In this case, it was a custom function that was tripping things up.

 Looking through the code, I *think* this was the scenario:

 1) Fact = trusted['certname']
 2) Function call to parse_host($trusted['certname'])
 3) Parse host:

 host = args.first
 host.strip! # Here's the offender

 I also tried seeing if dup'ing the args would work, but it copies the
 frozen attribute with the String object (which makes sense).

 So, I don't know if I was trying to modify something from a Fact or 
if the
 object just happened to be carrying the 'frozen' status as it went 
along.

so do not use strip! use x = something.strip? this way you dont
try to
modify anything in scope.

--
You received this message because you are subscribed to the
Google Groups Puppet Developers group.
To unsubscribe from this group and stop receiving emails from
it, send an email to puppet-dev+unsubscr...@googlegroups.com
mailto:puppet-dev%2bunsubscr...@googlegroups.com.
To view this discussion on the web visit

https://groups.google.com/d/msgid/puppet-dev/1753817636.200946.1436986745705.JavaMail.zimbra%40devco.net.
For more options, visit https://groups.google.com/d/optout.




--
Trevor Vaughan
Vice President, Onyx Point, Inc
(410) 541-6699 tel:%28410%29%20541-6699

-- This account not approved for unencrypted proprietary information --

--
You received this message because you are subscribed to the Google
Groups Puppet Developers group.
To unsubscribe from this group and stop receiving emails from it,
send an email to puppet-dev+unsubscr...@googlegroups.com
mailto:puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit

https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoWKrP%2Btj%2BJsiENkam9q7ohLGQd5-Z5Gfa%3DoE%2BsSX2D%2Bvw%40mail.gmail.com

https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoWKrP%2Btj%2BJsiENkam9q7ohLGQd5-Z5Gfa%3DoE%2BsSX2D%2Bvw%40mail.gmail.com?utm_medium=emailutm_source=footer.

For more options, visit https://groups.google.com/d/optout.




--
*Join us at **PuppetConf 2015, October 5-9 in Portland, OR - *www.
http://www.google.com/url?q=http%3A%2F%2Fwww.sa=Dsntz=1usg=AFQjCNEnS7itqgvQV3E4Se1fu4Um_UapSw2015.puppetconf.com
http://www.google.com/url?q=http%3A%2F%2F2015.puppetconf.comsa=Dsntz=1usg=AFQjCNE1uQL4Sh23Vr-XkPLa4xfNcoXSog**
/Register early to save 40%!/

--
You received this message because you are subscribed to the Google
Groups Puppet Developers group.
To unsubscribe from this group and stop receiving emails from it, send
an email to puppet-dev+unsubscr...@googlegroups.com
mailto:puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-dev/CACZQQfOK4iJk7KcFT9_9n3pU-4U90Cxnhw2660FdafMCuYAggg%40mail.gmail.com
https://groups.google.com/d/msgid/puppet-dev/CACZQQfOK4iJk7KcFT9_9n3pU-4U90Cxnhw2660FdafMCuYAggg%40mail.gmail.com?utm_medium=emailutm_source=footer.
For more options, visit https://groups.google.com/d/optout.



--

Visit my Blog Puppet on the Edge
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups Puppet 
Developers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/mo79uh%24hq2%241%40ger.gmane.org.
For more 

[Puppet-dev] Re: Catalog Deserialization performance

[Henrik Lindberg]

On 2015-01-07 9:05, Romain F. wrote:


There is probably lots of duplicated work going on at the levels above
and those are causing those generic methods to light up (except
Puppet::Resource.initialize).

There is both the deserialization process as such to optimize, but also
the Resource implementation itself which is far from optimal.

The next thing would be to focus on Resource.initialize/from_data_hash


Agreed, but I can't do that on my own in timely fashion. We just want
puppet devs  to be aware that Puppet::Resource gets more features but
performances is not following anyhow, and it's getting worse in puppet
4. I don't really now how it can be adressed.


We recently discussed how to keep an eye on performance regressions like 
this one. We were very observant on the 4.x future parser changes and 
that they performed well - I guess that work overshadowed performance 
regressions in other areas.


I think the problem should be logged as a ticket with the findings so 
far (big thanks you for taking the time and doing the detective work and 
profiling!)




I think it is also relevant to establish some kind of world record -
say serializing and deserializing a hash using MsgPack; a hash of data
cannot be transported faster across the wire than that (unless also not
using Ruby objects to represent the data - with a lot of extra
complexity).

I mean, a hash of some complexity will always consume quite a bit of
processing and memory to get across the wire. Is it hitting the world
record enough?

That's not the point, like I said, this performance gap is coming from
the creation the Graph itself, not the deserialization from PSON or
MsgPack. In my case, in the 4 sec deserialization, 3.5 secs is the
from_data_hash.


ok, point taken, you already established that.


--

Visit my Blog Puppet on the Edge
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups Puppet 
Developers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/5595B6A5.8030804%40cloudsmith.com.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: Catalog Deserialization performance

[Henrik Lindberg]

On 2015-30-06 16:17, Romain F. wrote:

I've already benchmarked and profiled Catalog's from_data_hash and
to_data_hash methods using the benchmark framework.
Most of the time is spent in the from_data_hash (we already knew it) but
there is no big pitfalls where Ruby loses his time.

My callgrind file shows that the top 5 (in self time) is :
- Array.flatten (55000 calls)
- Array..each (115089 calls)
- Puppet::Resource.initialize (15000 calls)
- String.=~ (65045 calls)
- Hash[]= (115084 calls)

This top 5 is taking ~30% of the total time .

As you can see, it can be dificult to optimize this. IMHO, the
benchmark - tweak - benchmark way of optimizing is not sufficient
here. I think the way it (de)serialize a catalog needs a deep refactor.



There is probably lots of duplicated work going on at the levels above 
and those are causing those generic methods to light up (except 
Puppet::Resource.initialize).


There is both the deserialization process as such to optimize, but also 
the Resource implementation itself which is far from optimal.


The next thing would be to focus on Resource.initialize/from_data_hash

I think it is also relevant to establish some kind of world record - 
say serializing and deserializing a hash using MsgPack; a hash of data 
cannot be transported faster across the wire than that (unless also not 
using Ruby objects to represent the data - with a lot of extra complexity).


I mean, a hash of some complexity will always consume quite a bit of 
processing and memory to get across the wire. Is it hitting the world 
record enough?


- henrik


Cheers,

Le mardi 30 juin 2015 04:23:42 UTC+2, henrik lindberg a écrit :

On 2015-29-06 22:41, Trevor Vaughan wrote:
  If you get a profiling suite together (aka, bunch of random patches)
  could you release it?
 

It is not difficult actually. Look at the benchmarks in the puppet code
base. Many of them are suitable for profiling with a ruby profiler.
I don't think we have any benchmarks targeting the agent side
though, so
the first thing to do (for someone) is to write one.

What is more difficult is coming up with a benchmark that does not
involve real/complex resources - but deserialization and up to actually
applying should be possible to work with in a simple way.

Profiling is then just running that benchmark with the ruby profiler
turned on and analyzing the result, make changes, run again... (repeat
until happy).

- henrik


  I've been curious about this for quite some time but never quite got
  around to dealing with it.
 
  My concern is very much client side performance since the more you
  managing a client, the less the client gets to do it's actual job.
 
  Thanks,
 
  Trevor
 
  On Mon, Jun 29, 2015 at 4:35 PM, Henrik Lindberg
  henrik.@cloudsmith.com javascript:
mailto:henrik@cloudsmith.com javascript:
  wrote:
 
  On 2015-29-06 16 tel:2015-29-06%2016:48, Romain F. wrote:
 
  Hi everyone,
 
  I try to optimize our Puppet runs by running some
benchmarks and
  patching the puppet core (if possible).. But I have some
difficulties
  around the catalog serialization/deserialization.
 
  In fact, in 3.7.5 or 3.8.x, the Config Retrieval takes
roughly
  7secs and
  only 4 secs is on the master side. Same fact in 4.2 but
with 9
  secs of
  config retrieval and still 4 secs on the master side.
 
  My first thoughts was Okay, time to try MsgPack. No
improvements.
 
  I've instrumented a bit the code in the master branch around
  this, and
  I've found out that, on my 9secs of config retrieval,
3.61secs
  is lost
  in catalog deserialization, 2 secs is the catalog
conversion..
  But it's
  not the real deserialization (PSON to Hash) that takes
ages,
  it's the
  creation of the Catalog object itself (Hash to catalog).
Benchmarks
  shows that the time to deserialize MsgPack (or PSON) is
negligible
  compared to the catalog deserialization time.
 
  So here is my question : Is that a known issue ? Is there
any
  reason of
  the regression in 4.x (Future parser creating more
objects, ...) ?
 
  The parser=future setting only makes a difference when
compiling the
  catalog - the catalog itself does not contain more or
different data
  (except possibly using numbers instead of strings for some
attributes).
 
  The best way to optimize this is to write a benchmark using the
  benchmark framework and measure the time it takes to
deserialize a
  given catalog. Then run that benchmark

[Puppet-dev] Re: Catalog Deserialization performance

[Henrik Lindberg]

On 2015-29-06 16:48, Romain F. wrote:

Hi everyone,

I try to optimize our Puppet runs by running some benchmarks and
patching the puppet core (if possible). But I have some difficulties
around the catalog serialization/deserialization.

In fact, in 3.7.5 or 3.8.x, the Config Retrieval takes roughly 7secs and
only 4 secs is on the master side. Same fact in 4.2 but with 9 secs of
config retrieval and still 4 secs on the master side.

My first thoughts was Okay, time to try MsgPack. No improvements.

I've instrumented a bit the code in the master branch around this, and
I've found out that, on my 9secs of config retrieval, 3.61secs is lost
in catalog deserialization, 2 secs is the catalog conversion.. But it's
not the real deserialization (PSON to Hash) that takes ages, it's the
creation of the Catalog object itself (Hash to catalog). Benchmarks
shows that the time to deserialize MsgPack (or PSON) is negligible
compared to the catalog deserialization time.

So here is my question : Is that a known issue ? Is there any reason of
the regression in 4.x (Future parser creating more objects, ...) ?

The parser=future setting only makes a difference when compiling the 
catalog - the catalog itself does not contain more or different data 
(except possibly using numbers instead of strings for some attributes).


The best way to optimize this is to write a benchmark using the 
benchmark framework and measure the time it takes to deserialize a given 
catalog. Then run that benchmark with Ruby profiling turned on.


There are quite a few things going on at the agent side in addition to 
taking the catalog PSON and turning it into a catalog that it can apply 
(loading types, resolving providers, etc). Make sure to benchmark these 
separately if possible.


Regards
- henrik


Cheers,

--
You received this message because you are subscribed to the Google
Groups Puppet Developers group.
To unsubscribe from this group and stop receiving emails from it, send
an email to puppet-dev+unsubscr...@googlegroups.com
mailto:puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-dev/a5bf7422-6119-43ee-ba11-44001c1ce097%40googlegroups.com
https://groups.google.com/d/msgid/puppet-dev/a5bf7422-6119-43ee-ba11-44001c1ce097%40googlegroups.com?utm_medium=emailutm_source=footer.
For more options, visit https://groups.google.com/d/optout.



--

Visit my Blog Puppet on the Edge
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups Puppet 
Developers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/mmsa75%24edc%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: Catalog Deserialization performance

[Henrik Lindberg]

On 2015-29-06 22:41, Trevor Vaughan wrote:

If you get a profiling suite together (aka, bunch of random patches)
could you release it?



It is not difficult actually. Look at the benchmarks in the puppet code 
base. Many of them are suitable for profiling with a ruby profiler.
I don't think we have any benchmarks targeting the agent side though, so 
the first thing to do (for someone) is to write one.


What is more difficult is coming up with a benchmark that does not 
involve real/complex resources - but deserialization and up to actually 
applying should be possible to work with in a simple way.


Profiling is then just running that benchmark with the ruby profiler 
turned on and analyzing the result, make changes, run again... (repeat 
until happy).


- henrik



I've been curious about this for quite some time but never quite got
around to dealing with it.

My concern is very much client side performance since the more you
managing a client, the less the client gets to do it's actual job.

Thanks,

Trevor

On Mon, Jun 29, 2015 at 4:35 PM, Henrik Lindberg
henrik.lindb...@cloudsmith.com mailto:henrik.lindb...@cloudsmith.com
wrote:

On 2015-29-06 16 tel:2015-29-06%2016:48, Romain F. wrote:

Hi everyone,

I try to optimize our Puppet runs by running some benchmarks and
patching the puppet core (if possible). But I have some difficulties
around the catalog serialization/deserialization.

In fact, in 3.7.5 or 3.8.x, the Config Retrieval takes roughly
7secs and
only 4 secs is on the master side. Same fact in 4.2 but with 9
secs of
config retrieval and still 4 secs on the master side.

My first thoughts was Okay, time to try MsgPack. No improvements.

I've instrumented a bit the code in the master branch around
this, and
I've found out that, on my 9secs of config retrieval, 3.61secs
is lost
in catalog deserialization, 2 secs is the catalog conversion..
But it's
not the real deserialization (PSON to Hash) that takes ages,
it's the
creation of the Catalog object itself (Hash to catalog). Benchmarks
shows that the time to deserialize MsgPack (or PSON) is negligible
compared to the catalog deserialization time.

So here is my question : Is that a known issue ? Is there any
reason of
the regression in 4.x (Future parser creating more objects, ...) ?

The parser=future setting only makes a difference when compiling the
catalog - the catalog itself does not contain more or different data
(except possibly using numbers instead of strings for some attributes).

The best way to optimize this is to write a benchmark using the
benchmark framework and measure the time it takes to deserialize a
given catalog. Then run that benchmark with Ruby profiling turned on.

There are quite a few things going on at the agent side in addition
to taking the catalog PSON and turning it into a catalog that it can
apply (loading types, resolving providers, etc). Make sure to
benchmark these separately if possible.

Regards
- henrik

Cheers,

--
You received this message because you are subscribed to the Google
Groups Puppet Developers group.
To unsubscribe from this group and stop receiving emails from
it, send
an email to puppet-dev+unsubscr...@googlegroups.com
mailto:puppet-dev%2bunsubscr...@googlegroups.com
mailto:puppet-dev+unsubscr...@googlegroups.com
mailto:puppet-dev%2bunsubscr...@googlegroups.com.
To view this discussion on the web visit

https://groups.google.com/d/msgid/puppet-dev/a5bf7422-6119-43ee-ba11-44001c1ce097%40googlegroups.com

https://groups.google.com/d/msgid/puppet-dev/a5bf7422-6119-43ee-ba11-44001c1ce097%40googlegroups.com?utm_medium=emailutm_source=footer.
For more options, visit https://groups.google.com/d/optout.



--

Visit my Blog Puppet on the Edge
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google
Groups Puppet Developers group.
To unsubscribe from this group and stop receiving emails from it,
send an email to puppet-dev+unsubscr...@googlegroups.com
mailto:puppet-dev%2bunsubscr...@googlegroups.com.
To view this discussion on the web visit

https://groups.google.com/d/msgid/puppet-dev/mmsa75%24edc%241%40ger.gmane.org.

For more options, visit https://groups.google.com/d/optout.




--
Trevor Vaughan
Vice President, Onyx Point, Inc
(410) 541-6699

-- This account not approved for unencrypted proprietary information --

--
You received this message because you are subscribed to the Google
Groups Puppet Developers group.
To unsubscribe from this group and stop receiving emails from it, send
an email to puppet-dev+unsubscr...@googlegroups.com
mailto:puppet

[Puppet-dev] Re: relative paths to defines deprecated?

[Henrik Lindberg]

On 2015-25-06 11:54, Erik Dalén wrote:

I guess this was https://tickets.puppetlabs.com/browse/PUP-121. So the
change was intentional then.

Yes, the change was intentional, but all holes were not plugged right 
away IIRC.


- henrik


On Thu, 25 Jun 2015 at 11:49 Erik Dalén erik.gustav.da...@gmail.com
mailto:erik.gustav.da...@gmail.com wrote:

I've noticed that relative paths to defines don't work in Puppet
4.x, but I'm unsure if that change is intentional or not. They do
work in 3.x with future parser.

Here is a gist demonstrating what I mean:
https://gist.github.com/dalen/d83826294dd5ce538e83

--
You received this message because you are subscribed to the Google
Groups Puppet Developers group.
To unsubscribe from this group and stop receiving emails from it, send
an email to puppet-dev+unsubscr...@googlegroups.com
mailto:puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-dev/CAAAzDLcP3BcZM3NRggrdYDDV_XKGjQgtVBwfkVyXPqQo8-Yo9w%40mail.gmail.com
https://groups.google.com/d/msgid/puppet-dev/CAAAzDLcP3BcZM3NRggrdYDDV_XKGjQgtVBwfkVyXPqQo8-Yo9w%40mail.gmail.com?utm_medium=emailutm_source=footer.
For more options, visit https://groups.google.com/d/optout.



--

Visit my Blog Puppet on the Edge
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups Puppet 
Developers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/mmgo5f%24bts%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] is a change of fqdn_rand to return a numeric a breaking change?

[Henrik Lindberg]

In ticket https://tickets.puppetlabs.com/browse/PUP-4698 it is suggested 
that fqdn_rand should change its return value to be a numeric value 
instead of a string since it is common to compare it to a value later 
and string comparison never falls back to a numeric comparison with 
future parser/puppet 4.0.0.


In a strict sense, fixing this bug is a breaking change and would have 
to wait until puppet 5.0.0. OTOH, the consequence is that some may be 
silently bitten because there is no error, only that logic takes a 
different part because of a string compare instead of a numeric compare.


What says the people here? Is this a breaking change that has to wait 
until puppet 5.0.0, or can this be done for puppet 4.x ?


- henrik
--

Visit my Blog Puppet on the Edge
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups Puppet 
Developers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/mkqkf8%24rvb%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: Using functions inside functions in Puppet 4

[Henrik Lindberg]

On 2015-18-05 20:54, Alessandro Franceschi wrote:

Thank you for the useful info, going to dive into this, looks quite
powerful



I suppose you found the specification of the API: 
https://github.com/puppetlabs/puppet-specifications/blob/master/language/func-api.md


Regards
- henrik


On Thursday, May 14, 2015 at 1:51:49 PM UTC+2, henrik lindberg wrote:

On 2015-14-05 12:58, Alessandro Franceschi wrote:
  Hi all,
  sorry to bother, I should probably search around better but I'm
quite
  confident that I can find here a quick solution.
 
  I've this issue: https://github..com/example42/puppi/issues/123
https://github.com/example42/puppi/issues/123 in a
  function which tries to use the hiera function as follows:
 
  value = function_hiera([#{module_name}_#{var_name}, ''])
 
 
  The error message comes from the 3.x API  hiera class in
  lib/parser/functions/
 
  |Error: Evaluation Error: Error while evaluating a Function Call,
hiera() has been converted to 4x API|
 
 
  Wha's the syntax to use an API4 function inside a function?
  Maybe should I move the function in my module from
  lib/puppet/parser/functions to lib/puppet/functions and edit
something?

If you change your function to use the 4.x API then it can call any
function (3.x or 4.x) using the method call_function. This works from
3.7.5 (earlier version had a problem in calling a 3.x function from 4.x
- see https://tickets.puppetlabs.com/browse/PUP-3548
https://tickets.puppetlabs.com/browse/PUP-3548).

For the 4.x function API see the language specification.

You can support both 3.x. and 4.x by just adding a 4.x function with
the
same name. 4.x. functions have higher precedence when using future
parser (and 4.x. functions are not visible at all when not using future
parser).

If you really do not want to migrate the function to the 4.x. API then
you could write a utility class to perform the call - the logic to do
this is is here:

https://github.com/puppetlabs/puppet/blob/master/lib/puppet/pops/evaluator/runtime3_support.rb#L234

https://github.com/puppetlabs/puppet/blob/master/lib/puppet/pops/evaluator/runtime3_support.rb#L234


Regards
- henrik

  Thanks * for any help
  al
 
  --
  You received this message because you are subscribed to the Google
  Groups Puppet Developers group.
  To unsubscribe from this group and stop receiving emails from it,
send
  an email to puppet-dev+...@googlegroups.com javascript:
  mailto:puppet-dev+unsubscr...@googlegroups.com javascript:.
  To view this discussion on the web visit
 

https://groups.google.com/d/msgid/puppet-dev/d6e542ff-854f-4b46-a568-96f418586af5%40googlegroups.com

https://groups.google.com/d/msgid/puppet-dev/d6e542ff-854f-4b46-a568-96f418586af5%40googlegroups.com

 

https://groups.google.com/d/msgid/puppet-dev/d6e542ff-854f-4b46-a568-96f418586af5%40googlegroups.com?utm_medium=emailutm_source=footer

https://groups.google.com/d/msgid/puppet-dev/d6e542ff-854f-4b46-a568-96f418586af5%40googlegroups.com?utm_medium=emailutm_source=footer.

  For more options, visit https://groups.google.com/d/optout
https://groups.google.com/d/optout.


--

Visit my Blog Puppet on the Edge
http://puppet-on-the-edge.blogspot.se/
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google
Groups Puppet Developers group.
To unsubscribe from this group and stop receiving emails from it, send
an email to puppet-dev+unsubscr...@googlegroups.com
mailto:puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-dev/e5c6cfed-7a1e-40cd-9fc0-d711ffe559cd%40googlegroups.com
https://groups.google.com/d/msgid/puppet-dev/e5c6cfed-7a1e-40cd-9fc0-d711ffe559cd%40googlegroups.com?utm_medium=emailutm_source=footer.
For more options, visit https://groups.google.com/d/optout.



--

Visit my Blog Puppet on the Edge
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups Puppet 
Developers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/mje2u4%24ksu%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: puppet4 regressions or features?

[Henrik Lindberg]

On 2015-15-05 24:59, Joshua hoblitt wrote:

As I'm slowly updating my modules' testing boilerplate for puppet 4.x,
I've run into a couple cases of breakage that I'm unsure if they are
intentional changes in semantics or regressions.

When introspecting on a resource and the param is undefined, an empty
string is now being returned instead of my good friend `undef`.

I understand that '' is the same type of the param in this case but it
seems like a step backwards to have to go back to [the early days of]
comparing everything to '' instead of undef.

```
   $user_home = getparam(User[$user], 'home')

   $home = $user_home ? {
 ''  = /home/${user}, # puppet 4.0
 undef   = /home/${user}, # puppet 3.7
 default = $user_home,
   }
```

It appears that classes are no longer a first class resource.


There is a chain of things happening here. First User[$user] is 
evaluated, it is an instance of a resource type - and it gets translated 
to a Resource internally (with type and title). This is then given to 
the getparam function. If it does not find a value it returns an empty 
string 
(https://github.com/puppetlabs/puppetlabs-stdlib/blob/master/lib/puppet/parser/functions/getparam.rb#L34)


An empty string is truthy in 4.x and not the same as undef. The function 
should be updated (it should not have returned an empty string in the 
3.x version, and it should return nil in 4.x).


There is cleanup of stdlib under way.



```
   Class['port389::admin::ssl']{ notify =
Class['port389::admin::service'] }
```

```
Evaluation Error: Resource Override can only operate on
resources, got: Class[port389::admin::ssl]-Type at
/home/jhoblitt/github/puppet-port389/spec/fixtures/modules/port389/manifests/admin/ssl.pp:8:3
on node leo.tuc.noao.edu
```

That change is intentional. While classes behave as resources for some 
operations it is not true for all. It happened to work for forming 
relationships, but not for other things. Using the lazy evaluating 
relationship expressions is much better (i.e. - or ~ as Eric Dalén 
pointed out).


- henrik


--

Visit my Blog Puppet on the Edge
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups Puppet 
Developers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/mj4k0g%24gsf%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: Using functions inside functions in Puppet 4

[Henrik Lindberg]

On 2015-14-05 12:58, Alessandro Franceschi wrote:

Hi all,
sorry to bother, I should probably search around better but I'm quite
confident that I can find here a quick solution.

I've this issue: https://github.com/example42/puppi/issues/123 in a
function which tries to use the hiera function as follows:

value = function_hiera([#{module_name}_#{var_name}, ''])


The error message comes from the 3.x API  hiera class in
lib/parser/functions/

|Error: Evaluation Error: Error while evaluating a Function Call, hiera() has 
been converted to 4x API|


Wha's the syntax to use an API4 function inside a function?
Maybe should I move the function in my module from
lib/puppet/parser/functions to lib/puppet/functions and edit something?


If you change your function to use the 4.x API then it can call any 
function (3.x or 4.x) using the method call_function. This works from 
3.7.5 (earlier version had a problem in calling a 3.x function from 4.x 
- see https://tickets.puppetlabs.com/browse/PUP-3548).


For the 4.x function API see the language specification.

You can support both 3.x. and 4.x by just adding a 4.x function with the 
same name. 4.x. functions have higher precedence when using future 
parser (and 4.x. functions are not visible at all when not using future 
parser).


If you really do not want to migrate the function to the 4.x. API then 
you could write a utility class to perform the call - the logic to do 
this is is here: 
https://github.com/puppetlabs/puppet/blob/master/lib/puppet/pops/evaluator/runtime3_support.rb#L234


Regards
- henrik


Thanks * for any help
al

--
You received this message because you are subscribed to the Google
Groups Puppet Developers group.
To unsubscribe from this group and stop receiving emails from it, send
an email to puppet-dev+unsubscr...@googlegroups.com
mailto:puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-dev/d6e542ff-854f-4b46-a568-96f418586af5%40googlegroups.com
https://groups.google.com/d/msgid/puppet-dev/d6e542ff-854f-4b46-a568-96f418586af5%40googlegroups.com?utm_medium=emailutm_source=footer.
For more options, visit https://groups.google.com/d/optout.



--

Visit my Blog Puppet on the Edge
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups Puppet 
Developers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/mj227v%24oh3%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: Unit testing v4 functions in modules

[Henrik Lindberg]

On 2015-30-04 18:10, Joshua Hoblitt wrote:

The core v4 function unit tests appear to be using Puppet::Pops::Loaders
to find the function method.  E.g.

 Puppet.lookup(:loaders).puppet_system_loader.load(:function,
'regsubst').call({}, *args)

From:

https://github.com/puppetlabs/puppet/blob/master/spec/unit/functions/regsubst_spec.rb



There is some unused and experimental gem-loader code (GemBased 
ModuleLoader). The loading of things from the $RUBYLIB path in general 
opened a lot of questions about the visibility - are those system 
functions (like the ones shipped in puppet with the same privileges, 
what are the rules for name-spacing them, if they are in a namespace, 
are they in a module, what is the name of that module - the gem they are 
in, what if it is not a gem, but just some random files pointed to? How 
is the pointing done (or should it search all of the paths on the 
$RUBYLIB path - which is very expensive) etc. etc. Also, gems are not 
available everywhere.


Also note that the 4.x function API does not use ruby require file, it 
explicitly reads and evaluates the code to enable hot-reloading.


So, for the time being 4.x. functions can only be shipped in puppet modules.

I also want to learn more about use cases where it is wanted to ship 
functions (or indeed puppet modules as gems).


Regards

- henrik


-Josh

--
On 04/30/2015 05:17 AM, Erik Dalén wrote:

I'm not even getting the custom functions to load if I set $RUBYLIB,
set libdir or copy them manually to the default libdir (emulating
pluginsync). If I build and install this module it works as expected,
but at the moment I can't find any good way to test v4 functions
without installing them first.

Has anyone verified that pluginsync with v4 functions works at all?

On Wed, 29 Apr 2015 at 12:49 Erik Dalén erik.gustav.da...@gmail.com
mailto:erik.gustav.da...@gmail.com wrote:

 I'm not getting it to work. Here's a minimal example project with
 a simple max function: https://github.com/dalen/puppet-v4functiontest

 And here is the failing travis build for it:
 https://travis-ci.org/dalen/puppet-v4functiontest/builds/60517933

 This is requiring the latest rspec-puppet (2.1.0) and puppet
 4.0.0. What am I doing wrong?

 On Tue, 28 Apr 2015 at 17:00 Henrik Lindberg
 henrik.lindb...@cloudsmith.com
 mailto:henrik.lindb...@cloudsmith.com wrote:

 On 2015-28-04 16:53, Henrik Lindberg wrote:
  On 2015-28-04 15:16, Erik Dalén wrote:
  Do any of you have any tips in regards to unit testing
 function using
  the v4 API in modules?
 
  I would like to do something like this:
 
 
https://github.com/puppetlabs/puppet/blob/master/spec/unit/functions/scanf_spec.rb
 
 
  But PuppetSpec::Compiler  Matchers::Resource are in the
 spec/lib
  directory of puppet which I can't depend on from a module
 AFAIK.
 
  Is there something equivalent in puppetlabs_spec_helper
 that works for
  v4 API functions?
 
  There is an updated version of puppet-rspec that will help
 with this. (I
  heard about this yesterday). Hunner knows more.
 
 
https://github.com/rodjek/rspec-puppet/commit/a66f9654671f8cd935599209b56eee2768c7c8c1#diff-2374d7eefa56aafa2b25c2914973fecbR14


 - henrik
 --

 Visit my Blog Puppet on the Edge
 http://puppet-on-the-edge.blogspot.se/

 --
 You received this message because you are subscribed to the
 Google Groups Puppet Developers group.
 To unsubscribe from this group and stop receiving emails from
 it, send an email to puppet-dev+unsubscr...@googlegroups.com
 mailto:puppet-dev%2bunsubscr...@googlegroups.com.
 To view this discussion on the web visit
 
https://groups.google.com/d/msgid/puppet-dev/mho73o%24dv8%242%40ger.gmane.org.
 For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups Puppet Developers group.
To unsubscribe from this group and stop receiving emails from it, send
an email to puppet-dev+unsubscr...@googlegroups.com
mailto:puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-dev/CAAAzDLcnh-ha1s72tJwLBUkH%2BZDmnqAAN5uowz9n03R1afeU-A%40mail.gmail.com
https://groups.google.com/d/msgid/puppet-dev/CAAAzDLcnh-ha1s72tJwLBUkH%2BZDmnqAAN5uowz9n03R1afeU-A%40mail.gmail.com?utm_medium=emailutm_source=footer.
For more options, visit https://groups.google.com/d/optout.






--

Visit my Blog Puppet on the Edge
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups Puppet 
Developers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev

[Puppet-dev] Re: Unit testing v4 functions in modules

[Henrik Lindberg]

On 2015-28-04 15:16, Erik Dalén wrote:

Do any of you have any tips in regards to unit testing function using
the v4 API in modules?

I would like to do something like this:
https://github.com/puppetlabs/puppet/blob/master/spec/unit/functions/scanf_spec.rb

But PuppetSpec::Compiler  Matchers::Resource are in the spec/lib
directory of puppet which I can't depend on from a module AFAIK.

Is there something equivalent in puppetlabs_spec_helper that works for
v4 API functions?

There is an updated version of puppet-rspec that will help with this. (I 
heard about this yesterday). Hunner knows more.


- henrik

--

Visit my Blog Puppet on the Edge
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups Puppet 
Developers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/mho6tv%24dv8%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: Unit testing v4 functions in modules

[Henrik Lindberg]

On 2015-28-04 16:53, Henrik Lindberg wrote:

On 2015-28-04 15:16, Erik Dalén wrote:

Do any of you have any tips in regards to unit testing function using
the v4 API in modules?

I would like to do something like this:
https://github.com/puppetlabs/puppet/blob/master/spec/unit/functions/scanf_spec.rb


But PuppetSpec::Compiler  Matchers::Resource are in the spec/lib
directory of puppet which I can't depend on from a module AFAIK.

Is there something equivalent in puppetlabs_spec_helper that works for
v4 API functions?


There is an updated version of puppet-rspec that will help with this. (I
heard about this yesterday). Hunner knows more.


https://github.com/rodjek/rspec-puppet/commit/a66f9654671f8cd935599209b56eee2768c7c8c1#diff-2374d7eefa56aafa2b25c2914973fecbR14


- henrik
--

Visit my Blog Puppet on the Edge
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups Puppet 
Developers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/mho73o%24dv8%242%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: detecting manifest parser version

[Henrik Lindberg]

On 2015-21-04 20:59, Corey Osman wrote:

Is there a better way to detect which parser a manifest is using than
what I have come up with below?


https://gist.github.com/logicminds/d3f795f4fa28ed8d0be6



Your code basically runs puppet parser validate with or without --parser 
future afaict


Result of validation:

current | future | meaning
--- | -- | ---
Y   | Y  | ambiguous, evaluation may fail in either
Y   | N  | most likely not intended for use with future parser
N   | Y  | most likely intended for future parser
N   | N  | manifest is universally broken

Note that puppet validate only finds static parsing problems (lexical, 
syntactical, and a few semantic errors), but not evaluation problems.
If your intent is simply to rule out those manifests that could not 
possibly be used with the current parser, then your code is good. For 
other use cases, there is probably too much uncertainty to be of real 
practical value in a general way (maybe it is still useful to your code 
base because you can make certain assumptions).


Also note that Puppet Enterprise 3.8.0 contains a Catalog Preview tool 
that can help more precisely validation and point to evaluation 
problems. This tool is only available for Puppet Enterprise users.


regards
- henrik




Corey


--
You received this message because you are subscribed to the Google
Groups Puppet Developers group.
To unsubscribe from this group and stop receiving emails from it, send
an email to puppet-dev+unsubscr...@googlegroups.com
mailto:puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-dev/B04ECC75-1BB7-451C-9409-B7197C6D9FD2%40logicminds.biz
https://groups.google.com/d/msgid/puppet-dev/B04ECC75-1BB7-451C-9409-B7197C6D9FD2%40logicminds.biz?utm_medium=emailutm_source=footer.
For more options, visit https://groups.google.com/d/optout.



--

Visit my Blog Puppet on the Edge
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups Puppet 
Developers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/mhasvt%24v8o%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: detecting manifest parser version

[Henrik Lindberg]

On 2015-23-04 21:36, Corey Osman wrote:

Yea, since I don't actually run evaluation at this point I only need a
manifest to parse.

I am using this in my project:
http://github.com/logicminds/puppet-retrospec if you are curious.   I am
thinking I may need to hot swap parser versions during manifest
inspection in order to produce the same output for future parser code..

Thanks for taking the time to respond.  I will have a look at the
catalog preview tool which I assume is useful for comparing 4.x catalogs
with 3.x catalogs.

Yes, as well as 4x catalogs with eachother. This will be part of PE 3.8 
release, and it is a PE only feature.


- henrik


--
You received this message because you are subscribed to the Google Groups Puppet 
Developers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/mhbtr7%24c3a%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: Puppet environments implementation question - get_conf() location and result caching

[Henrik Lindberg]

On 2015-16-04 18:34, Bostjan Skufca wrote:



On Thursday, 16 April 2015 14:01:29 UTC+2, henrik lindberg wrote:

On 2015-16-04 4:30, Bostjan Skufca wrote:
  Just recently I was looking at the environments part of puppet
  implementation, and I have stumbled upon this curious gimmick.
 
  There is Puppet::Environments class, which is a class for general
  Environments management (searching, loaders, caching, etc).
  Then there is Puppet::Node::Environment which is instantiated for
every
  environent found. Logical up to this point.
 
  But, contrary to all OOP notions, get_conf() function is not a
method of
  Puppet::Node::Environment class.
  Rather, it is a lookup function in Puppet::Environments class which
  takes environment name as an argument and fetches
  Puppet::Settings::EnvironmentConf instance.
 
  Puppet::Environments has two function for environment and
environment
  data retrieval:
  - get() fetches Puppet::Node::Environment instance
  - get_conf() fetches Puppet::Settings::EnvironmentConf instance
 
  The curious thing is:
  - get() method operates with cache
  - get_conf() does not include any caching
 
  I tested this on a catalog of 1633 resources, and:
  - get() was called around 220 times
  - get_conf() was called around 25.000 times
 
  These are the comments above get_conf() in environments.rb:
   # This implementation evicts the cache, and always gets the
current
   # configuration of the environment
   #
   # TODO: While this is wasteful since it
   # needs to go on a search for the conf, it is too disruptive to
  optimize
   # this.
 
  My questions are:
  - why does get_conf() method belong to Environments instead of
  Node::Environment class?
  - why are get_conf() results not cached? Why is it too disruptive?
 

It should be fine to cache the env conf in an instance of an
environment, and it should be possible to obtain the config given
an instantiated environment. This is safe since the conf will then be
evicted when the environment is evicted (and while the environment is
cached it will not reload or change).


It would be more logical that way too, for the user. If there is
environment timeout, user would expect that all parts of environment
(env configuration, code, even manifests) are cached. At least that was
my understanding until I actually looked at the code.

Shall I create a Jira ticket and pull request with working code change?

Yes please. A ticket for fix-version PUP 4.x (we then update it with the 
actual release it will go into). I don't think it is worth the trouble 
of fixing this on 3.x. Target the code to the master branch.



I believe that the large number of calls
to get the conf could almost exclusively be for an already loaded
environment (I should be measured though).


Yes, this is true. All the calls are for the environment where the
node/agent resides. I tested this with single node only, though.


Good to know. Then this is worth doing.


All other calls would be for information purposes and those cannot be
cached unless the environment.conf file is watched (which is difficult
since it may not exist, come into existence, be removed etc.). For
those
calls, the cost of checking if the file is up to date is almost as
expensive as loading the conf.


All other calls - do you mean calls for other environments?
If so, as said above, I did not notice this on my setup (could not). But
I believe that if env caching follows the same pattern for all
environments, it should not be a problem.

I do mean calls for other environments than the one(s) currently loaded 
and in use/cached by the master. This happens for instance when 
commands/services query for available environments, asks which classes 
that are in an environment etc. These calls when made for environments 
that are loaded, should read the conf directly (they are almost never 
interested in loading the environment they want information about).




Since the 3.x implementation is very complex due to the dynamic
environments, I suggest that this should be fixed for 4.x.


This is probably for the best, true. If one wants to refrain from using
dynamic environments and wishes to use 3.7.x, a separate patch may be
provided.


Cheers, and thanks for taking a stab at improving this!

- henrik

--

Visit my Blog Puppet on the Edge
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups Puppet 
Developers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/mgr0bs%24b1c%241%40ger.gmane.org.
For more options, visit

[Puppet-dev] Re: Puppet environments implementation question - get_conf() location and result caching

[Henrik Lindberg]

On 2015-16-04 4:30, Bostjan Skufca wrote:

Just recently I was looking at the environments part of puppet
implementation, and I have stumbled upon this curious gimmick.

There is Puppet::Environments class, which is a class for general
Environments management (searching, loaders, caching, etc).
Then there is Puppet::Node::Environment which is instantiated for every
environent found. Logical up to this point.

But, contrary to all OOP notions, get_conf() function is not a method of
Puppet::Node::Environment class.
Rather, it is a lookup function in Puppet::Environments class which
takes environment name as an argument and fetches
Puppet::Settings::EnvironmentConf instance.

Puppet::Environments has two function for environment and environment
data retrieval:
- get() fetches Puppet::Node::Environment instance
- get_conf() fetches Puppet::Settings::EnvironmentConf instance

The curious thing is:
- get() method operates with cache
- get_conf() does not include any caching

I tested this on a catalog of 1633 resources, and:
- get() was called around 220 times
- get_conf() was called around 25.000 times

These are the comments above get_conf() in environments.rb:
 # This implementation evicts the cache, and always gets the current
 # configuration of the environment
 #
 # TODO: While this is wasteful since it
 # needs to go on a search for the conf, it is too disruptive to
optimize
 # this.

My questions are:
- why does get_conf() method belong to Environments instead of
Node::Environment class?
- why are get_conf() results not cached? Why is it too disruptive?



The path to the new environments implementation (supporting both 
directory environments, and the now deprecated (and in 4.0 removed) 
dynamic environments) was paved with problems as it affected more or

less the entire code base.

It should be fine to cache the env conf in an instance of an 
environment, and it should be possible to obtain the config given
an instantiated environment. This is safe since the conf will then be 
evicted when the environment is evicted (and while the environment is 
cached it will not reload or change).


I believe that the large number of calls
to get the conf could almost exclusively be for an already loaded 
environment (I should be measured though).
All other calls would be for information purposes and those cannot be 
cached unless the environment.conf file is watched (which is difficult 
since it may not exist, come into existence, be removed etc.). For those 
calls, the cost of checking if the file is up to date is almost as 
expensive as loading the conf.


Since the 3.x implementation is very complex due to the dynamic 
environments, I suggest that this should be fixed for 4.x.


- henrik


Thanks for answers,
b.

--
You received this message because you are subscribed to the Google
Groups Puppet Developers group.
To unsubscribe from this group and stop receiving emails from it, send
an email to puppet-dev+unsubscr...@googlegroups.com
mailto:puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-dev/53b560d5-1c8f-4b00-944c-3156343a175f%40googlegroups.com
https://groups.google.com/d/msgid/puppet-dev/53b560d5-1c8f-4b00-944c-3156343a175f%40googlegroups.com?utm_medium=emailutm_source=footer.
For more options, visit https://groups.google.com/d/optout.



--

Visit my Blog Puppet on the Edge
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups Puppet 
Developers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/mgo8ac%24hc5%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: yaml Kwalify schema for hiera files

[Henrik Lindberg]

On 2015-25-03 20:06, Corey Osman wrote:

Hi,

Has anybody created a yaml Kwalify schema for hiera files.  I would like to 
validate my yaml files against an official hiera schema using Kwalify.  Since 
the YAML spec varies greatly a schema is required to validate hiera specific 
keys.



Exactly which hiera files do you mean?

The format is very much up to the implementation of a particular backend.

I also wonder if what you are trying to achieve is possible to check 
with Kwalify - I expect it to load the yaml using a yaml implementation, 
and thus any lexical problems in your yaml files have already occured 
due to differences in yaml implementations.


The result could still be valid - only not what you intended.

This lexical problem is different than validating that the structural 
content follows a particular yaml schema.


- henrik



Corey




--

Visit my Blog Puppet on the Edge
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups Puppet 
Developers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/mevahl%24do3%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: Generate README.md from inline doc

[Henrik Lindberg]

On 2015-17-03 17:34, Raphaël Pinson wrote:

Thanks for the answer Henrik. That is kind of what it looked to me.

I think if I have to go this way, i'll probably make a gem for this,
since that would be easier to plug into modulesync than a Puppet module
actually.



Note that puppet-strings will be available as a gem as well.

--

Visit my Blog Puppet on the Edge
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups Puppet 
Developers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/me9paj%24cu3%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: Generate README.md from inline doc

[Henrik Lindberg]

On 2015-17-03 12:04, Raphaël Pinson wrote:

Hello!


I'd like to generate the README.md of my Puppet modules from modulesync,
by getting the docs from the classes themselves.

I've considered using puppet-strings, but I don't really see how I could
retrieve markdown from it.

Is there a way I could get markdown from the inline docs in
classes/defines using Ruby, so I can include them in my README?



Afaik, there is no feature available that extracts the documentation
from source and produces markdown.

The puppet-strings does this internally to extract the documentation
comments from the source. It then feeds this into yard to produce
interactive html pages.

If you want to implement your own tool, you could base it of the 
extraction that is done by puppet-strings. It is actually not that 
difficult to write a ruby snippet that parses a .pp file (with the 
future parser) and extracts documentation.


I can imagine accepting a PR with a command that extracts text from source.

Note that puppet-strings requires that the source can be parsed
with the future parser.

Regards
- henrik



Thanks,

Raphaël

--
You received this message because you are subscribed to the Google
Groups Puppet Developers group.
To unsubscribe from this group and stop receiving emails from it, send
an email to puppet-dev+unsubscr...@googlegroups.com
mailto:puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-dev/26e8c57e-f6c5-49ea-ad8c-7ece042b8ffd%40googlegroups.com
https://groups.google.com/d/msgid/puppet-dev/26e8c57e-f6c5-49ea-ad8c-7ece042b8ffd%40googlegroups.com?utm_medium=emailutm_source=footer.
For more options, visit https://groups.google.com/d/optout.



--

Visit my Blog Puppet on the Edge
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups Puppet 
Developers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/me9jjv%24ruc%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: Rspec tests on puppet master branch failing

[Henrik Lindberg]

On 2015-13-03 2:43, Trevor Vaughan wrote:

This file appears to be the issue according to the binary search (neat
tool BTW)

The culprit is
/home/thanos/Work/puppet/spec/integration/agent/logging_spec.rb.

However, when I run that file by hand, it works just fine.

The issues do specifically seem to be related to the Package tests from
what I can tell.

For completeness:

Fedora 20
Ruby 2.0.0-p353



The next step would be to do an rspec run of the alleged culprit 
followed by the test that was originally failing. That then becomes 
proof of guilt.


- henrik


Thanks,

Trevor

On Thu, Mar 12, 2015 at 3:14 PM, Kylo Ginsberg k...@puppetlabs.com
mailto:k...@puppetlabs.com wrote:

On Thu, Mar 12, 2015 at 9:01 AM, Trevor Vaughan
tvaug...@onyxpoint.com mailto:tvaug...@onyxpoint.com wrote:

No, if I run parallel::spec, I actually get different errors, so
I stopped trying to use it.

I just ran with a clean ruby 2.0.0-p353 with the same results.


As Henrik mentioned, it could be a spec order problem.

If you run:

bundle exec rake ci:spec

then you should get a shiny spec_order.txt.

Then you can run: util/binary_search_specs.rb full path to failing
spec

And it will (usually) tell you the test that makes the failing spec
fail. (The 'usually' caveat is because there can be spec failures
that require specific ordering between  2 spec files, and this tool
only handles the case for 2 spec files. The  2 case is super duper
fun.)

Clearly, this needs to be in our docs, PR coming.

Kylo


Not quite sure what's going on.

I'll poke around some more later since you can't seem to reproduce.

Trevor

On Thu, Mar 12, 2015 at 11:57 AM, Henrik Lindberg
henrik.lindb...@cloudsmith.com
mailto:henrik.lindb...@cloudsmith.com wrote:

On 2015-12-03 16:40, Trevor Vaughan wrote:

Hi Henrik,

I purged all of my gems removed my Gemfile.lock file and
then ran bundle
install followed by bundle update with the same result.


Does it work if you run the failing tests in isolation? (If
so it could be an ordering problem).

I just ran this on master:

* bundle update
* bundle exec rake parallel:spec[4]

= 18768 examples, 0 failures, 44 pending

- henrik

Thanks,

Trevor

On Thu, Mar 12, 2015 at 11:11 AM, Henrik Lindberg
henrik.lindberg@cloudsmith.__com
mailto:henrik.lindb...@cloudsmith.com
mailto:henrik.lindberg@__cloudsmith.com
mailto:henrik.lindb...@cloudsmith.com
wrote:

 On 2015-12-03 14:10, Trevor Vaughan wrote:

 Hi all,

 Some of the package spec tests are failing for
me under Ruby 1.9.3.

 The offending commit appears to be:

 commit 8eb9557420bd10cfb268c55a62698136338c9b79
 Merge: 0abb9a9 11bba86
 Author: Henrik Lindberg
henrik.lindberg@cloudsmith.com
 mailto:henrik.lindberg@__cloudsmith.com
mailto:henrik.lindb...@cloudsmith.com
 mailto:henrik.lindberg@
mailto:henrik.lindberg@__clou__dsmith.com
http://cloudsmith.com
 mailto:henrik.lindberg@__cloudsmith.com
mailto:henrik.lindb...@cloudsmith.com
 Date:   Thu Mar 5 22:50:14 2015 +0100

   Merge branch 'stable'
   Conflicts:
   lib/puppet/parser/scope.rb
   lib/puppet/type/package.rb
   spec/unit/type/package_spec.rb
   Changed be_true matcher to be_truthy
because of rspec
 version change

 Ruby Version: ruby-1.9.3-p551
 Gemfile: Stock in Puppet core
 Branch: master (stable is fine)


 You probably do not have rspec 3 which is now
required on master.
 Try a 'bundle update'

 - henrik

 Thanks,

 Trevor

 --
 Trevor Vaughan
 Vice President, Onyx Point, Inc
(410) 541-6699 tel:%28410%29%20541-6699
tel:%28410%29%20541-6699
tvaug...@onyxpoint.com mailto:tvaug...@onyxpoint.com
mailto:tvaug...@onyxpoint.com
mailto:tvaug...@onyxpoint.com__

[Puppet-dev] Re: Rspec tests on puppet master branch failing

[Henrik Lindberg]

On 2015-12-03 16:40, Trevor Vaughan wrote:

Hi Henrik,

I purged all of my gems removed my Gemfile.lock file and then ran bundle
install followed by bundle update with the same result.



Does it work if you run the failing tests in isolation? (If so it could 
be an ordering problem).


I just ran this on master:

* bundle update
* bundle exec rake parallel:spec[4]

= 18768 examples, 0 failures, 44 pending

- henrik


Thanks,

Trevor

On Thu, Mar 12, 2015 at 11:11 AM, Henrik Lindberg
henrik.lindb...@cloudsmith.com mailto:henrik.lindb...@cloudsmith.com
wrote:

On 2015-12-03 14:10, Trevor Vaughan wrote:

Hi all,

Some of the package spec tests are failing for me under Ruby 1.9.3.

The offending commit appears to be:

commit 8eb9557420bd10cfb268c55a626981__36338c9b79
Merge: 0abb9a9 11bba86
Author: Henrik Lindberg henrik.lindberg@cloudsmith.__com
mailto:henrik.lindb...@cloudsmith.com
mailto:henrik.lindberg@__cloudsmith.com
mailto:henrik.lindb...@cloudsmith.com
Date:   Thu Mar 5 22:50:14 2015 +0100

  Merge branch 'stable'
  Conflicts:
  lib/puppet/parser/scope.rb
  lib/puppet/type/package.rb
  spec/unit/type/package_spec.rb
  Changed be_true matcher to be_truthy because of rspec
version change

Ruby Version: ruby-1.9.3-p551
Gemfile: Stock in Puppet core
Branch: master (stable is fine)


You probably do not have rspec 3 which is now required on master.
Try a 'bundle update'

- henrik

Thanks,

Trevor

--
Trevor Vaughan
Vice President, Onyx Point, Inc
(410) 541-6699 tel:%28410%29%20541-6699
tvaug...@onyxpoint.com mailto:tvaug...@onyxpoint.com
mailto:tvaug...@onyxpoint.com mailto:tvaug...@onyxpoint.com__

-- This account not approved for unencrypted proprietary
information --

--
You received this message because you are subscribed to the Google
Groups Puppet Developers group.
To unsubscribe from this group and stop receiving emails from
it, send
an email to puppet-dev+unsubscribe@__googlegroups.com
mailto:puppet-dev%2bunsubscr...@googlegroups.com
mailto:puppet-dev+__unsubscr...@googlegroups.com
mailto:puppet-dev%2bunsubscr...@googlegroups.com.
To view this discussion on the web visit

https://groups.google.com/d/__msgid/puppet-dev/CANs%__2BFoXR9dtAJ5x4qk4vwrK14CRGxyT___mLucay%3D9oXe-kQU9PQ%40mail.__gmail.com

https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoXR9dtAJ5x4qk4vwrK14CRGxyT_mLucay%3D9oXe-kQU9PQ%40mail.gmail.com

https://groups.google.com/d/__msgid/puppet-dev/CANs%__2BFoXR9dtAJ5x4qk4vwrK14CRGxyT___mLucay%3D9oXe-kQU9PQ%40mail.__gmail.com?utm_medium=email__utm_source=footer

https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoXR9dtAJ5x4qk4vwrK14CRGxyT_mLucay%3D9oXe-kQU9PQ%40mail.gmail.com?utm_medium=emailutm_source=footer.
For more options, visit https://groups.google.com/d/__optout
https://groups.google.com/d/optout.



--

Visit my Blog Puppet on the Edge
http://puppet-on-the-edge.__blogspot.se/
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google
Groups Puppet Developers group.
To unsubscribe from this group and stop receiving emails from it,
send an email to puppet-dev+unsubscribe@__googlegroups.com
mailto:puppet-dev%2bunsubscr...@googlegroups.com.
To view this discussion on the web visit

https://groups.google.com/d/__msgid/puppet-dev/mdsab4%24uit%__241%40ger.gmane.org

https://groups.google.com/d/msgid/puppet-dev/mdsab4%24uit%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/__optout
https://groups.google.com/d/optout.




--
Trevor Vaughan
Vice President, Onyx Point, Inc
(410) 541-6699
tvaug...@onyxpoint.com mailto:tvaug...@onyxpoint.com

-- This account not approved for unencrypted proprietary information --

--
You received this message because you are subscribed to the Google
Groups Puppet Developers group.
To unsubscribe from this group and stop receiving emails from it, send
an email to puppet-dev+unsubscr...@googlegroups.com
mailto:puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoVKj5nYqz2Ogbn-LZi%3DMbpTOPftKr%2Bz6gCpmD9Ss-rkLA%40mail.gmail.com
https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoVKj5nYqz2Ogbn-LZi%3DMbpTOPftKr%2Bz6gCpmD9Ss-rkLA%40mail.gmail.com?utm_medium=emailutm_source=footer.
For more options, visit https://groups.google.com/d/optout.



--

Visit my Blog Puppet on the Edge
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups Puppet 
Developers group.
To unsubscribe

[Puppet-dev] Re: Environmentpath support for globbing

[Henrik Lindberg]

On 2015-08-03 20:10, Bostjan Skufca wrote:

Hi there,

I like to have my environments organized into multiple subdirectories.
Puppet configuration supports that by defining multiple paths for
environmentpath setting, separated by ':'.

However, I find it very convenient to use this simple configuration line:
environmentpath = /etc/puppet/env/*

instead of this one:
environmentpath =
/etc/puppet/env/SUB1:/etc/puppet/env/SUB2:/etc/puppet/env/SUB3:/etc/puppet/env/SUBx


The patch for this convenience is a simple one-line change:
---
--- puppet/settings/path_setting.rb.ORIG2015-03-08 19:05:33.758705374 +
+++ puppet/settings/path_setting.rb2015-03-07 01:59:31.127330981 +
@@ -1,7 +1,7 @@
  class Puppet::Settings::PathSetting  Puppet::Settings::StringSetting
def munge(value)
  if value.is_a?(String)
-  value = value.split(File::PATH_SEPARATOR).map { |d|
File.expand_path(d) }.join(File::PATH_SEPARATOR)
+  value = value.split(File::PATH_SEPARATOR).map { |d|
File.expand_path(d) }.map { |d| Dir..glob(d) }.join(File::PATH_SEPARATOR)
  end
  value
end
---

I haven't tested this with adding subdirs AFTER master is already
started, thought, but a simple 'touch puppetmaster.conf' should do the
trick if new directories are not picked up by default.

Would there be any interest for merging mentioned change upstream?

This convenience looks like a useful addition. Please log a feature 
request and submit a PR.


- henrik

--

Visit my Blog Puppet on the Edge
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups Puppet 
Developers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/mdk63b%244dh%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: Modulepath and manifest in environment.conf should interpolate $environment too

[Henrik Lindberg]

On 2015-08-03 20:27, Bostjan Skufca wrote:

Hi all,

this change I have not yet implemented, but I believe it might be nice
to have.

Currently in environment.conf we can specify modulepath with relative or
absolute paths to module directories, like this:
modulepath = modules/:/some/other/location/env_name/
manifest = /path/to/env_name/manifest2/

It would be splendid if we could define modulepath paths with
$environment as variable part of path, like this:
modulepath = /path/to/$environment/modules
manifest = /path/to/$environment/manifest2/

Would there be any interest for this feature?
If this is implemented for modulepath setting, maybe it should be
appropriate to implement it for manifest setting too?



This scares me a bit. It looks like it has potential to open the can of 
worms known as dynamic environments we managed to put the lid on with 
directory environments. Will have to discuss if this can used to do 
harmful things. If allowed it could only be allowed inside an absolute path.


Will see what Joshua Partlow has to say as well.

- henrik


b.


PS: Explanation why I find this useful, or from which use case I derived
it would be useful:
I have two puppets running on each system. Primary is for managing whole
system + secondary puppet, and secondary puppet is just for managing
primary puppet. This is useful for puppet upgrades - if something goes
wrong, secondary puppet still works and fixes primary puppet if
required, and vice versa.
For these two I use the same module repository (each puppet has own
module though) and I find it convenient to not have to clone module
repository for the second time, for each environment.

--
You received this message because you are subscribed to the Google
Groups Puppet Developers group.
To unsubscribe from this group and stop receiving emails from it, send
an email to puppet-dev+unsubscr...@googlegroups.com
mailto:puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-dev/d32b2c66-4dd2-4bf7-b0d7-05194b65dde4%40googlegroups.com
https://groups.google.com/d/msgid/puppet-dev/d32b2c66-4dd2-4bf7-b0d7-05194b65dde4%40googlegroups.com?utm_medium=emailutm_source=footer.
For more options, visit https://groups.google.com/d/optout.



--

Visit my Blog Puppet on the Edge
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups Puppet 
Developers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/mdk6f4%24fjt%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: Modulepath and manifest in environment.conf should interpolate $environment too

[Henrik Lindberg]

On 2015-09-03 14:54, Felix Frank wrote:

On 03/09/2015 02:16 PM, Henrik Lindberg wrote:

It would be splendid if we could define modulepath paths with
$environment as variable part of path, like this:
modulepath = /path/to/$environment/modules
manifest = /path/to/$environment/manifest2/

Would there be any interest for this feature?
If this is implemented for modulepath setting, maybe it should be
appropriate to implement it for manifest setting too?



This scares me a bit. It looks like it has potential to open the can of
worms known as dynamic environments we managed to put the lid on with
directory environments. Will have to discuss if this can used to do
harmful things. If allowed it could only be allowed inside an absolute
path.


Agreed, there might be security implications.

I also fail to see the value in that. Do you mean to allow an
environment to extend itself to a whole different file system tree?
Wouldn't that just be horrible for organizing things?



The use case was to use one location for two master instances without 
having to have two copies, and without having to have a modulepath that 
includes the name of the environment (which is possible now).


Not sure that use case is worth the potential problems.

- henrik


--

Visit my Blog Puppet on the Edge
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups Puppet 
Developers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/mdkgbj%242b4%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: Puppet provider - xml parsing - help needed

[Henrik Lindberg]

On 2015-25-02 18:17, Martin Alfke wrote:

Hi,

I am stuck at a puppet provider where I need to read data from xml.

Data which needs to get parsed:

 AR
   AR_ID![CDATA[8]]/AR_ID
   IP![CDATA[172.16.100.208]]/IP
   MAC![CDATA[02:00:ac:10:64:d0]]/MAC
   SIZE![CDATA[1]]/SIZE
   TYPE![CDATA[IP4]]/TYPE
   USED_LEASES0/USED_LEASES
   LEASES/
 /AR
 AR
   AR_ID![CDATA[9]]/AR_ID
   IP![CDATA[172.16.100.209]]/IP
   MAC![CDATA[02:00:ac:10:64:d1]]/MAC
   SIZE![CDATA[1]]/SIZE
   TYPE![CDATA[IP4]]/TYPE
   USED_LEASES0/USED_LEASES
   LEASES/
 /AR


Expected result:

:addressrange =
   { ‘1' =
 { ‘ip' = ‘172.16.100.208’, ‘mac' = ’02:00:ac:10:64:d0’ },
  ‘2' =
 { ‘ip' =  ‘172.16.100.209’, ‘mac' = ’02:00:ac:10:64:d1’ }
   }






What would be proper code to build the hash as expected?



That is not easy. Is the structure static and only varying where the 
actual data is? If so, you could hard code a solution. If you expect 
there to be variability in the tags/layout you should consider using an 
XML parser and get the data from it.


To do this with regular expressions you need to match the part:

   IP![CDATA[172.16.100.209]]/IP
   MAC![CDATA[02:00:ac:10:64:d1]]/MAC

with something like this (untested):

/IP!\[CDATA|[(.*)\]]/IP.*MAC!\[CDATA\[(.*)\]]/MAC/

Then apply that to iterate over matches - you need to do matching that
accepts newlines as a match for . etc.

Play with it in Rubular (online ruby regexp tester) until you get it right.

- henrik
--

Visit my Blog Puppet on the Edge
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups Puppet 
Developers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/mcl543%243jf%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: Clojure authorization system for SERVER-111...

[Henrik Lindberg]

On 2015-23-02 17:47, Chris Price wrote:

On Mon, Feb 23, 2015 at 7:09 AM, Trevor Vaughan tvaug...@onyxpoint.com
mailto:tvaug...@onyxpoint.com wrote:

Sorry to derail for the moment but HOCON + JSON + YAML + XML? Sounds
great..


Totally agree that we have too many formats.  That's why we tried to put
a lot of thought into picking one that we think is robust enough to
standardize on going forward.  :)  Also, the current auth.conf format is
none of the above, so moving it to any of the above would mean 'n - 1'
formats :)



Is there an overlap with Node Classifier and RBAC as they also specify 
rules? We would want to have a common way to handle rules in different 
domains.


- henrik



On Mon, Feb 23, 2015 at 9:57 AM, Chris Price ch...@puppetlabs..com
mailto:ch...@puppetlabs.com wrote:

On Sun, Feb 22, 2015 at 9:18 PM, Eric Sorenson
eric.soren...@puppetlabs.com
mailto:eric.soren...@puppetlabs.com wrote:


Hi Brice! This project is really cool, thanks for taking it
on. I have a few comments about requirements and design that
I hope can save some work and make it easier to include this
upstream once it's done.

I went back and surveyed redmine, jira, and ask.pl.com
http://ask..pl.com for bugs around auth.conf to see what
people have run into over the years (

https://www.google.com/search?q=site%3Apuppetlabs.com+auth.confgws_rd=ssl

https://www.google.com/search?q=site:puppetlabs.com+auth.confgws_rd=ssl ),
and from those results plus recalling conversations with
#puppet there seem to be a few general categories that we
should examine when designing a replacement

First, I don't think you need to try to make it compatible
with the existing auth.conf format. It'd be good to take the
opportunity to move to a structured data format that is
easier to read and write programmatically,


It would be cool if we could figure out a way to represent the
rules in HOCON, since that's the format we're using for pretty
much all of our new config files going forward.  That way, the
same modules and tooling that we're building up around that data
format could be used on the auth stuff, and the syntax would
start to look more consistent and familiar compared to other new
puppet config files.  Since HOCON is basically a superset of
JSON I'm thinking that maybe the rules could be written as
basically a big array of maps.  It'd be a little more verbose
than the existing syntax, but I think the tradeoffs might be
worth it.

(This is presuming, of course, that we don't find some other
existing model that we like, as Eric suggested.)



--
You received this message because you are subscribed to the
Google Groups Puppet Developers group.
To unsubscribe from this group and stop receiving emails from
it, send an email to puppet-dev+unsubscr...@googlegroups.com
mailto:puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit

https://groups.google.com/d/msgid/puppet-dev/CAMx1QfL9TvgyWJ5__utWk12CQ3y_q0Wk63uJr6efMxoEk4gLeA%40mail.gmail.com

https://groups.google.com/d/msgid/puppet-dev/CAMx1QfL9TvgyWJ5__utWk12CQ3y_q0Wk63uJr6efMxoEk4gLeA%40mail.gmail.com?utm_medium=emailutm_source=footer.

For more options, visit https://groups.google.com/d/optout.




--
Trevor Vaughan
Vice President, Onyx Point, Inc
(410) 541-6699 tel:%28410%29%20541-6699
tvaug...@onyxpoint.com mailto:tvaug...@onyxpoint.com

-- This account not approved for unencrypted proprietary information --

--
You received this message because you are subscribed to the Google
Groups Puppet Developers group.
To unsubscribe from this group and stop receiving emails from it,
send an email to puppet-dev+unsubscr...@googlegroups.com
mailto:puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit

https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoVgeG5fYRqa3xkj9%3DKEQBpwB%2BUv%2BbRJsY0LoPTL8BZQ%3DQ%40mail.gmail.com

https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoVgeG5fYRqa3xkj9%3DKEQBpwB%2BUv%2BbRJsY0LoPTL8BZQ%3DQ%40mail.gmail.com?utm_medium=emailutm_source=footer.

For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google
Groups Puppet Developers group.
To unsubscribe from this group and stop receiving emails from it, send
an email to puppet-dev+unsubscr...@googlegroups.com
mailto:puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-dev/CAMx1QfLpVd2swVDpqvX5Xgtq%3DL7txZTkYKUTHLdOX5vOGUh-4g%40mail.gmail.com

[Puppet-dev] Puppet Dev Status -- week ending Feb 21

[Henrik Lindberg]

We gave all of the yaks a mullet, now they need a shave.

Everyone is really busy with the AIO packaging and getting all the right 
bits through the testing gauntlet.


After much going back and forth on one of the details; what should the 
paths be for all the components in the AIO? we decided to write this up 
as a proper specification. Now there is one: 
https://github.com/puppetlabs/puppet-specifications/blob/master/file_paths.md


Making the changes for these paths, the tests pipelines, and doing 
everything in the right order and also fixing the problems that have 
occurred is taking longer than anticipated.


Old friends come back and bite us... as an example there are circular 
dependencies between hiera and puppet that makes testing interesting.
We also again experienced how difficult it is to add a dependency on an 
additional gem - now experienced again with the deep_merge gem (for 
merging arrays and hashes in hiera and in the the data in modules 
support). We simply have a lot of ways the software gets delivered, and 
in some cases it is difficult to get a gem in the right place. Thus we 
ended up vendoring (i.e. including a copy) the deep_merge gem like we do 
with some other gems. Long term we want to stop doing this - but for now 
there were too many places to deal with, and we did not want to add 
additional yaks to the line of yaks already waiting for their shave.


On the bright side, while waiting for the bits to fall into place, we 
have been fixing additional puppet and hiera tickets that we thought we 
would not get to in time for the release of Puppet 4.0/3.7.5 and friends.


Real soon now folks...

- henrik


--

Visit my Blog Puppet on the Edge
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups Puppet 
Developers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/mc8ne9%245e5%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: Thoughts on Module Namespace Conflicts

[Henrik Lindberg]

On 2015-20-02 20:51, Wil Cooley wrote:

In some ways, this is a lot like environments, but unlike environments,
which are unique at the node-level, these allow the use of multiple
module-sets and are scoping for entities at the manifest-level. As I
understand environments, they can (and probably usually should) be
relatively isolated from each other; the purpose of namespaced
module-sets is to allow addressing without isolation. Maybe this is what
Henrik's last message in this thread is getting at when he says:

I have claimed on many occasions that an environment is just like a
module (with higher privileges / precedence). The set of available
modules should be more of a repository kind of thing that the loader
resolves modules from.


I meant that an environment can be seen as a module. Instead of having a 
modulepath, it should list all the modules that it requires (those 
modules in turn can require other modules). When a module is needed it 
should be resolved against a repository of modules. Such a repository 
could be implemented many ways; one being that it is just the modules 
now on the modulepath.


The main difference is that the resolution from name to actual module is 
now free from implementation concerns.


Currently the dependencies that are expressed are between the containers 
of logic (the module depends on another specific module with a specific 
implementations e.g. puppetlabs/stdlib). This is simple but not very 
flexible. It would be far better if the requirements were established 
based on what is actually required from that module - say that it needs 
a function 'foo'. Then that could be declared and any module that 
happens to make that function available is a potential replacement. The 
same is applicable to other elements of the puppet language, classes, 
defines, types, etc.


Such a scheme requires something to select the modules that are actually 
wanted out of all potential modules. This could be the role of the 
environment module - it would have dependencies on the modules that 
should be used. It is not limited to that however since any module can 
act as a configuration (by having explicit dependencies on other 
modules) - such modules does not have to contain anything but the 
dependencies. They become reusable parts of configuration thus making it 
easier to set up many different environments.


While this increases flexibility it also adds complexity. It is also
not the complete solution to all the things discussed in this thread, 
but I believe it could be the foundation for most of them.


- henrik

--

Visit my Blog Puppet on the Edge
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups Puppet 
Developers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/mc8jqc%24hus%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: Thoughts on Module Namespace Conflicts

[Henrik Lindberg]

On 2015-12-02 15:12, Erik Dalén wrote:

A related ticket to the interface and redirect functionality is
https://tickets.puppetlabs.com/browse/FORGE-111

There I just suggest the feature to be able to say that module B
provides the interface of module A version X. So either module can
satisfy a dependency on module A version X.
Then if you want to have some sort of generic interface that other
modules can implement, you could have a sort of virtual module with he
version being the interface version. And then other modules provide or
depend on that virtual module. The meaning of virtual module here is the
same as for virtual packages in package managers, like
ruby-interpreter or smtp-server in Debian, and the implementation
would also be pretty similar.


Regarding having multiple variants or versions of a module being
installed at the same time I think we should change the environment
loader API to provide a list of modules instead of a list of module
paths. Then plugins could implement any sort of behaviour for this
without jumping through hoops.



I have claimed on many occasions that an environment is just like a 
module (with higher privileges / precedence). The set of available 
modules should be more of a repository kind of thing that the loader 
resolves modules from.


Essentially this is how the new loaders work (used only for 4x functions 
so far).


- henrik

--

Visit my Blog Puppet on the Edge
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups Puppet 
Developers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/mbii8r%24mae%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: Thoughts on Module Namespace Conflicts

[Henrik Lindberg]

On 2015-10-02 1:23, Trevor Vaughan wrote:

I was talking with a few folks today about potential resolutions to
module namespace issues.

== Fundamental Issue ==

puppetlabs_apache -- Installs To -- apache
example42_apache -- Installs To -- apache
theforeman_apache -- Installs To -- apache

You get my point...

== Current Solutions ==

Right now, there are two ways to solve this problem if you want some of
your nodes to use puppetlabs_apache and others to use theforeman_apache.

1) Modify the module and force the namespace:
   - puppetlabs_apache -- Installs To -- puppetlabs_apache
   - theforeman_apache -- Installs To -- theforeman_apache

   * Isssue: You can't just use outside code at will. You have to modify
*everything* that uses the above code.

2) You have to create a separate environment for each host group
(potentially each host) that you want to have use the differing code.

* Issue: This is a LOT of overhead for a couple of modules and may
have other ramifications in terms of performance as you get up in number.

So, would it be possible to allow multiple versions of a module to exist
in the same namespace but only use one during a given run?



Basically no, this is not possible without adding some kind of mechanism 
to filter out modules on the modulepath. This will need to be done per 
environment anyway. Remember that modules can contribute all sorts of 
extensions to puppet (faces, indirections, facts, functions, types, and 
with future parser also bindings). For performance reasons (and 
bootstrapping, etc.) these are scanned when the environment loads (some 
scans are lazy, but may take place before manifests really starts to be 
evaluated).


- henrik


== The First Proposal ==

Inspired by RVM Gemsets, how about allowing modules to declare which
version of a given module they will use?

/etc/puppet/modules/apache/puppetlabs/{files,modules,manifests}
/etc/puppet/modules/apache/theforeman/{files,modules,manifests}
/etc/puppet/modules/apache/author/{files,modules,manifests}

Then, use could be dictated by something like: include apache@puppetlabs.

Unfortunately, this really comes down to the parser being able to
understand the following:

include apache = See if there are any @'s and use that one
include apache@puppetlabs = include the Puppetlabs apache
include apache@puppetlabs  include apache@theforeman = Fail, conflict

== Alternative ==

One possible alternative is to just use the metadata.json file to
dictate which module will be used when loading other modules.

Again, if there is a conflict, that is a failure but *only* if the code
attempts to use both at the same time.

The benefit here is that it should make things very unambiguous while
the drawback (if it really is) is that you *must* put a metadata.json
file in every module that you create.

This is just a set of thoughts that I hope get us moving in a direction
where this type of thing is possible and I look forward to hearing what
people think (good, bad, or ugly)!

Thanks,

Trevor

--
Trevor Vaughan
Vice President, Onyx Point, Inc
(410) 541-6699
tvaug...@onyxpoint.com mailto:tvaug...@onyxpoint.com

-- This account not approved for unencrypted proprietary information --

--
You received this message because you are subscribed to the Google
Groups Puppet Developers group.
To unsubscribe from this group and stop receiving emails from it, send
an email to puppet-dev+unsubscr...@googlegroups.com
mailto:puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoU2unExPve9ig%3DinW9obDpwii7gBi6W4RkHycViibJp-g%40mail.gmail.com
https://groups.google.com/d/msgid/puppet-dev/CANs%2BFoU2unExPve9ig%3DinW9obDpwii7gBi6W4RkHycViibJp-g%40mail.gmail.com?utm_medium=emailutm_source=footer.
For more options, visit https://groups.google.com/d/optout.



--

Visit my Blog Puppet on the Edge
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups Puppet 
Developers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/mbdd0b%24epb%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: Thoughts on Module Namespace Conflicts

[Henrik Lindberg]

On 2015-10-02 21:17, Trevor Vaughan wrote:

To Henrik: Yes, this make sense, it would be nice to have some easy way
to keep the purely Puppet DSL portions separate though.



Yeah, but that is the way it works - so way to much to change to make 
that happen.


I guess it would be possible to have a special implementation of a 
module that is a proxy for a real selected module, and that the real 
module is not on the real module path - only the proxy. Still quite 
complex to achieve. Since the concept of a module is not that well 
encapsulated.


Another approach is to differentiate between available modules (more 
like a repo) and modules being used in an environment - having a 
distinct step in between that configures the modulepath for the runtime. 
Still the same issue with where the description of which modules to use 
is kept - probably ends up with the same amount of configuration as when 
just using directory environments.


You could write something yourself that does that for you - i.e. 
something that creates the environments you need/want.


- henrik


Walter: This would be nice, but variables are the...uh...variable here.

Many people only expose the variables that they need while others expose
every variable they can find. It would be an interesting experiment to
see if it would work though.

Trevor

On Tue, Feb 10, 2015 at 1:52 PM, Walter Heck walterh...@olindata.com
mailto:walterh...@olindata.com wrote:

I'm personally more of a fan of what some programming languages call
interfaces; a sort of contract if you will that a module implements.

So for instance olindata::galera could look for any class that
implements the IMysql interface. That way, I don't really need to
worry if the end-user is using puppetlabs::mysql or
example42::mysql, as long as they implement the IMysql interface I
can guarantee it does what I need.

This would have pretty heavy impact on how you write code right now,
but it would be a rather elegant solution imho.

interface IMysql::server identified by
'9ff3d80b-b02d-4994-b4da-e1ff205304ea' {
   Service['mysql']
   Package['mysql-server']
   File['apache2.conf']
}

class puppetlabs::mysql implements
IMysql['9ff3d80b-b02d-4994-b4da-e1ff205304ea'] {
   [.. SNIP..]
   # file, service, package here like normal
}

class example42::mysql implements
IMysql['9ff3d80b-b02d-4994-b4da-e1ff205304ea'] {
   [.. SNIP..]
   # file, service, package here like normal
}

class olindata::galera {
   package { 'galera':
 require = Package['IMysql::server::mysql-server']
   }
}

Or something along those lines. Question remains where the
IMysql::server interface then needs to be declared, and how we would
manage the GUID registration of interfaces.

Comments, thoughts?

Walter


On Tuesday, February 10, 2015 at 5:52:37 PM UTC+1, henrik lindberg
wrote:

On 2015-10-02 1:23, Trevor Vaughan wrote:
  I was talking with a few folks today about potential
resolutions to
  module namespace issues.
 
  == Fundamental Issue ==
 
  puppetlabs_apache -- Installs To -- apache
  example42_apache -- Installs To -- apache
  theforeman_apache -- Installs To -- apache
 
  You get my point...
 
  == Current Solutions ==
 
  Right now, there are two ways to solve this problem if you
want some of
  your nodes to use puppetlabs_apache and others to use
theforeman_apache.
 
  1) Modify the module and force the namespace:
 - puppetlabs_apache -- Installs To -- puppetlabs_apache
 - theforeman_apache -- Installs To -- theforeman_apache
 
 * Isssue: You can't just use outside code at will.. You
have to modify
  *everything* that uses the above code.
 
  2) You have to create a separate environment for each host group
  (potentially each host) that you want to have use the
differing code.
 
  * Issue: This is a LOT of overhead for a couple of
modules and may
  have other ramifications in terms of performance as you get
up in number.
 
  So, would it be possible to allow multiple versions of a
module to exist
  in the same namespace but only use one during a given run?
 

Basically no, this is not possible without adding some kind of
mechanism
to filter out modules on the modulepath. This will need to be
done per
environment anyway. Remember that modules can contribute all
sorts of
extensions to puppet (faces, indirections, facts, functions,
types, and
with future parser also bindings). For performance reasons (and
bootstrapping, etc.) these are scanned

[Puppet-dev] Re: Thoughts on Module Namespace Conflicts

[Henrik Lindberg]

On 2015-10-02 19:52, Walter Heck wrote:

I'm personally more of a fan of what some programming languages call
interfaces; a sort of contract if you will that a module implements.

Yes that is good - does not solve the issue of modules having the same 
name though - either the module name must change, or modules must be 
installed with both author and name and then referenced the same way.


- henrik


So for instance olindata::galera could look for any class that
implements the IMysql interface. That way, I don't really need to worry
if the end-user is using puppetlabs::mysql or example42::mysql, as long
as they implement the IMysql interface I can guarantee it does what I need.

This would have pretty heavy impact on how you write code right now, but
it would be a rather elegant solution imho.

interface IMysql::server identified by
'9ff3d80b-b02d-4994-b4da-e1ff205304ea' {
   Service['mysql']
   Package['mysql-server']
   File['apache2.conf']
}

class puppetlabs::mysql implements
IMysql['9ff3d80b-b02d-4994-b4da-e1ff205304ea'] {
   [.. SNIP..]
   # file, service, package here like normal
}

class example42::mysql implements
IMysql['9ff3d80b-b02d-4994-b4da-e1ff205304ea'] {
   [.. SNIP..]
   # file, service, package here like normal
}

class olindata::galera {
   package { 'galera':
 require = Package['IMysql::server::mysql-server']
   }
}

Or something along those lines. Question remains where the
IMysql::server interface then needs to be declared, and how we would
manage the GUID registration of interfaces.

Comments, thoughts?

Walter

On Tuesday, February 10, 2015 at 5:52:37 PM UTC+1, henrik lindberg wrote:

On 2015-10-02 1:23, Trevor Vaughan wrote:
  I was talking with a few folks today about potential resolutions to
  module namespace issues.
 
  == Fundamental Issue ==
 
  puppetlabs_apache -- Installs To -- apache
  example42_apache -- Installs To -- apache
  theforeman_apache -- Installs To -- apache
 
  You get my point...
 
  == Current Solutions ==
 
  Right now, there are two ways to solve this problem if you want
some of
  your nodes to use puppetlabs_apache and others to use
theforeman_apache.
 
  1) Modify the module and force the namespace:
 - puppetlabs_apache -- Installs To -- puppetlabs_apache
 - theforeman_apache -- Installs To -- theforeman_apache
 
 * Isssue: You can't just use outside code at will. You have to
modify
  *everything* that uses the above code.
 
  2) You have to create a separate environment for each host group
  (potentially each host) that you want to have use the differing
code.
 
  * Issue: This is a LOT of overhead for a couple of modules
and may
  have other ramifications in terms of performance as you get up in
number.
 
  So, would it be possible to allow multiple versions of a module
to exist
  in the same namespace but only use one during a given run?
 

Basically no, this is not possible without adding some kind of
mechanism
to filter out modules on the modulepath. This will need to be done per
environment anyway. Remember that modules can contribute all sorts of
extensions to puppet (faces, indirections, facts, functions, types, and
with future parser also bindings). For performance reasons (and
bootstrapping, etc.) these are scanned when the environment loads (some
scans are lazy, but may take place before manifests really starts to be
evaluated).

- henrik

  == The First Proposal ==
 
  Inspired by RVM Gemsets, how about allowing modules to declare which
  version of a given module they will use?
 
  /etc/puppet/modules/apache/puppetlabs/{files,modules,manifests}
  /etc/puppet/modules/apache/theforeman/{files,modules,manifests}
  /etc/puppet/modules/apache/author/{files,modules,manifests}
 
  Then, use could be dictated by something like: include
apache@puppetlabs.
 
  Unfortunately, this really comes down to the parser being able to
  understand the following:
 
  include apache = See if there are any @'s and use that one
  include apache@puppetlabs = include the Puppetlabs apache
  include apache@puppetlabs  include apache@theforeman = Fail,
conflict
 
  == Alternative ==
 
  One possible alternative is to just use the metadata.json file to
  dictate which module will be used when loading other modules.
 
  Again, if there is a conflict, that is a failure but *only* if
the code
  attempts to use both at the same time.
 
  The benefit here is that it should make things very unambiguous
while
  the drawback (if it really is) is that you *must* put a
metadata.json
  file in every module that you create.
 
  This is just a set of thoughts that I hope get us moving in a
direction
  where

[Puppet-dev] puppet dev status week ending feb 7

[Henrik Lindberg]

The Puppet Dev team had a crazy week with lots of things happening at 
the same time.


* FOSDEM, Config Management Camp and Contributor Summit
* Building and testing packages for the AIO (All in One Agent)
* Fixing the issues we find as we start building and testing the Puppet
  4.0.0 release including the family of other projects.

We are totally focused on the end game of getting Puppet 4.0.0 and 
friends released!


Any day now we may have all the bits in a row...

- henrik

--
You received this message because you are subscribed to the Google Groups Puppet 
Developers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/mb3j73%24r10%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: Puppet Experimental Features (like parser=future)

[Henrik Lindberg]

On 2015-21-01 11:23, shashank wrote:

Hi All,

I was using puppet since the release of puppet 2.6 version in various
projects.Now came across the Puppet Experimental Features in the
Puppet 3.x series  and some of which are really very useful for projects
I was taking care of.

My concern is that how much it is safe to use the Puppet Experimental
Features like parser=future in the production environment as I am using
puppet 3.7.1.I had already tested this feature on the testing
environment for a long time and used some of the features frequently
like (each($x) |$a | {} ).

Now my codes are ready for the production environment, please suggest me
whether I should go with the puppet parser future feature or should I
remove the codes supported by these feature from my manifests (Basically
the feature of iteration over hashes using 'each') ?

Thanks



This was appraently separately cross posted to Puppet User's google 
group. I responded there.


- henrik

--

Visit my Blog Puppet on the Edge
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups Puppet 
Developers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/m9o18p%24qb3%242%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: custom function and special variables

[Henrik Lindberg]

On 2015-08-01 21:33, Johan De Wit wrote:

Hi,

Sorry for the somewhat misleading title, but don't know better wording.

Is it possible to access special variables that are set outside the 
function ?  I'm not talking about config settings, but looking at 
something like
a variable that I could query to get the filename of the manifest that 
is calling the function.



No, that is not generally available.

For the new Function API (new in Puppet 4.0, and avail when using 
parser=future) we want to add support for things like this. We already 
have the mechanism in place for injecting additional parameters into a 
function calls in the new API, but no support for file/line of caller yet.


See https://tickets.puppetlabs.com/browse/PUP-2769
Background is the create_resources that is not filling the 
resource.file and resource.line, which makes catalog-diff not 
recognize the resource as a 'resource', thus giving false reports.
One way to avoid this problem is also to use the new resource Puppet 4.0 
syntax (available in future parser as well in 3x) for resource creation 
that makes it as powerful as the create_resources function call. That 
way, the resource creation is done by the language and the file line 
information is included.


e.g.
Resource[$the_type] {
  # this part can be omitted if defaults were not used in the 
create_resource call


  default: * = $defaults_hash;

  # The $title can naturally be an array if the same attributes
  # should be applied to multiple resources.
  # Use iteration around the entire resource expression to create
  # a sequence of resources with different attributes

  $title: * = $attributes_hash;
}

Hope that helps.
Regards

- henrik

--

Visit my Blog Puppet on the Edge
http://puppet-on-the-edge.blogspot.se/


--
You received this message because you are subscribed to the Google Groups Puppet 
Developers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/m8opou%242tj%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: Non-breaking changes and puppet 3.7.x

[Henrik Lindberg]

On 2014-05-12 3:56, Kylo Ginsberg wrote:
On Thu, Dec 4, 2014 at 3:46 PM, Henrik Lindberg 
henrik.lindb...@cloudsmith.com 
mailto:henrik.lindb...@cloudsmith.com wrote:


On 2014-04-12 22:24, Kylo Ginsberg wrote:

* non-breaking changes should default to 3.7.x until some time
passes after 4.0 is out

Eh, no, not in general please. We have lots of code removal and
anything that needs to go through the process of being
implemented both an old and a new way should not be done at all on
stable IMO except if there is burning need / bug.

We have removed lots of code to save us work, remove complexity etc.

I am fine with non-breaking /bug fixes/ should default to 3.7.x
until some time passes after 4.0 is out.


Ah yes, I agree. I was thinking bug fixes specifically (but the 
wrong words came out).


Maybe the criteria are more like non-breaking bug fixes, not in 
support of any deprecated features, and trivially merged up to 4.x or 
some such.



There is no need to rename branches.


I *think* we'd still need an additional branch if we want to support 
any level of changes to 3.7.x after 4.0 is released b/c we'd have:

* branch for 3.7.x
* branch for 4.0.x
* branch for 4.1/5.0
which could be named 3.7.x/stable/master respectively (although we 
should discuss Josh H's comment too). Or am I missing your point here?


You are right when we reach 4.0.0, we will use stable for 4.0.x, and 
master for 4.x, so we do need a 3.7.x branch. The workflow will be
backwards compatible bugfixes that do not involve removed features go 
to 3.7.x, then stable, and then master.


And as you said, the work is really setting up CI and have the extra 
3.7.x branch being managed.



- henrik


--
You received this message because you are subscribed to the Google Groups Puppet 
Developers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/m5snkc%24bn2%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: Non-breaking changes and puppet 3.7.x

[Henrik Lindberg]

On 2014-04-12 22:24, Kylo Ginsberg wrote:
* non-breaking changes should default to 3.7.x until some time passes 
after 4.0 is out
Eh, no, not in general please. We have lots of code removal and anything 
that needs to go through the process of being
implemented both an old and a new way should not be done at all on 
stable IMO except if there is burning need / bug.


We have removed lots of code to save us work, remove complexity etc.

I am fine with non-breaking /bug fixes/ should default to 3.7.x until 
some time passes after 4.0 is out.


There is no need to rename branches.

--

Visit my Blog Puppet on the Edge
http://puppet-on-the-edge.blogspot.se/


--
You received this message because you are subscribed to the Google Groups Puppet 
Developers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/m5qrof%24dsg%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: Use of private in module skeleton

[Henrik Lindberg]

On 2014-22-11 21:25, Felix Frank wrote:

On 11/22/2014 01:22 PM, Gareth Rushgrove wrote:

Hey all

I'd love some feedback on this PR, which adds the private function to
the puppet-module-skeleton that a bunch of people use.

https://github.com/garethr/puppet-module-skeleton/pull/43

A few people commented on Twitter and it's prety much 50/50.

https://twitter.com/garethr/status/535526875477839872

And I'm on the fence too. Any thoughts?

Gareth

+1 for helping module authors help their users avoid shooting their own
feet.

That's what this is about, right?
If they are creating classes that should be visible to others, they can 
just remove the private declaration.


A word of warning here. In puppet 4.0.0 the word private is reserved 
since the concept of being private is
going to be moved to the language. The function called private must 
therefore be renamed.


- henrik


--

Visit my Blog Puppet on the Edge
http://puppet-on-the-edge.blogspot.se/


--
You received this message because you are subscribed to the Google Groups Puppet 
Developers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/m4tlp5%24na5%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] puppet-dev status the week ending 14-11-14

[Henrik Lindberg]

 there is the
  need to again do language feature work and have a "new
  future")


  The
  old Puppet::Parser::Parser is removed (if you have written
  code using the parser directly you should have switched to
  using the ParserFactory a long time ago.


  The
  old Puppet::Parser::Lexer is removed - if you relied to on
  low level lexing you need to use the new Lexer instead.


  Almost
  all of the AST classes have been removed except those that
  are used by the compiler and some of the functions. (List
  below of what remains of the AST).


  The
  AstTransformer has shrunken to only deal with a few 4x to
  3x tasks. It may be refactored and go away completely.


  A
  large number of tests have been removed.


  RDoc
  support no longer produce output for any .pp files (to be
  replaced with the Puppet Strings module).


  Since
  we are also dropping support for Ruby 1.8.7 in Puppet
  4.0.0 the support for RDoc1 has been removed.

  
  
  Remaining
  AST:
  
  

  AST::Regexp
  - a class we would like to also remove in favor of a
  different construct, but it is currently expected for
  supporting Node.


  AST::BlockExpression
  - is used as the container of code in the top level
  constructs that the compiler cares about. Will stay around
  probably until the compiler is rewritten.


  AST::HostClass
  - a top level construct the compiler cares about.


  AST::AstArray
  - used in the top level constructs - I would like to see
  this go away, not sure when that will happen.


  AST::ResourceInstance,
  is currently used by the create_resources function to get
  the same behavior as in 3x Puppet Languag when creating a
  resource. May be rewritten to instead use the same logic
  used in the new evaluator.


  AST::ResourceParam,
  these are used to instantiate resource attributes, they
  are registered as part of overrides and defaults. These
  will stay around until the compiler is rewritten.


  AST::Branch,
  an abstact class used by ResourceParam (and others).


  AST::TopLevelConstruct,
  an abstract class for some of the others.

  
  
  ALL
  OTHER AST CLASSES HAVE BEEN REMOVED.
  If you have logic that makes use of those you need to do
  things differently. Please ping if you are
  in
  this situation and need help with the transition.
  
  Regards
      
      Henrik Lindberg
  henrik.lindb...@puppetlabs.com

  

-- 

Visit my Blog "Puppet on the Edge"
http://puppet-on-the-edge.blogspot.se/
  




-- 
You received this message because you are subscribed to the Google Groups Puppet Developers group.
To unsubscribe from this group and stop receiving emails from it, send an email to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-dev/m45g10%24j35%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: puppet-dev status the week ending 14-11-14

[Henrik Lindberg]

Sorry a bout the botched format... investigating

--
You received this message because you are subscribed to the Google Groups Puppet 
Developers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/m45g3v%24j35%242%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: puppet-dev status the week ending 14-11-14

[Henrik Lindberg]

** Next PR Triage Wednesday, November 19th @ 10:00 am Pacific. **

Priorities

   1. 
   
   Puppet 3.7.4
   2. 
   
   Native Facter 0.3.0 
   3. 
   
   Puppet Server 1.0.0
   4. 
   
   Code removal for puppet 4
   5. 
   
   New puppet doc implementation
   

Commentary

Puppet:

We are going to keep 3.7.4 in parallel with 4.0.0 to ensure we get a 3x 
release that contains

a future parser that is as close as possible to what is released in 4.0.0. 
It will be released in close proximity to 4.0.0 (unless there is a burning 
need to release it sooner).

Please note that some changes in puppet 4 mean that puppet 3 agents won’t 
be able to talk to puppet 4 masters (or vice-versa). This probably means 
you don’t want to be updating puppet itself with “ensure = latest” :)

One of these changes is for the URL structure for the master/agent HTTP 
API.  Specifically, we’d like to move the environment name away from being 
the first segment of the URL, so that the URL space is more well-contained 
and predictable.  This will allow us to begin separating the monolithic 
“master” service into more fine-grained services.

Code removals for puppet 4 is picking up more steam (yay, burn all the 
things…) - 17k lines removed so far. See a heads up below if you are doing 
advanced things with any of the classes under puppet/parser (an additional 
15k removal).

We are doing major triage for 4.0.0 as we have accumulated quite a bit of 
scope creep. You will be seeing tickets move to target 4x as we like to 
reserve the 4.0.0 tag for things that we feel are required for this 
release. This means that we may pull in tickets/PRs marked 4x, but we will 
not block the release if they do not make it in time.

We are also engaged in yet more work around environments - this time 
removing support for legacy environments.

Puppet Server:

Puppet Server 0.4.0 was released!  The changes were mostly plumbing; 
packaging improvements, tweaks to make the CA behavior more consistent with 
the old CA, and some improvements related to SSL transports.

We’re planning for the next release to be our 1.0 release.  It will include 
a few cool new features related to directory environment caching, plus a 
few minor tweaks and bug fixes.  The 1.0 series will be compatible with the 
Puppet 3.7.x, so next up will be to start working on Puppet Server 2.0 for 
Puppet 4.x compatibility.

Puppet Doc:

We have worked on the scope of the new puppet doc implementation (puppet 
strings), there are now tickets, and we are just starting to work on those.

Native Facter: 

Solaris, Windows, ec2/gce facts are all in - there are a handful of misc 
fix-it tickets remaining before we tag 0.3.0. 

Also, we now have Native Facter nightlies! 

   - 
   
   http://nightlies.puppetlabs.com/cfacter/
   - 
   
   http://nightlies.puppetlabs.com/cfacter-latest/
   

Right now it’s rhel6/rhel7/wheezy with more packages coming. Try it out and 
tell us what we’re missing :)

Other conversations of note:

   - 
   
   Henrik taking over after Andy 
   - 
  
  Yes, I am stepping in after Andy which means that I now will be 
  saying no to my own crazy ideas - or face the consequences :-)
  - 
   
   Rosetta lands Philae on Comet 67P
   

 

Heads up on Code Removals

With the fix in PUP-2906 being merged (reimplementation of collections) we 
now freed up for the removal of the old AST classes and work is now ongoing 
on ticket PUP-3274, which removes the AST and switches puppet 4.0.0 over to 
always using the future parser.

As a heads up (if you have written code that relies on any of the parser 
classes), this means that:


   - 
   
   The --parser setting will be removed (it is always parser=future from 
   this point. (Well unless there is the need to again do language feature 
   work and have a new future)
   - 
   
   The old Puppet::Parser::Parser is removed (if you have written code 
   using the parser directly you should have switched to using the 
   ParserFactory a long time ago.
   - 
   
   The old Puppet::Parser::Lexer is removed - if you relied to on low level 
   lexing you need to use the new Lexer instead.
   - 
   
   Almost all of the AST classes have been removed except those that are 
   used by the compiler and some of the functions. (List below of what remains 
   of the AST).
   - 
   
   The AstTransformer has shrunken to only deal with a few 4x to 3x tasks. 
   It may be refactored and go away completely.
   - 
   
   A large number of tests have been removed.
   - 
   
   RDoc support no longer produce output for any .pp files (to be replaced 
   with the Puppet Strings module).
   - 
   
   Since we are also dropping support for Ruby 1.8.7 in Puppet 4.0.0 the 
   support for RDoc1 has been removed.
   

Remaining AST:


   - 
   
   AST::Regexp - a class we would like to also remove in favor of a 
   different construct, but it is currently expected for supporting Node.
   - 
   
   AST::BlockExpression - is used as the 

[Puppet-dev] Re: Puppet PR Triage starting soon

[Henrik Lindberg]

notes from the meeting
https://github.com/puppet-community/community-triage/blob/master/core/notes/2014-11-12.md

Since we changed to taking notes in the Github PRs directly, you can see 
the triaged PRs that are still open (and where comments were added), and 
the triaged PRs that were closed by using these links:


Closed:
https://github.com/pulls?q=repo%3Apuppetlabs%2Fpuppet+repo%3Apuppetlabs%2Ffacter+repo%3Apuppetlabs%2Fhiera+is%3Apr+label%3ATriaged+updated%3A%222014-11-12+..+2014-11-18%22+is%3Aclosed

Commented:
https://github.com/pulls?q=repo%3Apuppetlabs%2Fpuppet+repo%3Apuppetlabs%2Ffacter+repo%3Apuppetlabs%2Fhiera+is%3Apr+label%3ATriaged+updated%3A%222014-11-12+..+2014-11-18%22+is%3Aopen+


Regards
- henrik

--

Visit my Blog Puppet on the Edge
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups Puppet 
Developers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/m40e1h%24r9u%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: Improving the Resources Resource Type

[Henrik Lindberg]

On 2014-07-11 17:29, John Bollinger wrote:


Having said all that, I can completely get behind Luke's idea to provide
for client-side queries.


I am +1 on that too. Much better idea.
You can forget my earlier proposals.

- henrik
--

Visit my Blog Puppet on the Edge
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups Puppet 
Developers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/m3jqeb%2480i%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: Improving the Resources Resource Type

[Henrik Lindberg]

On 2014-07-11 18:28, Erik Dalén wrote:

On Fri Nov 07 2014 at 5:29:15 PM John Bollinger
john.bollin...@stjude.org mailto:john.bollin...@stjude.org wrote:

Having said all that, I can completely get behind Luke's idea to
provide for client-side queries.  I am confused, however, as to why
the ideas for that seem to be running in directions such as
overloading resource titles and introducing new collector-like
forms, when the entire purpose of the Resources type is to serve
this kind of function.  Why not just add a parameter to Resources to
specify the a query predicate (the default being something like
!managed, and another to specify a hash of resource properties to
set, e.g. { ensure = absent }.  Deprecate the existing 'purge' and
'unless_system_user' parameters.  This would look similar to some of
the forms being proposed, but it would rely mostly on existing
facilities for everything other than actually performing queries.
Example:

resources {'remove-unmanaged-users':
   name ='user',
   match =uid  100  !managed,

set={ensure='absent'}
}

I admit that I'm a bit leery of the !managed bit of the predicate,
as its presence suggests that maybe it can be omitted to allow
whatever is implemented for this to modify the properties of
/managed/ resources, too.  This could be especially nasty if it were
possible to stack these things so that more than one applies to the
same resource. *I urge that that possibility be foreclosed*, at
least in the initial implementation.  It can be added later if there
turns out to be sufficient justification, but it cannot easily be
taken away once granted.


I think this is sensible, likely the parser for this query should be
implemented in the munging method of the type though, so it can happen
server side and turn it into some sort of AST like format (which is what
I meant with something looking a bit like the PuppetDB query API).

Possibly it should allow users to submit that AST format directly, so
then you could have other parser functions to implement different
syntaxes for the query language.

as an example:
instead of match = title='foo' and version='2.1.2'
allow something like [and, [=, title, foo], [=, version,
2.1.2]]
And then you could have a function that could return something like that
instead of having to turn it into a string that then gets parsed again.



That is what a query would produce as serialization if allowed as a 
value expression. e.g.


match = | title == foo and version = '2.1.2' |

the big benefit is that the parser can validate the query, all querying 
is expressed the same way, etc. Just as if this was done with a string,

any interpolation is done server side.

match = | title == foo and version = $ver |

i.e. $ver is evaluated and the result is stored in the resulting baked 
query that is sent to the agent.


As part of getting rid of the old 3x AST, we just finished the rewrite 
of the Collector. It is now much easier to do the collector expression 
encoding transformation to array form or ruby proc predicate (used 
internally for catalog collection and exported collection), and we can 
now easily add an array form to ruby proc predicate transformation 
that makes it really easy to use as a filter in a type.


- henrik
--

Visit my Blog Puppet on the Edge
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups Puppet 
Developers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/m3jrev%24mfb%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: Improving the Resources Resource Type

[Henrik Lindberg]

On 2014-06-11 12:04, Luke Kanies wrote:

A fourth option would be to combine the best of Resources and exported
resources:  Build a querying system akin to exported resources, but
against the current system, rather than against the catalog.  Or really,
take the exact same querying we already have, but provide the ability to
control where it's searching against, and what to do with the results.

This is my preferred solution.  It makes querying more powerful, and
separates the concept of querying from the source of the data.



I like this a lot. There are some design challenges though as this 
pushes the querying to the agent, and that this is probably best 
supported at the language level rather than being expressed as a 
function. (This because variables etc. in a query needs to be 
evaluated/resolved in scope when the catalog is produced, just like they 
are today in virtual/exported collection).



I really do think this kind of functionality should be supported at the
framework level, and more hacks like Resources aren't the best long term
solution.


Agree.


--

Visit my Blog Puppet on the Edge
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups Puppet 
Developers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/545BB100.8050805%40cloudsmith.com.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: Puppet PR Triage 2014-11-05

[Henrik Lindberg]

On 2014-05-11 18:00, Josh Cooper wrote:

Starting the PR triage for puppet/facter/hiera in an hour (10am PST) at
http://links.puppetlabs.com/__pr-triage
http://links.puppetlabs.com/pr-triage



Notes from the meeting are here: 
https://github.com/puppet-community/community-triage/blob/master/core/notes/2014-11-05.md


We decided to simplify the note taking by instead using the Github 
PR/issue tracker. We will mark all PRs with a label Triaged when the 
PR is discussed for the first time in the triage meeting. Notes from the 
triage meeting are then entered directly into the PR by the note taker.


This saves us a lot of manual housekeeping of what has been triaged, 
what was said last etc.


It also means we will not be sending out a list of all the PRs that were 
discussed and what was said. If someone is interested in staying on top 
of what all the PRs are and what is said then there is already 
functionality at github for getting notifications etc.


If you feel we overlooked some aspect of communication related to this, 
raise your voice...


We hope keeping triage notes in the PR/github issues will make it easier 
for everyone.


Regards
- henrik



Josh

--
Josh Cooper
Developer, Puppet Labs

*Join us at **PuppetConf 2015, October 5-9 in Portland, OR -
*http://2015.puppetconf.com http://2015.puppetconf.com/.
/Register early to save 40%!/

--
You received this message because you are subscribed to the Google
Groups Puppet Developers group.
To unsubscribe from this group and stop receiving emails from it, send
an email to puppet-dev+unsubscr...@googlegroups.com
mailto:puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-dev/CA%2Bu97umwDG-ABm%3DNUUQR5zm1tikHMEcE8UbC_%2B3Tm%3D30k%2BH0sA%40mail.gmail.com
https://groups.google.com/d/msgid/puppet-dev/CA%2Bu97umwDG-ABm%3DNUUQR5zm1tikHMEcE8UbC_%2B3Tm%3D30k%2BH0sA%40mail.gmail.com?utm_medium=emailutm_source=footer.
For more options, visit https://groups.google.com/d/optout.



--

Visit my Blog Puppet on the Edge
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups Puppet 
Developers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/m3dvvq%24jkp%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: The string to number torture never stops

[Henrik Lindberg]

On 2014-05-11 15:45, Joshua Hoblitt wrote:

On 11/04/2014 09:09 PM, Joshua Hoblitt wrote:

On 11/04/2014 01:04 PM, Andy Parker wrote:



It would also provide at lot more confidence that a manifest is
4x safe
if it passes `puppet parser validate`.  Without that you'd have
to worry
about conditional expressions silently changing their result
between 3x
and 4x.


The error would have to be at runtime, so puppet parser validate
would not uncover these kinds of issues (except perhaps in trivial
cases where it could be done). I have an experiment for creating a
type inference system, but it is no where near complete enough to be
a static type analysis system for puppet 4, and even then it would
have to give up in far too many cases without yet more changes to the
language.

Your absolutely right but  `puppet master --compile  localhost
--manifest manifests/role/db.pp` would work.  Similarly, rspec-puppet
tests would catch an exception from building the catalog even if there
isn't coverage of the expression result. Perhaps an option could be
added to the parser face to eval expressions?


Actually, I think I'm full of it.  It appears that although the master
face accepts the --manifest, --modulepath, etc. flags in combination
with --compile they don't have any visible effect on the generated
catalog (my catalogs are so large it was hard to tell). In any event,
the magic behavior I was hallucinating was parsing the supplied manifest
and then evaluating it with cached facts (is --compile pulling facts out
of inventory?).

Upon further consideration, there are significant problems with trying
to implement a generic parse+evaluate checker that can be used in the
same manner as `puppet parser validate` on arbitrary DSL code.   Defined
but not declared classes / defined types may fail due to mandatory
params, the use of evaluation order dependent functions (defined(),
getparam(), etc.) and the `$caller_module_name != $module_name` idiom
are all gotchas.

Still, I wonder if it's possible to use the parser to fish out the names
of classes without mandatory parameters from of a series of manifests
and automagically perform the equivalent of rspec-puppet|`it { should
compile }' or 'it { should compile.with_all_deps }`|?  I suppose facter
would either need to be invoked locally or an external corpus of facts
would need to be supplied.  Hmm.



At cloudsmith we patched rspec-puppet to be able to feed in a set of 
captured facts from a node. I think that is a feature that is missing in 
rspec-puppet.


You can indeed to introspection of what is being parsed with the future 
parser (the older parser is not completely side effect free and should 
not be used for things like this). While not difficult, it does require 
some coding to parse content and then find / filter the returned ast model.


You can also dump the parsed result as s-expressions (i.e. lisp like 
tree) that shows the content of the ast model using puppet parser dump


- henrik

--

Visit my Blog Puppet on the Edge
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups Puppet 
Developers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/m3e0ev%24tl7%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Improving the Resources Resource Type

[Henrik Lindberg]

Hi,
I am bringing up this topic because of a recent discussion and PR for 
the ticket https://tickets.puppetlabs.com/browse/PUP-1486


Here is a recap.

The Resources resource is used to manage unmanaged resources of a 
particular given type - e.g. to purge users, groups etc. that are not 
otherwise managed.


The issues are that managing the unmanaged may require additional 
attributes that are specific to the type that is referenced. The base 
type only has the attributes unless_system_user and unless_uid (se 
https://docs.puppetlabs.com/references/latest/type.html#resources for 
the documentation).


At times it is suggested that the Resources type should have additional 
attributes that are specific only to one type. This solution has the 
obvious problem that over time, the Resource type would then have a very 
large number of attributes for different types and the temptation will 
be great to borrow attributes and apply similar semantics. Over time 
this solution will become unmanageable.


Another solution was proposed in PR-3269 
(https://github.com/puppetlabs/puppet/pull/3269) which we closed for 
various reasons.


The approach in the PR was to add a newresourcesparam to Puppet::Type
allowing this to define additional parameters to the Resources type.
An obvious problem with this approach is the need to load all types in 
order to check the validity of attributes of a Resources instance.


Instead, I would like to see something where the concerns are separated 
(to avoid the problem of the base class knows about all the things). 
One good pattern is the strategy pattern / adapter pattern which is 
used internally in the future parser. (You can read about this here: 
http://puppet-on-the-edge.blogspot.se/2014/02/puppet-internals-adapter-pattern.html). 
TL;DR - an adapter is responsible for sticking a label on someone else 
without that someone having to be concerned about its existence, 
protocol etc.


How can we achieve something similar in a simple way in Puppet?

One very simple approach is to add the basic ability to handle such 
adapters, or extra data if you like, in the Resources type. We 
cannot use the actual adapter pattern since when it comes to resources, 
they are serialized and processed in ways that would drop such adapters 
(since those are really a runtime concept).


Proposal:

* Add the attribute options (with the type Array[Resource])
* Add the ability in a resource type to specify a list of type names 
that it allows instances of as options (typically one)

* Add the ability in type to get the options.

Here is a simulation of the above using just user defined types - a type 
base specifies that it takes an option that must be a reference to a 
Base_option:


define base_options($p1, $p2) {
}

define base(Resource[base_option] $option = undef) {
}

base { 'the_base':
  extraref = Base_options[the_options]
}

base_options { the_options:
  p1 = 'extra 1',
  p2 = 'extra 2'
}

This way, everything ends up in the catalog. The same using Resources 
would look like this:


resources { base:
  options = [Base_options[the_options]]
}

How the rest of the protocol between Resources and the types that 
support options should work is left to be designed, but basically; the 
Resources implementation should present the options to the type either 
for just asking if a particular resource should be purged or not (as it 
does now but done as a delegation to the particular type).


I think this design is:

* type safe - it is possible to statically check the combination of the 
managed type, the options it supports, and the type of the entries in 
options.
* The optional type can be a user defined type - all that is needed are 
the parameter values specified by the user.

* A user defined type can be name-spaced

Alternative - using the type system in Puppet 4.0

Another alternative is to simply use the new type system in Puppet 4.0.
A type that supports options to Resources returns a Struct type that 
describes the options. (A Struct is a detailed type specification of a 
hash). (Read more about the Struct type here: 
http://puppet-on-the-edge.blogspot.se/2014/02/adding-struct-and-tuple-to-puppet-type.html)


In the catalog a Struct type is simply encoded as a String, and it is 
easily converted back to an instance of the type which can be used for 
type checking the options received from the Resources type.


Thoughts?
- henrik

--

Visit my Blog Puppet on the Edge
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups Puppet 
Developers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/m3ems4%24qu3%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: The string to number torture never stops

[Henrik Lindberg]

On 2014-03-11 18:19, John Bollinger wrote:



On Friday, October 31, 2014 8:22:36 PM UTC-5, henrik lindberg wrote:

[...]

Yet again someone was bit by the automatic String to Numeric conversion
that is in Puppet (and also in --parser future).



I must confess to a certain dark amusement at Puppet struggling with its
weak-typing legacy and moving more and more in the direction of strong
typing.  Not that I am in any way happy about these difficulties, but
I'm an old-school dinosaur, and weak typing has never seemed like such a
great idea to me.



It is never a great idea to have weak / no typing. It is an even worse 
idea to have all types represented as strings...



[...]

We fixed PUP-3602 by not converting strings that are floating point 0
with exponential part and we also do not convert values that are
floating point infinite in Ruby (e.g. 4e999 and such). This is a crutch
though, and it is only a matter of time until someone stumbles over the
next SURPRISE !



Indeed so.  If you rely on heuristics to choose behavior then you have
to accept that sometimes the wrong behavior will be chosen.  In this
case, it is purely a guess that a string starting with 0e[digit] is
not meant to be interpreted as a number.



Yes, this was a bit of a panic fix. It really sucks.


The best cure is naturally to never do String to numeric conversion.



What about numeric to string conversion?  I guess Puppet doesn't do that
automatically now, so maybe this isn't the time to start, but that /is/
an alternative approach to mixed string/number comparison.


True, 4.0.0 will not do numeric to string automatically (because of
radix, precision and floating point formatting). I do not see that 
coming back.



And
we wonder what people feel about that. Should we go for this in Puppet
4.0.0 (and have a 3.7.4 release as the last of the 3x series where this
behavior is implemented when using --parser future).



I think avoiding automatic string to numeric conversion is consistent
with forbidding bare strings starting with a digit..


Good point.


It would be a lot
better, though, if in the context of the manifest it were clearer which
expressions are strings and which are numeric.  That's no problem for
literals, of course, but none of this is interesting in the case where
all the values involved are literals.  I (think I) understand that with
the P4 parser and evaluator it will be possible to declare types
specifically enough to address that issue, but it is also my
understanding that expressions won't /necessarily/ have formal types
specific enough for that.



It is a bit difficult since operators are overloaded on type. The good 
part is that if we stop transforming strings to numbers there will be 
errors for arithmetic expressions.


The bad part is that ==, != cannot raise errors (since a string is 
simply not equal to a number). Currently comparisons order all numbers 
to be smaller than all strings. We could change those to instead error 
if the types are not comparable to each other.



It is highly desirable to give manifest authors sufficient control over
conversions to avoid unwanted ones, but it is not altogether clear to me
whether the best approach is to nix all automatic string-to-number
conversions.  A lot of existing manifests rely on such conversions,
since they used to be the only alternative.  P4 is at liberty to break
backward compatibility, but maybe a little less breakage would be wiser?

While I am not so worried about the logic in the manifests themselves. 
There has not been that many problems reported with respect to the 
conversion in the other direction. For resource attributes however the
situation is worse since there are many types out there where it is 
unknown how they deal with data types, what sort of munging / processing 
they do of strings/numbers etc.


This would be the primary reason (IMO) to not do this until resource 
types can type their parameters. (Since typed parameters direct 
serialization, and there is no longer a question if a serialized 42 
should be a number or a string).



* Add === operator to compare both type and value. This is a slippery
slope since we probably want Integer and Float to compare equal - say
0.0 and 0. It adds yet another operator, and we have to decide what
case, selector and in should use since there is no way to specify if
one
or the other should be used.



I agree that as proposed, the '===' operator would be troublesome.
There is always the alternative, though, of keeping '==' as it is, and
making '===' simply perform a comparison without string/number
conversion.  I think 'case', selector, and 'in' behavior are
collectively a red herring, though: if '===' were adopted in any form
then 'case', selector, and 'in' behavior would still be whatever is
specified for them, whether that's their current behavior or a variant
one based on '==='.  There is no requirement

[Puppet-dev] Re: The string to number torture never stops

[Henrik Lindberg]

On 2014-04-11 18:28, Joshua Hoblitt wrote:

On 11/04/2014 09:46 AM, Henrik Lindberg wrote:

It is a bit difficult since operators are overloaded on type. The good
part is that if we stop transforming strings to numbers there will be
errors for arithmetic expressions.

The bad part is that ==, != cannot raise errors (since a string is
simply not equal to a number). Currently comparisons order all numbers
to be smaller than all strings. We could change those to instead error
if the types are not comparable to each other.


I think raising an exception when the types of operands to a comparison
operator are different is the least surprising behavior absent automatic
type conversion.

It would also provide at lot more confidence that a manifest is 4x safe
if it passes `puppet parser validate`.  Without that you'd have to worry
about conditional expressions silently changing their result between 3x
and 4x.

Remember though, that validation cannot catch results of functions and 
expressions - since the validation is static, and we cannot really do 
reasonable quality type inference (at least not yet).


- henrik


-Josh

--




--

Visit my Blog Puppet on the Edge
http://puppet-on-the-edge.blogspot.se/

--
You received this message because you are subscribed to the Google Groups Puppet 
Developers group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/m3bfr7%245j9%241%40ger.gmane.org.
For more options, visit https://groups.google.com/d/optout.