[Puppet-dev] Re: Puppet RFC 23 - XPP Files

[Henrik Lindberg]

On 30/03/16 23:12, Jeremiah Powell wrote:

ASTs don't need to be built on a node-by-node basis (unless you
meant manifest-by-manifest basis


Well, manifest-by-manifest where the manifest will vary depending on the
node the compile job is depending upon.  Reviewing the code I get the
impression I just don't understand the existing parser[1] enough today
to hold a valid opinion on this.  But honestly, I'm not trying to troll.



In puppet it is the "compilation of a catalog" that is specific to a 
node, not the parsing of the individual .pp files. The word "parsing" 
does not imply "evaluation" or "catalog compilation".


The puppet parser does the following:

* reads the source text
* builds a representation of this in memory (the Abstract Syntax Tree (AST)
* Performs static validation of the produced AST (to report semantic 
problems that are not covered by syntax alone.


The compiler does this:

* given a node, with its facts, settings from a node classifier etc. it 
is determined where it should start evaluating puppet code


* when it needs to evaluate something - e.g. "site.pp", it first needs 
to parse this file into AST (the steps above).


* When it has the AST, it starts evaluating the AST (expressions like 1 
+ 1, function calls, resource declarations etc).


* The result of the evaluation is that it has built up a catalog of 
resources.


* The catalog is typically sent to an agent for application, but can be 
written to disk etc.


The XPP PRFC is only for the three parsing steps. There will be no 
difference what so ever in what happens once the AST is loaded into 
memory. The only difference between the "Ruby parser parsing .pp file 
into AST" and the "reading an XPP containing an AST" is that we do not 
have to use the very slow Ruby runtime to do all the processing.




The goal of this particular initiative is to enable the C++ parser
(i.e. the frontend) to interop with the Ruby evaluation
implementation (i.e. the backend).  The Puppet code is not being
pre-compiled, but pre-parsed/pre-validated; the C++ implementation
will not (yet) evaluate any Puppet code or load custom types or
functions defined in Ruby.


How will this work with create_resources[2]?


They are completely unrelated. The call to the "create_resources" will 
take place in exactly the same way. The AST that is evaluated looks 
exactly the same if it came from an XPP file or if it was read in source 
form from a .pp file and parsed by the ruby parser.




In compiler terminology, there's a "frontend" and a "backend".


In compiler terminology the frontend is a scanner composed of a parser
and a lexer.  The front-end validates the parse of the code as a
side-effect.  This is beyond the scope of the discussion of the PRFC and
into a sizing competition about who's read Aho, Lam, Sethi and Ullman.

The only point form this is that this is not compiling but a partial
parsing. Some of my concerns cannot be raised until there is actual
output to examine.



Puppet does not have a compiler in the typical computer science sense. 
The puppet term "compiler" uses the word in an English/generic sense 
since what it is doing is "compiling a catalog" (putting a catalog 
together out of the pieces that are supposed to be in it).


Puppet is an interpreter that interprets the AST that is produced by the 
puppet parser. The Puppet compiler uses that interpreter when it is 
compiling the catalog.



The above seems to be confusing, understandably so, pre-compiling a
resource catalog with pre-parsing a manifest.  In terms of a
language like Python, the "pre-compiled" pyc files are simply a
representation of the source that is more efficient to load and
execute than having to parse the Python source again


That is because the Java code and CPython code is completely compiled
and ready to link in at runtime.In this the XPP proposal does not
appear similar to .pyc files or Java bytecode.



Java is compiled to byte code. This byte code is then either interpreted 
or "just in time" compiled into machine code.


The Puppet AST is to Puppet what the Java byte code is to a JVM.
(Although puppet it is not byte code based; we simply use the AST).

Byte code is typically for a virtual machine that is stack based.
As a simple example if you have the source:

  a = 2 + 3

Byte code may be something like

 Push Literal 2
 Push Literal 3
 Add
 Store a

Whereas the AST is a tree of nodes (which is hard to draw, so here using
a list notation. The same in puppet would be:

(AssignmentExpression a
  (ArithmeticExpression +
(Literal 2)
(Literal 3)))



It does appear to me too be very similar to the Ecore technology[3] from
Eclipse, and thus Geppetto as mentioned in the references on RGen in the
prior art section.   It does appears to be similar in how you write a
Coffeescript parser for grammars in Atom or languages in Sublime Text.
It is just that you plan to serialize the result to disk 

Re: [Puppet-dev] Re: Puppet RFC 23 - XPP Files

[Jeremiah Powell]

>
> ASTs don't need to be built on a node-by-node basis (unless you meant
> manifest-by-manifest basis
>

Well, manifest-by-manifest where the manifest will vary depending on the
node the compile job is depending upon.  Reviewing the code I get the
impression I just don't understand the existing parser[1] enough today to
hold a valid opinion on this.  But honestly, I'm not trying to troll.

The goal of this particular initiative is to enable the C++ parser (i.e.
> the frontend) to interop with the Ruby evaluation implementation (i.e. the
> backend).  The Puppet code is not being pre-compiled, but
> pre-parsed/pre-validated; the C++ implementation will not (yet) evaluate
> any Puppet code or load custom types or functions defined in Ruby.
>

How will this work with create_resources[2]?

In compiler terminology, there's a "frontend" and a "backend".
>

In compiler terminology the frontend is a scanner composed of a parser and
a lexer.  The front-end validates the parse of the code as a side-effect.
This is beyond the scope of the discussion of the PRFC and into a sizing
competition about who's read Aho, Lam, Sethi and Ullman.

The only point form this is that this is not compiling but a partial
parsing. Some of my concerns cannot be raised until there is actual output
to examine.

The above seems to be confusing, understandably so, pre-compiling a
> resource catalog with pre-parsing a manifest.  In terms of a language like
> Python, the "pre-compiled" pyc files are simply a representation of the
> source that is more efficient to load and execute than having to parse the
> Python source again
>

That is because the Java code and CPython code is completely compiled and
ready to link in at runtime.In this the XPP proposal does not appear
similar to .pyc files or Java bytecode.

It does appear to me too be very similar to the Ecore technology[3] from
Eclipse, and thus Geppetto as mentioned in the references on RGen in the
prior art section.   It does appears to be similar in how you write a
Coffeescript parser for grammars in Atom or languages in Sublime Text.  It
is just that you plan to serialize the result to disk instead of displaying
to the user.

I suggest you read more about the CPython implementation of .pyc files in
PEP 3147[4]. The PEP proposal is very well written, IMHO.  It covers a lot
of the questions that are being raised in comments on the PRFC. Like the
discussion of not using a shadow filesystem for the files.

An example from the PEP: will there be features like the ability to detect
if pre-parsing is available or in use?  Can I turn it off in code as a
developer or must I always use the --no-xpp command line as a user?  Would
that even be a good idea?

>From my understanding of the PRFC and XPP is a half-parsed file with
compiler warnings mixed in.This brings to mind the use of this for
created a blocking step in a code deployment process.  I've already
commented on that use in the document, tough.

1. https://github.com/puppetlabs/puppet/blob/master/lib/puppet/parser
2.
https://docs.puppetlabs.com/puppet/latest/reference/function.html#createresources
3.
http://puppet-on-the-edge.blogspot.com/2014/04/puppet-internals-introduction-to.html
4. https://www.python.org/dev/peps/pep-3147/

Jeremiah Powell
Seagate Technology - U.S. Business Data Center
http://www.seagate.com
email : jeremiah.pow...@seagate.com
phone: +1(405) 324-3238

WARNING:

The information contained in this transmission may be privileged or
confidential. It is intended only for the above-identified recipient(s). If
you are not the intended recipient, please forward this transmission to the
author and delete this transmission and all copies. Thank you.

On Wed, Mar 30, 2016 at 2:37 PM, Peter Huene 
wrote:

> Hi Jeremiah,
>
> On Wed, Mar 30, 2016 at 11:56 AM,  wrote:
>
>> I write a lot of native types for our internal use at work.  (I tell
>> people that if you are just using Exec in Puppet that ansible is one search
>> away in Google. ) Some of the Puppet code used with these types would be
>> very challenging to "pre-compile" in any way.
>>
>> I think a lot of my questions are just because I don't consider a .pp
>> file to usually be the unit of function in Puppet.  I only consider the
>> resource, which is usually implemented in Ruby, and everything else is
>> dressing.
>>
>> Reading over this proposal raises questions about how it will fit into
>> Puppet, a language and environment very different from Python, Java or
>> other languages (Puppet 4 long leaving the domain-specific title by the
>> wayside.)
>>
>> Since you have to build an AST on a node-by-node basis I am having a hard
>> time telling the value of this verses something like a marker that
>> indicates the code has been parser validated.   If your code doesn't need
>> node-by-node rebuild then perhaps the compiler could cache the results
>> instead?  I know the adage about about three hard problems in 

Re: [Puppet-dev] Re: Puppet RFC 23 - XPP Files

[Peter Huene]

Hi Jeremiah,

On Wed, Mar 30, 2016 at 11:56 AM,  wrote:

> I write a lot of native types for our internal use at work.  (I tell
> people that if you are just using Exec in Puppet that ansible is one search
> away in Google. ) Some of the Puppet code used with these types would be
> very challenging to "pre-compile" in any way.
>
> I think a lot of my questions are just because I don't consider a .pp file
> to usually be the unit of function in Puppet.  I only consider the
> resource, which is usually implemented in Ruby, and everything else is
> dressing.
>
> Reading over this proposal raises questions about how it will fit into
> Puppet, a language and environment very different from Python, Java or
> other languages (Puppet 4 long leaving the domain-specific title by the
> wayside.)
>
> Since you have to build an AST on a node-by-node basis I am having a hard
> time telling the value of this verses something like a marker that
> indicates the code has been parser validated.   If your code doesn't need
> node-by-node rebuild then perhaps the compiler could cache the results
> instead?  I know the adage about about three hard problems in Computer
> Science still applies but what this is doing is creating and managing a
> cache.
>

ASTs don't need to be built on a node-by-node basis (unless you meant
manifest-by-manifest basis); an AST is just a representation of the
manifest's source code. The XPP file format is simply an attempt to define
a serialization format for the AST itself so that, say, all the manifest
files in an environment could be parsed, validated, and the resulting ASTs
saved in a format that is faster to read when a catalog for a node is being
compiled all in one go and "upfront" (i.e. not during the catalog
compilation for any one node).

In compiler terminology, there's a "frontend" and a "backend".  The
frontend is responsible for producing an AST that the backend can evaluate
or generate code from (in the case of Puppet, the backend directly
evaluates ASTs to generate a catalog).  Having a well-defined AST
serialization format means we could potentially swap out another
implementation for the compiler's frontend, which is one of the goals of
this project and a first step towards a new compiler implementation.


> Pre-compiling is translating one language into another because compiling
> is changing one programing language, usually higher level, into another,
> usually lower level.   Puppet does something very different in my
> understanding given above.  It sounds as if the goat of the C++ parser is
> not to produce a catalog of resources but instead a ruby program for the
> agent to run.
>

The goal of this particular initiative is to enable the C++ parser (i.e.
the frontend) to interop with the Ruby evaluation implementation (i.e. the
backend).  The Puppet code is not being pre-compiled, but
pre-parsed/pre-validated; the C++ implementation will not (yet) evaluate
any Puppet code or load custom types or functions defined in Ruby.

The ultimate goal of the C++ implementation is to replace both the frontend
and the backend for catalog compilation, but that's a ways off still as
there needs to be design and implementation around loading custom types and
functions from Ruby source to maintain backwards compatibility.


>  From the statements about a C++ Puppet parser is the target still just a
> collection of resources?  Or is the goal to eventually spit out something
> other than what the Puppet server should send to a node?
>

The target is still the same thing we send to agents now: a resource
catalog.  That won't change when the C++ compiler implements the backend
too.


> Is the scope just 'closed' .pp files?  That is classes where all variables
> can be resolved without inquiry to facts?  The behavior of languages that
> support things like pre-compiled files is specific to how they do binding
> of missing data.  While this proposal punts on the 'serialization' format
> the handling of binding is pretty central. That raises questions like:
>
> How can a compiled format deal with Puppet features that call APIs?  Is
> this for the defined types?  How will it deal with the fact that the code
> could radically change based on node data not even present at the time the
> pre-compile is done?
>
> What happens if a precompiled module that depends on PuppetDB information
> is moved to an environment without it?  For that matter, is the format
> intended to move between Puppet installations or just very similar ones?
>
> My model of Puppet is that the compiled catalog is just an ordered list of
> resources which do not have any placeholders like variables or
> sub-dividable parts.  Classes in this model are just names that help you
> find the resources.
>
> This format is relevant to recent discussions among people on IRC and in
> the mailing lists and groups about getting and storing the catalog for a
> node.  This is historically something hard as an end user.

[Puppet-dev] Re: Puppet RFC 23 - XPP Files

[jeremiah . powell]

I write a lot of native types for our internal use at work.  (I tell people 
that if you are just using Exec in Puppet that ansible is one search away 
in Google. ) Some of the Puppet code used with these types would be very 
challenging to "pre-compile" in any way.

I think a lot of my questions are just because I don't consider a .pp file 
to usually be the unit of function in Puppet.  I only consider the 
resource, which is usually implemented in Ruby, and everything else is 
dressing.

Reading over this proposal raises questions about how it will fit into 
Puppet, a language and environment very different from Python, Java or 
other languages (Puppet 4 long leaving the domain-specific title by the 
wayside.)

Since you have to build an AST on a node-by-node basis I am having a hard 
time telling the value of this verses something like a marker that 
indicates the code has been parser validated.   If your code doesn't need 
node-by-node rebuild then perhaps the compiler could cache the results 
instead?  I know the adage about about three hard problems in Computer 
Science still applies but what this is doing is creating and managing a 
cache.

Pre-compiling is translating one language into another because compiling is 
changing one programing language, usually higher level, into another, 
usually lower level.   Puppet does something very different in my 
understanding given above.  It sounds as if the goat of the C++ parser is 
not to produce a catalog of resources but instead a ruby program for the 
agent to run.

 From the statements about a C++ Puppet parser is the target still just a 
collection of resources?  Or is the goal to eventually spit out something 
other than what the Puppet server should send to a node?

Is the scope just 'closed' .pp files?  That is classes where all variables 
can be resolved without inquiry to facts?  The behavior of languages that 
support things like pre-compiled files is specific to how they do binding 
of missing data.  While this proposal punts on the 'serialization' format 
the handling of binding is pretty central. That raises questions like:

How can a compiled format deal with Puppet features that call APIs?  Is 
this for the defined types?  How will it deal with the fact that the code 
could radically change based on node data not even present at the time the 
pre-compile is done?

What happens if a precompiled module that depends on PuppetDB information 
is moved to an environment without it?  For that matter, is the format 
intended to move between Puppet installations or just very similar ones?

My model of Puppet is that the compiled catalog is just an ordered list of 
resources which do not have any placeholders like variables or 
sub-dividable parts.  Classes in this model are just names that help you 
find the resources.

This format is relevant to recent discussions among people on IRC and in 
the mailing lists and groups about getting and storing the catalog for a 
node.  This is historically something hard as an end user.  

The context of any given resource is the whole catalog and at a minimum 
requires the dependent and depending classes in the graph tree.  Otherwise 
how does this deal with the unresolved parts?

On Wednesday, March 30, 2016 at 11:24:46 AM UTC-5, Eric Sorenson wrote:
>
> Hi, I've just posted a new Puppet RFC that describes pre-parsed and 
> pre-validated Puppet files, akin to '.pyc' files for Python. It's called 
> XPP and the doc is open for comments here:
>
>
> https://docs.google.com/document/d/17SFn_2PJYcO5HjgA4R65a5ynR6_bng_Ak5W53KjM4F8/edit?usp=sharing
>
> Please comment inline on the doc, or come back to this thread if the 
> conversation gets too involved (more than about 4-5 replies in a comment 
> box on google docs becomes unwieldy)
>
> Once the commenting tapers off we'll incorporate changes into the spec and 
> post it as markdown in the puppet-rfc repo: 
> https://github.com/puppetlabs/puppet-rfc
>
> --eric0
>

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/6e8b3ba1-e1a3-4008-9451-a4e0e65c5fcd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Puppet RFC 23 - XPP Files

[Eric Sorenson]

Hi, I've just posted a new Puppet RFC that describes pre-parsed and 
pre-validated Puppet files, akin to '.pyc' files for Python. It's called 
XPP and the doc is open for comments here:

https://docs.google.com/document/d/17SFn_2PJYcO5HjgA4R65a5ynR6_bng_Ak5W53KjM4F8/edit?usp=sharing

Please comment inline on the doc, or come back to this thread if the 
conversation gets too involved (more than about 4-5 replies in a comment 
box on google docs becomes unwieldy)

Once the commenting tapers off we'll incorporate changes into the spec and 
post it as markdown in the puppet-rfc repo: 
https://github.com/puppetlabs/puppet-rfc

--eric0

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/6eed4d1f-647c-47d2-aade-044535a9c1a4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Puppet-dev] Re: Debian Package Naming Paradigm: Static Codenames only on apt.puppetlabs.com

[Rick Sherman]

This confirms the removal of stable and testing from apt.puppetlabs.com.  All 
packages are available under their codenames (jessie, wheezy, etc).

--
Rick Sherman
Release Engineering, Puppet Labs


On Wednesday, February 24, 2016 at 5:42:09 PM UTC-6, Melissa Stone wrote:
> We've stopped using the rolling release names (stable, testing) for our 
> Debian packages and instead use the direct codenames. This maps better to our 
> processes and is more maintainable over time. What this means is that the 
> stable/testing portions of the apt repos will be removed. All packages will 
> still be around under their codenames (jessie, wheezy, etc).
> 
> 
> 
> The stable/testing portions of apt.puppetlabs.com will be removed on 
> 2016-03-23.
> -- 
> 
> 
> Melissa Stone
> Release Engineer, Puppet Labs
> --

-- 
You received this message because you are subscribed to the Google Groups 
"Puppet Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to puppet-dev+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/puppet-dev/240a1f6c-8c3f-4db9-a963-8ae5712ee6e9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.