Re: [Freeipa-devel] [DESIGN] Text-based rules for CSR autogeneration using Jinja2

[Ben Lipton]

On 09/02/2016 05:04 AM, Petr Spacek wrote:

On 2.9.2016 04:19, Ben Lipton wrote:

On 07/27/2016 02:42 PM, Ben Lipton wrote:

On 07/21/2016 11:43 AM, Petr Spacek wrote:

Besides this nit,
http://www.freeipa.org/page/V4/Automatic_Certificate_Request_Generation/Mapping_Rules#Planned_implementation

sounds reasonable. I like how it prevents bad data from template-injection.

That's what I like about it, too. It does turn out to make things a little
tricky when it comes to writing rules that won't render if the data they
depend on is unavailable. (Because instead of rendering individual rules
which we can drop if they're missing data, we build one big template that
has to handle missing data correctly on its own.) I think it's probably
still worth it, though. I added this to the "Alternatives considered"
section of the above document.

By the way, I just wrote a followup blog post on this subject: describing the
challenges I've had with suppressing rules when the data isn't available, and
wondering if it's worth it. The post is here:
http://blog.benjaminlipton.com/2016/09/01/rule-suppression.html. It might be a
bit of a dense read, but I wanted to have the considerations documented at
least. As always, please let me know if there's anything I can clarify. And if
you do happen to read it and it makes you prefer one solution over the others,
I'd love to hear your opinion.

Hello Ben,

my comments are in-line (text copied from the blog post):


Conclusions

The current implementation is working ok, but the “Declaring data dependencies” 
solution is also appealing. Recording in data rules what data they depend on is 
only slightly more involved than wrapping that reference in ipa.datafield(), 
and could also be useful for other purposes.

I agree that syntax with explicit "if"s is little bit more elaborate. On the
other hand, the explicit condition is easier to read (for me) because I can
see what it is doing directly - I do not remember meaning of magical IPA macros.

I.e. I like version with explicit "if"s more.


Plus, it would get rid of the empty sections in openssl configs, as well as

some of the complex macros.

+1


The extra templating and new tags required to get rid of extra commas and 
newlines don’t seem worth it to me, unless we discover a version of openssl or 
certutil that can’t consume the current output.

I definitely agree.



Finally, I think the number of hoops needing to be jumped through to fine-tune 
the output format hint at this “template interpolation” approach being less 
successful than originally expected. While it was expected that inserting data 
rule templates into syntax rule templates and rendering the whole thing would 
produce similar results to rendering data rules first and inserting the output 
into syntax rules, that is not turning out to be the case. It might be wise to 
reconsider the simpler option - it may be easier to implement reliable jinja2 
template markup escaping than to build templates smart enough to handle any 
combination of data that’s available.

This is certainly something to think about. Personally I think that version
with explicit "if"s is easy to understandad, write, and also it has no risk of
data injection (AFAIK).

Explicit escaping is usually very error prone... but I'm not in position to
judge how user-friendly it would be when compared with other solutions. After
all, goal of the feature is to make life of an average admin easier :-)


I hope this brain-dump will help you somehow :-) Have a nice day!
It very much does, thanks for reading! I was leaning towards the 
"explicit ifs" version but didn't want to add more changes unless the 
difference was meaningful. I will finish up the patch to convert to the 
"explicit ifs" version and make it available in the PR for discussion.


I also agree that implementing escaping for a format we don't control 
makes me nervous. One thing we could do is change the format of the data 
that goes into the final formatter step. 
http://blog.benjaminlipton.com/2016/07/19/csr-generation-templating.html#two-pass-data-interpolation 
assumed that would also be jinja2 (hence the {% section %} tag), but it 
wouldn't have to be. We could create/choose a different format with 
well-defined escaping routines. But it would need to be a structured 
format, so that the openssl formatter can figure out what things go into 
sections and where. And if we're implementing a structured format for 
defining CSR contents, maybe we should skip the openssl format entirely 
and go straight to templating the actual CSR structure, as discussed in 
this thread 
https://www.redhat.com/archives/freeipa-devel/2016-August/msg00652.html 
(towards the bottom of the email). These are just some ideas for the 
future; for now I'm going to set this aside because template 
interpolation seems to be working well enough.


--
Manage your subscription for the Freeipa-devel mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-devel
Contribute t

Re: [Freeipa-devel] [DESIGN] Text-based rules for CSR autogeneration using Jinja2

[Petr Spacek]

On 2.9.2016 04:19, Ben Lipton wrote:
> On 07/27/2016 02:42 PM, Ben Lipton wrote:
>> On 07/21/2016 11:43 AM, Petr Spacek wrote:
>>> Besides this nit,
>>> http://www.freeipa.org/page/V4/Automatic_Certificate_Request_Generation/Mapping_Rules#Planned_implementation
>>>
>>> sounds reasonable. I like how it prevents bad data from template-injection.
>>
>> That's what I like about it, too. It does turn out to make things a little
>> tricky when it comes to writing rules that won't render if the data they
>> depend on is unavailable. (Because instead of rendering individual rules
>> which we can drop if they're missing data, we build one big template that
>> has to handle missing data correctly on its own.) I think it's probably
>> still worth it, though. I added this to the "Alternatives considered"
>> section of the above document.
> 
> By the way, I just wrote a followup blog post on this subject: describing the
> challenges I've had with suppressing rules when the data isn't available, and
> wondering if it's worth it. The post is here:
> http://blog.benjaminlipton.com/2016/09/01/rule-suppression.html. It might be a
> bit of a dense read, but I wanted to have the considerations documented at
> least. As always, please let me know if there's anything I can clarify. And if
> you do happen to read it and it makes you prefer one solution over the others,
> I'd love to hear your opinion.

Hello Ben,

my comments are in-line (text copied from the blog post):

> Conclusions
> 
> The current implementation is working ok, but the “Declaring data 
> dependencies” solution is also appealing. Recording in data rules what data 
> they depend on is only slightly more involved than wrapping that reference in 
> ipa.datafield(), and could also be useful for other purposes.

I agree that syntax with explicit "if"s is little bit more elaborate. On the
other hand, the explicit condition is easier to read (for me) because I can
see what it is doing directly - I do not remember meaning of magical IPA macros.

I.e. I like version with explicit "if"s more.

> Plus, it would get rid of the empty sections in openssl configs, as well as
some of the complex macros.

+1

> The extra templating and new tags required to get rid of extra commas and 
> newlines don’t seem worth it to me, unless we discover a version of openssl 
> or certutil that can’t consume the current output.

I definitely agree.


> Finally, I think the number of hoops needing to be jumped through to 
> fine-tune the output format hint at this “template interpolation” approach 
> being less successful than originally expected. While it was expected that 
> inserting data rule templates into syntax rule templates and rendering the 
> whole thing would produce similar results to rendering data rules first and 
> inserting the output into syntax rules, that is not turning out to be the 
> case. It might be wise to reconsider the simpler option - it may be easier to 
> implement reliable jinja2 template markup escaping than to build templates 
> smart enough to handle any combination of data that’s available.

This is certainly something to think about. Personally I think that version
with explicit "if"s is easy to understandad, write, and also it has no risk of
data injection (AFAIK).

Explicit escaping is usually very error prone... but I'm not in position to
judge how user-friendly it would be when compared with other solutions. After
all, goal of the feature is to make life of an average admin easier :-)


I hope this brain-dump will help you somehow :-) Have a nice day!

-- 
Petr^2 Spacek

P.S. I will not be available in next two weeks, sorry!

-- 
Manage your subscription for the Freeipa-devel mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-devel
Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code

Re: [Freeipa-devel] [DESIGN] Text-based rules for CSR autogeneration using Jinja2

[Ben Lipton]

On 07/27/2016 02:42 PM, Ben Lipton wrote:

On 07/21/2016 11:43 AM, Petr Spacek wrote:

Besides this nit,
http://www.freeipa.org/page/V4/Automatic_Certificate_Request_Generation/Mapping_Rules#Planned_implementation
sounds reasonable. I like how it prevents bad data from template-injection.


That's what I like about it, too. It does turn out to make things a 
little tricky when it comes to writing rules that won't render if the 
data they depend on is unavailable. (Because instead of rendering 
individual rules which we can drop if they're missing data, we build 
one big template that has to handle missing data correctly on its 
own.) I think it's probably still worth it, though. I added this to 
the "Alternatives considered" section of the above document.


By the way, I just wrote a followup blog post on this subject: 
describing the challenges I've had with suppressing rules when the data 
isn't available, and wondering if it's worth it. The post is here: 
http://blog.benjaminlipton.com/2016/09/01/rule-suppression.html. It 
might be a bit of a dense read, but I wanted to have the considerations 
documented at least. As always, please let me know if there's anything I 
can clarify. And if you do happen to read it and it makes you prefer one 
solution over the others, I'd love to hear your opinion.


Ben

--
Manage your subscription for the Freeipa-devel mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-devel
Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code

Re: [Freeipa-devel] [DESIGN] Text-based rules for CSR autogeneration using Jinja2

[Ben Lipton]

On 07/21/2016 11:43 AM, Petr Spacek wrote:

On 20.7.2016 19:25, Ben Lipton wrote:

On 07/20/2016 12:21 PM, Simo Sorce wrote:

On Wed, 2016-07-20 at 12:14 -0400, Ben Lipton wrote:

On 07/20/2016 10:37 AM, Simo Sorce wrote:

On Wed, 2016-07-20 at 10:17 -0400, Ben Lipton wrote:

On 07/20/2016 06:27 AM, Simo Sorce wrote:

On Tue, 2016-07-19 at 16:20 -0400, Ben Lipton wrote:

Hi,

I have updated the design page
http://www.freeipa.org/page/V4/Automatic_Certificate_Request_
Gene
rati
on/Mapping_Rules
with my plan for implementing user-configurable rules for
mapping
IPA
data into certificate requests. In brief: we will use Jinja2
for
templating. Data rules (which map individual data items) and
syntax
rules (which group them into certificate fields) will both be
snippets
of Jinja2 markup. The formatting process will be as follows:

I've finally found some time to read the sub-page Mapping_Rules and for me it
is kind of hard to follow. It would not be understandable without this e-mail
and the blog posts (BTW the blog articles are among best I have seen!).

Most importantly, the explanations in brackets above ["Data rules (which map
individual data items) and (which group them into certificate fields)"] are
missing in the wiki page itself :-)

Could you fold relevant parts of the e-mails and blogs back into the wiki page
so it is self-contained?


Very good point. I may have been assuming knowledge of the whole design 
when reading this document, which doesn't make sense. I did some 
cleanup, plus added some more detail about how things work in the 
implementation I just sent out for review. I hope that will clarify 
things somewhat.

Besides this nit,
http://www.freeipa.org/page/V4/Automatic_Certificate_Request_Generation/Mapping_Rules#Planned_implementation
sounds reasonable. I like how it prevents bad data from template-injection.


That's what I like about it, too. It does turn out to make things a 
little tricky when it comes to writing rules that won't render if the 
data they depend on is unavailable. (Because instead of rendering 
individual rules which we can drop if they're missing data, we build one 
big template that has to handle missing data correctly on its own.) I 
think it's probably still worth it, though. I added this to the 
"Alternatives considered" section of the above document.

Regarding
http://www.freeipa.org/page/V4/Automatic_Certificate_Request_Generation/Schema#Option_A
, I prefer Option A with separate object for each helper. It is somehow
cleaner and it might be useful to use distinct object classes for each helper 
etc.


+1, I think option B was a bit of premature optimization.



API for ipa cert-get-requestdata sounds good.
API for ipa cert-request makes sense to me as well.

In any case I would recommend you to consult API design with Jan Cholasta
  - he is our API custodian.


BTW I very much like "Alternatives considered" sections, we should have this
for each design!

Good work, I really like the dutiful analysis!

   

Thanks for the feedback, and the kind words!

Ben
-- 
Manage your subscription for the Freeipa-devel mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-devel
Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code

Re: [Freeipa-devel] [DESIGN] Text-based rules for CSR autogeneration using Jinja2

[Jan Cholasta]

On 21.7.2016 17:43, Petr Spacek wrote:

On 20.7.2016 19:25, Ben Lipton wrote:

On 07/20/2016 12:21 PM, Simo Sorce wrote:

On Wed, 2016-07-20 at 12:14 -0400, Ben Lipton wrote:

On 07/20/2016 10:37 AM, Simo Sorce wrote:

On Wed, 2016-07-20 at 10:17 -0400, Ben Lipton wrote:

On 07/20/2016 06:27 AM, Simo Sorce wrote:

On Tue, 2016-07-19 at 16:20 -0400, Ben Lipton wrote:

Hi,

I have updated the design page
http://www.freeipa.org/page/V4/Automatic_Certificate_Request_
Gene
rati
on/Mapping_Rules
with my plan for implementing user-configurable rules for
mapping
IPA
data into certificate requests. In brief: we will use Jinja2
for
templating. Data rules (which map individual data items) and
syntax
rules (which group them into certificate fields) will both be
snippets
of Jinja2 markup. The formatting process will be as follows:


I've finally found some time to read the sub-page Mapping_Rules and for me it
is kind of hard to follow. It would not be understandable without this e-mail
and the blog posts (BTW the blog articles are among best I have seen!).

Most importantly, the explanations in brackets above ["Data rules (which map
individual data items) and (which group them into certificate fields)"] are
missing in the wiki page itself :-)

Could you fold relevant parts of the e-mails and blogs back into the wiki page
so it is self-contained?

Besides this nit,
http://www.freeipa.org/page/V4/Automatic_Certificate_Request_Generation/Mapping_Rules#Planned_implementation
sounds reasonable. I like how it prevents bad data from template-injection.

Regarding
http://www.freeipa.org/page/V4/Automatic_Certificate_Request_Generation/Schema#Option_A
, I prefer Option A with separate object for each helper. It is somehow
cleaner and it might be useful to use distinct object classes for each helper 
etc.


API for ipa cert-get-requestdata sounds good.
API for ipa cert-request makes sense to me as well.

In any case I would recommend you to consult API design with Jan Cholasta
 - he is our API custodian.


See the other thread for my comments: 
.





BTW I very much like "Alternatives considered" sections, we should have this
for each design!

Good work, I really like the dutiful analysis!




1. Syntax rules will be rendered using Jinja2. Data rules
(rule
text,
not rendered) will be passed as the datarules attribute.
2. Rendered syntax rules will be processed by the Formatter
class
for
the selected CSR generation helper (e.g. openssl or
certutil).
The
formatter combines these partial rules into a full template
for
the
config.
3. The template will be rendered using Jinja2. Relevant data
from
the
IPA database will be available in the context for this
rendering.
4. The final rendered template will be returned to the
caller,
labeled
with its function (e.g. a command line or a config file).

Are there any comments or objections to this approach? Here's
an
example
to show what it might look like in practice.

Example data rules:
email={{subject.email}}
O={{config.ipacertificatesubjectbase}}\nCN={{subject.username
}}

Example syntax rule:
subjectAltName=@{% section %}{{datarules|join('\n')}}{%
endsection %}

Example composed config template:
[ req ]
prompt = no
encrypt_key = no

distinguished_name = {% section
%}O={{config.ipacertificatesubjectbase}}
CN={{subject.username}}{% endsection %}

req_extensions = exts

[ exts ]
subjectAltName=@{% section %}email={{subject.email}}{%
endsection
%}

There's a lot more information about the thinking behind this
at
http://blog.benjaminlipton.com/2016/07/19/csr-generation-temp
lati
ng.h
tml
if you're interested, as well.

Nice work Ben,
it's been really nice to be able to follow your notes on the
blog
post,
one question remains lingering in my head, why jinja2 ?
I know that engine relatively well as I used it in ipsilon, so
I am
not
questioning the choice just asking why specifically jinja2 and
not
something else, potentially language agnostic.

Simo.

Honestly, my reasoning didn't go very far beyond that it seems to
be
widely used and is compatible with python, which is the language
where
the implementation is taking place (in the IPA RPC server). I
thought
about using the built-in python format strings or creating a
simple
domain-specific language, but the likelihood of wanting the
built-in
text processing features (join, replace, maybe even for loops)
seemed
high, and I didn't want to reimplement those features.

Will the additional package dependency be a problem?

I am more concerned a out the ability to process the data (which I
guess is stored in LDAP) by another client, or in the CLI.
Other than that the dependency does not concern me too much
provided
jinja2 templating is stable and has some guarantee that it will be
supportable long term.

If that is not guaranteed it is a problem, we cannot easily swap
out
one language for another once data is stored and used by the
server.
So the most important consideration for 

Re: [Freeipa-devel] [DESIGN] Text-based rules for CSR autogeneration using Jinja2

[Petr Spacek]

On 20.7.2016 19:25, Ben Lipton wrote:
> On 07/20/2016 12:21 PM, Simo Sorce wrote:
>> On Wed, 2016-07-20 at 12:14 -0400, Ben Lipton wrote:
>>> On 07/20/2016 10:37 AM, Simo Sorce wrote:
 On Wed, 2016-07-20 at 10:17 -0400, Ben Lipton wrote:
> On 07/20/2016 06:27 AM, Simo Sorce wrote:
>> On Tue, 2016-07-19 at 16:20 -0400, Ben Lipton wrote:
>>> Hi,
>>>
>>> I have updated the design page
>>> http://www.freeipa.org/page/V4/Automatic_Certificate_Request_
>>> Gene
>>> rati
>>> on/Mapping_Rules
>>> with my plan for implementing user-configurable rules for
>>> mapping
>>> IPA
>>> data into certificate requests. In brief: we will use Jinja2
>>> for
>>> templating. Data rules (which map individual data items) and
>>> syntax
>>> rules (which group them into certificate fields) will both be
>>> snippets
>>> of Jinja2 markup. The formatting process will be as follows:

I've finally found some time to read the sub-page Mapping_Rules and for me it
is kind of hard to follow. It would not be understandable without this e-mail
and the blog posts (BTW the blog articles are among best I have seen!).

Most importantly, the explanations in brackets above ["Data rules (which map
individual data items) and (which group them into certificate fields)"] are
missing in the wiki page itself :-)

Could you fold relevant parts of the e-mails and blogs back into the wiki page
so it is self-contained?

Besides this nit,
http://www.freeipa.org/page/V4/Automatic_Certificate_Request_Generation/Mapping_Rules#Planned_implementation
sounds reasonable. I like how it prevents bad data from template-injection.

Regarding
http://www.freeipa.org/page/V4/Automatic_Certificate_Request_Generation/Schema#Option_A
, I prefer Option A with separate object for each helper. It is somehow
cleaner and it might be useful to use distinct object classes for each helper 
etc.


API for ipa cert-get-requestdata sounds good.
API for ipa cert-request makes sense to me as well.

In any case I would recommend you to consult API design with Jan Cholasta
 - he is our API custodian.


BTW I very much like "Alternatives considered" sections, we should have this
for each design!

Good work, I really like the dutiful analysis!



>>> 1. Syntax rules will be rendered using Jinja2. Data rules
>>> (rule
>>> text,
>>> not rendered) will be passed as the datarules attribute.
>>> 2. Rendered syntax rules will be processed by the Formatter
>>> class
>>> for
>>> the selected CSR generation helper (e.g. openssl or
>>> certutil).
>>> The
>>> formatter combines these partial rules into a full template
>>> for
>>> the
>>> config.
>>> 3. The template will be rendered using Jinja2. Relevant data
>>> from
>>> the
>>> IPA database will be available in the context for this
>>> rendering.
>>> 4. The final rendered template will be returned to the
>>> caller,
>>> labeled
>>> with its function (e.g. a command line or a config file).
>>>
>>> Are there any comments or objections to this approach? Here's
>>> an
>>> example
>>> to show what it might look like in practice.
>>>
>>> Example data rules:
>>> email={{subject.email}}
>>> O={{config.ipacertificatesubjectbase}}\nCN={{subject.username
>>> }}
>>>
>>> Example syntax rule:
>>> subjectAltName=@{% section %}{{datarules|join('\n')}}{%
>>> endsection %}
>>>
>>> Example composed config template:
>>> [ req ]
>>> prompt = no
>>> encrypt_key = no
>>>
>>> distinguished_name = {% section
>>> %}O={{config.ipacertificatesubjectbase}}
>>> CN={{subject.username}}{% endsection %}
>>>
>>> req_extensions = exts
>>>
>>> [ exts ]
>>> subjectAltName=@{% section %}email={{subject.email}}{%
>>> endsection
>>> %}
>>>
>>> There's a lot more information about the thinking behind this
>>> at
>>> http://blog.benjaminlipton.com/2016/07/19/csr-generation-temp
>>> lati
>>> ng.h
>>> tml
>>> if you're interested, as well.
>> Nice work Ben,
>> it's been really nice to be able to follow your notes on the
>> blog
>> post,
>> one question remains lingering in my head, why jinja2 ?
>> I know that engine relatively well as I used it in ipsilon, so
>> I am
>> not
>> questioning the choice just asking why specifically jinja2 and
>> not
>> something else, potentially language agnostic.
>>
>> Simo.
> Honestly, my reasoning didn't go very far beyond that it seems to
> be
> widely used and is compatible with python, which is the language
> where
> the implementation is taking place (in the IPA RPC server). I
> thought
> about using the built-in python format strings or creating a
> simple
> domain-specific language, but the likelihood of wanting the
> built-in
> text processing

Re: [Freeipa-devel] [DESIGN] Text-based rules for CSR autogeneration using Jinja2

[Ben Lipton]

On 07/20/2016 12:21 PM, Simo Sorce wrote:

On Wed, 2016-07-20 at 12:14 -0400, Ben Lipton wrote:

On 07/20/2016 10:37 AM, Simo Sorce wrote:

On Wed, 2016-07-20 at 10:17 -0400, Ben Lipton wrote:

On 07/20/2016 06:27 AM, Simo Sorce wrote:

On Tue, 2016-07-19 at 16:20 -0400, Ben Lipton wrote:

Hi,

I have updated the design page
http://www.freeipa.org/page/V4/Automatic_Certificate_Request_
Gene
rati
on/Mapping_Rules
with my plan for implementing user-configurable rules for
mapping
IPA
data into certificate requests. In brief: we will use Jinja2
for
templating. Data rules (which map individual data items) and
syntax
rules (which group them into certificate fields) will both be
snippets
of Jinja2 markup. The formatting process will be as follows:
1. Syntax rules will be rendered using Jinja2. Data rules
(rule
text,
not rendered) will be passed as the datarules attribute.
2. Rendered syntax rules will be processed by the Formatter
class
for
the selected CSR generation helper (e.g. openssl or
certutil).
The
formatter combines these partial rules into a full template
for
the
config.
3. The template will be rendered using Jinja2. Relevant data
from
the
IPA database will be available in the context for this
rendering.
4. The final rendered template will be returned to the
caller,
labeled
with its function (e.g. a command line or a config file).

Are there any comments or objections to this approach? Here's
an
example
to show what it might look like in practice.

Example data rules:
email={{subject.email}}
O={{config.ipacertificatesubjectbase}}\nCN={{subject.username
}}

Example syntax rule:
subjectAltName=@{% section %}{{datarules|join('\n')}}{%
endsection %}

Example composed config template:
[ req ]
prompt = no
encrypt_key = no

distinguished_name = {% section
%}O={{config.ipacertificatesubjectbase}}
CN={{subject.username}}{% endsection %}

req_extensions = exts

[ exts ]
subjectAltName=@{% section %}email={{subject.email}}{%
endsection
%}

There's a lot more information about the thinking behind this
at
http://blog.benjaminlipton.com/2016/07/19/csr-generation-temp
lati
ng.h
tml
if you're interested, as well.

Nice work Ben,
it's been really nice to be able to follow your notes on the
blog
post,
one question remains lingering in my head, why jinja2 ?
I know that engine relatively well as I used it in ipsilon, so
I am
not
questioning the choice just asking why specifically jinja2 and
not
something else, potentially language agnostic.

Simo.

Honestly, my reasoning didn't go very far beyond that it seems to
be
widely used and is compatible with python, which is the language
where
the implementation is taking place (in the IPA RPC server). I
thought
about using the built-in python format strings or creating a
simple
domain-specific language, but the likelihood of wanting the
built-in
text processing features (join, replace, maybe even for loops)
seemed
high, and I didn't want to reimplement those features.

Will the additional package dependency be a problem?

I am more concerned a out the ability to process the data (which I
guess is stored in LDAP) by another client, or in the CLI.
Other than that the dependency does not concern me too much
provided
jinja2 templating is stable and has some guarantee that it will be
supportable long term.

If that is not guaranteed it is a problem, we cannot easily swap
out
one language for another once data is stored and used by the
server.
So the most important consideration for me is whether we are
locking
ourselves into something that will be hard to deal with later or
not.

Should the jinja2 project fail by the wayside next year would we be
able to easily replace it with another engine without changing the
templates as stored ?

Simo.


Ah, ok, I understand the concern. For now, the plan is that the
server
will do all the text processing, so I don't really forsee a need for
any
other client to read the mapping rules from LDAP. However, it's true
that templates written in jinja2 would probably need at least minor
changes to be compatible with another templating engine. (Same goes
for
any other choice - a lot of these engines seem to have very similar,
but
not exactly compatible, syntax). I don't really know how to judge
the
long-term viability of the jinja2 project, though it seems to be
recognized by lots of projects (ansible[1], openstack[2], flask[3],
even
django[4] which has its own templating engine).

In any case, if the team prefers it, I'd be comfortable going with a
more minimal DSL that only has the features we know we need. It
might
slightly limit the types of certs that can be generated, but that can
be
iterated on. But it would be another thing to design, build and
maintain. Let me know what you think.

I am ok using jinja2 as long as we realize we may be on the hook for
maintaining it ourselves in the long term. It's probably easier to do
that than to write our own anyway.

Simo.
It might also be worth pointing out that although the examples in the 
document are ba

Re: [Freeipa-devel] [DESIGN] Text-based rules for CSR autogeneration using Jinja2

[Simo Sorce]

On Wed, 2016-07-20 at 12:14 -0400, Ben Lipton wrote:
> On 07/20/2016 10:37 AM, Simo Sorce wrote:
> > 
> > On Wed, 2016-07-20 at 10:17 -0400, Ben Lipton wrote:
> > > 
> > > On 07/20/2016 06:27 AM, Simo Sorce wrote:
> > > > 
> > > > On Tue, 2016-07-19 at 16:20 -0400, Ben Lipton wrote:
> > > > > 
> > > > > Hi,
> > > > > 
> > > > > I have updated the design page
> > > > > http://www.freeipa.org/page/V4/Automatic_Certificate_Request_
> > > > > Gene
> > > > > rati
> > > > > on/Mapping_Rules
> > > > > with my plan for implementing user-configurable rules for
> > > > > mapping
> > > > > IPA
> > > > > data into certificate requests. In brief: we will use Jinja2
> > > > > for
> > > > > templating. Data rules (which map individual data items) and
> > > > > syntax
> > > > > rules (which group them into certificate fields) will both be
> > > > > snippets
> > > > > of Jinja2 markup. The formatting process will be as follows:
> > > > > 1. Syntax rules will be rendered using Jinja2. Data rules
> > > > > (rule
> > > > > text,
> > > > > not rendered) will be passed as the datarules attribute.
> > > > > 2. Rendered syntax rules will be processed by the Formatter
> > > > > class
> > > > > for
> > > > > the selected CSR generation helper (e.g. openssl or
> > > > > certutil).
> > > > > The
> > > > > formatter combines these partial rules into a full template
> > > > > for
> > > > > the
> > > > > config.
> > > > > 3. The template will be rendered using Jinja2. Relevant data
> > > > > from
> > > > > the
> > > > > IPA database will be available in the context for this
> > > > > rendering.
> > > > > 4. The final rendered template will be returned to the
> > > > > caller,
> > > > > labeled
> > > > > with its function (e.g. a command line or a config file).
> > > > > 
> > > > > Are there any comments or objections to this approach? Here's
> > > > > an
> > > > > example
> > > > > to show what it might look like in practice.
> > > > > 
> > > > > Example data rules:
> > > > > email={{subject.email}}
> > > > > O={{config.ipacertificatesubjectbase}}\nCN={{subject.username
> > > > > }}
> > > > > 
> > > > > Example syntax rule:
> > > > > subjectAltName=@{% section %}{{datarules|join('\n')}}{%
> > > > > endsection %}
> > > > > 
> > > > > Example composed config template:
> > > > > [ req ]
> > > > > prompt = no
> > > > > encrypt_key = no
> > > > > 
> > > > > distinguished_name = {% section
> > > > > %}O={{config.ipacertificatesubjectbase}}
> > > > > CN={{subject.username}}{% endsection %}
> > > > > 
> > > > > req_extensions = exts
> > > > > 
> > > > > [ exts ]
> > > > > subjectAltName=@{% section %}email={{subject.email}}{%
> > > > > endsection
> > > > > %}
> > > > > 
> > > > > There's a lot more information about the thinking behind this
> > > > > at
> > > > > http://blog.benjaminlipton.com/2016/07/19/csr-generation-temp
> > > > > lati
> > > > > ng.h
> > > > > tml
> > > > > if you're interested, as well.
> > > > Nice work Ben,
> > > > it's been really nice to be able to follow your notes on the
> > > > blog
> > > > post,
> > > > one question remains lingering in my head, why jinja2 ?
> > > > I know that engine relatively well as I used it in ipsilon, so
> > > > I am
> > > > not
> > > > questioning the choice just asking why specifically jinja2 and
> > > > not
> > > > something else, potentially language agnostic.
> > > > 
> > > > Simo.
> > > Honestly, my reasoning didn't go very far beyond that it seems to
> > > be
> > > widely used and is compatible with python, which is the language
> > > where
> > > the implementation is taking place (in the IPA RPC server). I
> > > thought
> > > about using the built-in python format strings or creating a
> > > simple
> > > domain-specific language, but the likelihood of wanting the
> > > built-in
> > > text processing features (join, replace, maybe even for loops)
> > > seemed
> > > high, and I didn't want to reimplement those features.
> > > 
> > > Will the additional package dependency be a problem?
> > I am more concerned a out the ability to process the data (which I
> > guess is stored in LDAP) by another client, or in the CLI.
> > Other than that the dependency does not concern me too much
> > provided
> > jinja2 templating is stable and has some guarantee that it will be
> > supportable long term.
> > 
> > If that is not guaranteed it is a problem, we cannot easily swap
> > out
> > one language for another once data is stored and used by the
> > server.
> > So the most important consideration for me is whether we are
> > locking
> > ourselves into something that will be hard to deal with later or
> > not.
> > 
> > Should the jinja2 project fail by the wayside next year would we be
> > able to easily replace it with another engine without changing the
> > templates as stored ?
> > 
> > Simo.
> > 
> Ah, ok, I understand the concern. For now, the plan is that the
> server 
> will do all the text processing, so I don't really forsee a need for
> any 
> other client to read the mapping rules from LDAP

Re: [Freeipa-devel] [DESIGN] Text-based rules for CSR autogeneration using Jinja2

[Ben Lipton]

On 07/20/2016 10:37 AM, Simo Sorce wrote:

On Wed, 2016-07-20 at 10:17 -0400, Ben Lipton wrote:

On 07/20/2016 06:27 AM, Simo Sorce wrote:

On Tue, 2016-07-19 at 16:20 -0400, Ben Lipton wrote:

Hi,

I have updated the design page
http://www.freeipa.org/page/V4/Automatic_Certificate_Request_Gene
rati
on/Mapping_Rules
with my plan for implementing user-configurable rules for mapping
IPA
data into certificate requests. In brief: we will use Jinja2 for
templating. Data rules (which map individual data items) and
syntax
rules (which group them into certificate fields) will both be
snippets
of Jinja2 markup. The formatting process will be as follows:
1. Syntax rules will be rendered using Jinja2. Data rules (rule
text,
not rendered) will be passed as the datarules attribute.
2. Rendered syntax rules will be processed by the Formatter class
for
the selected CSR generation helper (e.g. openssl or certutil).
The
formatter combines these partial rules into a full template for
the
config.
3. The template will be rendered using Jinja2. Relevant data from
the
IPA database will be available in the context for this rendering.
4. The final rendered template will be returned to the caller,
labeled
with its function (e.g. a command line or a config file).

Are there any comments or objections to this approach? Here's an
example
to show what it might look like in practice.

Example data rules:
email={{subject.email}}
O={{config.ipacertificatesubjectbase}}\nCN={{subject.username}}

Example syntax rule:
subjectAltName=@{% section %}{{datarules|join('\n')}}{%
endsection %}

Example composed config template:
[ req ]
prompt = no
encrypt_key = no

distinguished_name = {% section
%}O={{config.ipacertificatesubjectbase}}
CN={{subject.username}}{% endsection %}

req_extensions = exts

[ exts ]
subjectAltName=@{% section %}email={{subject.email}}{% endsection
%}

There's a lot more information about the thinking behind this at
http://blog.benjaminlipton.com/2016/07/19/csr-generation-templati
ng.h
tml
if you're interested, as well.

Nice work Ben,
it's been really nice to be able to follow your notes on the blog
post,
one question remains lingering in my head, why jinja2 ?
I know that engine relatively well as I used it in ipsilon, so I am
not
questioning the choice just asking why specifically jinja2 and not
something else, potentially language agnostic.

Simo.

Honestly, my reasoning didn't go very far beyond that it seems to be
widely used and is compatible with python, which is the language
where
the implementation is taking place (in the IPA RPC server). I
thought
about using the built-in python format strings or creating a simple
domain-specific language, but the likelihood of wanting the built-in
text processing features (join, replace, maybe even for loops)
seemed
high, and I didn't want to reimplement those features.

Will the additional package dependency be a problem?

I am more concerned a out the ability to process the data (which I
guess is stored in LDAP) by another client, or in the CLI.
Other than that the dependency does not concern me too much provided
jinja2 templating is stable and has some guarantee that it will be
supportable long term.

If that is not guaranteed it is a problem, we cannot easily swap out
one language for another once data is stored and used by the server.
So the most important consideration for me is whether we are locking
ourselves into something that will be hard to deal with later or not.

Should the jinja2 project fail by the wayside next year would we be
able to easily replace it with another engine without changing the
templates as stored ?

Simo.

Ah, ok, I understand the concern. For now, the plan is that the server 
will do all the text processing, so I don't really forsee a need for any 
other client to read the mapping rules from LDAP. However, it's true 
that templates written in jinja2 would probably need at least minor 
changes to be compatible with another templating engine. (Same goes for 
any other choice - a lot of these engines seem to have very similar, but 
not exactly compatible, syntax). I don't really know how to judge the 
long-term viability of the jinja2 project, though it seems to be 
recognized by lots of projects (ansible[1], openstack[2], flask[3], even 
django[4] which has its own templating engine).


In any case, if the team prefers it, I'd be comfortable going with a 
more minimal DSL that only has the features we know we need. It might 
slightly limit the types of certs that can be generated, but that can be 
iterated on. But it would be another thing to design, build and 
maintain. Let me know what you think.


Ben

[1] 
http://docs.ansible.com/ansible/playbooks_variables.html#using-variables-about-jinja2

[2] http://lists.openstack.org/pipermail/openstack-dev/2013-July/012016.html
[3] http://flask.pocoo.org/docs/0.11/templating/
[4] 
https://docs.djangoproject.com/en/1.9/topics/templates/#django.template.backends.jinja2.Jinja2


--
Manage your subscription 

Re: [Freeipa-devel] [DESIGN] Text-based rules for CSR autogeneration using Jinja2

[Simo Sorce]

On Wed, 2016-07-20 at 10:17 -0400, Ben Lipton wrote:
> On 07/20/2016 06:27 AM, Simo Sorce wrote:
> > 
> > On Tue, 2016-07-19 at 16:20 -0400, Ben Lipton wrote:
> > > 
> > > Hi,
> > > 
> > > I have updated the design page
> > > http://www.freeipa.org/page/V4/Automatic_Certificate_Request_Gene
> > > rati
> > > on/Mapping_Rules
> > > with my plan for implementing user-configurable rules for mapping
> > > IPA
> > > data into certificate requests. In brief: we will use Jinja2 for
> > > templating. Data rules (which map individual data items) and
> > > syntax
> > > rules (which group them into certificate fields) will both be
> > > snippets
> > > of Jinja2 markup. The formatting process will be as follows:
> > > 1. Syntax rules will be rendered using Jinja2. Data rules (rule
> > > text,
> > > not rendered) will be passed as the datarules attribute.
> > > 2. Rendered syntax rules will be processed by the Formatter class
> > > for
> > > the selected CSR generation helper (e.g. openssl or certutil).
> > > The
> > > formatter combines these partial rules into a full template for
> > > the
> > > config.
> > > 3. The template will be rendered using Jinja2. Relevant data from
> > > the
> > > IPA database will be available in the context for this rendering.
> > > 4. The final rendered template will be returned to the caller,
> > > labeled
> > > with its function (e.g. a command line or a config file).
> > > 
> > > Are there any comments or objections to this approach? Here's an
> > > example
> > > to show what it might look like in practice.
> > > 
> > > Example data rules:
> > > email={{subject.email}}
> > > O={{config.ipacertificatesubjectbase}}\nCN={{subject.username}}
> > > 
> > > Example syntax rule:
> > > subjectAltName=@{% section %}{{datarules|join('\n')}}{%
> > > endsection %}
> > > 
> > > Example composed config template:
> > > [ req ]
> > > prompt = no
> > > encrypt_key = no
> > > 
> > > distinguished_name = {% section
> > > %}O={{config.ipacertificatesubjectbase}}
> > > CN={{subject.username}}{% endsection %}
> > > 
> > > req_extensions = exts
> > > 
> > > [ exts ]
> > > subjectAltName=@{% section %}email={{subject.email}}{% endsection
> > > %}
> > > 
> > > There's a lot more information about the thinking behind this at
> > > http://blog.benjaminlipton.com/2016/07/19/csr-generation-templati
> > > ng.h
> > > tml
> > > if you're interested, as well.
> > Nice work Ben,
> > it's been really nice to be able to follow your notes on the blog
> > post,
> > one question remains lingering in my head, why jinja2 ?
> > I know that engine relatively well as I used it in ipsilon, so I am
> > not
> > questioning the choice just asking why specifically jinja2 and not
> > something else, potentially language agnostic.
> > 
> > Simo.
> Honestly, my reasoning didn't go very far beyond that it seems to be 
> widely used and is compatible with python, which is the language
> where 
> the implementation is taking place (in the IPA RPC server). I
> thought 
> about using the built-in python format strings or creating a simple 
> domain-specific language, but the likelihood of wanting the built-in 
> text processing features (join, replace, maybe even for loops)
> seemed 
> high, and I didn't want to reimplement those features.
> 
> Will the additional package dependency be a problem?

I am more concerned a out the ability to process the data (which I
guess is stored in LDAP) by another client, or in the CLI.
Other than that the dependency does not concern me too much provided
jinja2 templating is stable and has some guarantee that it will be
supportable long term.

If that is not guaranteed it is a problem, we cannot easily swap out
one language for another once data is stored and used by the server.
So the most important consideration for me is whether we are locking
ourselves into something that will be hard to deal with later or not.

Should the jinja2 project fail by the wayside next year would we be
able to easily replace it with another engine without changing the
templates as stored ?

Simo.

-- 
Manage your subscription for the Freeipa-devel mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-devel
Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code

Re: [Freeipa-devel] [DESIGN] Text-based rules for CSR autogeneration using Jinja2

[Ben Lipton]

On 07/20/2016 06:27 AM, Simo Sorce wrote:

On Tue, 2016-07-19 at 16:20 -0400, Ben Lipton wrote:

Hi,

I have updated the design page
http://www.freeipa.org/page/V4/Automatic_Certificate_Request_Generati
on/Mapping_Rules
with my plan for implementing user-configurable rules for mapping
IPA
data into certificate requests. In brief: we will use Jinja2 for
templating. Data rules (which map individual data items) and syntax
rules (which group them into certificate fields) will both be
snippets
of Jinja2 markup. The formatting process will be as follows:
1. Syntax rules will be rendered using Jinja2. Data rules (rule
text,
not rendered) will be passed as the datarules attribute.
2. Rendered syntax rules will be processed by the Formatter class
for
the selected CSR generation helper (e.g. openssl or certutil). The
formatter combines these partial rules into a full template for the
config.
3. The template will be rendered using Jinja2. Relevant data from
the
IPA database will be available in the context for this rendering.
4. The final rendered template will be returned to the caller,
labeled
with its function (e.g. a command line or a config file).

Are there any comments or objections to this approach? Here's an
example
to show what it might look like in practice.

Example data rules:
email={{subject.email}}
O={{config.ipacertificatesubjectbase}}\nCN={{subject.username}}

Example syntax rule:
subjectAltName=@{% section %}{{datarules|join('\n')}}{% endsection %}

Example composed config template:
[ req ]
prompt = no
encrypt_key = no

distinguished_name = {% section
%}O={{config.ipacertificatesubjectbase}}
CN={{subject.username}}{% endsection %}

req_extensions = exts

[ exts ]
subjectAltName=@{% section %}email={{subject.email}}{% endsection %}

There's a lot more information about the thinking behind this at
http://blog.benjaminlipton.com/2016/07/19/csr-generation-templating.h
tml
if you're interested, as well.

Nice work Ben,
it's been really nice to be able to follow your notes on the blog post,
one question remains lingering in my head, why jinja2 ?
I know that engine relatively well as I used it in ipsilon, so I am not
questioning the choice just asking why specifically jinja2 and not
something else, potentially language agnostic.

Simo.
Honestly, my reasoning didn't go very far beyond that it seems to be 
widely used and is compatible with python, which is the language where 
the implementation is taking place (in the IPA RPC server). I thought 
about using the built-in python format strings or creating a simple 
domain-specific language, but the likelihood of wanting the built-in 
text processing features (join, replace, maybe even for loops) seemed 
high, and I didn't want to reimplement those features.


Will the additional package dependency be a problem?

Ben

--
Manage your subscription for the Freeipa-devel mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-devel
Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code

Re: [Freeipa-devel] [DESIGN] Text-based rules for CSR autogeneration using Jinja2

[Simo Sorce]

On Tue, 2016-07-19 at 16:20 -0400, Ben Lipton wrote:
> Hi,
> 
> I have updated the design page 
> http://www.freeipa.org/page/V4/Automatic_Certificate_Request_Generati
> on/Mapping_Rules 
> with my plan for implementing user-configurable rules for mapping
> IPA 
> data into certificate requests. In brief: we will use Jinja2 for 
> templating. Data rules (which map individual data items) and syntax 
> rules (which group them into certificate fields) will both be
> snippets 
> of Jinja2 markup. The formatting process will be as follows:
> 1. Syntax rules will be rendered using Jinja2. Data rules (rule
> text, 
> not rendered) will be passed as the datarules attribute.
> 2. Rendered syntax rules will be processed by the Formatter class
> for 
> the selected CSR generation helper (e.g. openssl or certutil). The 
> formatter combines these partial rules into a full template for the
> config.
> 3. The template will be rendered using Jinja2. Relevant data from
> the 
> IPA database will be available in the context for this rendering.
> 4. The final rendered template will be returned to the caller,
> labeled 
> with its function (e.g. a command line or a config file).
> 
> Are there any comments or objections to this approach? Here's an
> example 
> to show what it might look like in practice.
> 
> Example data rules:
> email={{subject.email}}
> O={{config.ipacertificatesubjectbase}}\nCN={{subject.username}}
> 
> Example syntax rule:
> subjectAltName=@{% section %}{{datarules|join('\n')}}{% endsection %}
> 
> Example composed config template:
> [ req ]
> prompt = no
> encrypt_key = no
> 
> distinguished_name = {% section
> %}O={{config.ipacertificatesubjectbase}}
> CN={{subject.username}}{% endsection %}
> 
> req_extensions = exts
> 
> [ exts ]
> subjectAltName=@{% section %}email={{subject.email}}{% endsection %}
> 
> There's a lot more information about the thinking behind this at 
> http://blog.benjaminlipton.com/2016/07/19/csr-generation-templating.h
> tml 
> if you're interested, as well.

Nice work Ben,
it's been really nice to be able to follow your notes on the blog post,
one question remains lingering in my head, why jinja2 ?
I know that engine relatively well as I used it in ipsilon, so I am not
questioning the choice just asking why specifically jinja2 and not
something else, potentially language agnostic.

Simo.

-- 
Manage your subscription for the Freeipa-devel mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-devel
Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code

[Freeipa-devel] [DESIGN] Text-based rules for CSR autogeneration using Jinja2

[Ben Lipton]

Hi,

I have updated the design page 
http://www.freeipa.org/page/V4/Automatic_Certificate_Request_Generation/Mapping_Rules 
with my plan for implementing user-configurable rules for mapping IPA 
data into certificate requests. In brief: we will use Jinja2 for 
templating. Data rules (which map individual data items) and syntax 
rules (which group them into certificate fields) will both be snippets 
of Jinja2 markup. The formatting process will be as follows:
1. Syntax rules will be rendered using Jinja2. Data rules (rule text, 
not rendered) will be passed as the datarules attribute.
2. Rendered syntax rules will be processed by the Formatter class for 
the selected CSR generation helper (e.g. openssl or certutil). The 
formatter combines these partial rules into a full template for the config.
3. The template will be rendered using Jinja2. Relevant data from the 
IPA database will be available in the context for this rendering.
4. The final rendered template will be returned to the caller, labeled 
with its function (e.g. a command line or a config file).


Are there any comments or objections to this approach? Here's an example 
to show what it might look like in practice.


Example data rules:
email={{subject.email}}
O={{config.ipacertificatesubjectbase}}\nCN={{subject.username}}

Example syntax rule:
subjectAltName=@{% section %}{{datarules|join('\n')}}{% endsection %}

Example composed config template:
[ req ]
prompt = no
encrypt_key = no

distinguished_name = {% section %}O={{config.ipacertificatesubjectbase}}
CN={{subject.username}}{% endsection %}

req_extensions = exts

[ exts ]
subjectAltName=@{% section %}email={{subject.email}}{% endsection %}

There's a lot more information about the thinking behind this at 
http://blog.benjaminlipton.com/2016/07/19/csr-generation-templating.html 
if you're interested, as well.


Thanks,
Ben

--
Manage your subscription for the Freeipa-devel mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-devel
Contribute to FreeIPA: http://www.freeipa.org/page/Contribute/Code