Re: [openstack-dev] [kolla] prototype of a DSL for generating Dockerfiles

[Ryan Hallisey]

Looking at some of the capabilities jinja2 has, it's hard to justify changing 
the method already in place.
I think jinja2 can provide a clear and operational way for operators to 
customize the dockerfiles as needed.
Kolla just hasn't applied them yet.

I'm extremely hesitant to agree on changing this because I think kolla can 
solve these issues without having
the overhead that will come with this change.

-Ryan

- Original Message -
From: "Michał Jastrzębski" <inc...@gmail.com>
To: "OpenStack Development Mailing List (not for usage questions)" 
<openstack-dev@lists.openstack.org>
Sent: Wednesday, June 1, 2016 4:55:50 PM
Subject: Re: [openstack-dev] [kolla] prototype of a DSL for generating  
Dockerfiles

Aaaand correct link: https://review.openstack.org/#/c/323589/ sorry
for pastefail.

On 1 June 2016 at 15:55, Michał Jastrzębski <inc...@gmail.com> wrote:
> So this is prototype of working template overrides:
> https://review.openstack.org/#/c/323612/
>
> Pass --template-overrides=path-to-file to build.py
> in file override you can add any custom code/logic/dockerfile stuff to
> any of hooks we provide in Dockerfiles, and we'll provide a lot of
> them as it's free and non breaking operation. With enough block you'll
> be able to do virtually anything with any of the containers.
>
> This one is already working. Only work needed is to provide more
> hooks/continue with refactoring of dockerfiles.
>
> Cheers,
> Michal
>
> On 31 May 2016 at 19:36, Steven Dake (stdake) <std...@cisco.com> wrote:
>>
>>
>> On 5/31/16, 1:42 PM, "Michał Jastrzębski" <inc...@gmail.com> wrote:
>>
>>>I am opposed to this idea as I don't think we need this. We can solve
>>>many problems by using jinja2 to greater extend. I'll publish demo of
>>>few improvements soon, please bear with me before we make a
>>>arch-changing call.
>>
>> Can you make a specification please as you have asked me to do?
>>
>>>
>>>On 29 May 2016 at 14:41, Steven Dake (stdake) <std...@cisco.com> wrote:
>>>>
>>>>>On 5/27/16, 1:58 AM, "Steven Dake (stdake)" <std...@cisco.com> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>>On 5/26/16, 8:45 PM, "Swapnil Kulkarni (coolsvap)" <m...@coolsvap.net>
>>>>>>wrote:
>>>>>>
>>>>>>>On Fri, May 27, 2016 at 8:35 AM, Steven Dake (stdake)
>>>>>>><std...@cisco.com>
>>>>>>>wrote:
>>>>>>>> Hey folks,
>>>>>>>>
>>>>>>>> While Swapnil has been busy churning the dockerfile.j2 files to all
>>>>>>>>match
>>>>>>>> the same style, and we also had summit where we declared we would
>>>>>>>>solve
>>>>>>>>the
>>>>>>>> plugin problem, I have decided to begin work on a DSL prototype.
>>>>>>>>
>>>>>>>> Here are the problems I want to solve in order of importance by this
>>>>>>>>work:
>>>>>>>>
>>>>>>>> Build CentOS, Ubuntu, Oracle Linux, Debian, Fedora containers
>>>>>>>> Provide a programmatic way to manage Dockerfile construction rather
>>>>>>>>then a
>>>>>>>> manual (with vi or emacs or the like) mechanism
>>>>>>>> Allow complete overrides of every facet of Dockerfile construction,
>>>>>>>>most
>>>>>>>> especially repositories per container (rather than in the base
>>>>>>>>container) to
>>>>>>>> permit the use case of dependencies from one version with
>>>>>>>>dependencies
>>>>>>>>in
>>>>>>>> another version of a different service
>>>>>>>> Get out of the business of maintaining 100+ dockerfiles but instead
>>>>>>>>maintain
>>>>>>>> one master file which defines the data that needs to be used to
>>>>>>>>construct
>>>>>>>> Dockerfiles
>>>>>>>> Permit different types of optimizations or Dockerfile building by
>>>>>>>>changing
>>>>>>>> around the parser implementation ­ to allow layering of each
>>>>>>>>operation,
>>>>>>>>or
>>>>>>>> alternatively to merge layers as we do today
>>>>

Re: [openstack-dev] [kolla] prototype of a DSL for generating Dockerfiles

[Michał Jastrzębski]

Aaaand correct link: https://review.openstack.org/#/c/323589/ sorry
for pastefail.

On 1 June 2016 at 15:55, Michał Jastrzębski  wrote:
> So this is prototype of working template overrides:
> https://review.openstack.org/#/c/323612/
>
> Pass --template-overrides=path-to-file to build.py
> in file override you can add any custom code/logic/dockerfile stuff to
> any of hooks we provide in Dockerfiles, and we'll provide a lot of
> them as it's free and non breaking operation. With enough block you'll
> be able to do virtually anything with any of the containers.
>
> This one is already working. Only work needed is to provide more
> hooks/continue with refactoring of dockerfiles.
>
> Cheers,
> Michal
>
> On 31 May 2016 at 19:36, Steven Dake (stdake)  wrote:
>>
>>
>> On 5/31/16, 1:42 PM, "Michał Jastrzębski"  wrote:
>>
>>>I am opposed to this idea as I don't think we need this. We can solve
>>>many problems by using jinja2 to greater extend. I'll publish demo of
>>>few improvements soon, please bear with me before we make a
>>>arch-changing call.
>>
>> Can you make a specification please as you have asked me to do?
>>
>>>
>>>On 29 May 2016 at 14:41, Steven Dake (stdake)  wrote:

>On 5/27/16, 1:58 AM, "Steven Dake (stdake)"  wrote:
>
>>
>>
>>On 5/26/16, 8:45 PM, "Swapnil Kulkarni (coolsvap)" 
>>wrote:
>>
>>>On Fri, May 27, 2016 at 8:35 AM, Steven Dake (stdake)
>>>
>>>wrote:
 Hey folks,

 While Swapnil has been busy churning the dockerfile.j2 files to all
match
 the same style, and we also had summit where we declared we would
solve
the
 plugin problem, I have decided to begin work on a DSL prototype.

 Here are the problems I want to solve in order of importance by this
work:

 Build CentOS, Ubuntu, Oracle Linux, Debian, Fedora containers
 Provide a programmatic way to manage Dockerfile construction rather
then a
 manual (with vi or emacs or the like) mechanism
 Allow complete overrides of every facet of Dockerfile construction,
most
 especially repositories per container (rather than in the base
container) to
 permit the use case of dependencies from one version with
dependencies
in
 another version of a different service
 Get out of the business of maintaining 100+ dockerfiles but instead
maintain
 one master file which defines the data that needs to be used to
construct
 Dockerfiles
 Permit different types of optimizations or Dockerfile building by
changing
 around the parser implementation ­ to allow layering of each
operation,
or
 alternatively to merge layers as we do today

 I don't believe we can proceed with both binary and source plugins
given our
 current implementation of Dockerfiles in any sane way.

 I further don't believe it is possible to customize repositories &
installed
 files per container, which I receive increasing requests for
offline.

 To that end, I've created a very very rough prototype which builds
the
base
 container as well as a mariadb container.  The mariadb container
builds
and
 I suspect would work.

 An example of the DSL usage is here:
 https://review.openstack.org/#/c/321468/4/dockerdsl/dsl.yml

 A very poorly written parser is here:
 https://review.openstack.org/#/c/321468/4/dockerdsl/load.py

 I played around with INI as a format, to take advantage of
oslo.config
and
 kolla-build.conf, but that didn't work out.  YML is the way to go.

 I'd appreciate reviews on the YML implementation especially.

 How I see this work progressing is as follows:

 A yml file describing all docker containers for all distros is
placed
in
 kolla/docker
 The build tool adds an option ‹use-yml which uses the YML file
 A parser (such as load.py above) is integrated into build.py to lay
down he
 Dockerfiles
 Wait 4-6 weeks for people to find bugs and complain
 Make the ‹use-yml the default for 4-6 weeks
 Once we feel confident in the yml implementation, remove all
Dockerfile.j2
 files
 Remove ‹use-yml option
 Remove all jinja2-isms from build.py

 This is similar to the work that took place to convert from raw
Dockerfiles
 to Dockerfile.j2 files.  We are just reusing that pattern.
Hopefully
this
 will be the last major 

Re: [openstack-dev] [kolla] prototype of a DSL for generating Dockerfiles

[Michał Jastrzębski]

So this is prototype of working template overrides:
https://review.openstack.org/#/c/323612/

Pass --template-overrides=path-to-file to build.py
in file override you can add any custom code/logic/dockerfile stuff to
any of hooks we provide in Dockerfiles, and we'll provide a lot of
them as it's free and non breaking operation. With enough block you'll
be able to do virtually anything with any of the containers.

This one is already working. Only work needed is to provide more
hooks/continue with refactoring of dockerfiles.

Cheers,
Michal

On 31 May 2016 at 19:36, Steven Dake (stdake)  wrote:
>
>
> On 5/31/16, 1:42 PM, "Michał Jastrzębski"  wrote:
>
>>I am opposed to this idea as I don't think we need this. We can solve
>>many problems by using jinja2 to greater extend. I'll publish demo of
>>few improvements soon, please bear with me before we make a
>>arch-changing call.
>
> Can you make a specification please as you have asked me to do?
>
>>
>>On 29 May 2016 at 14:41, Steven Dake (stdake)  wrote:
>>>
On 5/27/16, 1:58 AM, "Steven Dake (stdake)"  wrote:

>
>
>On 5/26/16, 8:45 PM, "Swapnil Kulkarni (coolsvap)" 
>wrote:
>
>>On Fri, May 27, 2016 at 8:35 AM, Steven Dake (stdake)
>>
>>wrote:
>>> Hey folks,
>>>
>>> While Swapnil has been busy churning the dockerfile.j2 files to all
>>>match
>>> the same style, and we also had summit where we declared we would
>>>solve
>>>the
>>> plugin problem, I have decided to begin work on a DSL prototype.
>>>
>>> Here are the problems I want to solve in order of importance by this
>>>work:
>>>
>>> Build CentOS, Ubuntu, Oracle Linux, Debian, Fedora containers
>>> Provide a programmatic way to manage Dockerfile construction rather
>>>then a
>>> manual (with vi or emacs or the like) mechanism
>>> Allow complete overrides of every facet of Dockerfile construction,
>>>most
>>> especially repositories per container (rather than in the base
>>>container) to
>>> permit the use case of dependencies from one version with
>>>dependencies
>>>in
>>> another version of a different service
>>> Get out of the business of maintaining 100+ dockerfiles but instead
>>>maintain
>>> one master file which defines the data that needs to be used to
>>>construct
>>> Dockerfiles
>>> Permit different types of optimizations or Dockerfile building by
>>>changing
>>> around the parser implementation ­ to allow layering of each
>>>operation,
>>>or
>>> alternatively to merge layers as we do today
>>>
>>> I don't believe we can proceed with both binary and source plugins
>>>given our
>>> current implementation of Dockerfiles in any sane way.
>>>
>>> I further don't believe it is possible to customize repositories &
>>>installed
>>> files per container, which I receive increasing requests for
>>>offline.
>>>
>>> To that end, I've created a very very rough prototype which builds
>>>the
>>>base
>>> container as well as a mariadb container.  The mariadb container
>>>builds
>>>and
>>> I suspect would work.
>>>
>>> An example of the DSL usage is here:
>>> https://review.openstack.org/#/c/321468/4/dockerdsl/dsl.yml
>>>
>>> A very poorly written parser is here:
>>> https://review.openstack.org/#/c/321468/4/dockerdsl/load.py
>>>
>>> I played around with INI as a format, to take advantage of
>>>oslo.config
>>>and
>>> kolla-build.conf, but that didn't work out.  YML is the way to go.
>>>
>>> I'd appreciate reviews on the YML implementation especially.
>>>
>>> How I see this work progressing is as follows:
>>>
>>> A yml file describing all docker containers for all distros is
>>>placed
>>>in
>>> kolla/docker
>>> The build tool adds an option ‹use-yml which uses the YML file
>>> A parser (such as load.py above) is integrated into build.py to lay
>>>down he
>>> Dockerfiles
>>> Wait 4-6 weeks for people to find bugs and complain
>>> Make the ‹use-yml the default for 4-6 weeks
>>> Once we feel confident in the yml implementation, remove all
>>>Dockerfile.j2
>>> files
>>> Remove ‹use-yml option
>>> Remove all jinja2-isms from build.py
>>>
>>> This is similar to the work that took place to convert from raw
>>>Dockerfiles
>>> to Dockerfile.j2 files.  We are just reusing that pattern.
>>>Hopefully
>>>this
>>> will be the last major refactor of the dockerfiles unless someone
>>>has
>>>some
>>> significant complaints about the approach.
>>>
>>> Regards
>>> -steve
>
> Hey folks,
>
> I have produced a specification for Kolla's DSL (which I call Elemental).
> The spec is ready for review here:
> 

Re: [openstack-dev] [kolla] prototype of a DSL for generating Dockerfiles

[Steven Dake (stdake)]

On 5/31/16, 1:42 PM, "Michał Jastrzębski"  wrote:

>I am opposed to this idea as I don't think we need this. We can solve
>many problems by using jinja2 to greater extend. I'll publish demo of
>few improvements soon, please bear with me before we make a
>arch-changing call.

Can you make a specification please as you have asked me to do?

>
>On 29 May 2016 at 14:41, Steven Dake (stdake)  wrote:
>>
>>>On 5/27/16, 1:58 AM, "Steven Dake (stdake)"  wrote:
>>>


On 5/26/16, 8:45 PM, "Swapnil Kulkarni (coolsvap)" 
wrote:

>On Fri, May 27, 2016 at 8:35 AM, Steven Dake (stdake)
>
>wrote:
>> Hey folks,
>>
>> While Swapnil has been busy churning the dockerfile.j2 files to all
>>match
>> the same style, and we also had summit where we declared we would
>>solve
>>the
>> plugin problem, I have decided to begin work on a DSL prototype.
>>
>> Here are the problems I want to solve in order of importance by this
>>work:
>>
>> Build CentOS, Ubuntu, Oracle Linux, Debian, Fedora containers
>> Provide a programmatic way to manage Dockerfile construction rather
>>then a
>> manual (with vi or emacs or the like) mechanism
>> Allow complete overrides of every facet of Dockerfile construction,
>>most
>> especially repositories per container (rather than in the base
>>container) to
>> permit the use case of dependencies from one version with
>>dependencies
>>in
>> another version of a different service
>> Get out of the business of maintaining 100+ dockerfiles but instead
>>maintain
>> one master file which defines the data that needs to be used to
>>construct
>> Dockerfiles
>> Permit different types of optimizations or Dockerfile building by
>>changing
>> around the parser implementation ­ to allow layering of each
>>operation,
>>or
>> alternatively to merge layers as we do today
>>
>> I don't believe we can proceed with both binary and source plugins
>>given our
>> current implementation of Dockerfiles in any sane way.
>>
>> I further don't believe it is possible to customize repositories &
>>installed
>> files per container, which I receive increasing requests for
>>offline.
>>
>> To that end, I've created a very very rough prototype which builds
>>the
>>base
>> container as well as a mariadb container.  The mariadb container
>>builds
>>and
>> I suspect would work.
>>
>> An example of the DSL usage is here:
>> https://review.openstack.org/#/c/321468/4/dockerdsl/dsl.yml
>>
>> A very poorly written parser is here:
>> https://review.openstack.org/#/c/321468/4/dockerdsl/load.py
>>
>> I played around with INI as a format, to take advantage of
>>oslo.config
>>and
>> kolla-build.conf, but that didn't work out.  YML is the way to go.
>>
>> I'd appreciate reviews on the YML implementation especially.
>>
>> How I see this work progressing is as follows:
>>
>> A yml file describing all docker containers for all distros is
>>placed
>>in
>> kolla/docker
>> The build tool adds an option ‹use-yml which uses the YML file
>> A parser (such as load.py above) is integrated into build.py to lay
>>down he
>> Dockerfiles
>> Wait 4-6 weeks for people to find bugs and complain
>> Make the ‹use-yml the default for 4-6 weeks
>> Once we feel confident in the yml implementation, remove all
>>Dockerfile.j2
>> files
>> Remove ‹use-yml option
>> Remove all jinja2-isms from build.py
>>
>> This is similar to the work that took place to convert from raw
>>Dockerfiles
>> to Dockerfile.j2 files.  We are just reusing that pattern.
>>Hopefully
>>this
>> will be the last major refactor of the dockerfiles unless someone
>>has
>>some
>> significant complaints about the approach.
>>
>> Regards
>> -steve

Hey folks,

I have produced a specification for Kolla's DSL (which I call Elemental).
The spec is ready for review here:
https://review.openstack.org/#/c/323612/


Regards
-steve

>>
>>
>> On 5/27/16, 3:44 AM, "Britt Houser (bhouser)"  wrote:
>>
>>>I admit I'm not as knowledgable about the Kolla codebase as I'd like to
>>>be, so most of what you're saying is going over my head.  I think mainly
>>>I don't understand the problem statement.  It looks like you're pulling
>>>all the "hard coded" things out of the docker files, and making them
>>>user
>>>replaceable?  So the dockerfiles just become a list of required steps,
>>>and the user can change how each step is implemented?  Would this also
>>>unify the dockefiles so there wouldn't be a huge if statements between
>>>Centos and Ubuntu?
>>>
>>>Thx,
>>>Britt
>>>
>>
>> What is being pulled out is all of 

Re: [openstack-dev] [kolla] prototype of a DSL for generating Dockerfiles

[Michał Jastrzębski]

I am opposed to this idea as I don't think we need this. We can solve
many problems by using jinja2 to greater extend. I'll publish demo of
few improvements soon, please bear with me before we make a
arch-changing call.

On 29 May 2016 at 14:41, Steven Dake (stdake)  wrote:
>
>>On 5/27/16, 1:58 AM, "Steven Dake (stdake)"  wrote:
>>
>>>
>>>
>>>On 5/26/16, 8:45 PM, "Swapnil Kulkarni (coolsvap)" 
>>>wrote:
>>>
On Fri, May 27, 2016 at 8:35 AM, Steven Dake (stdake) 
wrote:
> Hey folks,
>
> While Swapnil has been busy churning the dockerfile.j2 files to all
>match
> the same style, and we also had summit where we declared we would
>solve
>the
> plugin problem, I have decided to begin work on a DSL prototype.
>
> Here are the problems I want to solve in order of importance by this
>work:
>
> Build CentOS, Ubuntu, Oracle Linux, Debian, Fedora containers
> Provide a programmatic way to manage Dockerfile construction rather
>then a
> manual (with vi or emacs or the like) mechanism
> Allow complete overrides of every facet of Dockerfile construction,
>most
> especially repositories per container (rather than in the base
>container) to
> permit the use case of dependencies from one version with dependencies
>in
> another version of a different service
> Get out of the business of maintaining 100+ dockerfiles but instead
>maintain
> one master file which defines the data that needs to be used to
>construct
> Dockerfiles
> Permit different types of optimizations or Dockerfile building by
>changing
> around the parser implementation ­ to allow layering of each
>operation,
>or
> alternatively to merge layers as we do today
>
> I don't believe we can proceed with both binary and source plugins
>given our
> current implementation of Dockerfiles in any sane way.
>
> I further don't believe it is possible to customize repositories &
>installed
> files per container, which I receive increasing requests for offline.
>
> To that end, I've created a very very rough prototype which builds the
>base
> container as well as a mariadb container.  The mariadb container
>builds
>and
> I suspect would work.
>
> An example of the DSL usage is here:
> https://review.openstack.org/#/c/321468/4/dockerdsl/dsl.yml
>
> A very poorly written parser is here:
> https://review.openstack.org/#/c/321468/4/dockerdsl/load.py
>
> I played around with INI as a format, to take advantage of oslo.config
>and
> kolla-build.conf, but that didn't work out.  YML is the way to go.
>
> I'd appreciate reviews on the YML implementation especially.
>
> How I see this work progressing is as follows:
>
> A yml file describing all docker containers for all distros is placed
>in
> kolla/docker
> The build tool adds an option ‹use-yml which uses the YML file
> A parser (such as load.py above) is integrated into build.py to lay
>down he
> Dockerfiles
> Wait 4-6 weeks for people to find bugs and complain
> Make the ‹use-yml the default for 4-6 weeks
> Once we feel confident in the yml implementation, remove all
>Dockerfile.j2
> files
> Remove ‹use-yml option
> Remove all jinja2-isms from build.py
>
> This is similar to the work that took place to convert from raw
>Dockerfiles
> to Dockerfile.j2 files.  We are just reusing that pattern.  Hopefully
>this
> will be the last major refactor of the dockerfiles unless someone has
>some
> significant complaints about the approach.
>
> Regards
> -steve
>
>
> On 5/27/16, 3:44 AM, "Britt Houser (bhouser)"  wrote:
>
>>I admit I'm not as knowledgable about the Kolla codebase as I'd like to
>>be, so most of what you're saying is going over my head.  I think mainly
>>I don't understand the problem statement.  It looks like you're pulling
>>all the "hard coded" things out of the docker files, and making them user
>>replaceable?  So the dockerfiles just become a list of required steps,
>>and the user can change how each step is implemented?  Would this also
>>unify the dockefiles so there wouldn't be a huge if statements between
>>Centos and Ubuntu?
>>
>>Thx,
>>Britt
>>
>
> What is being pulled out is all of the metadata used by the Dockerfiles or
> Kolla in general.  This metadata, being structured either as a dictionary
> or ordered list, can be manipulated by simple python tools to do things
> like merge sections and override sections or optimize the built images.
> FWIW it looks without even trying the Dockerfiles produce a 50MB smaller
> image produced by the parser.  The jinja2 templates we have today cannot
> be easily overridden.  We have to provide a new key for each type of
> 

Re: [openstack-dev] [kolla] prototype of a DSL for generating Dockerfiles

[Steven Dake (stdake)]

>On 5/27/16, 1:58 AM, "Steven Dake (stdake)"  wrote:
>
>>
>>
>>On 5/26/16, 8:45 PM, "Swapnil Kulkarni (coolsvap)" 
>>wrote:
>>
>>>On Fri, May 27, 2016 at 8:35 AM, Steven Dake (stdake) 
>>>wrote:
 Hey folks,

 While Swapnil has been busy churning the dockerfile.j2 files to all
match
 the same style, and we also had summit where we declared we would
solve
the
 plugin problem, I have decided to begin work on a DSL prototype.

 Here are the problems I want to solve in order of importance by this
work:

 Build CentOS, Ubuntu, Oracle Linux, Debian, Fedora containers
 Provide a programmatic way to manage Dockerfile construction rather
then a
 manual (with vi or emacs or the like) mechanism
 Allow complete overrides of every facet of Dockerfile construction,
most
 especially repositories per container (rather than in the base
container) to
 permit the use case of dependencies from one version with dependencies
in
 another version of a different service
 Get out of the business of maintaining 100+ dockerfiles but instead
maintain
 one master file which defines the data that needs to be used to
construct
 Dockerfiles
 Permit different types of optimizations or Dockerfile building by
changing
 around the parser implementation ­ to allow layering of each
operation,
or
 alternatively to merge layers as we do today

 I don't believe we can proceed with both binary and source plugins
given our
 current implementation of Dockerfiles in any sane way.

 I further don't believe it is possible to customize repositories &
installed
 files per container, which I receive increasing requests for offline.

 To that end, I've created a very very rough prototype which builds the
base
 container as well as a mariadb container.  The mariadb container
builds
and
 I suspect would work.

 An example of the DSL usage is here:
 https://review.openstack.org/#/c/321468/4/dockerdsl/dsl.yml

 A very poorly written parser is here:
 https://review.openstack.org/#/c/321468/4/dockerdsl/load.py

 I played around with INI as a format, to take advantage of oslo.config
and
 kolla-build.conf, but that didn't work out.  YML is the way to go.

 I'd appreciate reviews on the YML implementation especially.

 How I see this work progressing is as follows:

 A yml file describing all docker containers for all distros is placed
in
 kolla/docker
 The build tool adds an option ‹use-yml which uses the YML file
 A parser (such as load.py above) is integrated into build.py to lay
down he
 Dockerfiles
 Wait 4-6 weeks for people to find bugs and complain
 Make the ‹use-yml the default for 4-6 weeks
 Once we feel confident in the yml implementation, remove all
Dockerfile.j2
 files
 Remove ‹use-yml option
 Remove all jinja2-isms from build.py

 This is similar to the work that took place to convert from raw
Dockerfiles
 to Dockerfile.j2 files.  We are just reusing that pattern.  Hopefully
this
 will be the last major refactor of the dockerfiles unless someone has
some
 significant complaints about the approach.

 Regards
 -steve


On 5/27/16, 3:44 AM, "Britt Houser (bhouser)"  wrote:

>I admit I'm not as knowledgable about the Kolla codebase as I'd like to
>be, so most of what you're saying is going over my head.  I think mainly
>I don't understand the problem statement.  It looks like you're pulling
>all the "hard coded" things out of the docker files, and making them user
>replaceable?  So the dockerfiles just become a list of required steps,
>and the user can change how each step is implemented?  Would this also
>unify the dockefiles so there wouldn't be a huge if statements between
>Centos and Ubuntu?
>
>Thx,
>Britt
>

What is being pulled out is all of the metadata used by the Dockerfiles or
Kolla in general.  This metadata, being structured either as a dictionary
or ordered list, can be manipulated by simple python tools to do things
like merge sections and override sections or optimize the built images.
FWIW it looks without even trying the Dockerfiles produce a 50MB smaller
image produced by the parser.  The jinja2 templates we have today cannot
be easily overridden.  We have to provide a new key for each type of
override, which is super onerous on the build.py tool.

To your question of docker files being a list of required steps, with this
method Dockerfile.j2 would go away permanently.  On each bulid, as is done
now to process a jinja2 file into a Dockerfile, the elemental.yml file
would be processed into a Dockerfile for that particular set of build
options. 
>




 

Re: [openstack-dev] [kolla] prototype of a DSL for generating Dockerfiles

[Britt Houser (bhouser)]

I admit I'm not as knowledgable about the Kolla codebase as I'd like to be, so 
most of what you're saying is going over my head.  I think mainly I don't 
understand the problem statement.  It looks like you're pulling all the "hard 
coded" things out of the docker files, and making them user replaceable?  So 
the dockerfiles just become a list of required steps, and the user can change 
how each step is implemented?  Would this also unify the dockefiles so there 
wouldn't be a huge if statements between Centos and Ubuntu?

Thx,
Britt



On 5/27/16, 1:58 AM, "Steven Dake (stdake)"  wrote:

>
>
>On 5/26/16, 8:45 PM, "Swapnil Kulkarni (coolsvap)"  wrote:
>
>>On Fri, May 27, 2016 at 8:35 AM, Steven Dake (stdake) 
>>wrote:
>>> Hey folks,
>>>
>>> While Swapnil has been busy churning the dockerfile.j2 files to all
>>>match
>>> the same style, and we also had summit where we declared we would solve
>>>the
>>> plugin problem, I have decided to begin work on a DSL prototype.
>>>
>>> Here are the problems I want to solve in order of importance by this
>>>work:
>>>
>>> Build CentOS, Ubuntu, Oracle Linux, Debian, Fedora containers
>>> Provide a programmatic way to manage Dockerfile construction rather
>>>then a
>>> manual (with vi or emacs or the like) mechanism
>>> Allow complete overrides of every facet of Dockerfile construction, most
>>> especially repositories per container (rather than in the base
>>>container) to
>>> permit the use case of dependencies from one version with dependencies
>>>in
>>> another version of a different service
>>> Get out of the business of maintaining 100+ dockerfiles but instead
>>>maintain
>>> one master file which defines the data that needs to be used to
>>>construct
>>> Dockerfiles
>>> Permit different types of optimizations or Dockerfile building by
>>>changing
>>> around the parser implementation ­ to allow layering of each operation,
>>>or
>>> alternatively to merge layers as we do today
>>>
>>> I don't believe we can proceed with both binary and source plugins
>>>given our
>>> current implementation of Dockerfiles in any sane way.
>>>
>>> I further don't believe it is possible to customize repositories &
>>>installed
>>> files per container, which I receive increasing requests for offline.
>>>
>>> To that end, I've created a very very rough prototype which builds the
>>>base
>>> container as well as a mariadb container.  The mariadb container builds
>>>and
>>> I suspect would work.
>>>
>>> An example of the DSL usage is here:
>>> https://review.openstack.org/#/c/321468/4/dockerdsl/dsl.yml
>>>
>>> A very poorly written parser is here:
>>> https://review.openstack.org/#/c/321468/4/dockerdsl/load.py
>>>
>>> I played around with INI as a format, to take advantage of oslo.config
>>>and
>>> kolla-build.conf, but that didn't work out.  YML is the way to go.
>>>
>>> I'd appreciate reviews on the YML implementation especially.
>>>
>>> How I see this work progressing is as follows:
>>>
>>> A yml file describing all docker containers for all distros is placed in
>>> kolla/docker
>>> The build tool adds an option ‹use-yml which uses the YML file
>>> A parser (such as load.py above) is integrated into build.py to lay
>>>down he
>>> Dockerfiles
>>> Wait 4-6 weeks for people to find bugs and complain
>>> Make the ‹use-yml the default for 4-6 weeks
>>> Once we feel confident in the yml implementation, remove all
>>>Dockerfile.j2
>>> files
>>> Remove ‹use-yml option
>>> Remove all jinja2-isms from build.py
>>>
>>> This is similar to the work that took place to convert from raw
>>>Dockerfiles
>>> to Dockerfile.j2 files.  We are just reusing that pattern.  Hopefully
>>>this
>>> will be the last major refactor of the dockerfiles unless someone has
>>>some
>>> significant complaints about the approach.
>>>
>>> Regards
>>> -steve
>>>
>>>
>>> 
>>>_
>>>_
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: 
>>>openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>>The DSL template to generate the Dockerfile seems way better than the
>>jinja templates in terms of extension which is currently a major
>>bottleneck in the plugin implementation. I am +2+W on this plan of
>>action to test it for next 4-6 weeks and see thereon.
>>
>>Swapnil
>>
>
>Agree.
>
>Customization and plugins are the trigger for the work.  I was thinking of
>the following:
>
>Elemental.yml (ships with Kolla)
>Elemental-merge.yml (operator provides in /etc/kolla, this file is yaml
>merged with elemental.yml)
>Elemental-override.yml (operator provides in /etc/kolla, this file
>overrides any YAML sections defined)
>
>I think merging and overriding the yaml files should be pretty easy,
>compared to jinja2, where I don't even know where to begin in a way that
>the operator doesn't have to have deep 

Re: [openstack-dev] [kolla] prototype of a DSL for generating Dockerfiles

[Steven Dake (stdake)]

On 5/26/16, 8:45 PM, "Swapnil Kulkarni (coolsvap)"  wrote:

>On Fri, May 27, 2016 at 8:35 AM, Steven Dake (stdake) 
>wrote:
>> Hey folks,
>>
>> While Swapnil has been busy churning the dockerfile.j2 files to all
>>match
>> the same style, and we also had summit where we declared we would solve
>>the
>> plugin problem, I have decided to begin work on a DSL prototype.
>>
>> Here are the problems I want to solve in order of importance by this
>>work:
>>
>> Build CentOS, Ubuntu, Oracle Linux, Debian, Fedora containers
>> Provide a programmatic way to manage Dockerfile construction rather
>>then a
>> manual (with vi or emacs or the like) mechanism
>> Allow complete overrides of every facet of Dockerfile construction, most
>> especially repositories per container (rather than in the base
>>container) to
>> permit the use case of dependencies from one version with dependencies
>>in
>> another version of a different service
>> Get out of the business of maintaining 100+ dockerfiles but instead
>>maintain
>> one master file which defines the data that needs to be used to
>>construct
>> Dockerfiles
>> Permit different types of optimizations or Dockerfile building by
>>changing
>> around the parser implementation ­ to allow layering of each operation,
>>or
>> alternatively to merge layers as we do today
>>
>> I don't believe we can proceed with both binary and source plugins
>>given our
>> current implementation of Dockerfiles in any sane way.
>>
>> I further don't believe it is possible to customize repositories &
>>installed
>> files per container, which I receive increasing requests for offline.
>>
>> To that end, I've created a very very rough prototype which builds the
>>base
>> container as well as a mariadb container.  The mariadb container builds
>>and
>> I suspect would work.
>>
>> An example of the DSL usage is here:
>> https://review.openstack.org/#/c/321468/4/dockerdsl/dsl.yml
>>
>> A very poorly written parser is here:
>> https://review.openstack.org/#/c/321468/4/dockerdsl/load.py
>>
>> I played around with INI as a format, to take advantage of oslo.config
>>and
>> kolla-build.conf, but that didn't work out.  YML is the way to go.
>>
>> I'd appreciate reviews on the YML implementation especially.
>>
>> How I see this work progressing is as follows:
>>
>> A yml file describing all docker containers for all distros is placed in
>> kolla/docker
>> The build tool adds an option ‹use-yml which uses the YML file
>> A parser (such as load.py above) is integrated into build.py to lay
>>down he
>> Dockerfiles
>> Wait 4-6 weeks for people to find bugs and complain
>> Make the ‹use-yml the default for 4-6 weeks
>> Once we feel confident in the yml implementation, remove all
>>Dockerfile.j2
>> files
>> Remove ‹use-yml option
>> Remove all jinja2-isms from build.py
>>
>> This is similar to the work that took place to convert from raw
>>Dockerfiles
>> to Dockerfile.j2 files.  We are just reusing that pattern.  Hopefully
>>this
>> will be the last major refactor of the dockerfiles unless someone has
>>some
>> significant complaints about the approach.
>>
>> Regards
>> -steve
>>
>>
>> 
>>_
>>_
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: 
>>openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>The DSL template to generate the Dockerfile seems way better than the
>jinja templates in terms of extension which is currently a major
>bottleneck in the plugin implementation. I am +2+W on this plan of
>action to test it for next 4-6 weeks and see thereon.
>
>Swapnil
>

Agree.

Customization and plugins are the trigger for the work.  I was thinking of
the following:

Elemental.yml (ships with Kolla)
Elemental-merge.yml (operator provides in /etc/kolla, this file is yaml
merged with elemental.yml)
Elemental-override.yml (operator provides in /etc/kolla, this file
overrides any YAML sections defined)

I think merging and overriding the yaml files should be pretty easy,
compared to jinja2, where I don't even know where to begin in a way that
the operator doesn't have to have deep knowledge of Kolla's internal
implementation.

Regards
-steve
  
>__
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Re: [openstack-dev] [kolla] prototype of a DSL for generating Dockerfiles

[Swapnil Kulkarni (coolsvap)]

On Fri, May 27, 2016 at 8:35 AM, Steven Dake (stdake)  wrote:
> Hey folks,
>
> While Swapnil has been busy churning the dockerfile.j2 files to all match
> the same style, and we also had summit where we declared we would solve the
> plugin problem, I have decided to begin work on a DSL prototype.
>
> Here are the problems I want to solve in order of importance by this work:
>
> Build CentOS, Ubuntu, Oracle Linux, Debian, Fedora containers
> Provide a programmatic way to manage Dockerfile construction rather then a
> manual (with vi or emacs or the like) mechanism
> Allow complete overrides of every facet of Dockerfile construction, most
> especially repositories per container (rather than in the base container) to
> permit the use case of dependencies from one version with dependencies in
> another version of a different service
> Get out of the business of maintaining 100+ dockerfiles but instead maintain
> one master file which defines the data that needs to be used to construct
> Dockerfiles
> Permit different types of optimizations or Dockerfile building by changing
> around the parser implementation – to allow layering of each operation, or
> alternatively to merge layers as we do today
>
> I don't believe we can proceed with both binary and source plugins given our
> current implementation of Dockerfiles in any sane way.
>
> I further don't believe it is possible to customize repositories & installed
> files per container, which I receive increasing requests for offline.
>
> To that end, I've created a very very rough prototype which builds the base
> container as well as a mariadb container.  The mariadb container builds and
> I suspect would work.
>
> An example of the DSL usage is here:
> https://review.openstack.org/#/c/321468/4/dockerdsl/dsl.yml
>
> A very poorly written parser is here:
> https://review.openstack.org/#/c/321468/4/dockerdsl/load.py
>
> I played around with INI as a format, to take advantage of oslo.config and
> kolla-build.conf, but that didn't work out.  YML is the way to go.
>
> I'd appreciate reviews on the YML implementation especially.
>
> How I see this work progressing is as follows:
>
> A yml file describing all docker containers for all distros is placed in
> kolla/docker
> The build tool adds an option —use-yml which uses the YML file
> A parser (such as load.py above) is integrated into build.py to lay down he
> Dockerfiles
> Wait 4-6 weeks for people to find bugs and complain
> Make the —use-yml the default for 4-6 weeks
> Once we feel confident in the yml implementation, remove all Dockerfile.j2
> files
> Remove —use-yml option
> Remove all jinja2-isms from build.py
>
> This is similar to the work that took place to convert from raw Dockerfiles
> to Dockerfile.j2 files.  We are just reusing that pattern.  Hopefully this
> will be the last major refactor of the dockerfiles unless someone has some
> significant complaints about the approach.
>
> Regards
> -steve
>
>
> __
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

The DSL template to generate the Dockerfile seems way better than the
jinja templates in terms of extension which is currently a major
bottleneck in the plugin implementation. I am +2+W on this plan of
action to test it for next 4-6 weeks and see thereon.

Swapnil

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

[openstack-dev] [kolla] prototype of a DSL for generating Dockerfiles

[Steven Dake (stdake)]

Hey folks,

While Swapnil has been busy churning the dockerfile.j2 files to all match the 
same style, and we also had summit where we declared we would solve the plugin 
problem, I have decided to begin work on a DSL prototype.

Here are the problems I want to solve in order of importance by this work:

  1.  Build CentOS, Ubuntu, Oracle Linux, Debian, Fedora containers
  2.  Provide a programmatic way to manage Dockerfile construction rather then 
a manual (with vi or emacs or the like) mechanism
  3.  Allow complete overrides of every facet of Dockerfile construction, most 
especially repositories per container (rather than in the base container) to 
permit the use case of dependencies from one version with dependencies in 
another version of a different service
  4.  Get out of the business of maintaining 100+ dockerfiles but instead 
maintain one master file which defines the data that needs to be used to 
construct Dockerfiles
  5.  Permit different types of optimizations or Dockerfile building by 
changing around the parser implementation - to allow layering of each 
operation, or alternatively to merge layers as we do today

I don't believe we can proceed with both binary and source plugins given our 
current implementation of Dockerfiles in any sane way.

I further don't believe it is possible to customize repositories & installed 
files per container, which I receive increasing requests for offline.

To that end, I've created a very very rough prototype which builds the base 
container as well as a mariadb container.  The mariadb container builds and I 
suspect would work.

An example of the DSL usage is here:
https://review.openstack.org/#/c/321468/4/dockerdsl/dsl.yml

A very poorly written parser is here:
https://review.openstack.org/#/c/321468/4/dockerdsl/load.py

I played around with INI as a format, to take advantage of oslo.config and 
kolla-build.conf, but that didn't work out.  YML is the way to go.

I'd appreciate reviews on the YML implementation especially.

How I see this work progressing is as follows:

  1.  A yml file describing all docker containers for all distros is placed in 
kolla/docker
  2.  The build tool adds an option -use-yml which uses the YML file
  3.  A parser (such as load.py above) is integrated into build.py to lay down 
he Dockerfiles
  4.  Wait 4-6 weeks for people to find bugs and complain
  5.  Make the -use-yml the default for 4-6 weeks
  6.  Once we feel confident in the yml implementation, remove all 
Dockerfile.j2 files
  7.  Remove -use-yml option
  8.  Remove all jinja2-isms from build.py

This is similar to the work that took place to convert from raw Dockerfiles to 
Dockerfile.j2 files.  We are just reusing that pattern.  Hopefully this will be 
the last major refactor of the dockerfiles unless someone has some significant 
complaints about the approach.

Regards
-steve

__
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev