Re: [openstack-dev] [kolla] prototype of a DSL for generating Dockerfiles
Looking at some of the capabilities jinja2 has, it's hard to justify changing the method already in place. I think jinja2 can provide a clear and operational way for operators to customize the dockerfiles as needed. Kolla just hasn't applied them yet. I'm extremely hesitant to agree on changing this because I think kolla can solve these issues without having the overhead that will come with this change. -Ryan - Original Message - From: "Michał Jastrzębski" To: "OpenStack Development Mailing List (not for usage questions)" Sent: Wednesday, June 1, 2016 4:55:50 PM Subject: Re: [openstack-dev] [kolla] prototype of a DSL for generating Dockerfiles Aaaand correct link: https://review.openstack.org/#/c/323589/ sorry for pastefail. On 1 June 2016 at 15:55, Michał Jastrzębski wrote: > So this is prototype of working template overrides: > https://review.openstack.org/#/c/323612/ > > Pass --template-overrides=path-to-file to build.py > in file override you can add any custom code/logic/dockerfile stuff to > any of hooks we provide in Dockerfiles, and we'll provide a lot of > them as it's free and non breaking operation. With enough block you'll > be able to do virtually anything with any of the containers. > > This one is already working. Only work needed is to provide more > hooks/continue with refactoring of dockerfiles. > > Cheers, > Michal > > On 31 May 2016 at 19:36, Steven Dake (stdake) wrote: >> >> >> On 5/31/16, 1:42 PM, "Michał Jastrzębski" wrote: >> >>>I am opposed to this idea as I don't think we need this. We can solve >>>many problems by using jinja2 to greater extend. I'll publish demo of >>>few improvements soon, please bear with me before we make a >>>arch-changing call. >> >> Can you make a specification please as you have asked me to do? >> >>> >>>On 29 May 2016 at 14:41, Steven Dake (stdake) wrote: >>>> >>>>>On 5/27/16, 1:58 AM, "Steven Dake (stdake)" wrote: >>>>> >>>>>> >>>>>> >>>>>>On 5/26/16, 8:45 PM, "Swapnil Kulkarni (coolsvap)" >>>>>>wrote: >>>>>> >>>>>>>On Fri, May 27, 2016 at 8:35 AM, Steven Dake (stdake) >>>>>>> >>>>>>>wrote: >>>>>>>> Hey folks, >>>>>>>> >>>>>>>> While Swapnil has been busy churning the dockerfile.j2 files to all >>>>>>>>match >>>>>>>> the same style, and we also had summit where we declared we would >>>>>>>>solve >>>>>>>>the >>>>>>>> plugin problem, I have decided to begin work on a DSL prototype. >>>>>>>> >>>>>>>> Here are the problems I want to solve in order of importance by this >>>>>>>>work: >>>>>>>> >>>>>>>> Build CentOS, Ubuntu, Oracle Linux, Debian, Fedora containers >>>>>>>> Provide a programmatic way to manage Dockerfile construction rather >>>>>>>>then a >>>>>>>> manual (with vi or emacs or the like) mechanism >>>>>>>> Allow complete overrides of every facet of Dockerfile construction, >>>>>>>>most >>>>>>>> especially repositories per container (rather than in the base >>>>>>>>container) to >>>>>>>> permit the use case of dependencies from one version with >>>>>>>>dependencies >>>>>>>>in >>>>>>>> another version of a different service >>>>>>>> Get out of the business of maintaining 100+ dockerfiles but instead >>>>>>>>maintain >>>>>>>> one master file which defines the data that needs to be used to >>>>>>>>construct >>>>>>>> Dockerfiles >>>>>>>> Permit different types of optimizations or Dockerfile building by >>>>>>>>changing >>>>>>>> around the parser implementation to allow layering of each >>>>>>>>operation, >>>>>>>>or >>>>>>>> alternatively to merge layers as we do today >>>>>>>> >>>>>>>> I don't believe we can proceed with both binary and source plugins >>>>>>>>given our >>>>>&
Re: [openstack-dev] [kolla] prototype of a DSL for generating Dockerfiles
Aaaand correct link: https://review.openstack.org/#/c/323589/ sorry for pastefail. On 1 June 2016 at 15:55, Michał Jastrzębski wrote: > So this is prototype of working template overrides: > https://review.openstack.org/#/c/323612/ > > Pass --template-overrides=path-to-file to build.py > in file override you can add any custom code/logic/dockerfile stuff to > any of hooks we provide in Dockerfiles, and we'll provide a lot of > them as it's free and non breaking operation. With enough block you'll > be able to do virtually anything with any of the containers. > > This one is already working. Only work needed is to provide more > hooks/continue with refactoring of dockerfiles. > > Cheers, > Michal > > On 31 May 2016 at 19:36, Steven Dake (stdake) wrote: >> >> >> On 5/31/16, 1:42 PM, "Michał Jastrzębski" wrote: >> >>>I am opposed to this idea as I don't think we need this. We can solve >>>many problems by using jinja2 to greater extend. I'll publish demo of >>>few improvements soon, please bear with me before we make a >>>arch-changing call. >> >> Can you make a specification please as you have asked me to do? >> >>> >>>On 29 May 2016 at 14:41, Steven Dake (stdake) wrote: >On 5/27/16, 1:58 AM, "Steven Dake (stdake)" wrote: > >> >> >>On 5/26/16, 8:45 PM, "Swapnil Kulkarni (coolsvap)" >>wrote: >> >>>On Fri, May 27, 2016 at 8:35 AM, Steven Dake (stdake) >>> >>>wrote: Hey folks, While Swapnil has been busy churning the dockerfile.j2 files to all match the same style, and we also had summit where we declared we would solve the plugin problem, I have decided to begin work on a DSL prototype. Here are the problems I want to solve in order of importance by this work: Build CentOS, Ubuntu, Oracle Linux, Debian, Fedora containers Provide a programmatic way to manage Dockerfile construction rather then a manual (with vi or emacs or the like) mechanism Allow complete overrides of every facet of Dockerfile construction, most especially repositories per container (rather than in the base container) to permit the use case of dependencies from one version with dependencies in another version of a different service Get out of the business of maintaining 100+ dockerfiles but instead maintain one master file which defines the data that needs to be used to construct Dockerfiles Permit different types of optimizations or Dockerfile building by changing around the parser implementation to allow layering of each operation, or alternatively to merge layers as we do today I don't believe we can proceed with both binary and source plugins given our current implementation of Dockerfiles in any sane way. I further don't believe it is possible to customize repositories & installed files per container, which I receive increasing requests for offline. To that end, I've created a very very rough prototype which builds the base container as well as a mariadb container. The mariadb container builds and I suspect would work. An example of the DSL usage is here: https://review.openstack.org/#/c/321468/4/dockerdsl/dsl.yml A very poorly written parser is here: https://review.openstack.org/#/c/321468/4/dockerdsl/load.py I played around with INI as a format, to take advantage of oslo.config and kolla-build.conf, but that didn't work out. YML is the way to go. I'd appreciate reviews on the YML implementation especially. How I see this work progressing is as follows: A yml file describing all docker containers for all distros is placed in kolla/docker The build tool adds an option ‹use-yml which uses the YML file A parser (such as load.py above) is integrated into build.py to lay down he Dockerfiles Wait 4-6 weeks for people to find bugs and complain Make the ‹use-yml the default for 4-6 weeks Once we feel confident in the yml implementation, remove all Dockerfile.j2 files Remove ‹use-yml option Remove all jinja2-isms from build.py This is similar to the work that took place to convert from raw Dockerfiles to Dockerfile.j2 files. We are just reusing that pattern. Hopefully this will be the last major refactor of the dockerfiles unless someone has some significant complaints about the approach. >>>
Re: [openstack-dev] [kolla] prototype of a DSL for generating Dockerfiles
So this is prototype of working template overrides: https://review.openstack.org/#/c/323612/ Pass --template-overrides=path-to-file to build.py in file override you can add any custom code/logic/dockerfile stuff to any of hooks we provide in Dockerfiles, and we'll provide a lot of them as it's free and non breaking operation. With enough block you'll be able to do virtually anything with any of the containers. This one is already working. Only work needed is to provide more hooks/continue with refactoring of dockerfiles. Cheers, Michal On 31 May 2016 at 19:36, Steven Dake (stdake) wrote: > > > On 5/31/16, 1:42 PM, "Michał Jastrzębski" wrote: > >>I am opposed to this idea as I don't think we need this. We can solve >>many problems by using jinja2 to greater extend. I'll publish demo of >>few improvements soon, please bear with me before we make a >>arch-changing call. > > Can you make a specification please as you have asked me to do? > >> >>On 29 May 2016 at 14:41, Steven Dake (stdake) wrote: >>> On 5/27/16, 1:58 AM, "Steven Dake (stdake)" wrote: > > >On 5/26/16, 8:45 PM, "Swapnil Kulkarni (coolsvap)" >wrote: > >>On Fri, May 27, 2016 at 8:35 AM, Steven Dake (stdake) >> >>wrote: >>> Hey folks, >>> >>> While Swapnil has been busy churning the dockerfile.j2 files to all >>>match >>> the same style, and we also had summit where we declared we would >>>solve >>>the >>> plugin problem, I have decided to begin work on a DSL prototype. >>> >>> Here are the problems I want to solve in order of importance by this >>>work: >>> >>> Build CentOS, Ubuntu, Oracle Linux, Debian, Fedora containers >>> Provide a programmatic way to manage Dockerfile construction rather >>>then a >>> manual (with vi or emacs or the like) mechanism >>> Allow complete overrides of every facet of Dockerfile construction, >>>most >>> especially repositories per container (rather than in the base >>>container) to >>> permit the use case of dependencies from one version with >>>dependencies >>>in >>> another version of a different service >>> Get out of the business of maintaining 100+ dockerfiles but instead >>>maintain >>> one master file which defines the data that needs to be used to >>>construct >>> Dockerfiles >>> Permit different types of optimizations or Dockerfile building by >>>changing >>> around the parser implementation to allow layering of each >>>operation, >>>or >>> alternatively to merge layers as we do today >>> >>> I don't believe we can proceed with both binary and source plugins >>>given our >>> current implementation of Dockerfiles in any sane way. >>> >>> I further don't believe it is possible to customize repositories & >>>installed >>> files per container, which I receive increasing requests for >>>offline. >>> >>> To that end, I've created a very very rough prototype which builds >>>the >>>base >>> container as well as a mariadb container. The mariadb container >>>builds >>>and >>> I suspect would work. >>> >>> An example of the DSL usage is here: >>> https://review.openstack.org/#/c/321468/4/dockerdsl/dsl.yml >>> >>> A very poorly written parser is here: >>> https://review.openstack.org/#/c/321468/4/dockerdsl/load.py >>> >>> I played around with INI as a format, to take advantage of >>>oslo.config >>>and >>> kolla-build.conf, but that didn't work out. YML is the way to go. >>> >>> I'd appreciate reviews on the YML implementation especially. >>> >>> How I see this work progressing is as follows: >>> >>> A yml file describing all docker containers for all distros is >>>placed >>>in >>> kolla/docker >>> The build tool adds an option ‹use-yml which uses the YML file >>> A parser (such as load.py above) is integrated into build.py to lay >>>down he >>> Dockerfiles >>> Wait 4-6 weeks for people to find bugs and complain >>> Make the ‹use-yml the default for 4-6 weeks >>> Once we feel confident in the yml implementation, remove all >>>Dockerfile.j2 >>> files >>> Remove ‹use-yml option >>> Remove all jinja2-isms from build.py >>> >>> This is similar to the work that took place to convert from raw >>>Dockerfiles >>> to Dockerfile.j2 files. We are just reusing that pattern. >>>Hopefully >>>this >>> will be the last major refactor of the dockerfiles unless someone >>>has >>>some >>> significant complaints about the approach. >>> >>> Regards >>> -steve > > Hey folks, > > I have produced a specification for Kolla's DSL (which I call Elemental). > The spec is ready for review here: > https://review.openstack.org/#/c/323612/ > > > Regards > -steve > >>> >>> >>> On 5/27/16, 3:44 AM, "Britt Houser (bhouser)
Re: [openstack-dev] [kolla] prototype of a DSL for generating Dockerfiles
On 5/31/16, 1:42 PM, "Michał Jastrzębski" wrote: >I am opposed to this idea as I don't think we need this. We can solve >many problems by using jinja2 to greater extend. I'll publish demo of >few improvements soon, please bear with me before we make a >arch-changing call. Can you make a specification please as you have asked me to do? > >On 29 May 2016 at 14:41, Steven Dake (stdake) wrote: >> >>>On 5/27/16, 1:58 AM, "Steven Dake (stdake)" wrote: >>> On 5/26/16, 8:45 PM, "Swapnil Kulkarni (coolsvap)" wrote: >On Fri, May 27, 2016 at 8:35 AM, Steven Dake (stdake) > >wrote: >> Hey folks, >> >> While Swapnil has been busy churning the dockerfile.j2 files to all >>match >> the same style, and we also had summit where we declared we would >>solve >>the >> plugin problem, I have decided to begin work on a DSL prototype. >> >> Here are the problems I want to solve in order of importance by this >>work: >> >> Build CentOS, Ubuntu, Oracle Linux, Debian, Fedora containers >> Provide a programmatic way to manage Dockerfile construction rather >>then a >> manual (with vi or emacs or the like) mechanism >> Allow complete overrides of every facet of Dockerfile construction, >>most >> especially repositories per container (rather than in the base >>container) to >> permit the use case of dependencies from one version with >>dependencies >>in >> another version of a different service >> Get out of the business of maintaining 100+ dockerfiles but instead >>maintain >> one master file which defines the data that needs to be used to >>construct >> Dockerfiles >> Permit different types of optimizations or Dockerfile building by >>changing >> around the parser implementation to allow layering of each >>operation, >>or >> alternatively to merge layers as we do today >> >> I don't believe we can proceed with both binary and source plugins >>given our >> current implementation of Dockerfiles in any sane way. >> >> I further don't believe it is possible to customize repositories & >>installed >> files per container, which I receive increasing requests for >>offline. >> >> To that end, I've created a very very rough prototype which builds >>the >>base >> container as well as a mariadb container. The mariadb container >>builds >>and >> I suspect would work. >> >> An example of the DSL usage is here: >> https://review.openstack.org/#/c/321468/4/dockerdsl/dsl.yml >> >> A very poorly written parser is here: >> https://review.openstack.org/#/c/321468/4/dockerdsl/load.py >> >> I played around with INI as a format, to take advantage of >>oslo.config >>and >> kolla-build.conf, but that didn't work out. YML is the way to go. >> >> I'd appreciate reviews on the YML implementation especially. >> >> How I see this work progressing is as follows: >> >> A yml file describing all docker containers for all distros is >>placed >>in >> kolla/docker >> The build tool adds an option ‹use-yml which uses the YML file >> A parser (such as load.py above) is integrated into build.py to lay >>down he >> Dockerfiles >> Wait 4-6 weeks for people to find bugs and complain >> Make the ‹use-yml the default for 4-6 weeks >> Once we feel confident in the yml implementation, remove all >>Dockerfile.j2 >> files >> Remove ‹use-yml option >> Remove all jinja2-isms from build.py >> >> This is similar to the work that took place to convert from raw >>Dockerfiles >> to Dockerfile.j2 files. We are just reusing that pattern. >>Hopefully >>this >> will be the last major refactor of the dockerfiles unless someone >>has >>some >> significant complaints about the approach. >> >> Regards >> -steve Hey folks, I have produced a specification for Kolla's DSL (which I call Elemental). The spec is ready for review here: https://review.openstack.org/#/c/323612/ Regards -steve >> >> >> On 5/27/16, 3:44 AM, "Britt Houser (bhouser)" wrote: >> >>>I admit I'm not as knowledgable about the Kolla codebase as I'd like to >>>be, so most of what you're saying is going over my head. I think mainly >>>I don't understand the problem statement. It looks like you're pulling >>>all the "hard coded" things out of the docker files, and making them >>>user >>>replaceable? So the dockerfiles just become a list of required steps, >>>and the user can change how each step is implemented? Would this also >>>unify the dockefiles so there wouldn't be a huge if statements between >>>Centos and Ubuntu? >>> >>>Thx, >>>Britt >>> >> >> What is being pulled out is all of the metadata used by the Dockerfiles >>or >> Kolla in general. This metadata, being structured either as a >>dic
Re: [openstack-dev] [kolla] prototype of a DSL for generating Dockerfiles
I am opposed to this idea as I don't think we need this. We can solve many problems by using jinja2 to greater extend. I'll publish demo of few improvements soon, please bear with me before we make a arch-changing call. On 29 May 2016 at 14:41, Steven Dake (stdake) wrote: > >>On 5/27/16, 1:58 AM, "Steven Dake (stdake)" wrote: >> >>> >>> >>>On 5/26/16, 8:45 PM, "Swapnil Kulkarni (coolsvap)" >>>wrote: >>> On Fri, May 27, 2016 at 8:35 AM, Steven Dake (stdake) wrote: > Hey folks, > > While Swapnil has been busy churning the dockerfile.j2 files to all >match > the same style, and we also had summit where we declared we would >solve >the > plugin problem, I have decided to begin work on a DSL prototype. > > Here are the problems I want to solve in order of importance by this >work: > > Build CentOS, Ubuntu, Oracle Linux, Debian, Fedora containers > Provide a programmatic way to manage Dockerfile construction rather >then a > manual (with vi or emacs or the like) mechanism > Allow complete overrides of every facet of Dockerfile construction, >most > especially repositories per container (rather than in the base >container) to > permit the use case of dependencies from one version with dependencies >in > another version of a different service > Get out of the business of maintaining 100+ dockerfiles but instead >maintain > one master file which defines the data that needs to be used to >construct > Dockerfiles > Permit different types of optimizations or Dockerfile building by >changing > around the parser implementation to allow layering of each >operation, >or > alternatively to merge layers as we do today > > I don't believe we can proceed with both binary and source plugins >given our > current implementation of Dockerfiles in any sane way. > > I further don't believe it is possible to customize repositories & >installed > files per container, which I receive increasing requests for offline. > > To that end, I've created a very very rough prototype which builds the >base > container as well as a mariadb container. The mariadb container >builds >and > I suspect would work. > > An example of the DSL usage is here: > https://review.openstack.org/#/c/321468/4/dockerdsl/dsl.yml > > A very poorly written parser is here: > https://review.openstack.org/#/c/321468/4/dockerdsl/load.py > > I played around with INI as a format, to take advantage of oslo.config >and > kolla-build.conf, but that didn't work out. YML is the way to go. > > I'd appreciate reviews on the YML implementation especially. > > How I see this work progressing is as follows: > > A yml file describing all docker containers for all distros is placed >in > kolla/docker > The build tool adds an option ‹use-yml which uses the YML file > A parser (such as load.py above) is integrated into build.py to lay >down he > Dockerfiles > Wait 4-6 weeks for people to find bugs and complain > Make the ‹use-yml the default for 4-6 weeks > Once we feel confident in the yml implementation, remove all >Dockerfile.j2 > files > Remove ‹use-yml option > Remove all jinja2-isms from build.py > > This is similar to the work that took place to convert from raw >Dockerfiles > to Dockerfile.j2 files. We are just reusing that pattern. Hopefully >this > will be the last major refactor of the dockerfiles unless someone has >some > significant complaints about the approach. > > Regards > -steve > > > On 5/27/16, 3:44 AM, "Britt Houser (bhouser)" wrote: > >>I admit I'm not as knowledgable about the Kolla codebase as I'd like to >>be, so most of what you're saying is going over my head. I think mainly >>I don't understand the problem statement. It looks like you're pulling >>all the "hard coded" things out of the docker files, and making them user >>replaceable? So the dockerfiles just become a list of required steps, >>and the user can change how each step is implemented? Would this also >>unify the dockefiles so there wouldn't be a huge if statements between >>Centos and Ubuntu? >> >>Thx, >>Britt >> > > What is being pulled out is all of the metadata used by the Dockerfiles or > Kolla in general. This metadata, being structured either as a dictionary > or ordered list, can be manipulated by simple python tools to do things > like merge sections and override sections or optimize the built images. > FWIW it looks without even trying the Dockerfiles produce a 50MB smaller > image produced by the parser. The jinja2 templates we have today cannot > be easily overridden. We have to provide a new key for each type of > override, which is super onerous on the build.py tool. > > To your question of docker files being
Re: [openstack-dev] [kolla] prototype of a DSL for generating Dockerfiles
>On 5/27/16, 1:58 AM, "Steven Dake (stdake)" wrote: > >> >> >>On 5/26/16, 8:45 PM, "Swapnil Kulkarni (coolsvap)" >>wrote: >> >>>On Fri, May 27, 2016 at 8:35 AM, Steven Dake (stdake) >>>wrote: Hey folks, While Swapnil has been busy churning the dockerfile.j2 files to all match the same style, and we also had summit where we declared we would solve the plugin problem, I have decided to begin work on a DSL prototype. Here are the problems I want to solve in order of importance by this work: Build CentOS, Ubuntu, Oracle Linux, Debian, Fedora containers Provide a programmatic way to manage Dockerfile construction rather then a manual (with vi or emacs or the like) mechanism Allow complete overrides of every facet of Dockerfile construction, most especially repositories per container (rather than in the base container) to permit the use case of dependencies from one version with dependencies in another version of a different service Get out of the business of maintaining 100+ dockerfiles but instead maintain one master file which defines the data that needs to be used to construct Dockerfiles Permit different types of optimizations or Dockerfile building by changing around the parser implementation to allow layering of each operation, or alternatively to merge layers as we do today I don't believe we can proceed with both binary and source plugins given our current implementation of Dockerfiles in any sane way. I further don't believe it is possible to customize repositories & installed files per container, which I receive increasing requests for offline. To that end, I've created a very very rough prototype which builds the base container as well as a mariadb container. The mariadb container builds and I suspect would work. An example of the DSL usage is here: https://review.openstack.org/#/c/321468/4/dockerdsl/dsl.yml A very poorly written parser is here: https://review.openstack.org/#/c/321468/4/dockerdsl/load.py I played around with INI as a format, to take advantage of oslo.config and kolla-build.conf, but that didn't work out. YML is the way to go. I'd appreciate reviews on the YML implementation especially. How I see this work progressing is as follows: A yml file describing all docker containers for all distros is placed in kolla/docker The build tool adds an option ‹use-yml which uses the YML file A parser (such as load.py above) is integrated into build.py to lay down he Dockerfiles Wait 4-6 weeks for people to find bugs and complain Make the ‹use-yml the default for 4-6 weeks Once we feel confident in the yml implementation, remove all Dockerfile.j2 files Remove ‹use-yml option Remove all jinja2-isms from build.py This is similar to the work that took place to convert from raw Dockerfiles to Dockerfile.j2 files. We are just reusing that pattern. Hopefully this will be the last major refactor of the dockerfiles unless someone has some significant complaints about the approach. Regards -steve On 5/27/16, 3:44 AM, "Britt Houser (bhouser)" wrote: >I admit I'm not as knowledgable about the Kolla codebase as I'd like to >be, so most of what you're saying is going over my head. I think mainly >I don't understand the problem statement. It looks like you're pulling >all the "hard coded" things out of the docker files, and making them user >replaceable? So the dockerfiles just become a list of required steps, >and the user can change how each step is implemented? Would this also >unify the dockefiles so there wouldn't be a huge if statements between >Centos and Ubuntu? > >Thx, >Britt > What is being pulled out is all of the metadata used by the Dockerfiles or Kolla in general. This metadata, being structured either as a dictionary or ordered list, can be manipulated by simple python tools to do things like merge sections and override sections or optimize the built images. FWIW it looks without even trying the Dockerfiles produce a 50MB smaller image produced by the parser. The jinja2 templates we have today cannot be easily overridden. We have to provide a new key for each type of override, which is super onerous on the build.py tool. To your question of docker files being a list of required steps, with this method Dockerfile.j2 would go away permanently. On each bulid, as is done now to process a jinja2 file into a Dockerfile, the elemental.yml file would be processed into a Dockerfile for that particular set of build options. > ___ __ _ OpenStack Development Mailing List
Re: [openstack-dev] [kolla] prototype of a DSL for generating Dockerfiles
I admit I'm not as knowledgable about the Kolla codebase as I'd like to be, so most of what you're saying is going over my head. I think mainly I don't understand the problem statement. It looks like you're pulling all the "hard coded" things out of the docker files, and making them user replaceable? So the dockerfiles just become a list of required steps, and the user can change how each step is implemented? Would this also unify the dockefiles so there wouldn't be a huge if statements between Centos and Ubuntu? Thx, Britt On 5/27/16, 1:58 AM, "Steven Dake (stdake)" wrote: > > >On 5/26/16, 8:45 PM, "Swapnil Kulkarni (coolsvap)" wrote: > >>On Fri, May 27, 2016 at 8:35 AM, Steven Dake (stdake) >>wrote: >>> Hey folks, >>> >>> While Swapnil has been busy churning the dockerfile.j2 files to all >>>match >>> the same style, and we also had summit where we declared we would solve >>>the >>> plugin problem, I have decided to begin work on a DSL prototype. >>> >>> Here are the problems I want to solve in order of importance by this >>>work: >>> >>> Build CentOS, Ubuntu, Oracle Linux, Debian, Fedora containers >>> Provide a programmatic way to manage Dockerfile construction rather >>>then a >>> manual (with vi or emacs or the like) mechanism >>> Allow complete overrides of every facet of Dockerfile construction, most >>> especially repositories per container (rather than in the base >>>container) to >>> permit the use case of dependencies from one version with dependencies >>>in >>> another version of a different service >>> Get out of the business of maintaining 100+ dockerfiles but instead >>>maintain >>> one master file which defines the data that needs to be used to >>>construct >>> Dockerfiles >>> Permit different types of optimizations or Dockerfile building by >>>changing >>> around the parser implementation to allow layering of each operation, >>>or >>> alternatively to merge layers as we do today >>> >>> I don't believe we can proceed with both binary and source plugins >>>given our >>> current implementation of Dockerfiles in any sane way. >>> >>> I further don't believe it is possible to customize repositories & >>>installed >>> files per container, which I receive increasing requests for offline. >>> >>> To that end, I've created a very very rough prototype which builds the >>>base >>> container as well as a mariadb container. The mariadb container builds >>>and >>> I suspect would work. >>> >>> An example of the DSL usage is here: >>> https://review.openstack.org/#/c/321468/4/dockerdsl/dsl.yml >>> >>> A very poorly written parser is here: >>> https://review.openstack.org/#/c/321468/4/dockerdsl/load.py >>> >>> I played around with INI as a format, to take advantage of oslo.config >>>and >>> kolla-build.conf, but that didn't work out. YML is the way to go. >>> >>> I'd appreciate reviews on the YML implementation especially. >>> >>> How I see this work progressing is as follows: >>> >>> A yml file describing all docker containers for all distros is placed in >>> kolla/docker >>> The build tool adds an option ‹use-yml which uses the YML file >>> A parser (such as load.py above) is integrated into build.py to lay >>>down he >>> Dockerfiles >>> Wait 4-6 weeks for people to find bugs and complain >>> Make the ‹use-yml the default for 4-6 weeks >>> Once we feel confident in the yml implementation, remove all >>>Dockerfile.j2 >>> files >>> Remove ‹use-yml option >>> Remove all jinja2-isms from build.py >>> >>> This is similar to the work that took place to convert from raw >>>Dockerfiles >>> to Dockerfile.j2 files. We are just reusing that pattern. Hopefully >>>this >>> will be the last major refactor of the dockerfiles unless someone has >>>some >>> significant complaints about the approach. >>> >>> Regards >>> -steve >>> >>> >>> >>>_ >>>_ >>> OpenStack Development Mailing List (not for usage questions) >>> Unsubscribe: >>>openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >> >>The DSL template to generate the Dockerfile seems way better than the >>jinja templates in terms of extension which is currently a major >>bottleneck in the plugin implementation. I am +2+W on this plan of >>action to test it for next 4-6 weeks and see thereon. >> >>Swapnil >> > >Agree. > >Customization and plugins are the trigger for the work. I was thinking of >the following: > >Elemental.yml (ships with Kolla) >Elemental-merge.yml (operator provides in /etc/kolla, this file is yaml >merged with elemental.yml) >Elemental-override.yml (operator provides in /etc/kolla, this file >overrides any YAML sections defined) > >I think merging and overriding the yaml files should be pretty easy, >compared to jinja2, where I don't even know where to begin in a way that >the operator doesn't have to have deep knowledge of Kolla's internal >implementation. > >Regards
Re: [openstack-dev] [kolla] prototype of a DSL for generating Dockerfiles
On 5/26/16, 8:45 PM, "Swapnil Kulkarni (coolsvap)" wrote: >On Fri, May 27, 2016 at 8:35 AM, Steven Dake (stdake) >wrote: >> Hey folks, >> >> While Swapnil has been busy churning the dockerfile.j2 files to all >>match >> the same style, and we also had summit where we declared we would solve >>the >> plugin problem, I have decided to begin work on a DSL prototype. >> >> Here are the problems I want to solve in order of importance by this >>work: >> >> Build CentOS, Ubuntu, Oracle Linux, Debian, Fedora containers >> Provide a programmatic way to manage Dockerfile construction rather >>then a >> manual (with vi or emacs or the like) mechanism >> Allow complete overrides of every facet of Dockerfile construction, most >> especially repositories per container (rather than in the base >>container) to >> permit the use case of dependencies from one version with dependencies >>in >> another version of a different service >> Get out of the business of maintaining 100+ dockerfiles but instead >>maintain >> one master file which defines the data that needs to be used to >>construct >> Dockerfiles >> Permit different types of optimizations or Dockerfile building by >>changing >> around the parser implementation to allow layering of each operation, >>or >> alternatively to merge layers as we do today >> >> I don't believe we can proceed with both binary and source plugins >>given our >> current implementation of Dockerfiles in any sane way. >> >> I further don't believe it is possible to customize repositories & >>installed >> files per container, which I receive increasing requests for offline. >> >> To that end, I've created a very very rough prototype which builds the >>base >> container as well as a mariadb container. The mariadb container builds >>and >> I suspect would work. >> >> An example of the DSL usage is here: >> https://review.openstack.org/#/c/321468/4/dockerdsl/dsl.yml >> >> A very poorly written parser is here: >> https://review.openstack.org/#/c/321468/4/dockerdsl/load.py >> >> I played around with INI as a format, to take advantage of oslo.config >>and >> kolla-build.conf, but that didn't work out. YML is the way to go. >> >> I'd appreciate reviews on the YML implementation especially. >> >> How I see this work progressing is as follows: >> >> A yml file describing all docker containers for all distros is placed in >> kolla/docker >> The build tool adds an option ‹use-yml which uses the YML file >> A parser (such as load.py above) is integrated into build.py to lay >>down he >> Dockerfiles >> Wait 4-6 weeks for people to find bugs and complain >> Make the ‹use-yml the default for 4-6 weeks >> Once we feel confident in the yml implementation, remove all >>Dockerfile.j2 >> files >> Remove ‹use-yml option >> Remove all jinja2-isms from build.py >> >> This is similar to the work that took place to convert from raw >>Dockerfiles >> to Dockerfile.j2 files. We are just reusing that pattern. Hopefully >>this >> will be the last major refactor of the dockerfiles unless someone has >>some >> significant complaints about the approach. >> >> Regards >> -steve >> >> >> >>_ >>_ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: >>openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > >The DSL template to generate the Dockerfile seems way better than the >jinja templates in terms of extension which is currently a major >bottleneck in the plugin implementation. I am +2+W on this plan of >action to test it for next 4-6 weeks and see thereon. > >Swapnil > Agree. Customization and plugins are the trigger for the work. I was thinking of the following: Elemental.yml (ships with Kolla) Elemental-merge.yml (operator provides in /etc/kolla, this file is yaml merged with elemental.yml) Elemental-override.yml (operator provides in /etc/kolla, this file overrides any YAML sections defined) I think merging and overriding the yaml files should be pretty easy, compared to jinja2, where I don't even know where to begin in a way that the operator doesn't have to have deep knowledge of Kolla's internal implementation. Regards -steve >__ >OpenStack Development Mailing List (not for usage questions) >Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [kolla] prototype of a DSL for generating Dockerfiles
On Fri, May 27, 2016 at 8:35 AM, Steven Dake (stdake) wrote: > Hey folks, > > While Swapnil has been busy churning the dockerfile.j2 files to all match > the same style, and we also had summit where we declared we would solve the > plugin problem, I have decided to begin work on a DSL prototype. > > Here are the problems I want to solve in order of importance by this work: > > Build CentOS, Ubuntu, Oracle Linux, Debian, Fedora containers > Provide a programmatic way to manage Dockerfile construction rather then a > manual (with vi or emacs or the like) mechanism > Allow complete overrides of every facet of Dockerfile construction, most > especially repositories per container (rather than in the base container) to > permit the use case of dependencies from one version with dependencies in > another version of a different service > Get out of the business of maintaining 100+ dockerfiles but instead maintain > one master file which defines the data that needs to be used to construct > Dockerfiles > Permit different types of optimizations or Dockerfile building by changing > around the parser implementation – to allow layering of each operation, or > alternatively to merge layers as we do today > > I don't believe we can proceed with both binary and source plugins given our > current implementation of Dockerfiles in any sane way. > > I further don't believe it is possible to customize repositories & installed > files per container, which I receive increasing requests for offline. > > To that end, I've created a very very rough prototype which builds the base > container as well as a mariadb container. The mariadb container builds and > I suspect would work. > > An example of the DSL usage is here: > https://review.openstack.org/#/c/321468/4/dockerdsl/dsl.yml > > A very poorly written parser is here: > https://review.openstack.org/#/c/321468/4/dockerdsl/load.py > > I played around with INI as a format, to take advantage of oslo.config and > kolla-build.conf, but that didn't work out. YML is the way to go. > > I'd appreciate reviews on the YML implementation especially. > > How I see this work progressing is as follows: > > A yml file describing all docker containers for all distros is placed in > kolla/docker > The build tool adds an option —use-yml which uses the YML file > A parser (such as load.py above) is integrated into build.py to lay down he > Dockerfiles > Wait 4-6 weeks for people to find bugs and complain > Make the —use-yml the default for 4-6 weeks > Once we feel confident in the yml implementation, remove all Dockerfile.j2 > files > Remove —use-yml option > Remove all jinja2-isms from build.py > > This is similar to the work that took place to convert from raw Dockerfiles > to Dockerfile.j2 files. We are just reusing that pattern. Hopefully this > will be the last major refactor of the dockerfiles unless someone has some > significant complaints about the approach. > > Regards > -steve > > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > The DSL template to generate the Dockerfile seems way better than the jinja templates in terms of extension which is currently a major bottleneck in the plugin implementation. I am +2+W on this plan of action to test it for next 4-6 weeks and see thereon. Swapnil __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [kolla] prototype of a DSL for generating Dockerfiles
Hey folks, While Swapnil has been busy churning the dockerfile.j2 files to all match the same style, and we also had summit where we declared we would solve the plugin problem, I have decided to begin work on a DSL prototype. Here are the problems I want to solve in order of importance by this work: 1. Build CentOS, Ubuntu, Oracle Linux, Debian, Fedora containers 2. Provide a programmatic way to manage Dockerfile construction rather then a manual (with vi or emacs or the like) mechanism 3. Allow complete overrides of every facet of Dockerfile construction, most especially repositories per container (rather than in the base container) to permit the use case of dependencies from one version with dependencies in another version of a different service 4. Get out of the business of maintaining 100+ dockerfiles but instead maintain one master file which defines the data that needs to be used to construct Dockerfiles 5. Permit different types of optimizations or Dockerfile building by changing around the parser implementation - to allow layering of each operation, or alternatively to merge layers as we do today I don't believe we can proceed with both binary and source plugins given our current implementation of Dockerfiles in any sane way. I further don't believe it is possible to customize repositories & installed files per container, which I receive increasing requests for offline. To that end, I've created a very very rough prototype which builds the base container as well as a mariadb container. The mariadb container builds and I suspect would work. An example of the DSL usage is here: https://review.openstack.org/#/c/321468/4/dockerdsl/dsl.yml A very poorly written parser is here: https://review.openstack.org/#/c/321468/4/dockerdsl/load.py I played around with INI as a format, to take advantage of oslo.config and kolla-build.conf, but that didn't work out. YML is the way to go. I'd appreciate reviews on the YML implementation especially. How I see this work progressing is as follows: 1. A yml file describing all docker containers for all distros is placed in kolla/docker 2. The build tool adds an option -use-yml which uses the YML file 3. A parser (such as load.py above) is integrated into build.py to lay down he Dockerfiles 4. Wait 4-6 weeks for people to find bugs and complain 5. Make the -use-yml the default for 4-6 weeks 6. Once we feel confident in the yml implementation, remove all Dockerfile.j2 files 7. Remove -use-yml option 8. Remove all jinja2-isms from build.py This is similar to the work that took place to convert from raw Dockerfiles to Dockerfile.j2 files. We are just reusing that pattern. Hopefully this will be the last major refactor of the dockerfiles unless someone has some significant complaints about the approach. Regards -steve __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev