On 19 October 2017 at 13:37, Michał Jastrzębski <inc...@gmail.com> wrote: > On 19 October 2017 at 13:32, Joshua Harlow <harlo...@fastmail.com> wrote: >> This reminded me of something I wanted to ask. >> >> Is it true to state that only way to get 'fully' shared-base layers is to >> have `kolla-build` build all the projects (that a person/company/other may >> use) in one invocation? (in part because of the jinja2 template generation >> which would cause differences in dockerfiles?...) > > Well jinja2 should render same dockerfile no matter when you call it, > so it should be fine. Alternatively you can run something like > kolla-build nova --skip-parents - this call will try to build all > images with "nova" in them while not rebuilding openstack-base and > base image. > >> I was pretty sure this was the case (unless things have changed), but just >> wanting to check since that question seems (somewhat) on-topic... >> >> At godaddy we build individual projects using `kolla-build` (in part because >> it makes it easy to rebuild + test + deply a single project with either an >> update or a patch or ...) and I suspect others are doing this also (after >> all the kolla-build command does take a regex of projects to build) - though >> doing it in this way does seem like it would not reuse (all the layers >> outside of the base operating system) layers 'optimally'? >> >> Thoughts? >> >> -Josh >> >> Sam Yaple wrote: >>> >>> docker_image wouldn't be the best place for that. Buf if you are looking >>> for a quicker solution, kolla_docker was written specifically to be >>> license compatible for openstack. its structure should make it easily >>> adapted to delete an image. And you can copy it and cut it up thanks to >>> the license. >>> >>> Are you pushing images with no shared base layers at all (300MB >>> compressed image is no shared base layers)? With shared base layers a >>> full image set of Kolla images should be much smaller than the numbers >>> you posted. >>> >>> Thanks, >>> SamYaple >>> >>> On Thu, Oct 19, 2017 at 11:03 AM, Gabriele Cerami <gcer...@redhat.com >>> <mailto:gcer...@redhat.com>> wrote: >>> >>> Hi, >>> >>> our CI scripts are now automatically building, testing and pushing >>> approved openstack/RDO services images to public repositories in >>> dockerhub using ansible docker_image module. >>> >>> Promotions have had some hiccups, but we're starting to regularly >>> upload >>> new images every 4 hours. >>> >>> When we'll get at full speed, we'll potentially have >>> - 3-4 different sets of images, one per release of openstack (counting >>> a >>> EOL release grace period) >>> - 90-100 different services images per release >>> - 4-6 different versions of the same image ( keeping older promoted >>> images for a while ) >>> >>> At around 300MB per image a possible grand total is around 650GB of >>> space used.
That doesn't sound correct as images share a lot - full registry of single type/distro (like centos source) is ~10gig >>> We don't know if this is acceptable usage of dockerhub space and for >>> this we already sent a similar email the to docker support to ask >>> specifically if the user would get penalty in any way (e.g. enforcing >>> quotas, rete limiting, blocking). We're still waiting for a reply. >>> >>> In any case it's critical to keep the usage around the estimate, and >>> to >>> achieve this we need a way to automatically delete the older images. >>> docker_image module does not provide this functionality, and we think >>> the only way is issuing direct calls to dockerhub API >>> >>> https://docs.docker.com/registry/spec/api/#deleting-an-image >>> <https://docs.docker.com/registry/spec/api/#deleting-an-image> >>> >>> docker_image module structure doesn't seem to encourage the addition >>> of >>> such functionality directly in it, so we may be forced to use the uri >>> module. >>> With new images uploaded potentially every 4 hours, this will become a >>> problem to be solved within the next two weeks. >>> >>> We'd appreciate any input for an existing, in progress and/or better >>> solution for bulk deletion, and issues that may arise with our space >>> usage in dockerhub >>> >>> Thanks >>> >>> >>> __________________________________________________________________________ >>> OpenStack Development Mailing List (not for usage questions) >>> Unsubscribe: >>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>> <http://openstack-dev-requ...@lists.openstack.org?subject:unsubscribe> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev> >>> >>> >>> __________________________________________________________________________ >>> OpenStack Development Mailing List (not for usage questions) >>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> >> __________________________________________________________________________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev