Still confused by: [base] -> [service] -> [+ puppet] not: [base] -> [puppet] and [base] -> [service] ?
Thanks, Kevin ________________________________________ From: Bogdan Dobrelya [bdobr...@redhat.com] Sent: Friday, November 30, 2018 5:31 AM To: Dan Prince; openstack-dev@lists.openstack.org; openstack-disc...@lists.openstack.org Subject: Re: [openstack-dev] [TripleO][Edge] Reduce base layer of containers for security and size of images (maintenance) sakes On 11/30/18 1:52 PM, Dan Prince wrote: > On Fri, 2018-11-30 at 10:31 +0100, Bogdan Dobrelya wrote: >> On 11/29/18 6:42 PM, Jiří Stránský wrote: >>> On 28. 11. 18 18:29, Bogdan Dobrelya wrote: >>>> On 11/28/18 6:02 PM, Jiří Stránský wrote: >>>>> <snip> >>>>> >>>>>> Reiterating again on previous points: >>>>>> >>>>>> -I'd be fine removing systemd. But lets do it properly and >>>>>> not via 'rpm >>>>>> -ev --nodeps'. >>>>>> -Puppet and Ruby *are* required for configuration. We can >>>>>> certainly put >>>>>> them in a separate container outside of the runtime service >>>>>> containers >>>>>> but doing so would actually cost you much more >>>>>> space/bandwidth for each >>>>>> service container. As both of these have to get downloaded to >>>>>> each node >>>>>> anyway in order to generate config files with our current >>>>>> mechanisms >>>>>> I'm not sure this buys you anything. >>>>> >>>>> +1. I was actually under the impression that we concluded >>>>> yesterday on >>>>> IRC that this is the only thing that makes sense to seriously >>>>> consider. >>>>> But even then it's not a win-win -- we'd gain some security by >>>>> leaner >>>>> production images, but pay for it with space+bandwidth by >>>>> duplicating >>>>> image content (IOW we can help achieve one of the goals we had >>>>> in mind >>>>> by worsening the situation w/r/t the other goal we had in >>>>> mind.) >>>>> >>>>> Personally i'm not sold yet but it's something that i'd >>>>> consider if we >>>>> got measurements of how much more space/bandwidth usage this >>>>> would >>>>> consume, and if we got some further details/examples about how >>>>> serious >>>>> are the security concerns if we leave config mgmt tools in >>>>> runtime >>>>> images. >>>>> >>>>> IIRC the other options (that were brought forward so far) were >>>>> already >>>>> dismissed in yesterday's IRC discussion and on the reviews. >>>>> Bin/lib bind >>>>> mounting being too hacky and fragile, and nsenter not really >>>>> solving the >>>>> problem (because it allows us to switch to having different >>>>> bins/libs >>>>> available, but it does not allow merging the availability of >>>>> bins/libs >>>>> from two containers into a single context). >>>>> >>>>>> We are going in circles here I think.... >>>>> >>>>> +1. I think too much of the discussion focuses on "why it's bad >>>>> to have >>>>> config tools in runtime images", but IMO we all sorta agree >>>>> that it >>>>> would be better not to have them there, if it came at no cost. >>>>> >>>>> I think to move forward, it would be interesting to know: if we >>>>> do this >>>>> (i'll borrow Dan's drawing): >>>>> >>>>>> base container| --> |service container| --> |service >>>>>> container w/ >>>>> Puppet installed| >>>>> >>>>> How much more space and bandwidth would this consume per node >>>>> (e.g. >>>>> separately per controller, per compute). This could help with >>>>> decision >>>>> making. >>>> >>>> As I've already evaluated in the related bug, that is: >>>> >>>> puppet-* modules and manifests ~ 16MB >>>> puppet with dependencies ~61MB >>>> dependencies of the seemingly largest a dependency, systemd >>>> ~190MB >>>> >>>> that would be an extra layer size for each of the container >>>> images to be >>>> downloaded/fetched into registries. >>> >>> Thanks, i tried to do the math of the reduction vs. inflation in >>> sizes >>> as follows. I think the crucial point here is the layering. If we >>> do >>> this image layering: >>> >>>> base| --> |+ service| --> |+ Puppet| >>> >>> we'd drop ~267 MB from base image, but we'd be installing that to >>> the >>> topmost level, per-component, right? >> >> Given we detached systemd from puppet, cronie et al, that would be >> 267-190MB, so the math below would be looking much better > > Would it be worth writing a spec that summarizes what action items are > bing taken to optimize our base image with regards to the systemd? Perhaps it would be. But honestly, I see nothing biggie to require a full blown spec. Just changing RPM deps and layers for containers images. I'm tracking systemd changes here [0],[1],[2], btw (if accepted, it should be working as of fedora28(or 29) I hope) [0] https://review.rdoproject.org/r/#/q/topic:base-container-reduction [1] https://bugzilla.redhat.com/show_bug.cgi?id=1654659 [2] https://bugzilla.redhat.com/show_bug.cgi?id=1654672 > > It seems like the general consenses is that cleaning up some of the RPM > dependencies so that we don't install Systemd is the biggest win. > > What confuses me is why are there still patches posted to move Puppet > out of the base layer when we agree moving it out of the base layer > would actually cause our resulting container image set to be larger in > size. > > Dan > > >> >>> In my basic deployment, undercloud seems to have 17 "components" >>> (49 >>> containers), overcloud controller 15 components (48 containers), >>> and >>> overcloud compute 4 components (7 containers). Accounting for >>> overlaps, >>> the total number of "components" used seems to be 19. (By >>> "components" >>> here i mean whatever uses a different ConfigImage than other >>> services. I >>> just eyeballed it but i think i'm not too far off the correct >>> number.) >>> >>> So we'd subtract 267 MB from base image and add that to 19 leaf >>> images >>> used in this deployment. That means difference of +4.8 GB to the >>> current >>> image sizes. My /var/lib/registry dir on undercloud with all the >>> images >>> currently has 5.1 GB. We'd almost double that to 9.9 GB. >>> >>> Going from 5.1 to 9.9 GB seems like a lot of extra traffic for the >>> CDNs >>> (both external and e.g. internal within OpenStack Infra CI clouds). >>> >>> And for internal traffic between local registry and overcloud >>> nodes, it >>> gives +3.7 GB per controller and +800 MB per compute. That may not >>> be so >>> critical but still feels like a considerable downside. >>> >>> Another gut feeling is that this way of image layering would take >>> longer >>> time to build and to run the modify-image Ansible role which we use >>> in >>> CI, so that could endanger how our CI jobs fit into the time limit. >>> We >>> could also probably measure this but i'm not sure if it's worth >>> spending >>> the time. >>> >>> All in all i'd argue we should be looking at different options >>> still. >>> >>>> Given that we should decouple systemd from all/some of the >>>> dependencies >>>> (an example topic for RDO [0]), that could save a 190MB. But it >>>> seems we >>>> cannot break the love of puppet and systemd as it heavily relies >>>> on the >>>> latter and changing packaging like that would higly likely affect >>>> baremetal deployments with puppet and systemd co-operating. >>> >>> Ack :/ >>> >>>> Long story short, we cannot shoot both rabbits with a single >>>> shot, not >>>> with puppet :) May be we could with ansible replacing puppet >>>> fully... >>>> So splitting config and runtime images is the only choice yet to >>>> address >>>> the raised security concerns. And let's forget about edge cases >>>> for now. >>>> Tossing around a pair of extra bytes over 40,000 WAN-distributed >>>> computes ain't gonna be our the biggest problem for sure. >>>> >>>> [0] >>>> https://review.rdoproject.org/r/#/q/topic:base-container-reduction >>>> >>>>>> Dan >>>>>> >>>>> >>>>> Thanks >>>>> >>>>> Jirka >>>>> >>>>> _______________________________________________________________ >>>>> ___________ >>>>> >>>>> OpenStack Development Mailing List (not for usage questions) >>>>> Unsubscribe: >>>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>> ___________________________________________________________________ >>> _______ >>> OpenStack Development Mailing List (not for usage questions) >>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsu >>> bscribe >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > -- Best regards, Bogdan Dobrelya, Irc #bogdando __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev