Re: [openstack-dev] Disk space requirement - any way to lower it a little?
On 07/20/2018 09:49 AM, Cédric Jeanneret wrote: > > > On 07/19/2018 06:55 PM, Paul Belanger wrote: >> On Thu, Jul 19, 2018 at 05:30:27PM +0200, Cédric Jeanneret wrote: >>> Hello, >>> >>> While trying to get a new validation¹ in the undercloud preflight >>> checks, I hit an (not so) unexpected issue with the CI: >>> it doesn't provide flavors with the minimal requirements, at least >>> regarding the disk space. >>> >>> A quick-fix is to disable the validations in the CI - Wes has already >>> pushed a patch for that in the upstream CI: >>> https://review.openstack.org/#/c/583275/ >>> We can consider this as a quick'n'temporary fix². >>> >>> The issue is on the RDO CI: apparently, they provide instances with >>> "only" 55G of free space, making the checks fail: >>> https://logs.rdoproject.org/17/582917/3/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/c9cf398/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2018-07-17_10_23_46 >>> >>> So, the question is: would it be possible to lower the requirment to, >>> let's say, 50G? Where does that 60G³ come from? >>> >>> Thanks for your help/feedback. >>> >>> Cheers, >>> >>> C. >>> >>> >>> >>> ¹ https://review.openstack.org/#/c/582917/ >>> >>> ² as you might know, there's a BP for a unified validation framework, >>> and it will allow to get injected configuration in CI env in order to >>> lower the requirements if necessary: >>> https://blueprints.launchpad.net/tripleo/+spec/validation-framework >>> >>> ³ >>> http://tripleo.org/install/environments/baremetal.html#minimum-system-requirements >>> >> Keep in mind, upstream we don't really have control over partitions of >> nodes, in >> some case it is a single, other multiple. I'd suggest looking more at: > > After some checks on y locally deployed containerized undercloud (hence, > Rocky) without real activity, here's what I could get: > - most data are located in /var - this explains the current check. > > If we go a bit deeper, here are the "actually used" directory in /var/lib: > 20K alternatives > 36K certmonger > 4.0K chrony > 1.2G config-data > 4.0K dhclient > 6.0G docker > 28K docker-config-scripts > 92K docker-container-startup-configs.json > 44K docker-puppet > 592K heat-config > 832K ironic > 4.0K ironic-inspector > 236K kolla > 4.0K logrotate > 286M mysql > 48K neutron > 4.0K ntp > 4.0K postfix > 872K puppet > 3.8M rabbitmq > 59M rpm > 4.0K rsyslog > 64K systemd > 20K tripleo > 236K tripleo-config > 9.8M yum > 7.5G total > > Most of the "default installer" partition schema don't go further than > putting /var, /tmp, /home and /usr in dedicated volumes - of course, > end-user can chose to ignore that and provide a custom schema. > > That said, we can get the "used" paths. In addition to /var/lib, there's > obviously /usr. > > We might want to: > - loop on known locations > - check if they are on dedicated mount points > - check the available disk space on those mount points. > > An interesting thing in bash: > df /var/lib/docker > Filesystem 1K-blocks Used Available Use% Mounted on > /dev/sda1 104846316 10188828 94657488 10% / > > This allows to get: > - actual volume > - free space on the volume. > > More than that, we might also try to figure out some pattern. For > instance, "docker" seems to be a pretty good candidate for space, as it > will get the images and container data. This is probably even the > biggest eater, at least on the undercloud - as well as the logs (/var/logs). > > We might do a check ensuring we can, at least, DEPLOY the app. This > would require far less than the required 60G, and with a proper doc > announcing that, we can get a functional test, aiming on its purpose: > ensure we can deploy (so asking, let's say, 10G in /var/lib/docker, 5G > in /var/lib/config-data, 5G in /usr, 1G in /var/log) and, later, upgrade > (requiring the same amount of *free* space). > > That would require some changes in the validation check of course. But > at least, we might get a pretty nice covering, while allowing it to run > smoothly in the CI. > But, as said: proper documentation should be set, and the "60G minimum > required" should be rephrased in order to point the locations needing > space (with the appropriate warning about "none exhaustiveness" and the > like). > > Would that suit better the actual needs, and allow to get a proper disk > space check/validation? > > Cheers, > > C. Following those thoughts, here's a proposal, to be discussed, augmented, enhanced: https://review.openstack.org/#/c/584314/ This should allow to get a really nice space check, and in addition allow ops to create a layout suited for the undercloud if they want - getting dedicated volumes for specific uses, allowing to get a smart monitoring of the disk usage per resources is always good. It kind of also allow to sort out the issue of the CI, providing we update the doc to reflect the "new" reality of this
Re: [openstack-dev] Disk space requirement - any way to lower it a little?
On 07/19/2018 06:55 PM, Paul Belanger wrote: > On Thu, Jul 19, 2018 at 05:30:27PM +0200, Cédric Jeanneret wrote: >> Hello, >> >> While trying to get a new validation¹ in the undercloud preflight >> checks, I hit an (not so) unexpected issue with the CI: >> it doesn't provide flavors with the minimal requirements, at least >> regarding the disk space. >> >> A quick-fix is to disable the validations in the CI - Wes has already >> pushed a patch for that in the upstream CI: >> https://review.openstack.org/#/c/583275/ >> We can consider this as a quick'n'temporary fix². >> >> The issue is on the RDO CI: apparently, they provide instances with >> "only" 55G of free space, making the checks fail: >> https://logs.rdoproject.org/17/582917/3/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/c9cf398/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2018-07-17_10_23_46 >> >> So, the question is: would it be possible to lower the requirment to, >> let's say, 50G? Where does that 60G³ come from? >> >> Thanks for your help/feedback. >> >> Cheers, >> >> C. >> >> >> >> ¹ https://review.openstack.org/#/c/582917/ >> >> ² as you might know, there's a BP for a unified validation framework, >> and it will allow to get injected configuration in CI env in order to >> lower the requirements if necessary: >> https://blueprints.launchpad.net/tripleo/+spec/validation-framework >> >> ³ >> http://tripleo.org/install/environments/baremetal.html#minimum-system-requirements >> > Keep in mind, upstream we don't really have control over partitions of nodes, > in > some case it is a single, other multiple. I'd suggest looking more at: After some checks on y locally deployed containerized undercloud (hence, Rocky) without real activity, here's what I could get: - most data are located in /var - this explains the current check. If we go a bit deeper, here are the "actually used" directory in /var/lib: 20K alternatives 36K certmonger 4.0Kchrony 1.2Gconfig-data 4.0Kdhclient 6.0Gdocker 28K docker-config-scripts 92K docker-container-startup-configs.json 44K docker-puppet 592Kheat-config 832Kironic 4.0Kironic-inspector 236Kkolla 4.0Klogrotate 286Mmysql 48K neutron 4.0Kntp 4.0Kpostfix 872Kpuppet 3.8Mrabbitmq 59M rpm 4.0Krsyslog 64K systemd 20K tripleo 236Ktripleo-config 9.8Myum 7.5Gtotal Most of the "default installer" partition schema don't go further than putting /var, /tmp, /home and /usr in dedicated volumes - of course, end-user can chose to ignore that and provide a custom schema. That said, we can get the "used" paths. In addition to /var/lib, there's obviously /usr. We might want to: - loop on known locations - check if they are on dedicated mount points - check the available disk space on those mount points. An interesting thing in bash: df /var/lib/docker Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda1 104846316 10188828 94657488 10% / This allows to get: - actual volume - free space on the volume. More than that, we might also try to figure out some pattern. For instance, "docker" seems to be a pretty good candidate for space, as it will get the images and container data. This is probably even the biggest eater, at least on the undercloud - as well as the logs (/var/logs). We might do a check ensuring we can, at least, DEPLOY the app. This would require far less than the required 60G, and with a proper doc announcing that, we can get a functional test, aiming on its purpose: ensure we can deploy (so asking, let's say, 10G in /var/lib/docker, 5G in /var/lib/config-data, 5G in /usr, 1G in /var/log) and, later, upgrade (requiring the same amount of *free* space). That would require some changes in the validation check of course. But at least, we might get a pretty nice covering, while allowing it to run smoothly in the CI. But, as said: proper documentation should be set, and the "60G minimum required" should be rephrased in order to point the locations needing space (with the appropriate warning about "none exhaustiveness" and the like). Would that suit better the actual needs, and allow to get a proper disk space check/validation? Cheers, C. > > https://docs.openstack.org/infra/manual/testing.html > > As for downstream RDO, the same is going to apply once we start adding more > cloud providers. I would look to see if you actually need that much space for > deployments, and make try to mock the testing of that logic. > > - Paul > > __ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > -- Cédric Jeanneret Software Engineer DFG:DF signature.asc Description: OpenPGP digital signature
Re: [openstack-dev] Disk space requirement - any way to lower it a little?
On 07/19/2018 11:55 AM, Paul Belanger wrote: On Thu, Jul 19, 2018 at 05:30:27PM +0200, Cédric Jeanneret wrote: Hello, While trying to get a new validation¹ in the undercloud preflight checks, I hit an (not so) unexpected issue with the CI: it doesn't provide flavors with the minimal requirements, at least regarding the disk space. A quick-fix is to disable the validations in the CI - Wes has already pushed a patch for that in the upstream CI: https://review.openstack.org/#/c/583275/ We can consider this as a quick'n'temporary fix². The issue is on the RDO CI: apparently, they provide instances with "only" 55G of free space, making the checks fail: https://logs.rdoproject.org/17/582917/3/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/c9cf398/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2018-07-17_10_23_46 So, the question is: would it be possible to lower the requirment to, let's say, 50G? Where does that 60G³ come from? Thanks for your help/feedback. Cheers, C. ¹ https://review.openstack.org/#/c/582917/ ² as you might know, there's a BP for a unified validation framework, and it will allow to get injected configuration in CI env in order to lower the requirements if necessary: https://blueprints.launchpad.net/tripleo/+spec/validation-framework ³ http://tripleo.org/install/environments/baremetal.html#minimum-system-requirements Keep in mind, upstream we don't really have control over partitions of nodes, in some case it is a single, other multiple. I'd suggest looking more at: https://docs.openstack.org/infra/manual/testing.html And this isn't just a testing thing. As I mentioned in the previous thread, real-world users often use separate partitions for some data (logs, for example). Looking at the existing validation[1] I don't know that it would handle multiple partitions sufficiently well to turn it on by default. It's only checking /var and /, and I've seen much more complex partition layouts than that. 1: https://github.com/openstack/tripleo-validations/blob/master/validations/tasks/disk_space.yaml As for downstream RDO, the same is going to apply once we start adding more cloud providers. I would look to see if you actually need that much space for deployments, and make try to mock the testing of that logic. It's also worth noting that what we can get away with in ci is not necessarily appropriate for production. Being able to run a short-lived, single-use deployment in 50 GB doesn't mean that you could realistically run that on a long-lived production cloud. Log and database storage tends to increase over time. There should be a ceiling to how large that all grows if rotation and db cleanup is configured correctly, but that ceiling is much higher than anything ci is ever going to hit. Anecdotally, I bumped my development flavor disk space to >50 GB because I ran out of space when I built containers locally. I don't know if that's something we expect users to be doing, but it is definitely possible to exhaust 50 GB in a short period of time. __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] Disk space requirement - any way to lower it a little?
On Thu, Jul 19, 2018 at 05:30:27PM +0200, Cédric Jeanneret wrote: > Hello, > > While trying to get a new validation¹ in the undercloud preflight > checks, I hit an (not so) unexpected issue with the CI: > it doesn't provide flavors with the minimal requirements, at least > regarding the disk space. > > A quick-fix is to disable the validations in the CI - Wes has already > pushed a patch for that in the upstream CI: > https://review.openstack.org/#/c/583275/ > We can consider this as a quick'n'temporary fix². > > The issue is on the RDO CI: apparently, they provide instances with > "only" 55G of free space, making the checks fail: > https://logs.rdoproject.org/17/582917/3/openstack-check/legacy-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/c9cf398/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2018-07-17_10_23_46 > > So, the question is: would it be possible to lower the requirment to, > let's say, 50G? Where does that 60G³ come from? > > Thanks for your help/feedback. > > Cheers, > > C. > > > > ¹ https://review.openstack.org/#/c/582917/ > > ² as you might know, there's a BP for a unified validation framework, > and it will allow to get injected configuration in CI env in order to > lower the requirements if necessary: > https://blueprints.launchpad.net/tripleo/+spec/validation-framework > > ³ > http://tripleo.org/install/environments/baremetal.html#minimum-system-requirements > Keep in mind, upstream we don't really have control over partitions of nodes, in some case it is a single, other multiple. I'd suggest looking more at: https://docs.openstack.org/infra/manual/testing.html As for downstream RDO, the same is going to apply once we start adding more cloud providers. I would look to see if you actually need that much space for deployments, and make try to mock the testing of that logic. - Paul __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
