> 4) do lrzipping as weak as possible during the development phase and lrzip it strongly only when we do release
We create lrzip archive with compression level "2" (from 1-9 range, default is 7). So we don't have much potential for improvement here. Here are some tests: == Bare metal 12 cpus, 32 G RAM == level 1 decompression total time: 00:00:27.70 level 2 decompression total time: 00:00:29.64 == Virtual server 1 cpu, 1G RAM == level 1 decompression total time: 00:05:17.12 level 2 decompression total time: 00:05:22.86 I did some further research on this matter, increasing RAM to 4G did not help much as well: == Virtual server 1 cpu, 4G RAM == level 1 decompression total time: 00:05:08.70 level 2 decompression total time: 00:05:12.57 So it looks like 'lrzuntar' command itself causes this problem on a weak hardware, because it uses "lrzcat | tar" which works very fast if we have enough memory. If we extract archive in two separate steps (lrzip -d && tar -xf) instead of single "lrzuntar", then they take only ~1m30s summary (comparing to 5+ minutes with single lrzuntar command). Here are some test results for lrzip+tar commands: == Bare metal 12 cpus, 32 G RAM == level 2 decompression total time: 00:00:50.12 == Virtual server 1 cpu, 1G RAM == level 2 decompression total time: 00:01:35.95 So "lrzuntar" works faster on a powerful hardware, lrzip+tar works faster on a weak hardware (like VMs). I suggest to switch to lrzip+tar. On Sat, May 10, 2014 at 8:01 PM, Dmitry Borodaenko <dborodae...@mirantis.com > wrote: > FWIW 1GB works fine for me on my laptop, I run the master setup manually. > So I'm against increasing RAM requirement, we have better things to spend > that RAM on. > On May 10, 2014 1:37 AM, "Mike Scherbakov" <mscherba...@mirantis.com> > wrote: > >> It is not related to RAM or CPU. I run installation on my Mac with 1Gb of >> RAM for master node, and experience the following: >> >> - yes, it needs time to bootstrap admin node >> - As soon as I have message that master node is installed, I >> immediately open 10.20.0.2:8000 and try to generate diag snapshot. >> And it is failed. >> - If I wait a few more minutes, and try again - it is passed. >> >> It actually seems to me that we simply still do not have >> https://bugs.launchpad.net/fuel/+bug/1315865 fixed, I'll add more >> details there as well as logs. >> >> When I checked logs, I saw: >> >> - for about a minute, astute was not able to connect to MQ. It means >> it is still started before MQ is ready? >> - shotgun -c /tmp/dump_config >> /var/log/dump.log 2>&1 && cat >> /var/www/nailgun/dump/last returned 1 >> >> When I tried to run diag_snapshot for a second time, the command above >> succeeded with 0 return code. >> >> So it obviously needs further debugging and in my opinion even if we need >> to increase VCPU or RAM, then no more than 2 VCPU / 2 Gb. >> >> Vladimir, as you and Matt were changing the code which should run >> containers in a certain order, I'm looking forward to hear from both of you >> suggestions on where and how we should hack it. >> >> Thanks, >> >> >> On Sat, May 10, 2014 at 1:04 AM, Vladimir Kuklin <vkuk...@mirantis.com>wrote: >> >>> Hi all >>> >>> We are still experiencing some issues with master node bootstrapping >>> after moving to container-based installation. >>> >>> First of all, these issues are related to our system tests. We have >>> rather small nodes running as master node - only 1 GB of RAM and 1 virtual >>> CPU. As we are using strongly lrzipped archive, this seems quite not enough >>> and leads to timeouts during deployment of the master node. >>> >>> I have several suggestions: >>> >>> 1) Increase amount of RAM for master node to at least 8 Gigabytes (or >>> do some pci virtual memory hotplug during master node bootstrapping) and >>> add additional vCPU for the master node. >>> 2) Run system tests with non-containerized environment (env variable >>> PRODUCTION=prod set) >>> 3) Split our system tests in that way not allowing more than 2 master >>> nodes to bootstrap simulteneously on the single hardware node. >>> 4) do lrzipping as weak as possible during the development phase and >>> lrzip it strongly only when we do release >>> 5) increase bootstrap timeout for the master node in system tests >>> >>> >>> Any input would be appreciated. >>> >>> -- >>> Yours Faithfully, >>> Vladimir Kuklin, >>> Fuel Library Tech Lead, >>> Mirantis, Inc. >>> +7 (495) 640-49-04 >>> +7 (926) 702-39-68 >>> Skype kuklinvv >>> 45bk3, Vorontsovskaya Str. >>> Moscow, Russia, >>> www.mirantis.com <http://www.mirantis.ru/> >>> www.mirantis.ru >>> vkuk...@mirantis.com >>> >>> _______________________________________________ >>> OpenStack-dev mailing list >>> OpenStack-dev@lists.openstack.org >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >>> >> >> >> -- >> Mike Scherbakov >> #mihgen >> >> >> _______________________________________________ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.org >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> >> > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev