Re: how to speed up mock?
On Mon, Jan 18, 2010 at 17:30, Ville Skyttä ville.sky...@iki.fi wrote: On Monday 18 January 2010, Seth Vidal wrote: On Mon, 18 Jan 2010, Farkas Levente wrote: the real bottleneck is not the rpmbuild itself (with ccache it cab be very fast), but the mock surroundings. suppose there is build which takes about 2 minutes and in mock it takes about 5 minutes:-( most of the time is in yum, python tar, gzip etc which all use only one cpu/core and it's very slow! the tar and gzip are mostly BUILDING the cache. I've been using lzop as the root cache compressor, it makes a difference over gzip here, especially when compressing the cache, but somewhat also when decompressing it. Add this to /etc/mock/site-defaults.cfg to try it out: config_opts['plugin_conf']['root_cache_opts']['compress_program'] = 'lzop' config_opts['plugin_conf']['root_cache_opts']['extension'] = '.lzo' For boxes with multiple cores, pigz (and maybe even pbzip2) might be worth looking into. I haven't tried these myself. why we compress at all? disk space is cheep. can we speed up mock to somehow totally disable compressing root_cache? it used the case but store and restore in an uncompressed way? thanks. -- Levente Si vis pacem para bellum! -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: how to speed up mock?
Le 03/04/2011 12:31, Farkas Levente a écrit : the real bottleneck is not the rpmbuild itself (with ccache it cab be very fast), but the mock surroundings. suppose there is build which takes about 2 minutes and in mock it takes about 5 minutes:-( I use for a while a full /dev/shm mock configuration which is really fast (minimum overhead of ~10), read: http://blog.famillecollet.com/post/2011/01/11/new-builder-new-mock-configuration Of course I need a big /dev/shm (6Gio here) Remi. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: how to speed up mock?
Am 03.04.11 13:17, schrieb Remi Collet: Le 03/04/2011 12:31, Farkas Levente a écrit : the real bottleneck is not the rpmbuild itself (with ccache it cab be very fast), but the mock surroundings. suppose there is build which takes about 2 minutes and in mock it takes about 5 minutes:-( I use for a while a full /dev/shm mock configuration which is really fast (minimum overhead of ~10), read: http://blog.famillecollet.com/post/2011/01/11/new-builder-new-mock-configuration Of course I need a big /dev/shm (6Gio here) Remi. config_opts['plugin_conf']['ccache_opts']['dir'] = /dev/shm/ccache.fc14x/ Oh.. /dev/shm is the wrong place here, I think... Should better be /tmp with a tmpfs mounted on. -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: how to speed up mock?
On 01/18/2010 04:10 PM, Seth Vidal wrote: On Mon, 18 Jan 2010, Farkas Levente wrote: the real bottleneck is not the rpmbuild itself (with ccache it cab be very fast), but the mock surroundings. suppose there is build which takes about 2 minutes and in mock it takes about 5 minutes:-( most of the time is in yum, python tar, gzip etc which all use only one cpu/core and it's very slow! the tar and gzip are mostly BUILDING the cache. no tar and gzip used unpacking root cache. Mock currently makes a cached copy of the buildroot it just created so it doesn't have to redepsolve and download/install the rpms each time. but have to run yum each time for the package specific depsolve and yum installs (ps axufww:) --- \_ /usr/bin/python -tt /usr/sbin/mock -r testing-i386 --define rhel 5 --define el5 1 --define dist .el5 --rebuild /home/robot/rpm/SRPMS/xyz-4.2.2-5280.el5.src.rpm root 28319 49.5 0.8 255000 34292 pts/1D16:15 0:00 | \_ /usr/bin/python /usr/bin/yum --installroot /var/lib/mock/testing-i386/root/ install ccache rsync zip --- and it much slower then the compile itself. it's very annoying when i try to rebuild only a dozen of packages most of the time. eg in this output: --- INFO: mock.py version 1.0.2 starting... State Changed: init plugins State Changed: start INFO: Start(/home/robot/rpm/SRPMS/xyz-4.2.2-5280.el5.src.rpm) Config(testing-i386) State Changed: lock buildroot State Changed: clean State Changed: init State Changed: lock buildroot Mock Version: 1.0.2 INFO: Mock Version: 1.0.2 INFO: enabled root cache State Changed: unpacking root cache INFO: enabled yum cache State Changed: cleaning yum metadata INFO: enabled ccache State Changed: running yum State Changed: setup State Changed: build INFO: Done(/home/robot/rpm/SRPMS/xyz-4.2.2-5280.el5.src.rpm) Config(testing-i386) 3 minutes 37 seconds INFO: Results and/or logs in: /var/lib/mock/testing-i386/result --- the time reach the State Changed: build is usually more then all other stuff before it. So the first time you run it makes a cache. You aren't clearing out the cache each time, are you? That would definitely eat up a lot of time. of course not:-) -- Levente Si vis pacem para bellum! -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: how to speed up mock?
On Mon, 18 Jan 2010, Farkas Levente wrote: the tar and gzip are mostly BUILDING the cache. no tar and gzip used unpacking root cache. How slow are your disks? You're not doing any of this to nfs are you? but have to run yum each time for the package specific depsolve and yum installs (ps axufww:) --- \_ /usr/bin/python -tt /usr/sbin/mock -r testing-i386 --define rhel 5 --define el5 1 --define dist .el5 --rebuild /home/robot/rpm/SRPMS/xyz-4.2.2-5280.el5.src.rpm root 28319 49.5 0.8 255000 34292 pts/1D16:15 0:00 | \_ /usr/bin/python /usr/bin/yum --installroot /var/lib/mock/testing-i386/root/ install ccache rsync zip --- and it much slower then the compile itself. it's very annoying when i try to rebuild only a dozen of packages most of the time. eg in this output: How much of this is network access and how much is disk? B/c I doubt very much that you're cpu bound at all. -sv -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: how to speed up mock?
Farkas, Don't email just to me offlist. Keep this onlist. How much of this is network access and how much is disk? B/c I doubt very much that you're cpu bound at all. everything is on the local mirror server which is on a gigabit lan. is there any way to banchmark mock and different part of it? in the mock config you're using (/etc/mock/.) look for: config_opts['yum.conf'] = [main] cachedir=/var/cache/yum debuglevel=1 reposdir=/dev/null logfile=/var/log/yum.log retries=20 obsoletes=1 gpgcheck=0 assumeyes=1 and change debuglevel=1 to debuglevel=3 and then look at the root.log and look for lines that contain: time: that will tell you how long is being spent in each section post those times. -sv -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: how to speed up mock?
On Monday 18 January 2010, Seth Vidal wrote: On Mon, 18 Jan 2010, Farkas Levente wrote: the real bottleneck is not the rpmbuild itself (with ccache it cab be very fast), but the mock surroundings. suppose there is build which takes about 2 minutes and in mock it takes about 5 minutes:-( most of the time is in yum, python tar, gzip etc which all use only one cpu/core and it's very slow! the tar and gzip are mostly BUILDING the cache. I've been using lzop as the root cache compressor, it makes a difference over gzip here, especially when compressing the cache, but somewhat also when decompressing it. Add this to /etc/mock/site-defaults.cfg to try it out: config_opts['plugin_conf']['root_cache_opts']['compress_program'] = 'lzop' config_opts['plugin_conf']['root_cache_opts']['extension'] = '.lzo' For boxes with multiple cores, pigz (and maybe even pbzip2) might be worth looking into. I haven't tried these myself. Another thing I've been doing is to increase the max age for various root caches, for example to 60 for all EPEL roots, and 30 for released Fedora versions, i.e. something like this in the respective distro config in /etc/mock/*.cfg: config_opts['plugin_conf']['root_cache_opts']['max_age_days'] = 60 -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: how to speed up mock?
On 10-01-18 11:34:44, Ville Skyttä wrote: ... So instead of modifying specfiles, one can do something like this in /etc/mock/site-defaults.cfg: config_opts['macros']['%_smp_mflags'] = '-j3' Unless `rpmbuild --showrc` shows a bad definition for _smp_mflags, you're probably better off letting the macro decide. IME, using a number larger than the actual number of CPUs slows the build a lot, due to cache thrashing. Modern CPUs perform well only with hot caches, and each switch to the other build process will replace the cache contents. The OP said that rpmbuild was using the CPUs properly and ran fast, but earlier parts of mock were not and ran slowly, and take most of the time. He indicated that most of the time is spent extracting the build root from the cache. -- TonyN.:' mailto:tonynel...@georgeanelson.com ' http://www.georgeanelson.com/ -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: how to speed up mock?
On Monday 18 January 2010, Tony Nelson wrote: On 10-01-18 11:34:44, Ville Skyttä wrote: ... So instead of modifying specfiles, one can do something like this in /etc/mock/site-defaults.cfg: config_opts['macros']['%_smp_mflags'] = '-j3' Unless `rpmbuild --showrc` shows a bad definition for _smp_mflags, you're probably better off letting the macro decide. Possibly, but this was just to point out that there's no need to modify spec files if you want to have a custom value for it. Anyway, for single core machines, it's better to set %_smp_mflags manually to -jX (X 1) in any case, not only for build speed reasons, see https://fedoraproject.org/wiki/Packaging/Guidelines#Parallel_make -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: how to speed up mock?
On Mon, 2010-01-18 at 10:10 -0500, Seth Vidal wrote: So the first time you run it makes a cache. You aren't clearing out the cache each time, are you? That would definitely eat up a lot of time. Or running builds a long time apart, as the cache gets aged out. On my system (an overclocked Q6600, 4GB RAM) I can get through a trivial build in about 1:30 in mock if it uses a cached environment; minimum of 5 minutes if it doesn't. -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Re: how to speed up mock?
On Mon, 2010-01-18 at 15:30 -0500, Tony Nelson wrote: On 10-01-18 11:34:44, Ville Skyttä wrote: ... So instead of modifying specfiles, one can do something like this in /etc/mock/site-defaults.cfg: config_opts['macros']['%_smp_mflags'] = '-j3' Unless `rpmbuild --showrc` shows a bad definition for _smp_mflags, or, more easily: rpm --eval %_smp_mflags -- Adam Williamson Fedora QA Community Monkey IRC: adamw | Fedora Talk: adamwill AT fedoraproject DOT org http://www.happyassassin.net -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
