Re: Koji build root caching
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Dne 14.1.2015 v 23:46 Dennis Gilmore napsal(a): On Wed, 14 Jan 2015 16:40:35 +0100 Vít Ondruch vondr...@redhat.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Dne 14.1.2015 v 16:00 Dennis Gilmore napsal(a): On Wed, 14 Jan 2015 14:57:59 +0100 Miroslav Suchý msu...@redhat.com wrote: On 01/13/2015 06:01 PM, Dennis Gilmore wrote: that all being said. koji doesn't use any caching and will not use the lvm plugin. we make every buildroot from scratch using a fully clean environment to help with ensuring reproducability. You can cache and still preserve reproducability. What I'm planning for Copr is to do (every week/month) for chroot in fedora-20-x86_64 fedora-21_86_64 ... ; do mock --init $chroot done take snapshot of that. I plan to do that on VM level. And when new task come, I will just restore from that snapshot. And mock will start with already populated cache. So I will have better caching and yet reproducability. you really can't. you would need to make a new cache any time one of the packages in the minimal buildroot changes. Actually this is not anytime. newRepo has to be run, which is not run more then 4 times per hour I'd say. If the new snapshot is prepared as part of newRepo task, the mock could reuse the snapshot. ok I have a data point here https://ausil.fedorapeople.org/buildroots the file in the link is the output of a sql query on koji's db it gives the number of times a unique repo was used on a builder to do a build. Sorry, but would you mind to explain what the columns actually mean? Thanks. Vít so basically it tells you if we cached the buildroots somehow how many times they would get used. $ grep ^ 19 | buildroots|wc -l 1 $ grep ^ 18 | buildroots|wc -l 0 $ grep ^ 17 | buildroots|wc -l 0 $ grep ^ 16 | buildroots|wc -l 2 $ grep ^ 15 | buildroots|wc -l 1 $ grep ^ 14 | buildroots|wc -l 0 $ grep ^ 13 | buildroots|wc -l 8 $ grep ^ 12 | buildroots|wc -l 13 $ grep ^ 11 | buildroots|wc -l 14 $ grep ^ 10 | buildroots|wc -l 15 $ grep ^ 9 | buildroots|wc -l 20 $ grep ^ 8 | buildroots|wc -l 24 $ grep ^ 7 | buildroots|wc -l 40 $ grep ^ 6 | buildroots|wc -l 59 $ grep ^ 5 | buildroots|wc -l 84 $ grep ^ 4 | buildroots|wc -l 155 $ grep ^ 3 | buildroots|wc -l 409 $ grep ^ 2 | buildroots|wc -l 1446 $ grep ^ 1 | buildroots|wc -l 4794 What I get from this is that in the majority of cases caching won't help at all. The cost of creating the cache is higher than the benefit we would get. Dennis -BEGIN PGP SIGNATURE- Version: GnuPG v1 iQIcBAEBAgAGBQJUt9uZAAoJEAzgnueZF7h8NmoP/3igDT3GJBBNDttOMjF2aIKA y+F9ZyEMorp4nGYN2MK7tcwMC8fdwhBeNNI7OYkJqLru1dhsHszLj1VpOIE5gGaV gQmoULyDpklDBa0JCa+2kwI3LQGad9m6KjTXo3jhBrOhy+5wJnzxKIdi8GdZzDTL giHtnxMjt1JncbD0ve7jhx8QPb+x4ZssDG1KuXMOKl7Skz78qPH6ZVGJ3FP1fBnA 4KrBNKRgy2AtB0dqLIvr2m6itUVb2DCNVbqvoIWfKJJM6k+bPt5e90efCsb2ooxD gNCsjayzRqyMoFSQW+E4sLss+vV0vXajPIsb5eyWhIiEwbTcQQlqGKb1d5PjeVzn f4tCEVOr5jf0Mx7OLumqdpHPvnPI8ymZt1ZvF3S4SZxy2iiCgR3a40hc0oXoztRd 7LazKk5CKH+W0zhbXMOiKpCk85h8ysFXxcurOBQnRUdGiEgMLxLX5cMfcl41uVko PAU4n/QAh+6hWoA+ILWglUOrBlmX2XESTSDiUsxfDUpuxIm/amsq/+UD+K3z6EXO Bv9FbqLb9qVn10kJUDVE7vokBQ75RNGWCvmC3SkyN/gpXX3Ht/whqYTbiQ35iWWo jN8RGvDvdx4y/WV0RdmqmWphQDMr3K5DEIlKbbknqKyi2s3vk3wge8ApEkOtNVAn TbtDNyDokF0x6jj6/gy4 =k3fH -END PGP SIGNATURE- -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Koji build root caching
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Thu, 15 Jan 2015 16:24:09 +0100 Vít Ondruch vondr...@redhat.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Dne 14.1.2015 v 23:46 Dennis Gilmore napsal(a): On Wed, 14 Jan 2015 16:40:35 +0100 Vít Ondruch vondr...@redhat.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Dne 14.1.2015 v 16:00 Dennis Gilmore napsal(a): On Wed, 14 Jan 2015 14:57:59 +0100 Miroslav Suchý msu...@redhat.com wrote: On 01/13/2015 06:01 PM, Dennis Gilmore wrote: that all being said. koji doesn't use any caching and will not use the lvm plugin. we make every buildroot from scratch using a fully clean environment to help with ensuring reproducability. You can cache and still preserve reproducability. What I'm planning for Copr is to do (every week/month) for chroot in fedora-20-x86_64 fedora-21_86_64 ... ; do mock --init $chroot done take snapshot of that. I plan to do that on VM level. And when new task come, I will just restore from that snapshot. And mock will start with already populated cache. So I will have better caching and yet reproducability. you really can't. you would need to make a new cache any time one of the packages in the minimal buildroot changes. Actually this is not anytime. newRepo has to be run, which is not run more then 4 times per hour I'd say. If the new snapshot is prepared as part of newRepo task, the mock could reuse the snapshot. ok I have a data point here https://ausil.fedorapeople.org/buildroots the file in the link is the output of a sql query on koji's db it gives the number of times a unique repo was used on a builder to do a build. Sorry, but would you mind to explain what the columns actually mean? Sure The first column is the number of times a repo has been used on a individual builder The second column is the tag name that populates the buildroot The third column is the repo id each time koji runs a newRepo task the resulting repo gets a repo_id that koji uses to identify the repo in use for a given task. this way we can easily go back to a given repo to reproduce a build the fourth column is the host id each host in koji has a unique id. the result is that the the greatest bulk of repos are used once or twice on a given host and that we would get no benefit but a greater cost in caching minimal buildroots Dennis -BEGIN PGP SIGNATURE- Version: GnuPG v2 iQIcBAEBAgAGBQJUuE4sAAoJEH7ltONmPFDRZR8QAL+VCeF1VPI7WSc4Th+ZdvmR JQrEqJSud5y8gucuTyU2R+sYcLFTaZR2qcjWaPdzQS7XiIIhv68KlKbganq6/r4g bHEytdcG9mZXxV69Uojq7hbfpDvd0EJu6sKCAy/uz2fuOJt/mdvxld5nVIpHBvJH vd8j+XN6jFOjb3FTVCrW/gxXEKW2+1eA7XsECu+e7XS1rBNC04FkFlZEbK9thMoQ K0ICXhrOtLXaara7ZuC7afP+xBvuTHjUQiFWrUDPrCtPKZXiXjXfk0rRFfhY+VHO ZJP7OR/RkLQ3dDVgRnHsVTxu5He0XdVnrsGmnPBneS+ThH46HEfVTMOjs4dUVcbW EjSzsYlPhiHCBMOi21NPgBH0XhB6x/9mCi6NYUTCFHkP8OHLQTVtcbA37gp6weqh 0UEhHtnmPCC/cD0UbmOhRD4ac75RQAMLt8vQr98g2HHzsis30gW1/A6KlBxZvhe5 d0ESfQEp3o/ex1wsw/b3oVuI9fPSoYH/c9nU8yEqZhx9bP8CoSXmvgHxxRBooZgf /7Uc+jANIJx0vFL8BPPPOhgGhOGe1VsJpUZ4gIH5Ee+vmXMqQOa4rOcWTLlmUjUD t60DuORK0gQdiXh6sljpXnyDYcJXleDXqq89hztdnCtaN+pUQsPhzawTnibjDlTL 1mGTikF3qAVK2EPJtOTN =I42g -END PGP SIGNATURE- -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
Re: Koji build root caching
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On Wed, 14 Jan 2015 16:40:35 +0100 Vít Ondruch vondr...@redhat.com wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Dne 14.1.2015 v 16:00 Dennis Gilmore napsal(a): On Wed, 14 Jan 2015 14:57:59 +0100 Miroslav Suchý msu...@redhat.com wrote: On 01/13/2015 06:01 PM, Dennis Gilmore wrote: that all being said. koji doesn't use any caching and will not use the lvm plugin. we make every buildroot from scratch using a fully clean environment to help with ensuring reproducability. You can cache and still preserve reproducability. What I'm planning for Copr is to do (every week/month) for chroot in fedora-20-x86_64 fedora-21_86_64 ... ; do mock --init $chroot done take snapshot of that. I plan to do that on VM level. And when new task come, I will just restore from that snapshot. And mock will start with already populated cache. So I will have better caching and yet reproducability. you really can't. you would need to make a new cache any time one of the packages in the minimal buildroot changes. Actually this is not anytime. newRepo has to be run, which is not run more then 4 times per hour I'd say. If the new snapshot is prepared as part of newRepo task, the mock could reuse the snapshot. ok I have a data point here https://ausil.fedorapeople.org/buildroots the file in the link is the output of a sql query on koji's db it gives the number of times a unique repo was used on a builder to do a build. so basically it tells you if we cached the buildroots somehow how many times they would get used. $ grep ^ 19 | buildroots|wc -l 1 $ grep ^ 18 | buildroots|wc -l 0 $ grep ^ 17 | buildroots|wc -l 0 $ grep ^ 16 | buildroots|wc -l 2 $ grep ^ 15 | buildroots|wc -l 1 $ grep ^ 14 | buildroots|wc -l 0 $ grep ^ 13 | buildroots|wc -l 8 $ grep ^ 12 | buildroots|wc -l 13 $ grep ^ 11 | buildroots|wc -l 14 $ grep ^ 10 | buildroots|wc -l 15 $ grep ^ 9 | buildroots|wc -l 20 $ grep ^ 8 | buildroots|wc -l 24 $ grep ^ 7 | buildroots|wc -l 40 $ grep ^ 6 | buildroots|wc -l 59 $ grep ^ 5 | buildroots|wc -l 84 $ grep ^ 4 | buildroots|wc -l 155 $ grep ^ 3 | buildroots|wc -l 409 $ grep ^ 2 | buildroots|wc -l 1446 $ grep ^ 1 | buildroots|wc -l 4794 What I get from this is that in the majority of cases caching won't help at all. The cost of creating the cache is higher than the benefit we would get. Dennis -BEGIN PGP SIGNATURE- Version: GnuPG v2 iQIcBAEBAgAGBQJUtvHgAAoJEH7ltONmPFDRP/sP/1iteUyvThkZLQGH4o+MY4uX oIwdgwQQwkkjvpCyjFP1yyO5aJa/BTiOGSMtT0vQtnfAl+qYs7IiqGH6VJkNt+Io Lmqr0yItXLr6ZyTwuKha2XYa88g+q657VmaJT/OmXGqi0Ezx8Rj/6NLXcV6twgKW LD3YjxtPmyxX79p7RrkrI5+iJ+Iyw48HLepZup+XOfydOJ0RWZccRVlm0c0MYzt1 P6H0M+iiDikLl2NE1/jCTx+/P9yQNSQ8zhMVCjhevGiHSHmhIZmcpPib/IYGfRBN OmgTaQ9XSaePNNrE4iqq6bQQ9fKqpA1FDlj1U2IaEHTh0CQ9zLCqKBuOl0ky98zY /m3obzNSYu441tl/1Ig6imzRXga/iQ0jo8WnQoW8Grh3JtGpNM6NVPEUkkyKWCxE XuagkPGWqP7x8YT/czn0Y9ZGjA+jIijzhdcAJ8R9R94wuhy+wvY5F69z6FKSycX6 Mho6yGj6cOrIB+F3Vs2+zn4rmtsU8jT7uhYc81CHJ0pEilcnINvFW9VDw7gYKq1E LFzxMeixQugtGO39PqVFdG7WErqX2pjInJLu/CZIVz0bofQ17RuE006yl1pS4+jG HoF6pCbZlapsRWS6TqXiMPhTSruszbZ733zihOgbDNSB6kIXZgI9nqj/jN8s8Dw1 ikGLTDN8nrnFxWMSt9pY =/MGg -END PGP SIGNATURE- -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel Fedora Code of Conduct: http://fedoraproject.org/code-of-conduct
