Re: [openstack-dev] [Fuel] Use lrzip for upgrade tarball - reject?
I've updated the spec: https://review.openstack.org/#/c/116874/ Major change in this spec: get rid of unpacked upgrade tarball. Use only lrzipped archives. It will save disk space and network traffic, it will make upgrade process longer, it will make our upgrade tests longer as well, it will make things simpler. Code is already merged, docs are on review, we need to update our system tests and jenkins jobs. It will be done after merge of the spec. On Tue, Aug 26, 2014 at 6:07 PM, Aleksandra Fedorova afedor...@mirantis.com wrote: As an update, please check and review commit [1] to fuel-specs with detailed feature description. According to this feature, we are going to switch our CI system to lrzipped tarballs. [1] https://review.openstack.org/#/c/116874/ On Thu, Aug 21, 2014 at 5:50 PM, Dmitry Pyzhov dpyz...@mirantis.com wrote: Fuelers, Our upgrade tarball for 5.1 is more than 4.5Gb. We can reduce it size by 2Gb with lrzip tool (ticket https://bugs.launchpad.net/fuel/+bug/1356813, change in build system https://review.openstack.org/#/c/114201/, change in docs https://review.openstack.org/#/c/115331/), but it will dramatically increase unpacking time. I've run unpack on my virtualbox environment and got this result: [root@fuel var]# lrzuntar fuel-5.1-upgrade.tar.lrz Decompressing... 100%7637.48 / 7637.48 MB Average DeCompression Speed: 8.014MB/s [OK] - 8008478720 bytes Total time: 00:15:52.93 My suggestion is to reject this change, release 5.1 with big tarball and find another solution in next release. Any objections? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Aleksandra Fedorova bookwar ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] Use lrzip for upgrade tarball - reject?
As an update, please check and review commit [1] to fuel-specs with detailed feature description. According to this feature, we are going to switch our CI system to lrzipped tarballs. [1] https://review.openstack.org/#/c/116874/ On Thu, Aug 21, 2014 at 5:50 PM, Dmitry Pyzhov dpyz...@mirantis.com wrote: Fuelers, Our upgrade tarball for 5.1 is more than 4.5Gb. We can reduce it size by 2Gb with lrzip tool (ticket https://bugs.launchpad.net/fuel/+bug/1356813, change in build system https://review.openstack.org/#/c/114201/, change in docs https://review.openstack.org/#/c/115331/), but it will dramatically increase unpacking time. I've run unpack on my virtualbox environment and got this result: [root@fuel var]# lrzuntar fuel-5.1-upgrade.tar.lrz Decompressing... 100%7637.48 / 7637.48 MB Average DeCompression Speed: 8.014MB/s [OK] - 8008478720 bytes Total time: 00:15:52.93 My suggestion is to reject this change, release 5.1 with big tarball and find another solution in next release. Any objections? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Aleksandra Fedorova bookwar ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] Use lrzip for upgrade tarball - reject?
On 21 August 2014 18:47, Sergii Golovatiuk sgolovat...@mirantis.com wrote: I think 15 minutes is not too bad. Additionally, it will reduce download time and price for bandwidth. It's worth to leave lrzip for customers, as upgrade is one time operation so user can wait for a while. For development it would be nice to have the fastest solution to boost development time. I would agree. Perhaps for 6 an option can be made to allow the Fuel master to use package repositories instead of an upgrade file - and the option can be used for development and production? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] Use lrzip for upgrade tarball - reject?
Dmitry, we already use lrzuntar in deploying Docker containers. Use a lower compression ratio and it will decompress faster on virtual envs and it takes under two mins on my virtual env. Compress: https://github.com/stackforge/fuel-main/blob/master/docker/module.mk#L27 Decompress: https://github.com/stackforge/fuel-main/blob/master/iso/bootstrap_admin_node.docker.sh#L63 On Aug 21, 2014 5:54 PM, Dmitry Pyzhov dpyz...@mirantis.com wrote: Fuelers, Our upgrade tarball for 5.1 is more than 4.5Gb. We can reduce it size by 2Gb with lrzip tool (ticket https://bugs.launchpad.net/fuel/+bug/1356813, change in build system https://review.openstack.org/#/c/114201/, change in docs https://review.openstack.org/#/c/115331/), but it will dramatically increase unpacking time. I've run unpack on my virtualbox environment and got this result: [root@fuel var]# lrzuntar fuel-5.1-upgrade.tar.lrz Decompressing... 100%7637.48 / 7637.48 MB Average DeCompression Speed: 8.014MB/s [OK] - 8008478720 bytes Total time: 00:15:52.93 My suggestion is to reject this change, release 5.1 with big tarball and find another solution in next release. Any objections? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] Use lrzip for upgrade tarball - reject?
I think 15 minutes is not too bad. Additionally, it will reduce download time and price for bandwidth. +1 But let's see if we can find some other solution still in 5.1 (hardlinks, whatever else), and we obviously need to seriously address it in next release. Perhaps for 6 an option can be made to allow the Fuel master to use package repositories instead of an upgrade file - and the option can be used for development and production? Jessee, could please clarify this? Do you mean to use remote repository with packages, instead of tarballing everything into single bundle? On Fri, Aug 22, 2014 at 12:39 PM, Matthew Mosesohn mmoses...@mirantis.com wrote: Dmitry, we already use lrzuntar in deploying Docker containers. Use a lower compression ratio and it will decompress faster on virtual envs and it takes under two mins on my virtual env. Compress: https://github.com/stackforge/fuel-main/blob/master/docker/module.mk#L27 Decompress: https://github.com/stackforge/fuel-main/blob/master/iso/bootstrap_admin_node.docker.sh#L63 On Aug 21, 2014 5:54 PM, Dmitry Pyzhov dpyz...@mirantis.com wrote: Fuelers, Our upgrade tarball for 5.1 is more than 4.5Gb. We can reduce it size by 2Gb with lrzip tool (ticket https://bugs.launchpad.net/fuel/+bug/1356813, change in build system https://review.openstack.org/#/c/114201/, change in docs https://review.openstack.org/#/c/115331/), but it will dramatically increase unpacking time. I've run unpack on my virtualbox environment and got this result: [root@fuel var]# lrzuntar fuel-5.1-upgrade.tar.lrz Decompressing... 100%7637.48 / 7637.48 MB Average DeCompression Speed: 8.014MB/s [OK] - 8008478720 bytes Total time: 00:15:52.93 My suggestion is to reject this change, release 5.1 with big tarball and find another solution in next release. Any objections? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Mike Scherbakov #mihgen ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] Use lrzip for upgrade tarball - reject?
Mike, others, I believe Jesse was proposing an upgrade that downloads all the files separately on the Fuel Master itself. This is a move that we've gone away from since Fuel 2.0 because of intermittent issues with 3rd party mirrors. It's often better to consume 1 large file that has everything and can be verified than to try to pull hundreds of separate bits that can't be verified, plus trying to track down errors when something small doesn't work. Locking down to a single install base really contributed to Fuel's success early on, so I don't think moving back to separate file downloads is a good idea. However, clearly we should do our best to compress the upgrade package file as best as possible so it is less expensive to transfer and also consume. On Fri, Aug 22, 2014 at 1:43 PM, Mike Scherbakov mscherba...@mirantis.com wrote: I think 15 minutes is not too bad. Additionally, it will reduce download time and price for bandwidth. +1 But let's see if we can find some other solution still in 5.1 (hardlinks, whatever else), and we obviously need to seriously address it in next release. Perhaps for 6 an option can be made to allow the Fuel master to use package repositories instead of an upgrade file - and the option can be used for development and production? Jessee, could please clarify this? Do you mean to use remote repository with packages, instead of tarballing everything into single bundle? On Fri, Aug 22, 2014 at 12:39 PM, Matthew Mosesohn mmoses...@mirantis.com wrote: Dmitry, we already use lrzuntar in deploying Docker containers. Use a lower compression ratio and it will decompress faster on virtual envs and it takes under two mins on my virtual env. Compress: https://github.com/stackforge/fuel-main/blob/master/docker/module.mk#L27 Decompress: https://github.com/stackforge/fuel-main/blob/master/iso/bootstrap_admin_node.docker.sh#L63 On Aug 21, 2014 5:54 PM, Dmitry Pyzhov dpyz...@mirantis.com wrote: Fuelers, Our upgrade tarball for 5.1 is more than 4.5Gb. We can reduce it size by 2Gb with lrzip tool (ticket, change in build system, change in docs), but it will dramatically increase unpacking time. I've run unpack on my virtualbox environment and got this result: [root@fuel var]# lrzuntar fuel-5.1-upgrade.tar.lrz Decompressing... 100%7637.48 / 7637.48 MB Average DeCompression Speed: 8.014MB/s [OK] - 8008478720 bytes Total time: 00:15:52.93 My suggestion is to reject this change, release 5.1 with big tarball and find another solution in next release. Any objections? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Mike Scherbakov #mihgen ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
[openstack-dev] [Fuel] Use lrzip for upgrade tarball - reject?
Fuelers, Our upgrade tarball for 5.1 is more than 4.5Gb. We can reduce it size by 2Gb with lrzip tool (ticket https://bugs.launchpad.net/fuel/+bug/1356813, change in build system https://review.openstack.org/#/c/114201/, change in docs https://review.openstack.org/#/c/115331/), but it will dramatically increase unpacking time. I've run unpack on my virtualbox environment and got this result: [root@fuel var]# lrzuntar fuel-5.1-upgrade.tar.lrz Decompressing... 100%7637.48 / 7637.48 MB Average DeCompression Speed: 8.014MB/s [OK] - 8008478720 bytes Total time: 00:15:52.93 My suggestion is to reject this change, release 5.1 with big tarball and find another solution in next release. Any objections? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] Use lrzip for upgrade tarball - reject?
What are other possible solutions to this issue? On Thu, Aug 21, 2014 at 5:50 PM, Dmitry Pyzhov dpyz...@mirantis.com wrote: Fuelers, Our upgrade tarball for 5.1 is more than 4.5Gb. We can reduce it size by 2Gb with lrzip tool (ticket https://bugs.launchpad.net/fuel/+bug/1356813, change in build system https://review.openstack.org/#/c/114201/, change in docs https://review.openstack.org/#/c/115331/), but it will dramatically increase unpacking time. I've run unpack on my virtualbox environment and got this result: [root@fuel var]# lrzuntar fuel-5.1-upgrade.tar.lrz Decompressing... 100%7637.48 / 7637.48 MB Average DeCompression Speed: 8.014MB/s [OK] - 8008478720 bytes Total time: 00:15:52.93 My suggestion is to reject this change, release 5.1 with big tarball and find another solution in next release. Any objections? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Mike Scherbakov #mihgen ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] Use lrzip for upgrade tarball - reject?
I see no other quick solutions in 5.1. We can find the difference in packages between 5.0 and 5.0.2, put only updated packages in tarball and get missed packages from existing repos on master node. On Thu, Aug 21, 2014 at 5:55 PM, Mike Scherbakov mscherba...@mirantis.com wrote: What are other possible solutions to this issue? On Thu, Aug 21, 2014 at 5:50 PM, Dmitry Pyzhov dpyz...@mirantis.com wrote: Fuelers, Our upgrade tarball for 5.1 is more than 4.5Gb. We can reduce it size by 2Gb with lrzip tool (ticket https://bugs.launchpad.net/fuel/+bug/1356813, change in build system https://review.openstack.org/#/c/114201/, change in docs https://review.openstack.org/#/c/115331/), but it will dramatically increase unpacking time. I've run unpack on my virtualbox environment and got this result: [root@fuel var]# lrzuntar fuel-5.1-upgrade.tar.lrz Decompressing... 100%7637.48 / 7637.48 MB Average DeCompression Speed: 8.014MB/s [OK] - 8008478720 bytes Total time: 00:15:52.93 My suggestion is to reject this change, release 5.1 with big tarball and find another solution in next release. Any objections? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Mike Scherbakov #mihgen ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] Use lrzip for upgrade tarball - reject?
Hi, Hmm.. I think ~15 minutes isn't long enough to skip this approach in production. What about using lrzip only for end-users, but keep regular tarball for CI and internal usage? Thanks, Igor On Thu, Aug 21, 2014 at 5:22 PM, Dmitry Pyzhov dpyz...@mirantis.com wrote: I see no other quick solutions in 5.1. We can find the difference in packages between 5.0 and 5.0.2, put only updated packages in tarball and get missed packages from existing repos on master node. On Thu, Aug 21, 2014 at 5:55 PM, Mike Scherbakov mscherba...@mirantis.com wrote: What are other possible solutions to this issue? On Thu, Aug 21, 2014 at 5:50 PM, Dmitry Pyzhov dpyz...@mirantis.com wrote: Fuelers, Our upgrade tarball for 5.1 is more than 4.5Gb. We can reduce it size by 2Gb with lrzip tool (ticket, change in build system, change in docs), but it will dramatically increase unpacking time. I've run unpack on my virtualbox environment and got this result: [root@fuel var]# lrzuntar fuel-5.1-upgrade.tar.lrz Decompressing... 100%7637.48 / 7637.48 MB Average DeCompression Speed: 8.014MB/s [OK] - 8008478720 bytes Total time: 00:15:52.93 My suggestion is to reject this change, release 5.1 with big tarball and find another solution in next release. Any objections? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Mike Scherbakov #mihgen ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] Use lrzip for upgrade tarball - reject?
Hi, I think 15 minutes is not too bad. Additionally, it will reduce download time and price for bandwidth. It's worth to leave lrzip for customers, as upgrade is one time operation so user can wait for a while. For development it would be nice to have the fastest solution to boost development time. -- Best regards, Sergii Golovatiuk, Skype #golserge IRC #holser On Thu, Aug 21, 2014 at 5:40 PM, Igor Kalnitsky ikalnit...@mirantis.com wrote: Hi, Hmm.. I think ~15 minutes isn't long enough to skip this approach in production. What about using lrzip only for end-users, but keep regular tarball for CI and internal usage? Thanks, Igor On Thu, Aug 21, 2014 at 5:22 PM, Dmitry Pyzhov dpyz...@mirantis.com wrote: I see no other quick solutions in 5.1. We can find the difference in packages between 5.0 and 5.0.2, put only updated packages in tarball and get missed packages from existing repos on master node. On Thu, Aug 21, 2014 at 5:55 PM, Mike Scherbakov mscherba...@mirantis.com wrote: What are other possible solutions to this issue? On Thu, Aug 21, 2014 at 5:50 PM, Dmitry Pyzhov dpyz...@mirantis.com wrote: Fuelers, Our upgrade tarball for 5.1 is more than 4.5Gb. We can reduce it size by 2Gb with lrzip tool (ticket, change in build system, change in docs), but it will dramatically increase unpacking time. I've run unpack on my virtualbox environment and got this result: [root@fuel var]# lrzuntar fuel-5.1-upgrade.tar.lrz Decompressing... 100%7637.48 / 7637.48 MB Average DeCompression Speed: 8.014MB/s [OK] - 8008478720 bytes Total time: 00:15:52.93 My suggestion is to reject this change, release 5.1 with big tarball and find another solution in next release. Any objections? ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Mike Scherbakov #mihgen ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
