Re: [openstack-dev] [Fuel] Cache for packages on master node
There where some questions regarding direction for this on the fuel meeting today. Can you elaborate on the status? On Thu, Feb 12, 2015 at 12:01 PM, Andrew Woodward xar...@gmail.com wrote: On Thu, Feb 12, 2015 at 3:59 AM, Tomasz Napierala tnapier...@mirantis.com wrote: On 10 Feb 2015, at 23:02, Andrew Woodward xar...@gmail.com wrote: previously we used squid in 3.0 and before. The main problem is that the deployment would proceeded even if not all the packages where cached or even available on the remote. This often lead to broken deployments that where hard to debug and a waste of alot of time. This _MUST_ be resolved or we will re-introduce this horrible work flow that we had placed all the packages on the system for to begin with. Anyway we need to ensure our QA is run against fresh mirror, that would prevent a lot of problems. We also think about how situation in the field can differ from our labs and QA infra - there might be differences indeed. I think we need to add a requirements that we need to be able to: a) pre-populate the cacher b) we need to not start the deployment until we either have every package in the chache (eiew) or at least know every package is reachable currently (or allow the user to select either as a deployment criteria) This sounds for me like creating local mirror ;) We don’t want to do this. We are thinking about mirror verification tool, it was mentioned by eifferent guys already. Do you really think we should prepopulate cache? by pre-populate, I mean that we need to start some form of task that can be started to create a repo/mirror of the packages we know we need for the installation. The source of where this would be built from could be an ISO, or equally any other mirror site. The user should be able to use this as a base population for the packages. If the mirror is incomplete this should be OK also as long as the user is told that their nodes will attempt to get the remaining packages from the internet. The task should be able to be run at any time, and if desired the user would be able to make the deployment require it to finish first. so yes, we need both a repo/mirror like now, with a passthrough that might use a squid proxy to help with multiple access. Keep in mind that the squid proxy would have to work with the virtual router for nodes bp [1] I hink first node deployment will fetch a lot of packages, and other nodes will be easier. Once we have prototype, we will see some number. The first OS install will fetch packages, then later the fist of each roll will fetch different packages, it's possible we could get all the way to compute and fail there because we cant get a package. I can personally promise that without something else this will have problems with this the same as we did before with 3.0 (I could run two squid layers one in my host and one on my fuel vm and still have problems (usually cache misses)). When this occurs the result is terrible, hard for not fuel people to realize, and you will end up restarting the whole deployment. the user experience (UX) from this is horrible. We need the tools to prevent this from occurring at all. Regards, -- Tomasz 'Zen' Napierala Sr. OpenStack Engineer tnapier...@mirantis.com __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev [1] https://blueprints.launchpad.net/fuel/+spec/virtual-router-for-env-nodes -- Andrew Mirantis Fuel community ambassador Ceph community -- Andrew Mirantis Fuel community ambassador Ceph community __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] Cache for packages on master node
On Thu, Feb 12, 2015 at 3:59 AM, Tomasz Napierala tnapier...@mirantis.com wrote: On 10 Feb 2015, at 23:02, Andrew Woodward xar...@gmail.com wrote: previously we used squid in 3.0 and before. The main problem is that the deployment would proceeded even if not all the packages where cached or even available on the remote. This often lead to broken deployments that where hard to debug and a waste of alot of time. This _MUST_ be resolved or we will re-introduce this horrible work flow that we had placed all the packages on the system for to begin with. Anyway we need to ensure our QA is run against fresh mirror, that would prevent a lot of problems. We also think about how situation in the field can differ from our labs and QA infra - there might be differences indeed. I think we need to add a requirements that we need to be able to: a) pre-populate the cacher b) we need to not start the deployment until we either have every package in the chache (eiew) or at least know every package is reachable currently (or allow the user to select either as a deployment criteria) This sounds for me like creating local mirror ;) We don’t want to do this. We are thinking about mirror verification tool, it was mentioned by eifferent guys already. Do you really think we should prepopulate cache? by pre-populate, I mean that we need to start some form of task that can be started to create a repo/mirror of the packages we know we need for the installation. The source of where this would be built from could be an ISO, or equally any other mirror site. The user should be able to use this as a base population for the packages. If the mirror is incomplete this should be OK also as long as the user is told that their nodes will attempt to get the remaining packages from the internet. The task should be able to be run at any time, and if desired the user would be able to make the deployment require it to finish first. so yes, we need both a repo/mirror like now, with a passthrough that might use a squid proxy to help with multiple access. Keep in mind that the squid proxy would have to work with the virtual router for nodes bp [1] I hink first node deployment will fetch a lot of packages, and other nodes will be easier. Once we have prototype, we will see some number. The first OS install will fetch packages, then later the fist of each roll will fetch different packages, it's possible we could get all the way to compute and fail there because we cant get a package. I can personally promise that without something else this will have problems with this the same as we did before with 3.0 (I could run two squid layers one in my host and one on my fuel vm and still have problems (usually cache misses)). When this occurs the result is terrible, hard for not fuel people to realize, and you will end up restarting the whole deployment. the user experience (UX) from this is horrible. We need the tools to prevent this from occurring at all. Regards, -- Tomasz 'Zen' Napierala Sr. OpenStack Engineer tnapier...@mirantis.com __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev [1] https://blueprints.launchpad.net/fuel/+spec/virtual-router-for-env-nodes -- Andrew Mirantis Fuel community ambassador Ceph community __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] Cache for packages on master node
On 02/10/2015 03:24 PM, Tomasz Napierala wrote: Hi, We are currently redesigning our apporach to upstream distributions and obviusly we will need some cache system for packages on master node. It should work for deb and rpm packages, and be able to serve up to 200 nodes. I know we had bad experience in the past, can you guys share your thought on that? I just collected what was mentioned in other discussions: - approx - squid - apt-cacher-ng - ? Regards, Yesterday I have tested apt-cacher-ng on my personal laptop with help of bunch of virtual machines running Ubuntu 14.04 and a proxy on the host system. When 14 nodes started to request packages simultaneously, apt-cacher-ng decided to get stuck and packages installation failed with timeout. Approx doesn't seem to be developed actively and won't give us any advantage if we decide to use similiar approach for CentOS. I vote for squid as a transparent proxy. Cheers, Bartłomiej __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] Cache for packages on master node
On 10 Feb 2015, at 23:02, Andrew Woodward xar...@gmail.com wrote: previously we used squid in 3.0 and before. The main problem is that the deployment would proceeded even if not all the packages where cached or even available on the remote. This often lead to broken deployments that where hard to debug and a waste of alot of time. This _MUST_ be resolved or we will re-introduce this horrible work flow that we had placed all the packages on the system for to begin with. Anyway we need to ensure our QA is run against fresh mirror, that would prevent a lot of problems. We also think about how situation in the field can differ from our labs and QA infra - there might be differences indeed. I think we need to add a requirements that we need to be able to: a) pre-populate the cacher b) we need to not start the deployment until we either have every package in the chache (eiew) or at least know every package is reachable currently (or allow the user to select either as a deployment criteria) This sounds for me like creating local mirror ;) We don’t want to do this. We are thinking about mirror verification tool, it was mentioned by eifferent guys already. Do you really think we should prepopulate cache? I hink first node deployment will fetch a lot of packages, and other nodes will be easier. Once we have prototype, we will see some number. Regards, -- Tomasz 'Zen' Napierala Sr. OpenStack Engineer tnapier...@mirantis.com __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] Cache for packages on master node
Hello Tomasz, In a previous life, I used squid to speed up packages downloads and it worked just fine... Simon On Tue, Feb 10, 2015 at 3:24 PM, Tomasz Napierala tnapier...@mirantis.com wrote: Hi, We are currently redesigning our apporach to upstream distributions and obviusly we will need some cache system for packages on master node. It should work for deb and rpm packages, and be able to serve up to 200 nodes. I know we had bad experience in the past, can you guys share your thought on that? I just collected what was mentioned in other discussions: - approx - squid - apt-cacher-ng - ? Regards, -- Tomasz 'Zen' Napierala Sr. OpenStack Engineer tnapier...@mirantis.com __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] Cache for packages on master node
On Tue, 2015-02-10 at 15:24 +0100, Tomasz Napierala wrote: Hi, We are currently redesigning our apporach to upstream distributions and obviusly we will need some cache system for packages on master node. It should work for deb and rpm packages, and be able to serve up to 200 nodes. I know we had bad experience in the past, can you guys share your thought on that? I just collected what was mentioned in other discussions: - approx - squid - apt-cacher-ng - ? As this should work for both .rpm/.deb packages, i think that squid (probably configured as transparent proxy, but not necessarily, we can explicitly set FMN as http/https proxy on deployed nodes) could be easiest to setup. http://codepoets.co.uk/2014/squid-3-4-x-with-ssl-for-debian-wheezy/ - example how to setup squid as transparent proxy also for https . -- regards jell __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [Fuel] Cache for packages on master node
previously we used squid in 3.0 and before. The main problem is that the deployment would proceeded even if not all the packages where cached or even available on the remote. This often lead to broken deployments that where hard to debug and a waste of alot of time. This _MUST_ be resolved or we will re-introduce this horrible work flow that we had placed all the packages on the system for to begin with. I think we need to add a requirements that we need to be able to: a) pre-populate the cacher b) we need to not start the deployment until we either have every package in the chache (eiew) or at least know every package is reachable currently (or allow the user to select either as a deployment criteria) On Tue, Feb 10, 2015 at 7:19 AM, Skamruk, Piotr piotr.skam...@intel.com wrote: On Tue, 2015-02-10 at 15:24 +0100, Tomasz Napierala wrote: Hi, We are currently redesigning our apporach to upstream distributions and obviusly we will need some cache system for packages on master node. It should work for deb and rpm packages, and be able to serve up to 200 nodes. I know we had bad experience in the past, can you guys share your thought on that? I just collected what was mentioned in other discussions: - approx - squid - apt-cacher-ng - ? As this should work for both .rpm/.deb packages, i think that squid (probably configured as transparent proxy, but not necessarily, we can explicitly set FMN as http/https proxy on deployed nodes) could be easiest to setup. http://codepoets.co.uk/2014/squid-3-4-x-with-ssl-for-debian-wheezy/ - example how to setup squid as transparent proxy also for https . -- regards jell __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev -- Andrew Mirantis Fuel community ambassador Ceph community __ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
