Changing the download code to use the mirror is quite involved. Should we just disable (remove from templates.sql) download of default templates and advise people to use the method outlined here. http://shankerbalan.net/blog/seed-cloudstack-templates-offline/
(actually, we could write a script similar to cloud-install-sys-tmplt to install default templates) On Wed, May 18, 2016 at 12:14 PM, Chiradeep Vittal <chirade...@gmail.com> wrote: > Mirrors still require a "seed" site, right? E.g., msync.centos.org is the > seed for CentOS. Depending on the number of mirrors, this seed site would > still require a non-trivial bandwidth limit and cap. > How soon can the mirror be set up? This week? :) > > > On Wed, May 18, 2016 at 10:12 AM, Will Stevens <wstev...@cloudops.com> > wrote: > >> Inline... >> >> *Will STEVENS* >> Lead Developer >> >> *CloudOps* *| *Cloud Solutions Experts >> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 >> w cloudops.com *|* tw @CloudOps_ >> >> On Wed, May 18, 2016 at 1:08 PM, Chiradeep Vittal <chirade...@gmail.com> >> wrote: >> >> > The mirrolist is a great idea. There's 2 issues though: >> > >> > 1. Already released ACS releases in the wild will continue to attempt to >> > download from download.cloud.com whenever a zone is created. These old >> > releases cannot understand mirror lists. But users can be advised to >> patch >> > templates.sql to point to an alternate location. The question is what is >> > this alternate location? I propose "templates.cloudstack.org/old" >> > >> >> If they patch, maybe they can pull one of the URLs from the mirror list? >> That way we don't have to hardcode a specific mirror into the patch >> procedure. >> >> >> > >> > 2. New releases of ACS. The next release of ACS has to support the >> mirror >> > list, otherwise we're kicking the can too far down the road. >> > >> >> I agree, we will need to get this feature into ACS ASAP. >> >> >> > >> > For (1) >> > * templates.cloudstack.org/old could redirect (A record) to >> > cloudstack.apt-get.eu . Or the project could use AWS S3 (as >> > download.cloud.com is currently hosted in S3). Current bill is $40-50 >> per >> > month (between 7GB-40GB downloaded per day). I like the S3 approach >> myself >> > but it needs the project to cough up some money or a benevolent sponsor. >> > There's quite a few organizations making money off of ACS, so this >> > shouldn't be a problem? >> > >> > >> >> >> > For (2) >> > * Assuming somebody steps up to write the new code, we need to find >> > sponsors that can host the mirrors. >> > >> > As somebody said earlier, this needs to happen much sooner than later. >> > >> > On Wed, May 18, 2016 at 1:06 AM, Paul Angus <paul.an...@shapeblue.com> >> > wrote: >> > >> > > Pulling a few threads together - if the ASF would create a >> > > download.cloudstack.org page on ASF infra with the mirrorlist json >> in it >> > > as suggested by Nux, we can maintain just that page and add/remove >> > mirrors >> > > at will. >> > > >> > > >> > > Kind regards, >> > > >> > > Paul Angus >> > > >> > > paul.an...@shapeblue.com >> > > www.shapeblue.com >> > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK >> > > @shapeblue >> > > >> > > >> > > >> > > -----Original Message----- >> > > From: Wido den Hollander [mailto:w...@widodh.nl] >> > > Sent: 18 May 2016 08:37 >> > > To: Giles Sirett <giles.sir...@shapeblue.com>; >> dev@cloudstack.apache.org >> > > Subject: RE: Migrating CloudStack content from download.cloud.com >> > > >> > > >> > > > Op 17 mei 2016 om 10:34 schreef Giles Sirett < >> > giles.sir...@shapeblue.com >> > > >: >> > > > >> > > > >> > > > This is a MAJOR problem and needs to be figured out before the >> current >> > > download site is removed - @Raja - have Citrix committed to it being >> > there >> > > another year ? >> > > > >> > > > It is my understanding that Cloudstack relies on this download at >> > > > initial build/startup and then periodically if people do things like >> > > > build a new zone >> > > > >> > > > So, users environments aren't going to stop if its pulled, but >> nobody >> > > > would be a able to build/install cloudstack and it may cause issues >> as >> > > > and when users make config changes >> > > > >> > > > The problem is, for new builds, its hard coded : you start the Sec >> > > > Storage VM, CloudStack will try to download the built in template >> from >> > > > download.cloud.com >> > > > >> > > > My understanding is that its simple db change to make once an >> > > > environment is running (exisiting users) but *may* require a code >> > > > patch to fix the initial install >> > > > >> > > > #notideal >> > > > >> > > > Really, this should be maintained by ASF/ the project as its key to >> > > > the software working, however my understanding is it involves >> binaries >> > > > which ASF is never keen on (and may be some 3rd party binaries) - so >> > > > that just wont happen >> > > > >> > > > >> > > > >> > > > Apt-get is controlled by Wido et al (for which we all owe him a >> massive >> > > thanks) - but you are right, it is a 3rd party dependency and >> arguably a >> > > SPOF. But, if ASF wont host binaries, we're always going to have this >> > > issue. >> > > > >> > > >> > > Fully agree with you. I'am happy to host this, but for the project it >> is >> > a >> > > danger. The domain name should be controlled by the project. >> > > >> > > > So, we could patch the code to point somewhere else - but where ? - >> by >> > > > definition its got to be something maintained by a 3rd party >> > > > >> > > > >> > > > We had this debate some time ago when talking about the >> repositories. >> > We >> > > (ShapeBlue) maintain a repo for our customers, which wev'e always >> opened >> > up >> > > as public. However, there were concerns about having a company domain >> > name >> > > (which I understand) - so apt-get became our preferred repo. >> > > > >> > > > >> > > > Thoughts on how to fix this: >> > > > >> > > > 1. if ASF would allow us to create the host entry >> > > > download.cloudstack.org. Its under their domain, but with the site >> > > > being with a 3rd party it could be allowed >> > > > >> > > >> > > I think that would be the best. A CNAME pointing to >> > cloudstack.apt-get.eu >> > > for now might be enough. >> > > >> > > > 2. maybe Citrix could commit to pointing the current DNS entry to a >> > > > new location (would be simplest but may be legally complex and >> relies >> > > > on ongoing goodwill) >> > > > >> > > > 3. make a mod to allow users to choose the download location on >> > > > install. We can then maintain apt-get. To remove the SPOF, we could >> > > > then get the templates copied to a number of 3rd party locations and >> > > > allow people to choose which one they use >> > > > >> > > > >> > > > >> > > > >> > > > Kind Regards >> > > > Giles >> > > > >> > > > >> > > > giles.sir...@shapeblue.com >> > > > www.shapeblue.com >> > > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue >> > > > >> > > > >> > > > >> > > > -----Original Message----- >> > > > From: williamstev...@gmail.com [mailto:williamstev...@gmail.com] On >> > > Behalf Of Will Stevens >> > > > Sent: 16 May 2016 18:59 >> > > > To: dev@cloudstack.apache.org >> > > > Cc: Wido den Hollander <w...@widodh.nl> >> > > > Subject: Re: Migrating CloudStack content from download.cloud.com >> > > > >> > > > @Ian, yes, but I think there is more to it than that. We can >> change it >> > > going forward and we can move everything from there over to a new >> domain. >> > > > I believe the problem is with all of the existing ACS or CCP >> installs >> > > out there currently pointing at the download.cloud.com URL, so when >> it >> > > goes down, those installs have to be updated. >> > > > >> > > > Maybe I don't understand exactly, but that is the impression I am >> > > getting from this thread. Can this be confirmed? >> > > > >> > > > *Will STEVENS* >> > > > Lead Developer >> > > > >> > > > *CloudOps* *| *Cloud Solutions Experts >> > > > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* >> tw >> > > @CloudOps_ >> > > > >> > > > On Mon, May 16, 2016 at 1:38 PM, Ian Rae <i...@cloudops.com> wrote: >> > > > >> > > > > Hey guys, who controls apt-get.eu and given the (rather >> concerning) >> > > > > level of dependency on the operation of the download.cloud.com - >> > would >> > > > > it not make sense to use a domain that is under the control of >> party >> > > > > whose alignment with CloudStack will not shift. I hesitate to >> > > > > implicate the red tape of Apache Foundation, but perhaps the >> fabled >> > > > > CloudStack alliance organization would be a good way to handle >> this >> > > > > long term. >> > > > > >> > > > > On Mon, May 16, 2016 at 1:32 PM, Raja Pullela >> > > > > <raja.pull...@accelerite.com> wrote: >> > > > > > Paul, >> > > > > > >> > > > > > trust me, it won’t be go away overnight. We are talking to >> Citrix >> > > > > > and >> > > > > requested them to keep it active for another year. >> > > > > > >> > > > > > Once we have the content copied over (I will work with Wido and >> > > > > > verify >> > > > > this) to cloudstack.apt-get.eu and update the documentation (user >> > > > > guides update and create a wiki to show how an existing >> installation >> > > > > can fixed to point to new URLs) we don’t have a dependency on “ >> > > download.cloud.com” – >> > > > > right? sorry, if I am missing something here? Please let me >> know, >> > > > > > >> > > > > > best, >> > > > > > Raja >> > > > > > Senior Manager, Product Development >> > > > > > Accelerite, >> > > > > > 2055, Laurelwood Road, Santa Clara, CA 95054, USA >> > > > > > Phone: 1-408-216-7010, www.accelerite.com, @accelerite >> > > > > > >> > > > > > >> > > > > > On 5/16/16, 9:38 PM, "Paul Angus" <paul.an...@shapeblue.com> >> > wrote: >> > > > > >>Thanks Raja, >> > > > > >> >> > > > > >>May I humbly suggest that someone at Accelerite speaks to Citrix >> > > > > >>about >> > > > > how long the content will remain on download.cloud.com? >> > > > > >>Simply losing the content on download.cloud.com overnight will >> > make >> > > > > everyone look bad >> > > > > >> >> > > > > >>Kind regards, >> > > > > >> >> > > > > >>Paul Angus >> > > > > >> >> > > > > >>paul.an...@shapeblue.com >> > > > > >>www.shapeblue.com >> > > > > >>53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue >> > > > > >> >> > > > > >>-----Original Message----- >> > > > > >>From: Raja Pullela [mailto:raja.pull...@accelerite.com] >> > > > > >>Sent: 16 May 2016 16:44 >> > > > > >>To: dev@cloudstack.apache.org >> > > > > >>Subject: Re: Migrating CloudStack content from >> download.cloud.com >> > > > > >> >> > > > > >>Paul, >> > > > > >> >> > > > > >>Yes, you are right, this change impacts both CloudStack and >> > > > > CloudPlatform customers. Not sure, if I understand you correctly, >> > the >> > > > > move away from “download.cloud.com” is permanent. I don’t know >> how >> > > > > long the content on download.cloud.com will be up/available. >> > > > > >> >> > > > > >>Based on Wido’s availability, we can get rest of the content to >> > > > > cloudstack.apt-get.eu, hopefully, by this week ? BTW, the >> content >> > > on >> > > > > download.cloud.com is left as is and will stay as is until Citrix >> > > > > decides on what to do with it. >> > > > > >> >> > > > > >>Best, >> > > > > >>Raja >> > > > > >>Senior Manager, Product Development >> > > > > >>Accelerite, >> > > > > >>2055, Laurelwood Road, Santa Clara, CA 95054, USA >> > > > > >>Phone: 1-408-216-7010, www.accelerite.com, @accelerite >> > > > > >> >> > > > > >> >> > > > > >>On 5/16/16, 6:36 PM, "Paul Angus" <paul.an...@shapeblue.com> >> > wrote: >> > > > > >> >> > > > > >>>Raja, >> > > > > >>> >> > > > > >>>As this effects every copy of CloudStack and CloudPlatform >> there >> > > > > >>>ever >> > > > > was. Please may I ask how long we have to manage a migration away >> > > > > from download.cloud.com? >> > > > > >>> >> > > > > >>> >> > > > > >>> >> > > > > >>>Kind regards, >> > > > > >>> >> > > > > >>>Paul Angus >> > > > > >>> >> > > > > >>>Regards, >> > > > > >>> >> > > > > >>>Paul Angus >> > > > > >>> >> > > > > >>>paul.an...@shapeblue.com >> > > > > >>>www.shapeblue.com >> > > > > >>>53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue >> > > > > >>> >> > > > > >>>-----Original Message----- >> > > > > >>>From: Raja Pullela [mailto:raja.pull...@accelerite.com] >> > > > > >>>Sent: 16 May 2016 13:53 >> > > > > >>>To: dev@cloudstack.apache.org >> > > > > >>>Subject: Re: Migrating CloudStack content from >> download.cloud.com >> > > > > >>> >> > > > > >>>Hi Paul, >> > > > > >>> >> > > > > >>>Thank you for the additional file details and I will request >> wido >> > > > > >>>to >> > > > > copy these as well. >> > > > > >>> >> > > > > >>>BTW, I agree with you there is every copy of CloudStack out >> there >> > > > > >>>and >> > > > > reason why I have requested Wido to download all the old files as >> > well. >> > > > > Looks like we missed few and get these uploaded as well. >> > > > > >>> >> > > > > >>>Please note that Citrix owns the “download.cloud.com” address >> and >> > > > > >>>we >> > > > > have to get all the CloudStack related content from there. Once >> we >> > > > > confirm on the content move Citrix may choose to remove the >> content >> > > > > from the location. >> > > > > >>> >> > > > > >>>That said, I will get in touch with Wido again and do a >> complete >> > > > > >>>copy >> > > > > such that we have everything. >> > > > > >>> >> > > > > >>>Best, >> > > > > >>>Raja >> > > > > >>>Senior Manager, Product Development Accelerite, 2055, >> Laurelwood >> > > > > >>>Road, Santa Clara, CA 95054, USA >> > > > > >>>Phone: 1-408-216-7010, www.accelerite.com, @accelerite >> > > > > >>> >> > > > > >>>On 5/14/16, 7:12 AM, "Paul Angus" <paul.an...@shapeblue.com> >> > wrote: >> > > > > >>>>Raja, >> > > > > >>>> >> > > > > >>>>I've just picked up on this. >> > > > > >>>> >> > > > > >>>>There are a number of URLs which are hardcoded into the >> > CloudStack >> > > > > database for the small built-in templates. These are in just >> about >> > > > > every copy of CloudStack out there. >> > > > > >>>> >> > > > > >>>> >> > http://download.cloud.com/templates/builtin/f59f18fb-ae94-4f97-afd >> > > > > >>>>2-f8 >> > > > > >>>>4 >> > > > > >>>>755767aca.vhd.bz2 >> > > > > >>>> >> > http://download.cloud.com/releases/2.2.0/eec2209b-9875-3c8d-92be-c >> > > > > >>>>001b >> > > > > >>>>d >> > > > > >>>>8a0faf.qcow2.bz2 >> > > > > >>>> >> > http://download.cloud.com/templates/builtin/centos56-x86_64.vhd.bz >> > > > > >>>>2 >> > > > > >>>> >> http://download.cloud.com/releases/4.3/centos6_4_64bit.vhd.bz2 >> > > > > >>>>http://download.cloud.com/releases/2.2.0/CentOS5.3-x86_64.ova >> > > > > >>>> >> > http://download.cloud.com/templates/builtin/centos-7-x86_64.tar.gz >> > > > > >>>> >> > > > > >>>> >> > > > > >>>>These will need copying and the CloudStack code updating to >> the >> > > > > >>>>new >> > > > > location. >> > > > > >>>> >> > > > > >>>> >> > > > > >>>>I don't think that you can deprecate download.cloud.com or >> it's >> > > > > contents for some time, without causing a lot of issues for users. >> > > > > >>>> >> > > > > >>>> >> > > > > >>>> >> > > > > >>>> >> > > > > >>>> >> > > > > >>>> >> > > > > >>>> >> > > > > >>>> >> > > > > >>>> >> > > > > >>>>Hi, >> > > > > >>>> >> > > > > >>>> >> > > > > >>>> >> > > > > >>>>Short Term - all the content has moved to >> cloudstack.apt-get.eu >> > > > > >>>> >> > > > > >>>> >> > > > > >>>> >> > > > > >>>>Which sections to be modified? >> > > > > >>>> >> > > > > >>>> >> > > > > >>>> >> > > > > >>>>The Install Guide needs to be modified. Sample links/details >> are >> > > > > >>>>mentioned below for 4.6 and same needs to be updated for other >> > > > > >>>>Released versions of Cloudstack documentation: >> > > > > >>>> >> > > > > >>>> >> > http://docs.cloudstack.apache.org/projects/cloudstack-installation >> > > > > >>>>/en/ >> > > > > >>>>4 .6/management-server/#downloading-vhd-util >> > > > > >>>> >> > > > > >>>>Section - Downloading vhd-util¶ >> > > > > >>>> >> > > > > >>>> >> > > > > >>>> >> > > > > >>>>What to do modified ? >> > > > > >>>> >> > > > > >>>>modify URL from >> > > > > >>>>http://download.cloud.com.s3.amazonaws.com/tools/vhd-util to >> > > > > >>>>http://cloudstack.apt-get.eu/tools/vhd-util >> > > > > >>>> >> > > > > >>>> >> > > > > >>>> >> > > > > >>>>@Rajsekhar can you please update the sections ? >> > > > > >>>> >> > > > > >>>> >> > > > > >>>> >> > > > > >>>>Thanks >> > > > > >>>> >> > > > > >>>>Raja >> > > > > >>>> >> > > > > >>>>Senior Manager, Product Development >> > > > > >>>> >> > > > > >>>>Accelerite, >> > > > > >>>> >> > > > > >>>>2055, Laurelwood Road, Santa Clara, CA 95054, USA >> > > > > >>>> >> > > > > >>>>Phone: 1-408-216-7010, www.accelerite.com, @accelerite >> > > > > >>>> >> > > > > >>>> >> > > > > >>>>Kind regards, >> > > > > >>>> >> > > > > >>>>Paul Angus >> > > > > >>>> >> > > > > >>>> >> > > > > >>>>Regards, >> > > > > >>>> >> > > > > >>>>Paul Angus >> > > > > >>>> >> > > > > >>>>paul.an...@shapeblue.com >> > > > > >>>>www.shapeblue.com >> > > > > >>>>53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue >> > > > > >>> >> > > > > >>> >> > > > > >>> >> > > > > >>> >> > > > > >>>DISCLAIMER >> > > > > >>>========== >> > > > > >>>This e-mail may contain privileged and confidential information >> > > > > >>>which >> > > > > is the property of Accelerite, a Persistent Systems business. It >> is >> > > > > intended only for the use of the individual or entity to which it >> is >> > > > > addressed. If you are not the intended recipient, you are not >> > > > > authorized to read, retain, copy, print, distribute or use this >> > > > > message. If you have received this communication in error, please >> > > > > notify the sender and delete all copies of this message. >> Accelerite, >> > a >> > > > > Persistent Systems business does not accept any liability for >> virus >> > > infected mails. >> > > > > >> >> > > > > >> >> > > > > >> >> > > > > >> >> > > > > >>DISCLAIMER >> > > > > >>========== >> > > > > >>This e-mail may contain privileged and confidential information >> > > > > >>which is >> > > > > the property of Accelerite, a Persistent Systems business. It is >> > > > > intended only for the use of the individual or entity to which it >> is >> > > > > addressed. If you are not the intended recipient, you are not >> > > > > authorized to read, retain, copy, print, distribute or use this >> > > > > message. If you have received this communication in error, please >> > > > > notify the sender and delete all copies of this message. >> Accelerite, >> > a >> > > > > Persistent Systems business does not accept any liability for >> virus >> > > infected mails. >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > DISCLAIMER >> > > > > > ========== >> > > > > > This e-mail may contain privileged and confidential information >> > > > > > which is >> > > > > the property of Accelerite, a Persistent Systems business. It is >> > > > > intended only for the use of the individual or entity to which it >> is >> > > > > addressed. If you are not the intended recipient, you are not >> > > > > authorized to read, retain, copy, print, distribute or use this >> > > > > message. If you have received this communication in error, please >> > > > > notify the sender and delete all copies of this message. >> Accelerite, >> > a >> > > > > Persistent Systems business does not accept any liability for >> virus >> > > infected mails. >> > > > > >> > > > > >> > > > > >> > > > > -- >> > > > > Ian Rae >> > > > > CEO | PDG >> > > > > c: 514.944.4008 >> > > > > >> > > > > CloudOps | Cloud Infrastructure and Networking Solutions >> > > > > www.cloudops.com | 420 rue Guy | Montreal | Canada | H3J 1S6 >> > > > > >> > > >> > >> > >
