+1 for disabling the default template download, it's the easiest/simplest way.
With regards to seeding if we decide to go with that, I'm sure Wido's cloudstack.apt-get.eu has sufficient bw, already several community members have access there (I think) and it's running an rsync server, too. -- Sent from the Delta quadrant using Borg technology! Nux! www.nux.ro ----- Original Message ----- > From: "Chiradeep Vittal" <chirade...@gmail.com> > To: dev@cloudstack.apache.org > Sent: Thursday, 19 May, 2016 06:22:45 > Subject: Re: Migrating CloudStack content from download.cloud.com > Changing the download code to use the mirror is quite involved. Should we > just disable (remove from templates.sql) download of default templates and > advise people to use the method outlined here. > http://shankerbalan.net/blog/seed-cloudstack-templates-offline/ > > (actually, we could write a script similar to cloud-install-sys-tmplt to > install default templates) > > On Wed, May 18, 2016 at 12:14 PM, Chiradeep Vittal <chirade...@gmail.com> > wrote: > >> Mirrors still require a "seed" site, right? E.g., msync.centos.org is the >> seed for CentOS. Depending on the number of mirrors, this seed site would >> still require a non-trivial bandwidth limit and cap. >> How soon can the mirror be set up? This week? :) >> >> >> On Wed, May 18, 2016 at 10:12 AM, Will Stevens <wstev...@cloudops.com> >> wrote: >> >>> Inline... >>> >>> *Will STEVENS* >>> Lead Developer >>> >>> *CloudOps* *| *Cloud Solutions Experts >>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 >>> w cloudops.com *|* tw @CloudOps_ >>> >>> On Wed, May 18, 2016 at 1:08 PM, Chiradeep Vittal <chirade...@gmail.com> >>> wrote: >>> >>> > The mirrolist is a great idea. There's 2 issues though: >>> > >>> > 1. Already released ACS releases in the wild will continue to attempt to >>> > download from download.cloud.com whenever a zone is created. These old >>> > releases cannot understand mirror lists. But users can be advised to >>> patch >>> > templates.sql to point to an alternate location. The question is what is >>> > this alternate location? I propose "templates.cloudstack.org/old" >>> > >>> >>> If they patch, maybe they can pull one of the URLs from the mirror list? >>> That way we don't have to hardcode a specific mirror into the patch >>> procedure. >>> >>> >>> > >>> > 2. New releases of ACS. The next release of ACS has to support the >>> mirror >>> > list, otherwise we're kicking the can too far down the road. >>> > >>> >>> I agree, we will need to get this feature into ACS ASAP. >>> >>> >>> > >>> > For (1) >>> > * templates.cloudstack.org/old could redirect (A record) to >>> > cloudstack.apt-get.eu . Or the project could use AWS S3 (as >>> > download.cloud.com is currently hosted in S3). Current bill is $40-50 >>> per >>> > month (between 7GB-40GB downloaded per day). I like the S3 approach >>> myself >>> > but it needs the project to cough up some money or a benevolent sponsor. >>> > There's quite a few organizations making money off of ACS, so this >>> > shouldn't be a problem? >>> > >>> > >>> >>> >>> > For (2) >>> > * Assuming somebody steps up to write the new code, we need to find >>> > sponsors that can host the mirrors. >>> > >>> > As somebody said earlier, this needs to happen much sooner than later. >>> > >>> > On Wed, May 18, 2016 at 1:06 AM, Paul Angus <paul.an...@shapeblue.com> >>> > wrote: >>> > >>> > > Pulling a few threads together - if the ASF would create a >>> > > download.cloudstack.org page on ASF infra with the mirrorlist json >>> in it >>> > > as suggested by Nux, we can maintain just that page and add/remove >>> > mirrors >>> > > at will. >>> > > >>> > > >>> > > Kind regards, >>> > > >>> > > Paul Angus >>> > > >>> > > paul.an...@shapeblue.com >>> > > www.shapeblue.com >>> > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK >>> > > @shapeblue >>> > > >>> > > >>> > > >>> > > -----Original Message----- >>> > > From: Wido den Hollander [mailto:w...@widodh.nl] >>> > > Sent: 18 May 2016 08:37 >>> > > To: Giles Sirett <giles.sir...@shapeblue.com>; >>> dev@cloudstack.apache.org >>> > > Subject: RE: Migrating CloudStack content from download.cloud.com >>> > > >>> > > >>> > > > Op 17 mei 2016 om 10:34 schreef Giles Sirett < >>> > giles.sir...@shapeblue.com >>> > > >: >>> > > > >>> > > > >>> > > > This is a MAJOR problem and needs to be figured out before the >>> current >>> > > download site is removed - @Raja - have Citrix committed to it being >>> > there >>> > > another year ? >>> > > > >>> > > > It is my understanding that Cloudstack relies on this download at >>> > > > initial build/startup and then periodically if people do things like >>> > > > build a new zone >>> > > > >>> > > > So, users environments aren't going to stop if its pulled, but >>> nobody >>> > > > would be a able to build/install cloudstack and it may cause issues >>> as >>> > > > and when users make config changes >>> > > > >>> > > > The problem is, for new builds, its hard coded : you start the Sec >>> > > > Storage VM, CloudStack will try to download the built in template >>> from >>> > > > download.cloud.com >>> > > > >>> > > > My understanding is that its simple db change to make once an >>> > > > environment is running (exisiting users) but *may* require a code >>> > > > patch to fix the initial install >>> > > > >>> > > > #notideal >>> > > > >>> > > > Really, this should be maintained by ASF/ the project as its key to >>> > > > the software working, however my understanding is it involves >>> binaries >>> > > > which ASF is never keen on (and may be some 3rd party binaries) - so >>> > > > that just wont happen >>> > > > >>> > > > >>> > > > >>> > > > Apt-get is controlled by Wido et al (for which we all owe him a >>> massive >>> > > thanks) - but you are right, it is a 3rd party dependency and >>> arguably a >>> > > SPOF. But, if ASF wont host binaries, we're always going to have this >>> > > issue. >>> > > > >>> > > >>> > > Fully agree with you. I'am happy to host this, but for the project it >>> is >>> > a >>> > > danger. The domain name should be controlled by the project. >>> > > >>> > > > So, we could patch the code to point somewhere else - but where ? - >>> by >>> > > > definition its got to be something maintained by a 3rd party >>> > > > >>> > > > >>> > > > We had this debate some time ago when talking about the >>> repositories. >>> > We >>> > > (ShapeBlue) maintain a repo for our customers, which wev'e always >>> opened >>> > up >>> > > as public. However, there were concerns about having a company domain >>> > name >>> > > (which I understand) - so apt-get became our preferred repo. >>> > > > >>> > > > >>> > > > Thoughts on how to fix this: >>> > > > >>> > > > 1. if ASF would allow us to create the host entry >>> > > > download.cloudstack.org. Its under their domain, but with the site >>> > > > being with a 3rd party it could be allowed >>> > > > >>> > > >>> > > I think that would be the best. A CNAME pointing to >>> > cloudstack.apt-get.eu >>> > > for now might be enough. >>> > > >>> > > > 2. maybe Citrix could commit to pointing the current DNS entry to a >>> > > > new location (would be simplest but may be legally complex and >>> relies >>> > > > on ongoing goodwill) >>> > > > >>> > > > 3. make a mod to allow users to choose the download location on >>> > > > install. We can then maintain apt-get. To remove the SPOF, we could >>> > > > then get the templates copied to a number of 3rd party locations and >>> > > > allow people to choose which one they use >>> > > > >>> > > > >>> > > > >>> > > > >>> > > > Kind Regards >>> > > > Giles >>> > > > >>> > > > >>> > > > giles.sir...@shapeblue.com >>> > > > www.shapeblue.com >>> > > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue >>> > > > >>> > > > >>> > > > >>> > > > -----Original Message----- >>> > > > From: williamstev...@gmail.com [mailto:williamstev...@gmail.com] On >>> > > Behalf Of Will Stevens >>> > > > Sent: 16 May 2016 18:59 >>> > > > To: dev@cloudstack.apache.org >>> > > > Cc: Wido den Hollander <w...@widodh.nl> >>> > > > Subject: Re: Migrating CloudStack content from download.cloud.com >>> > > > >>> > > > @Ian, yes, but I think there is more to it than that. We can >>> change it >>> > > going forward and we can move everything from there over to a new >>> domain. >>> > > > I believe the problem is with all of the existing ACS or CCP >>> installs >>> > > out there currently pointing at the download.cloud.com URL, so when >>> it >>> > > goes down, those installs have to be updated. >>> > > > >>> > > > Maybe I don't understand exactly, but that is the impression I am >>> > > getting from this thread. Can this be confirmed? >>> > > > >>> > > > *Will STEVENS* >>> > > > Lead Developer >>> > > > >>> > > > *CloudOps* *| *Cloud Solutions Experts >>> > > > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6 w cloudops.com *|* >>> tw >>> > > @CloudOps_ >>> > > > >>> > > > On Mon, May 16, 2016 at 1:38 PM, Ian Rae <i...@cloudops.com> wrote: >>> > > > >>> > > > > Hey guys, who controls apt-get.eu and given the (rather >>> concerning) >>> > > > > level of dependency on the operation of the download.cloud.com - >>> > would >>> > > > > it not make sense to use a domain that is under the control of >>> party >>> > > > > whose alignment with CloudStack will not shift. I hesitate to >>> > > > > implicate the red tape of Apache Foundation, but perhaps the >>> fabled >>> > > > > CloudStack alliance organization would be a good way to handle >>> this >>> > > > > long term. >>> > > > > >>> > > > > On Mon, May 16, 2016 at 1:32 PM, Raja Pullela >>> > > > > <raja.pull...@accelerite.com> wrote: >>> > > > > > Paul, >>> > > > > > >>> > > > > > trust me, it won’t be go away overnight. We are talking to >>> Citrix >>> > > > > > and >>> > > > > requested them to keep it active for another year. >>> > > > > > >>> > > > > > Once we have the content copied over (I will work with Wido and >>> > > > > > verify >>> > > > > this) to cloudstack.apt-get.eu and update the documentation (user >>> > > > > guides update and create a wiki to show how an existing >>> installation >>> > > > > can fixed to point to new URLs) we don’t have a dependency on “ >>> > > download.cloud.com” – >>> > > > > right? sorry, if I am missing something here? Please let me >>> know, >>> > > > > > >>> > > > > > best, >>> > > > > > Raja >>> > > > > > Senior Manager, Product Development >>> > > > > > Accelerite, >>> > > > > > 2055, Laurelwood Road, Santa Clara, CA 95054, USA >>> > > > > > Phone: 1-408-216-7010, www.accelerite.com, @accelerite >>> > > > > > >>> > > > > > >>> > > > > > On 5/16/16, 9:38 PM, "Paul Angus" <paul.an...@shapeblue.com> >>> > wrote: >>> > > > > >>Thanks Raja, >>> > > > > >> >>> > > > > >>May I humbly suggest that someone at Accelerite speaks to Citrix >>> > > > > >>about >>> > > > > how long the content will remain on download.cloud.com? >>> > > > > >>Simply losing the content on download.cloud.com overnight will >>> > make >>> > > > > everyone look bad >>> > > > > >> >>> > > > > >>Kind regards, >>> > > > > >> >>> > > > > >>Paul Angus >>> > > > > >> >>> > > > > >>paul.an...@shapeblue.com >>> > > > > >>www.shapeblue.com >>> > > > > >>53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue >>> > > > > >> >>> > > > > >>-----Original Message----- >>> > > > > >>From: Raja Pullela [mailto:raja.pull...@accelerite.com] >>> > > > > >>Sent: 16 May 2016 16:44 >>> > > > > >>To: dev@cloudstack.apache.org >>> > > > > >>Subject: Re: Migrating CloudStack content from >>> download.cloud.com >>> > > > > >> >>> > > > > >>Paul, >>> > > > > >> >>> > > > > >>Yes, you are right, this change impacts both CloudStack and >>> > > > > CloudPlatform customers. Not sure, if I understand you correctly, >>> > the >>> > > > > move away from “download.cloud.com” is permanent. I don’t know >>> how >>> > > > > long the content on download.cloud.com will be up/available. >>> > > > > >> >>> > > > > >>Based on Wido’s availability, we can get rest of the content to >>> > > > > cloudstack.apt-get.eu, hopefully, by this week ? BTW, the >>> content >>> > > on >>> > > > > download.cloud.com is left as is and will stay as is until Citrix >>> > > > > decides on what to do with it. >>> > > > > >> >>> > > > > >>Best, >>> > > > > >>Raja >>> > > > > >>Senior Manager, Product Development >>> > > > > >>Accelerite, >>> > > > > >>2055, Laurelwood Road, Santa Clara, CA 95054, USA >>> > > > > >>Phone: 1-408-216-7010, www.accelerite.com, @accelerite >>> > > > > >> >>> > > > > >> >>> > > > > >>On 5/16/16, 6:36 PM, "Paul Angus" <paul.an...@shapeblue.com> >>> > wrote: >>> > > > > >> >>> > > > > >>>Raja, >>> > > > > >>> >>> > > > > >>>As this effects every copy of CloudStack and CloudPlatform >>> there >>> > > > > >>>ever >>> > > > > was. Please may I ask how long we have to manage a migration away >>> > > > > from download.cloud.com? >>> > > > > >>> >>> > > > > >>> >>> > > > > >>> >>> > > > > >>>Kind regards, >>> > > > > >>> >>> > > > > >>>Paul Angus >>> > > > > >>> >>> > > > > >>>Regards, >>> > > > > >>> >>> > > > > >>>Paul Angus >>> > > > > >>> >>> > > > > >>>paul.an...@shapeblue.com >>> > > > > >>>www.shapeblue.com >>> > > > > >>>53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue >>> > > > > >>> >>> > > > > >>>-----Original Message----- >>> > > > > >>>From: Raja Pullela [mailto:raja.pull...@accelerite.com] >>> > > > > >>>Sent: 16 May 2016 13:53 >>> > > > > >>>To: dev@cloudstack.apache.org >>> > > > > >>>Subject: Re: Migrating CloudStack content from >>> download.cloud.com >>> > > > > >>> >>> > > > > >>>Hi Paul, >>> > > > > >>> >>> > > > > >>>Thank you for the additional file details and I will request >>> wido >>> > > > > >>>to >>> > > > > copy these as well. >>> > > > > >>> >>> > > > > >>>BTW, I agree with you there is every copy of CloudStack out >>> there >>> > > > > >>>and >>> > > > > reason why I have requested Wido to download all the old files as >>> > well. >>> > > > > Looks like we missed few and get these uploaded as well. >>> > > > > >>> >>> > > > > >>>Please note that Citrix owns the “download.cloud.com” address >>> and >>> > > > > >>>we >>> > > > > have to get all the CloudStack related content from there. Once >>> we >>> > > > > confirm on the content move Citrix may choose to remove the >>> content >>> > > > > from the location. >>> > > > > >>> >>> > > > > >>>That said, I will get in touch with Wido again and do a >>> complete >>> > > > > >>>copy >>> > > > > such that we have everything. >>> > > > > >>> >>> > > > > >>>Best, >>> > > > > >>>Raja >>> > > > > >>>Senior Manager, Product Development Accelerite, 2055, >>> Laurelwood >>> > > > > >>>Road, Santa Clara, CA 95054, USA >>> > > > > >>>Phone: 1-408-216-7010, www.accelerite.com, @accelerite >>> > > > > >>> >>> > > > > >>>On 5/14/16, 7:12 AM, "Paul Angus" <paul.an...@shapeblue.com> >>> > wrote: >>> > > > > >>>>Raja, >>> > > > > >>>> >>> > > > > >>>>I've just picked up on this. >>> > > > > >>>> >>> > > > > >>>>There are a number of URLs which are hardcoded into the >>> > CloudStack >>> > > > > database for the small built-in templates. These are in just >>> about >>> > > > > every copy of CloudStack out there. >>> > > > > >>>> >>> > > > > >>>> >>> > http://download.cloud.com/templates/builtin/f59f18fb-ae94-4f97-afd >>> > > > > >>>>2-f8 >>> > > > > >>>>4 >>> > > > > >>>>755767aca.vhd.bz2 >>> > > > > >>>> >>> > http://download.cloud.com/releases/2.2.0/eec2209b-9875-3c8d-92be-c >>> > > > > >>>>001b >>> > > > > >>>>d >>> > > > > >>>>8a0faf.qcow2.bz2 >>> > > > > >>>> >>> > http://download.cloud.com/templates/builtin/centos56-x86_64.vhd.bz >>> > > > > >>>>2 >>> > > > > >>>> >>> http://download.cloud.com/releases/4.3/centos6_4_64bit.vhd.bz2 >>> > > > > >>>>http://download.cloud.com/releases/2.2.0/CentOS5.3-x86_64.ova >>> > > > > >>>> >>> > http://download.cloud.com/templates/builtin/centos-7-x86_64.tar.gz >>> > > > > >>>> >>> > > > > >>>> >>> > > > > >>>>These will need copying and the CloudStack code updating to >>> the >>> > > > > >>>>new >>> > > > > location. >>> > > > > >>>> >>> > > > > >>>> >>> > > > > >>>>I don't think that you can deprecate download.cloud.com or >>> it's >>> > > > > contents for some time, without causing a lot of issues for users. >>> > > > > >>>> >>> > > > > >>>> >>> > > > > >>>> >>> > > > > >>>> >>> > > > > >>>> >>> > > > > >>>> >>> > > > > >>>> >>> > > > > >>>> >>> > > > > >>>> >>> > > > > >>>>Hi, >>> > > > > >>>> >>> > > > > >>>> >>> > > > > >>>> >>> > > > > >>>>Short Term - all the content has moved to >>> cloudstack.apt-get.eu >>> > > > > >>>> >>> > > > > >>>> >>> > > > > >>>> >>> > > > > >>>>Which sections to be modified? >>> > > > > >>>> >>> > > > > >>>> >>> > > > > >>>> >>> > > > > >>>>The Install Guide needs to be modified. Sample links/details >>> are >>> > > > > >>>>mentioned below for 4.6 and same needs to be updated for other >>> > > > > >>>>Released versions of Cloudstack documentation: >>> > > > > >>>> >>> > > > > >>>> >>> > http://docs.cloudstack.apache.org/projects/cloudstack-installation >>> > > > > >>>>/en/ >>> > > > > >>>>4 .6/management-server/#downloading-vhd-util >>> > > > > >>>> >>> > > > > >>>>Section - Downloading vhd-util¶ >>> > > > > >>>> >>> > > > > >>>> >>> > > > > >>>> >>> > > > > >>>>What to do modified ? >>> > > > > >>>> >>> > > > > >>>>modify URL from >>> > > > > >>>>http://download.cloud.com.s3.amazonaws.com/tools/vhd-util to >>> > > > > >>>>http://cloudstack.apt-get.eu/tools/vhd-util >>> > > > > >>>> >>> > > > > >>>> >>> > > > > >>>> >>> > > > > >>>>@Rajsekhar can you please update the sections ? >>> > > > > >>>> >>> > > > > >>>> >>> > > > > >>>> >>> > > > > >>>>Thanks >>> > > > > >>>> >>> > > > > >>>>Raja >>> > > > > >>>> >>> > > > > >>>>Senior Manager, Product Development >>> > > > > >>>> >>> > > > > >>>>Accelerite, >>> > > > > >>>> >>> > > > > >>>>2055, Laurelwood Road, Santa Clara, CA 95054, USA >>> > > > > >>>> >>> > > > > >>>>Phone: 1-408-216-7010, www.accelerite.com, @accelerite >>> > > > > >>>> >>> > > > > >>>> >>> > > > > >>>>Kind regards, >>> > > > > >>>> >>> > > > > >>>>Paul Angus >>> > > > > >>>> >>> > > > > >>>> >>> > > > > >>>>Regards, >>> > > > > >>>> >>> > > > > >>>>Paul Angus >>> > > > > >>>> >>> > > > > >>>>paul.an...@shapeblue.com >>> > > > > >>>>www.shapeblue.com >>> > > > > >>>>53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue >>> > > > > >>> >>> > > > > >>> >>> > > > > >>> >>> > > > > >>> >>> > > > > >>>DISCLAIMER >>> > > > > >>>========== >>> > > > > >>>This e-mail may contain privileged and confidential information >>> > > > > >>>which >>> > > > > is the property of Accelerite, a Persistent Systems business. It >>> is >>> > > > > intended only for the use of the individual or entity to which it >>> is >>> > > > > addressed. If you are not the intended recipient, you are not >>> > > > > authorized to read, retain, copy, print, distribute or use this >>> > > > > message. If you have received this communication in error, please >>> > > > > notify the sender and delete all copies of this message. >>> Accelerite, >>> > a >>> > > > > Persistent Systems business does not accept any liability for >>> virus >>> > > infected mails. >>> > > > > >> >>> > > > > >> >>> > > > > >> >>> > > > > >> >>> > > > > >>DISCLAIMER >>> > > > > >>========== >>> > > > > >>This e-mail may contain privileged and confidential information >>> > > > > >>which is >>> > > > > the property of Accelerite, a Persistent Systems business. It is >>> > > > > intended only for the use of the individual or entity to which it >>> is >>> > > > > addressed. If you are not the intended recipient, you are not >>> > > > > authorized to read, retain, copy, print, distribute or use this >>> > > > > message. If you have received this communication in error, please >>> > > > > notify the sender and delete all copies of this message. >>> Accelerite, >>> > a >>> > > > > Persistent Systems business does not accept any liability for >>> virus >>> > > infected mails. >>> > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> > > > > > DISCLAIMER >>> > > > > > ========== >>> > > > > > This e-mail may contain privileged and confidential information >>> > > > > > which is >>> > > > > the property of Accelerite, a Persistent Systems business. It is >>> > > > > intended only for the use of the individual or entity to which it >>> is >>> > > > > addressed. If you are not the intended recipient, you are not >>> > > > > authorized to read, retain, copy, print, distribute or use this >>> > > > > message. If you have received this communication in error, please >>> > > > > notify the sender and delete all copies of this message. >>> Accelerite, >>> > a >>> > > > > Persistent Systems business does not accept any liability for >>> virus >>> > > infected mails. >>> > > > > >>> > > > > >>> > > > > >>> > > > > -- >>> > > > > Ian Rae >>> > > > > CEO | PDG >>> > > > > c: 514.944.4008 >>> > > > > >>> > > > > CloudOps | Cloud Infrastructure and Networking Solutions >>> > > > > www.cloudops.com | 420 rue Guy | Montreal | Canada | H3J 1S6 >>> > > > > >>> > > >>> > >>> >>
