Erik, yes I think this is the right approach and covers the main problem case. I can start the conversation with infra unless you would like to. Let me know.
We did get some statistics from the other mirrors last year, but I don't think we ever had visibility into the cloud.com repository traffic. On Mar 3, 2017 6:42 PM, "Erik Weber" <terbol...@gmail.com> wrote: It'll most likely only be an issue in the case where a mirror goes down without any immediate chance of getting back up, right? Could we check with ASF Infra beforehand if such case is OK to warrant an urgent Jira ticket to resolve - should it happen? We can document all the mirrors and provide the howtos for manually downloading and seeding the image as a backup. Do we have any traffic statistics for the current solution, or is that not available at S3? -- Erik On Sat, Mar 4, 2017 at 12:28 AM, Will Stevens <williamstev...@gmail.com> wrote: > Yes Erik, I agree with you. The only thing that could add complexity in > this is the fact that we don't control the domain, the ASF does (from what > I understand). Do we expect this to be managed by ASF infra or would there > be another way? Ideally the domain is community controlled so we can adapt > in case of change without long delays. > > I am 100% in agreement that this is the shortest path and likely to be good > enough. > > On Mar 3, 2017 6:19 PM, "Erik Weber" <terbol...@gmail.com> wrote: > > IMHO; implement it as simple as possible, use DNS RR and assume/expect > mirror hosts to use the same path. > > > Yes I know; DNS RR isn't bulletproof, but as already mentioned in this > thread - mirrors rarely go down and they aren't used all that often > (primarily seeding). > We can keep a low TTL on the record so that we're able to remove a > mirror that goes down for a significant period of time (more than > hours). > > And yes I know, some operators might find it a burden having to set up > a new vhost with another document root, but tbh if they don't care > enough to do it we'll manage without that mirror. > > -- > Erik > > On Sat, Mar 4, 2017 at 12:09 AM, Will Stevens <williamstev...@gmail.com> > wrote: >> I agree with Paul. Look at the list of things they have to >> learn/master/care about. We don't want to add to that list, we want to >> remove from that list. Make the project/product more accessible. Reduce > the >> barrier to entry... >> >> On Mar 3, 2017 5:50 PM, "Chiradeep Vittal" <chirade...@gmail.com> wrote: >> >>> Seriously? >>> apt-get install apache2 >>> >>> To install CloudStack, they need to know >>> - DB installation, perhaps some SQL >>> - VLAN configuration on their switches >>> - Ins-and-outs between port forwarding, static ips, >>> - NFS >>> - package management >>> - VHDs, qcow2. vmdk >>> - hyperviosrs >>> - and on and on. >>> >>> If they can figure all that out and not figure out a web server, they can >>> always come to the mailing list. >>> >>> >>> >>> >>> On Fri, Mar 3, 2017 at 1:45 PM, Paul Angus <paul.an...@shapeblue.com> >>> wrote: >>> >>> > The issue is not with supporting highly experienced cloud operators. It >>> is >>> > an issue for new users and other 'relatively' inexperienced operators. >>> > >>> > I have helped enough newbies and cloud operators who have been running >>> > their cloud for a while to know that the barriers to entry are too high >>> as >>> > they are. And telling anyone that they need to create a web server so >>> that >>> > they can add their initial template to get started or in order to > create >>> a >>> > new zone, just isn't going to fly. >>> > >>> > I'm a huge advocate of the 'download.cloudstack.org' endpoint which the >>> > community can add/remove mirrors to or from for system VMs or built-in >>> > templates. Can the same system be used for binary repos ? although > there >>> > is an added complication of redist vs no-redist there... >>> > >>> > >>> > >>> > >>> > >>> > >>> > paul.an...@shapeblue.com >>> > www.shapeblue.com >>> > 53 Chandos Place, Covent Garden, London WC2N 4HSUK >>> > @shapeblue >>> > >>> > >>> > >>> > >>> > -----Original Message----- >>> > From: Chiradeep Vittal [mailto:chirade...@gmail.com] >>> > Sent: 03 March 2017 18:28 >>> > To: dev <dev@cloudstack.apache.org> >>> > Subject: Re: Modern template hosting >>> > >>> > I do feel like this is early optimization. Mirrors rarely fail. I'd >>> expect >>> > a single web server hosted on Apache Infra without any monitors to fail >>> > more often than a mirror. We already expect Wido's systemvm repository > to >>> > be up all the time. And it has been. Similarly, I don't believe Nux's >>> > repository has ever been down. And if Accelerite wants to host on S3, >>> that >>> > one is pretty solid as well. >>> > >>> > This is an infrequent operation in a cloud. After the cloud is > installed, >>> > the download servers are only needed for a new zone. If we trust the > user >>> > to run a cloud, surely he/she can run a web server to serve some > built-in >>> > templates. And if her cloud is successful, she needs to figure out how > to >>> > host her templates anyway and not rely on 3rd parties. >>> > >>> > >>> > On Fri, Mar 3, 2017 at 9:31 AM, Will Stevens <wstev...@cloudops.com> >>> > wrote: >>> > >>> > > 1) If the legacy implementations do not support redirects, that does >>> > > cause a problem. A potential solution in that case is to have the > web >>> > > server actually proxy the download, but that is not ideal and I would >>> > > like to avoid it if possible. Thanks for bringing that up Chiradeep. >>> > > >>> > > 2) I think we need to have a single URL which people can target. > Once >>> > > they make the switch to the new URL, we want the implementation to be >>> > > able to handle mirror failures without affecting the end client. We >>> > > want to avoid the situation where an ACS user will ever have to > change >>> > > this URL more than once. Mirror failures SHOULD NOT affect the ACS >>> > > users assuming there is still at least one mirror who can serve the >>> > requested resource. >>> > > >>> > > These are obviously my personal opinions and others will probably > have >>> > > differing opinions. >>> > > >>> > > *Will STEVENS* >>> > > Lead Developer >>> > > >>> > > <https://goo.gl/NYZ8KK> >>> > > >>> > > On Fri, Mar 3, 2017 at 12:23 PM, Chiradeep Vittal >>> > > <chirade...@gmail.com> >>> > > wrote: >>> > > >>> > > > 1. If you are targeting legacy installations, they are not able to >>> > > > follow redirects. The line of code that added this capability was >>> > > > added on >>> > > 11/16. >>> > > > 2. If you trust the users to edit the database to change the URL, >>> > > > you can trust them to change it to anything. Just document a known >>> > > > list of good template locations. >>> > > > >>> > > > On Fri, Mar 3, 2017 at 8:13 AM, Will Stevens < wstev...@cloudops.com >> >>> > > > wrote: >>> > > > >>> > > > > So the main issue I see with this is the following. >>> > > > > >>> > > > > This implementation is designed to target legacy installations >>> > > > > which >>> > > will >>> > > > > be affected when download.cloud.com disappears. These people > will >>> > > need >>> > > > to >>> > > > > make a DB change to replace the 'download.cloud.com' with some >>> > > > > other >>> > > url >>> > > > > (maybe 'download.cloudstack.org'). Once the DB has been updated, >>> > > > > we >>> > > can >>> > > > > not expect anything else of the client, they need to be able to >>> > > > > just continue operation as they were without any need to rebuild >>> > > > > or upgrade their ACS. >>> > > > > >>> > > > > If we try to force the decision to the client, then we break the >>> > > > > legacy implementations. Since the SSVM is likely going to be the >>> > > > > client in >>> > > some >>> > > > > cases, and since it does not already have the logic to handle the >>> > > > > 300 approach correctly, I feel like it is not a viable solution >>> > > > > for the >>> > > > legacy >>> > > > > deployments. >>> > > > > >>> > > > > In addition to that. I don't think the client is in any better >>> > > position >>> > > > to >>> > > > > make the mirror decision than I am. I am likely able to make a >>> > > > > more 'educated' decision than the client would be able to because >>> > > > > I can do different tests on the endpoint(s) before making a final >>> > decision. >>> > > > > >>> > > > > Does this make sense or am I missing something obvious? >>> > > > > >>> > > > > *Will STEVENS* >>> > > > > Lead Developer >>> > > > > >>> > > > > <https://goo.gl/NYZ8KK> >>> > > > > >>> > > > > On Fri, Mar 3, 2017 at 10:48 AM, Rafael Weingärtner < >>> > > > > rafaelweingart...@gmail.com> wrote: >>> > > > > >>> > > > > > I think I understood you, but I did not understand you. >>> > > > > > >>> > > > > > Let me see if I can create an example to illustrate how we > could >>> > > > > > do >>> > > > using >>> > > > > > the HTTP 300 code. >>> > > > > > >>> > > > > > The HTTP 300 code indicates clients that he/she/it has multiple >>> > > choices >>> > > > > to >>> > > > > > access the requested resource. If the request is a HEAD, this > is >>> > > > > > the >>> > > > only >>> > > > > > thing that the server returns. If the request if a GET, then > the >>> > > server >>> > > > > > returns the choices for the client. The response type is > defined >>> > > > > > on >>> > > the >>> > > > > > server based on the Content-Type, user agent and others. >>> > > > > > >>> > > > > > Let’s say the server receives a GET request and informs the >>> > > > content-type >>> > > > > as >>> > > > > > JSON (application/JSON). Your application could get the mirrors >>> > > > > > list, >>> > > > do >>> > > > > > the appending process for the requested resource path, and then >>> > > return >>> > > > a >>> > > > > > JSON array with the possible mirror the client can use. Then, >>> > > > > > the >>> > > > client >>> > > > > > gets this list and does the processing required to select a >>> mirror. >>> > > > > > >>> > > > > > Now let’s say that a human access the link using a browser. >>> > > > > > Then, the server should receive something like (text/plain) as >>> > > > > > the >>> > > content-type; >>> > > > I >>> > > > > > think depending on the browser this field may differ a little >>> > > > > > (it >>> > > would >>> > > > > > require some checking). Anyways, the server detects that it is > a >>> > > > “human” >>> > > > > > requesting the resource, then we could serve an HTML page with > a >>> > > > > Javascript >>> > > > > > that uses the mirror list. This Javascript could do some >>> > > > > > checking and choose the best mirror for that specific. >>> > > > > > >>> > > > > > Why do I think this approach is interesting? >>> > > > > > The application you developed would be used only to retrieve >>> > > > > > valid >>> > > and >>> > > > > > trusted sources of system VM images (a trusted repository for >>> > > > > > mirrors >>> > > > > that >>> > > > > > we as a community vouch for). Also, we shift the decision >>> > > > > > process regarding mirrors from the server to the client. Then, >>> > > > > > it is up to >>> > > > > clients >>> > > > > > to select mirrors, and not up to us (or some of our >>> > implementations). >>> > > > > > >>> > > > > > Did this help? >>> > > > > > BTW: I am not saying I am against the way you proposed, which >>> > > > > > would >>> > > > work >>> > > > > > fine. It is merely a suggestion using a different perspective. >>> > > > > > >>> > > > > > On Fri, Mar 3, 2017 at 10:23 AM, Will Stevens < >>> > > > williamstev...@gmail.com> >>> > > > > > wrote: >>> > > > > > >>> > > > > > > @rafael: in general, I don't think the client should ever be >>> > > making a >>> > > > > > > choice about a mirror. We have to assume we are working with > a >>> > > > scripted >>> > > > > > > application and anything hard coded in that implementation on >>> > > > > > > the >>> > > > > client >>> > > > > > > side is a risk. >>> > > > > > > >>> > > > > > > I may not be understanding the use of the 300 approach, so I >>> > > > > > > will >>> > > > > > research >>> > > > > > > it to see if I can make it fit. >>> > > > > > > >>> > > > > > > On Mar 3, 2017 9:59 AM, "Rafael Weingärtner" < >>> > > > > > rafaelweingart...@gmail.com> >>> > > > > > > wrote: >>> > > > > > > >>> > > > > > > Will great job. >>> > > > > > > >>> > > > > > > I had the same doubt as Daan. >>> > > > > > > >>> > > > > > > Have you considered using HTTP 300 working mode? Then, we >>> > > > > > > could let >>> > > > the >>> > > > > > > client decide which mirror is the best(closest?). Section >>> > “10.3.1” >>> > > of >>> > > > > [1] >>> > > > > > > talks about it; this would put the pressure on deciding upon > a >>> > > mirror >>> > > > > on >>> > > > > > > the client side, but I think it would be fair. >>> > > > > > > >>> > > > > > > [1] https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html >>> > > > > > > >>> > > > > > > PS: You really like very short variables names! >>> > > > > > > >>> > > > > > > On Fri, Mar 3, 2017 at 1:24 AM, Daan Hoogland < >>> > > > > > daan.hoogl...@shapeblue.com >>> > > > > > > > >>> > > > > > > wrote: >>> > > > > > > >>> > > > > > > > Nice little thing Will, >>> > > > > > > > >>> > > > > > > > One question: if I read the code correctly it ‘go’es and >>> > > > > > > > tries >>> > > all >>> > > > > > > mirrors >>> > > > > > > > at once and whichever responses first is redirected to the >>> > > client. >>> > > > > This >>> > > > > > > > might well be the same every time. This might be the one >>> > > > > > > > closest >>> > > to >>> > > > > the >>> > > > > > > > mirror302 and not to the client. Is that correct or did I >>> > > > > > > > miss a >>> > > > > > weighing >>> > > > > > > > algorithm hidden in there? >>> > > > > > > > >>> > > > > > > > Good coding, >>> > > > > > > > >>> > > > > > > > On 03/03/17 00:23, "Will Stevens" <sw...@apache.org> wrote: >>> > > > > > > > >>> > > > > > > > Hey All, >>> > > > > > > > Please review this repo and tell me what you think: >>> > > > > > > > https://github.com/swill/mirror302 >>> > > > > > > > >>> > > > > > > > Let me know if you have questions or you would like me >>> > > > > > > > to >>> > > make >>> > > > > > > changes >>> > > > > > > > before I bring this topic up with ASF Infra and open a >>> > > > > > > > ticket >>> > > > to >>> > > > > > get >>> > > > > > > > this >>> > > > > > > > implemented. >>> > > > > > > > >>> > > > > > > > Cheers, >>> > > > > > > > >>> > > > > > > > *Will Stevens* >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > On Wed, Mar 1, 2017 at 11:37 AM, Will Stevens < >>> > > > > > wstev...@cloudops.com >>> > > > > > > > >>> > > > > > > > wrote: >>> > > > > > > > >>> > > > > > > > > I am building a short term solution right now. I >>> > > > > > > > hopefully >>> > > > > will >>> > > > > > > > have the >>> > > > > > > > > building blocks in place by the weekend so we can >>> > > > > > > > start >>> > > > working >>> > > > > > > with >>> > > > > > > > Infra >>> > > > > > > > > to get it in place. >>> > > > > > > > > >>> > > > > > > > > We will have a web server which we can point > something >>> > like >>> > > > > > > > > downloads.cloudstack.org (or whatever url) at. Then >>> > > > > > > > we >>> > > will >>> > > > > > have >>> > > > > > > > an ASF >>> > > > > > > > > repo which tracks a mirror list and exposes a static >>> > > > > > > > site >>> > > to >>> > > > be >>> > > > > > > able >>> > > > > > > > to >>> > > > > > > > > query the mirror list. >>> > > > > > > > > >>> > > > > > > > > The web server will receive a request and will do a >>> > > > > > > > 302 >>> > > > > redirect >>> > > > > > to >>> > > > > > > > the >>> > > > > > > > > appropriate resource on one of the mirrors. >>> > > > > > > > > >>> > > > > > > > > This gets us started. Legacy environments will have >>> > > > > > > > to do >>> > > a >>> > > > DB >>> > > > > > > > change to >>> > > > > > > > > change from 'download.cloud.com' to ' >>> > > > downloads.cloudstack.org' >>> > > > > > (or >>> > > > > > > > > whatever), but otherwise it should be pretty simple. >>> > > > > > > > > >>> > > > > > > > > We can improve how we deliver templates going > forward, >>> > > > > > > > but >>> > > > this >>> > > > > > > > obviously >>> > > > > > > > > requires some discussion still. I will try to buy us >>> > > > > > > > some >>> > > > time >>> > > > > > > with >>> > > > > > > > an >>> > > > > > > > > implementation which solves our problems today... >>> > > > > > > > > >>> > > > > > > > > *Will STEVENS* >>> > > > > > > > > Lead Developer >>> > > > > > > > > >>> > > > > > > > > <https://goo.gl/NYZ8KK> >>> > > > > > > > > >>> > > > > > > > > On Wed, Mar 1, 2017 at 7:52 AM, Abhinandan Prateek < >>> > > > > > > > > abhinandan.prat...@shapeblue.com> wrote: >>> > > > > > > > > >>> > > > > > > > >> >>> > > > > > > > >> Initial seeding is a manual step and that is not >>> > > > > > > > going to >>> > > > > change >>> > > > > > > in >>> > > > > > > > near >>> > > > > > > > >> future. A handy list of official places from where >>> > > > > > > > these >>> > > > > > templates >>> > > > > > > > can be >>> > > > > > > > >> downloaded won’t harm this part of the installation. >>> > > > > > > > Just >>> > > > > allow >>> > > > > > > > anyone >>> > > > > > > > >> credible who follows democratic process to publish >>> > > > > > > > their >>> > > > > > templates >>> > > > > > > > with >>> > > > > > > > >> apache’s blessings. >>> > > > > > > > >> >>> > > > > > > > >> >>> > > > > > > > >> Coming to urls hard coded in db, the popular >>> > > > > > > > suggestion is >>> > > > to >>> > > > > > > point >>> > > > > > > > them >>> > > > > > > > >> to mirrors. This looks good but this is going to > take >>> > > > > > > > some >>> > > > > > effort >>> > > > > > > > that has >>> > > > > > > > >> not materialised yet. Once this change is made it >>> > > > > > > > will be >>> > > > > going >>> > > > > > in >>> > > > > > > > some >>> > > > > > > > >> future release like 4.11; the problem still remains >>> > > > > > > > for >>> > > the >>> > > > > > > > releases that >>> > > > > > > > >> are already out there. In case access to >>> > > download.cloud.com >>> > > > > is >>> > > > > > > > dropped >>> > > > > > > > >> at some point we will have to document the procedure >>> > > > > > > > such >>> > > > that >>> > > > > > > > people know >>> > > > > > > > >> how to make these stable releases work. That is > where >>> > > > > > > > the >>> > > > > > > procedure >>> > > > > > > > >> documented here https://shankerbalan.net/blog/ >>> > > > > > > > >> seed-cloudstack-templates-offline/ should be >>> > > > > > > > officially >>> > > > > adapted >>> > > > > > > > with a >>> > > > > > > > >> marketplace to pick up templates of choosing. Maybe >>> > > > > > > > we can >>> > > > > have >>> > > > > > a >>> > > > > > > > pre-setup >>> > > > > > > > >> script to make choices instead of written wiki. Note >>> > > > > > > > that >>> > > > this >>> > > > > > > will >>> > > > > > > > be >>> > > > > > > > >> anyway needed by the older releases. >>> > > > > > > > >> >>> > > > > > > > >> >>> > > > > > > > >> >>> > > > > > > > >> >>> > > > > > > > >> On 01/03/17, 4:24 PM, "Paul Angus" < >>> > > > paul.an...@shapeblue.com> >>> > > > > > > > wrote: >>> > > > > > > > >> >>> > > > > > > > >> >-1 >>> > > > > > > > >> > >>> > > > > > > > >> >We are trying to get MORE people to use CloudStack. >>> > > > > > > > And >>> > > > > > raising >>> > > > > > > > the >>> > > > > > > > >> barrier to entry does nothing to help that. >>> > > > > > > > >> > >>> > > > > > > > >> > >>> > > > > > > > >> >Kind regards, >>> > > > > > > > >> > >>> > > > > > > > >> >Paul Angus >>> > > > > > > > >> > >>> > > > > > > > >> >paul.an...@shapeblue.com >>> > > > > > > > >> >www.shapeblue.com >>> > > > > > > > >> >53 Chandos Place, Covent Garden, London WC2N 4HSUK >>> > > > > > > > >> >@shapeblue >>> > > > > > > > >> > >>> > > > > > > > >> > >>> > > > > > > > >> > >>> > > > > > > > >> > >>> > > > > > > > >> >-----Original Message----- >>> > > > > > > > >> >From: Abhinandan Prateek [mailto: > abhinandan.prateek@ >>> > > > > > > shapeblue.com] >>> > > > > > > > >> >Sent: 01 March 2017 09:39 >>> > > > > > > > >> >To: dev@cloudstack.apache.org >>> > > > > > > > >> >Subject: Re: Modern template hosting >>> > > > > > > > >> > >>> > > > > > > > >> >+1 >>> > > > > > > > >> > >>> > > > > > > > >> >After seeing this discussion running in circles >>> > > > > > > > several >>> > > > > times, >>> > > > > > I >>> > > > > > > > think >>> > > > > > > > >> we should at least get started with the simplest >>> option. >>> > > > > > > > >> > >>> > > > > > > > >> > >>> > > > > > > > >> > >>> > > > > > > > >> > >>> > > > > > > > >> >On 27/02/17, 11:54 PM, "Chiradeep Vittal" < >>> > > > > > chirade...@gmail.com> >>> > > > > > > > wrote: >>> > > > > > > > >> > >>> > > > > > > > >> >>The alternative foolproof way is to simply not >>> > > > > > > > provide >>> > > the >>> > > > > > > > automatic >>> > > > > > > > >> >>download. >>> > > > > > > > >> >> >>> > > > > > > > >> >>Just document the mirror list URL where the >>> > > > > > > > templates >>> > > can >>> > > > be >>> > > > > > > > found. I >>> > > > > > > > >> >>don't know why people reject this option. >>> > > > > > > > >> >> >>> > > > > > > > >> >>https://shankerbalan.net/blog/seed-cloudstack- >>> > > > > > > templates-offline/ >>> > > > > > > > >> >> >>> > > > > > > > >> >>On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens >>> > > > > > > > >> >><williamstev...@gmail.com> >>> > > > > > > > >> >>wrote: >>> > > > > > > > >> >> >>> > > > > > > > >> >>> OK. Thanks for the heads up. >>> > > > > > > > >> >>> >>> > > > > > > > >> >>> On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" < >>> > > > > > > > chirade...@gmail.com> >>> > > > > > > > >> wrote: >>> > > > > > > > >> >>> >>> > > > > > > > >> >>> > Sounds workable. The downloader code in the >>> > > > > > > > SSVM >>> > > won't >>> > > > > > > follow >>> > > > > > > > >> >>> > redirects I think. >>> > > > > > > > >> >>> > https://github.com/apache/cloudstack/blob/ >>> > > > > > > 5511065fc20787619d >>> > > > > > > > 9cd0444 >>> > > > > > > > >> >>> > a65a3 155fc9c921/core/src/com/cloud/ >>> > > storage/template/ >>> > > > > > > > >> >>> > HttpTemplateDownloader.java#L93 >>> > > > > > > > >> >>> > https://goo.gl/dSi0r5 >>> > > > > > > > >> >>> > Might need to add >>> > > > > > > > >> >>> > client.setRedirectStrategy(new >>> > > LaxRedirectStrategy()); >>> > > > > > > > >> >>> > >>> > > > > > > > >> >>> > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens >>> > > > > > > > >> >>> > <wstev...@cloudops.com> >>> > > > > > > > >> >>> > wrote: >>> > > > > > > > >> >>> > >>> > > > > > > > >> >>> > > We haven't opened a ticket yet because we >>> > > > > > > > don't >>> > > > have a >>> > > > > > > > strategy >>> > > > > > > > >> yet. >>> > > > > > > > >> >>> > > >>> > > > > > > > >> >>> > > What do you guys think of this: >>> > > > > > > > >> >>> > > - We setup a new github repo in the 'apache' >>> > > > > > > > org >>> > > > which >>> > > > > > > > consists >>> > > > > > > > >> >>> > > of a >>> > > > > > > > >> >>> > single >>> > > > > > > > >> >>> > > file with a list of active/supported > mirrors. >>> > > > > > > > >> >>> > > - I write a small web server, distributed as >>> > > > > > > > a >>> > > > binary, >>> > > > > > > > which can >>> > > > > > > > >> >>> > > be >>> > > > > > > > >> >>> > hosted >>> > > > > > > > >> >>> > > by ASF Infra. This web server will query > the >>> > > > current >>> > > > > > list >>> > > > > > > > of >>> > > > > > > > >> >>> > > mirrors >>> > > > > > > > >> >>> and >>> > > > > > > > >> >>> > > will select one and then do a 302 redirect > to >>> > > > > > > > that >>> > > > > > mirror. >>> > > > > > > > >> >>> > > >>> > > > > > > > >> >>> > > The act of 'choosing' a mirror could be done >>> > > > > > > > in a >>> > > > > number >>> > > > > > > of >>> > > > > > > > ways. >>> > > > > > > > >> >>> > > - If we want to define an order, then it >>> > > > > > > > could >>> > > just >>> > > > > try >>> > > > > > > > from the >>> > > > > > > > >> >>> > > top of >>> > > > > > > > >> >>> > the >>> > > > > > > > >> >>> > > list and work its way down. It would curl >>> > > > > > > > the >>> > > > target >>> > > > > to >>> > > > > > > > make >>> > > > > > > > >> >>> > > sure it >>> > > > > > > > >> >>> > gets >>> > > > > > > > >> >>> > > a 200 and if it does, it would do a 302 >>> > redirect. >>> > > > > > > > >> >>> > > - Or, if we want to distribute the load >>> > > > > > > > across the >>> > > > > > > mirrors, >>> > > > > > > > we >>> > > > > > > > >> >>> > > could >>> > > > > > > > >> >>> pick >>> > > > > > > > >> >>> > > from the list randomly. Again, doing a curl >>> > > > > > > > to >>> > > > verify >>> > > > > > the >>> > > > > > > > mirror >>> > > > > > > > >> >>> > > is up >>> > > > > > > > >> >>> > and >>> > > > > > > > >> >>> > > then doing a redirect. >>> > > > > > > > >> >>> > > - If we want to get fancy, we could do a >>> > > > > > > > reverse >>> > > IP >>> > > > > > lookup >>> > > > > > > > and >>> > > > > > > > >> >>> > > try to >>> > > > > > > > >> >>> > match >>> > > > > > > > >> >>> > > the requester with their closest > geographical >>> > > > mirror. >>> > > > > > > > >> >>> > > >>> > > > > > > > >> >>> > > Thoughts? >>> > > > > > > > >> >>> > > >>> > > > > > > > >> >>> > > *Will STEVENS* >>> > > > > > > > >> >>> > > Lead Developer >>> > > > > > > > >> >>> > > >>> > > > > > > > >> >>> > > <https://goo.gl/NYZ8KK> >>> > > > > > > > >> >>> > > >>> > > > > > > > >> >>> > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep >>> > > Vittal < >>> > > > > > > > >> >>> chirade...@gmail.com >>> > > > > > > > >> >>> > > >>> > > > > > > > >> >>> > > wrote: >>> > > > > > > > >> >>> > > >>> > > > > > > > >> >>> > > > What steps are needed to set up a mirror? >>> > > > > > > > What >>> > > > does >>> > > > > > > Infra >>> > > > > > > > need >>> > > > > > > > >> to do? >>> > > > > > > > >> >>> > Has >>> > > > > > > > >> >>> > > > anybody filed a ticket with Infra? >>> > > > > > > > >> >>> > > > >>> > > > > > > > >> >>> > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja >>> > Pullela < >>> > > > > > > > >> >>> > > > raja.pull...@accelerite.com> >>> > > > > > > > >> >>> > > > wrote: >>> > > > > > > > >> >>> > > > >>> > > > > > > > >> >>> > > > > Hi will, >>> > > > > > > > >> >>> > > > > >>> > > > > > > > >> >>> > > > > I believe, we didn’t get to close >>> > > > > > > > ‘getting a >>> > > > > mirror >>> > > > > > on >>> > > > > > > > Apache’ >>> > > > > > > > >> >>> > because >>> > > > > > > > >> >>> > > we >>> > > > > > > > >> >>> > > > > needed someone on the Apache Infra side >>> > > > > > > > to >>> > > close >>> > > > > > this. >>> > > > > > > > BTW, >>> > > > > > > > >> >>> > > > > cloudstack-apt.get.eu (I think Nux >>> > > > > > > > manages >>> > > > this?) >>> > > > > > has >>> > > > > > > > >> >>> > > > > all/most of >>> > > > > > > > >> >>> > the >>> > > > > > > > >> >>> > > > > content. Once we can close on the > Apache >>> > > mirror >>> > > > > for >>> > > > > > > > hosting >>> > > > > > > > >> >>> > > > > the >>> > > > > > > > >> >>> > > > content, I >>> > > > > > > > >> >>> > > > > can help assist getting the content > there. >>> > > > > > > > >> >>> > > > > >>> > > > > > > > >> >>> > > > > For now, we have replicated the >>> > > > > download.cloud.com >>> > > > > > > > content to >>> > > > > > > > >> >>> > > > > ‘ s3.download.accelerite.com’. >>> > > > > > > > >> >>> > > > > Also, we are working on a set of >>> > > steps/procedure >>> > > > > to >>> > > > > > > > help with >>> > > > > > > > >> >>> > > > > this change. I will update everyone in >>> > > > > > > > about >>> > > a >>> > > > > > week’s >>> > > > > > > > time >>> > > > > > > > >> >>> > > > > on the >>> > > > > > > > >> >>> > details. >>> > > > > > > > >> >>> > > > > >>> > > > > > > > >> >>> > > > > Best, >>> > > > > > > > >> >>> > > > > Raja Pullela >>> > > > > > > > >> >>> > > > > Engineering Team, >>> > > > > > > > >> >>> > > > > Accelerite, 2055 Laurelwood Road, Santa >>> > > > > > > > Clara, >>> > > > CA, >>> > > > > > > 95054 >>> > > > > > > > >> >>> > > > > >>> > > > > > > > >> >>> > > > > On 2/24/17, 11:23 PM, " >>> > > williamstev...@gmail.com >>> > > > > on >>> > > > > > > > behalf of >>> > > > > > > > >> >>> > > > > Will Stevens" <williamstev...@gmail.com >>> > > > > > > > on >>> > > > behalf >>> > > > > > of >>> > > > > > > > >> >>> > wstev...@cloudops.com> >>> > > > > > > > >> >>> > > > > wrote: >>> > > > > > > > >> >>> > > > > >>> > > > > > > > >> >>> > > > > unfortunately the template mirror >>> > > > > > > > conversation >>> > > > got >>> > > > > > > > caught up >>> > > > > > > > >> >>> > > > > in >>> > > > > > > > >> >>> > details >>> > > > > > > > >> >>> > > > and >>> > > > > > > > >> >>> > > > > nobody took the lead on implementing a >>> > > solution. >>> > > > > > > > >> >>> > > > > >>> > > > > > > > >> >>> > > > > citrix has been pinging me every couple >>> > > > > > > > months >>> > > > to >>> > > > > > say >>> > > > > > > > 'dude, >>> > > > > > > > >> >>> > > > > we >>> > > > > > > > >> >>> need >>> > > > > > > > >> >>> > to >>> > > > > > > > >> >>> > > > > remove the dependency on >>> > > > > > > > download.citrix.com >>> > > ', >>> > > > > but >>> > > > > > i >>> > > > > > > > have not >>> > > > > > > > >> >>> > > > > had >>> > > > > > > > >> >>> > the >>> > > > > > > > >> >>> > > > > cycles to get in and solve the problem. >>> > > > > > > > the >>> > > > > > shutdown >>> > > > > > > > of that >>> > > > > > > > >> >>> > > > > is >>> > > > > > > > >> >>> > > imminent >>> > > > > > > > >> >>> > > > > right now, so we need to solve it asap. >>> > > > > > > > >> >>> > > > > >>> > > > > > > > >> >>> > > > > >>> > > > > > > > >> >>> > > > > >>> > > > > > > > >> >>> > > > > *Will STEVENS* >>> > > > > > > > >> >>> > > > > Lead Developer >>> > > > > > > > >> >>> > > > > >>> > > > > > > > >> >>> > > > > <https://goo.gl/NYZ8KK> >>> > > > > > > > >> >>> > > > > >>> > > > > > > > >> >>> > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul >>> > Angus < >>> > > > > > > > >> >>> > paul.an...@shapeblue.com >>> > > > > > > > >> >>> > > > >>> > > > > > > > >> >>> > > > > wrote: >>> > > > > > > > >> >>> > > > > >>> > > > > > > > >> >>> > > > > > Hi Nathan, >>> > > > > > > > >> >>> > > > > > >>> > > > > > > > >> >>> > > > > > Ideally, if you put the template >>> > > > > > > > location in >>> > > > (or >>> > > > > > use >>> > > > > > > a >>> > > > > > > > >> >>> > > > > > template >>> > > > > > > > >> >>> > > defined >>> > > > > > > > >> >>> > > > > > in) test_data.py then the actual >>> > > > > > > > location >>> > > can >>> > > > > be >>> > > > > > > > >> >>> > > > > > overridden by >>> > > > > > > > >> >>> > > anyone >>> > > > > > > > >> >>> > > > > > testing. >>> > > > > > > > >> >>> > > > > > >>> > > > > > > > >> >>> > > > > > For Trillian, we've copied all of the >>> > > > templates >>> > > > > > that >>> > > > > > > > people >>> > > > > > > > >> >>> > > > > > have >>> > > > > > > > >> >>> > > define >>> > > > > > > > >> >>> > > > > to >>> > > > > > > > >> >>> > > > > > a local repo and then replace the URLs >>> > > > > > > > in test_data.py to >>> > > > > > > > >> >>> > > > > > reduce >>> > > > > > > > >> >>> > > > > bandwidth >>> > > > > > > > >> >>> > > > > > use and download times. >>> > > > > > > > >> >>> > > > > > >>> > > > > > > > >> >>> > > > > > Ie: >>> > > > > > > > >> >>> > > > > > >>> > > > > > > > >> >>> > > > > > "bootableIso": >>> > > > > > > > >> >>> > > > > > { >>> > > > > > > > >> >>> > > > > > "displaytext": > "Test >>> > > > > Bootable >>> > > > > > > > ISO", >>> > > > > > > > >> >>> > > > > > "name": "testISO", >>> > > > > > > > >> >>> > > > > > "bootable": True, >>> > > > > > > > >> >>> > > > > > "ispublic": False, >>> > > > > > > > >> >>> > > > > > "url": "{{ >>> > > > > > > marvin_images_location >>> > > > > > > > >> >>> > > > > > }}/TinyCore-current.iso", >>> > > > > > > > >> >>> > > > > > "ostype": 'Other >>> Linux >>> > > > > > > (64-bit)', >>> > > > > > > > >> >>> > > > > > "mode": >>> > 'HTTP_DOWNLOAD' >>> > > > > > > > >> >>> > > > > > }, >>> > > > > > > > >> >>> > > > > > >>> > > > > > > > >> >>> > > > > > >>> > > > > > > > >> >>> > > > > > >>> > > > > > > > >> >>> > > > > > I thought that we had come up with a >>> > > solution >>> > > > > for >>> > > > > > > > >> >>> > download.cloud.com >>> > > > > > > > >> >>> > > , >>> > > > > > > > >> >>> > > > by >>> > > > > > > > >> >>> > > > > > having a mirrorlist hosted in > Community >>> > > Apache >>> > > > > > > > 'space' with >>> > > > > > > > >> >>> anyone >>> > > > > > > > >> >>> > > able >>> > > > > > > > >> >>> > > > > to >>> > > > > > > > >> >>> > > > > > out themselves forward as a mirror. >>> > > > > > > > >> >>> > > > > > But I must admit I lost track of >>> > > > > > > > whether >>> > > > anyone >>> > > > > > made >>> > > > > > > > the >>> > > > > > > > >> >>> requisite >>> > > > > > > > >> >>> > > > > changes >>> > > > > > > > >> >>> > > > > > in code.... >>> > > > > > > > >> >>> > > > > > >>> > > > > > > > >> >>> > > > > > >>> > > > > > > > >> >>> > > > > > >>> > > > > > > > >> >>> > > > > > >>> > > > > > > > >> >>> > > > > > Kind regards, >>> > > > > > > > >> >>> > > > > > >>> > > > > > > > >> >>> > > > > > Paul Angus >>> > > > > > > > >> >>> > > > > > >>> > > > > > > > >> >>> > > > > > paul.an...@shapeblue.com >>> > > > > > > > >> >>> > > > > > www.shapeblue.com >>> > > > > > > > >> >>> > > > > > 53 Chandos Place, Covent Garden, > London >>> > > WC2N >>> > > > > > 4HSUK >>> > > > > > > > >> >>> > > > > > @shapeblue >>> > > > > > > > >> >>> > > > > > >>> > > > > > > > >> >>> > > > > > >>> > > > > > > > >> >>> > > > > > >>> > > > > > > > >> >>> > > > > > >>> > > > > > > > >> >>> > > > > > -----Original Message----- >>> > > > > > > > >> >>> > > > > > From: williamstev...@gmail.com >>> > > > > > > > >> >>> > > > > > [mailto:williamstev...@gmail.com] >>> > > > > > > > >> >>> > On >>> > > > > > > > >> >>> > > > > > Behalf Of Will Stevens >>> > > > > > > > >> >>> > > > > > Sent: 24 February 2017 16:30 >>> > > > > > > > >> >>> > > > > > To: dev@cloudstack.apache.org >>> > > > > > > > >> >>> > > > > > Subject: Re: Modern template hosting >>> > > > > > > > >> >>> > > > > > >>> > > > > > > > >> >>> > > > > > this is a hard questions. in general, >>> > > > > > > > we >>> > > > should >>> > > > > > be >>> > > > > > > > setting >>> > > > > > > > >> >>> > > > > > up a >>> > > > > > > > >> >>> > > mirror >>> > > > > > > > >> >>> > > > > on >>> > > > > > > > >> >>> > > > > > some cloudstack/apache domain and then >>> > > mirror >>> > > > to >>> > > > > > > other >>> > > > > > > > >> >>> > > > > > provided >>> > > > > > > > >> >>> > > > > templates. >>> > > > > > > > >> >>> > > > > > >>> > > > > > > > >> >>> > > > > > we MUST come up with a solution to >>> > deprecate >>> > > > > > > > >> >>> > > > > > 'download.cloud.com >>> > > > > > > > >> >>> ', >>> > > > > > > > >> >>> > > > that >>> > > > > > > > >> >>> > > > > > is going to be going away any day now. >>> > > > > > > > >> >>> > > > > > >>> > > > > > > > >> >>> > > > > > i don't know the right way to solve >>> > > > > > > > this to >>> > > be >>> > > > > > > > honest, but >>> > > > > > > > >> >>> > > > > > if you >>> > > > > > > > >> >>> > > have >>> > > > > > > > >> >>> > > > > > ideas, i am willing to help. >>> > > > > > > > >> >>> > > > > > >>> > > > > > > > >> >>> > > > > > >>> > > > > > > > >> >>> > > > > > >>> > > > > > > > >> >>> > > > > > *Will STEVENS* >>> > > > > > > > >> >>> > > > > > Lead Developer >>> > > > > > > > >> >>> > > > > > >>> > > > > > > > >> >>> > > > > > <https://goo.gl/NYZ8KK> >>> > > > > > > > >> >>> > > > > > >>> > > > > > > > >> >>> > > > > > On Fri, Feb 24, 2017 at 11:25 AM, >>> > > > > > > > Nathan >>> > > > > Johnson < >>> > > > > > > > >> >>> njohn...@ena.com >>> > > > > > > > >> >>> > > >>> > > > > > > > >> >>> > > > > wrote: >>> > > > > > > > >> >>> > > > > > >>> > > > > > > > >> >>> > > > > > > So not to re-open a can of worms, > but >>> > > > > > > > I’m >>> > > > in a >>> > > > > > > > situation >>> > > > > > > > >> >>> > > > > > > where >>> > > > > > > > >> >>> I >>> > > > > > > > >> >>> > > need >>> > > > > > > > >> >>> > > > > > > to come up with a Marvin component >>> > > > > > > > test >>> > > that >>> > > > > > > > depends on a >>> > > > > > > > >> >>> > template >>> > > > > > > > >> >>> > > > > > > based on a kernel that’s relatively >>> > > > > > > > new, >>> > > > i.e., >>> > > > > > > > newer than >>> > > > > > > > >> >>> Centos >>> > > > > > > > >> >>> > > 5.3 >>> > > > > > > > >> >>> > > > / >>> > > > > > > > >> >>> > > > > > Ubuntu 10.04 . >>> > > > > > > > >> >>> > > > > > > I see openvm.eu has a suitable >>> > > > > > > > template >>> > > > > (Ubuntu >>> > > > > > > > 16.0.4 >>> > > > > > > > >> >>> > > > > > > for >>> > > > > > > > >> >>> KVM), >>> > > > > > > > >> >>> > > but >>> > > > > > > > >> >>> > > > > > > from looking at the thread > "Migrating >>> > > > > CloudStack >>> > > > > > > > content >>> > > > > > > > >> >>> > > > > > > from download.cloud.com” it looks >>> > > > > > > > like >>> > > > there >>> > > > > is >>> > > > > > > > >> >>> > > > > > > resistance to using >>> > > > > > > > >> >>> > > this >>> > > > > > > > >> >>> > > > at >>> > > > > > > > >> >>> > > > > > > least for hosting system vm > templates >>> > > > > > > > over >>> > > > > > > concerns >>> > > > > > > > of >>> > > > > > > > >> >>> > neutrality. >>> > > > > > > > >> >>> > > > > > > Would this be suitable for a >>> > > > > > > > component >>> > > test? >>> > > > > If >>> > > > > > > > not, >>> > > > > > > > >> >>> > > > > > > what is a >>> > > > > > > > >> >>> > > > > “blessed” >>> > > > > > > > >> >>> > > > > > > template location? >>> > > > > > > > >> >>> > > > > > > >>> > > > > > > > >> >>> > > > > > > Thanks in advance! >>> > > > > > > > >> >>> > > > > > > >>> > > > > > > > >> >>> > > > > > > Nathan Johnson >>> > > > > > > > >> >>> > > > > > > R&D Engineer >>> > > > > > > > >> >>> > > > > > > >>> > > > > > > > >> >>> > > > > > > >>> > > > > > > > >> >>> > > > > > > >>> > > > > > > > >> >>> > > > > > > 618 Grassmere Park Drive, Suite 12 >>> > > > Nashville, >>> > > > > TN >>> > > > > > > > 37211 >>> > > > > > > > >> >>> > > > > > > General Office: 615-312-6000 >>> > > > > > > > >> >>> > > > > > > >>> > > > > > > > >> >>> > > > > > > website | blog | support >>> > > > > > > > >> >>> > > > > > > >>> > > > > > > > >> >>> > > > > > > >>> > > > > > > > >> >>> > > > > > > >>> > > > > > > > >> >>> > > > > > > >>> > > > > > > > >> >>> > > > > > > >>> > > > > > > > >> >>> > > > > > > >>> > > > > > > > >> >>> > > > > > > >>> > > > > > > > >> >>> > > > > > >>> > > > > > > > >> >>> > > > > >>> > > > > > > > >> >>> > > > > >>> > > > > > > > >> >>> > > > > >>> > > > > > > > >> >>> > > > > >>> > > > > > > > >> >>> > > > > >>> > > > > > > > >> >>> > > > > >>> > > > > > > > >> >>> > > > > DISCLAIMER >>> > > > > > > > >> >>> > > > > ========== >>> > > > > > > > >> >>> > > > > This e-mail may contain privileged and >>> > > > > confidential >>> > > > > > > > >> >>> > > > > information >>> > > > > > > > >> >>> which >>> > > > > > > > >> >>> > > is >>> > > > > > > > >> >>> > > > > the property of Accelerite, a Persistent >>> > > Systems >>> > > > > > > > business. It >>> > > > > > > > >> >>> > > > > is >>> > > > > > > > >> >>> > > intended >>> > > > > > > > >> >>> > > > > only for the use of the individual or >>> > > > > > > > entity >>> > > to >>> > > > > > which >>> > > > > > > > it is >>> > > > > > > > >> >>> > addressed. >>> > > > > > > > >> >>> > > If >>> > > > > > > > >> >>> > > > > you are not the intended recipient, you >>> > > > > > > > are >>> > > not >>> > > > > > > > authorized to >>> > > > > > > > >> >>> > > > > read, >>> > > > > > > > >> >>> > > > retain, >>> > > > > > > > >> >>> > > > > copy, print, distribute or use this >>> message. >>> > > If >>> > > > > you >>> > > > > > > have >>> > > > > > > > >> >>> > > > > received >>> > > > > > > > >> >>> > this >>> > > > > > > > >> >>> > > > > communication in error, please notify > the >>> > > sender >>> > > > > and >>> > > > > > > > delete >>> > > > > > > > >> >>> > > > > all >>> > > > > > > > >> >>> > copies >>> > > > > > > > >> >>> > > of >>> > > > > > > > >> >>> > > > > this message. Accelerite, a Persistent >>> > > > > > > > Systems >>> > > > > > > business >>> > > > > > > > does >>> > > > > > > > >> >>> > > > > not >>> > > > > > > > >> >>> > accept >>> > > > > > > > >> >>> > > > any >>> > > > > > > > >> >>> > > > > liability for virus infected mails. >>> > > > > > > > >> >>> > > > > >>> > > > > > > > >> >>> > > > >>> > > > > > > > >> >>> > > >>> > > > > > > > >> >>> > >>> > > > > > > > >> >>> >>> > > > > > > > >> > >>> > > > > > > > >> >abhinandan.prat...@shapeblue.com >>> > > > > > > > >> >www.shapeblue.com >>> > > > > > > > >> >53 Chandos Place, Covent Garden, London WC2N 4HSUK >>> > > > > @shapeblue >>> > > > > > > > >> > >>> > > > > > > > >> > >>> > > > > > > > >> > >>> > > > > > > > >> >>> > > > > > > > >> abhinandan.prat...@shapeblue.com >>> > > > > > > > >> www.shapeblue.com >>> > > > > > > > >> 53 Chandos Place, Covent Garden, London WC2N 4HSUK >>> > > > > > > > >> @shapeblue >>> > > > > > > > >> >>> > > > > > > > >> >>> > > > > > > > >> >>> > > > > > > > >> >>> > > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > daan.hoogl...@shapeblue.com >>> > > > > > > > www.shapeblue.com >>> > > > > > > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK >>> > > > > > > > @shapeblue >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > > >>> > > > > > > >>> > > > > > > >>> > > > > > > -- >>> > > > > > > Rafael Weingärtner >>> > > > > > > >>> > > > > > >>> > > > > > >>> > > > > > >>> > > > > > -- >>> > > > > > Rafael Weingärtner >>> > > > > > >>> > > > > >>> > > > >>> > > >>> > >>>
