Another thing, in my opinion, we should avoid these hashes (MD5, SHA1, and others) where hash collisions have already been found; if we will be working to create a new working flow for managing these system VMs templates, I think we should instead use SHA256 or SHA512.
On Mon, Feb 27, 2017 at 4:55 PM, Will Stevens <wstev...@cloudops.com> wrote: > @rafael: Yes, I think it makes sense to support sub-paths. It will make > things a lot easier to work with. The security question is an interesting > one. I am not sure I would be the best person to answer that. I would say > that we would want to have the MD5s hosted in the same repo as the mirror > list urls. > > I do like the idea of running a web server on the management server for > hosting the system vms though. It could be used to populate secondary > storage and could potentially give operators an easier (more efficient?) > workflow for introducing new templates. That area could use some work... > > *Will STEVENS* > Lead Developer > > <https://goo.gl/NYZ8KK> > > On Mon, Feb 27, 2017 at 4:36 PM, Chiradeep Vittal <chirade...@gmail.com> > wrote: > > > Hashes are checked (md5 IIRC) today. > > But given the issues, I think the project should steer away from hosting > > templates except the systemvm template. > > > > On Mon, Feb 27, 2017 at 1:31 PM, Rafael Weingärtner < > > rafaelweingart...@gmail.com> wrote: > > > > > Will, I think we could support different path structures. This can > > > facilitate different deployment of mirrors based on the structure the > > host > > > has. > > > > > > Could I add something else to the discussion? Have we discussed the > > > security impacts of setting up this mirrors approach? > > > I mean, if any of the mirrors gets corrupted (let`s say by a hacker), > and > > > the templates are injected with malicious code, an attacker could > > > potentially get un-monitored and unlimited access to a cloud > environment. > > > > > > If we assume that the mirror may get malicious (it is not that I do not > > > trust you guys, but bad things happen), we cannot host hashes there. > > Where > > > do you think we could store Sha512 or another hash type for these > > > templates? Could we host in the newly proposed Github repo or maybe > some > > > place in the ACS website? > > > > > > This would have an impact on clients (needing clear documentation) and > > our > > > code that automatically downloads System VM templates (does it check > > hashes > > > when automatically installing templates today? It may require > > > implementation changes). > > > > > > On Mon, Feb 27, 2017 at 3:48 PM, Will Stevens <wstev...@cloudops.com> > > > wrote: > > > > > > > so this is what I am looking to do. Please let me know if you have > > > > suggestions for me or think I should be solving the problem a > different > > > > way. > > > > > > > > - We request a new Github repository from the ASF at: > > > > 'apache/cloudstack-mirror-list' > > > > - In this repository we track a text file in the 'gh-pages' branch > > with a > > > > list of valid download mirrors. > > > > - I build a binary to be hosted by the ASF (or at least with the ASF > > > > pointing a domain at the binary and I could potentially host it). We > > > will > > > > see how they want to handle the hosting of the binary. > > > > > > > > The binary would expose a web server which would behave as follows: > > > > - When the 'client' requests a download url the following flow is > > kicked > > > > off: > > > > -- The mirror list is queried from github (or from a static site > hosted > > > on > > > > asf, as we see fit). > > > > -- The Lat/Lon of the 'client' is determined based on their IP. > > > > -- The Lat/Lon for each of the 'mirror's is determined based on an IP > > > > lookup of the hostname. > > > > -- The closest geographical mirror is determined, the target is > > validated > > > > to be available and the user is redirected. > > > > > > > > Some questions I have right now: > > > > - Will every mirror have the same path structure to access the > > equivalent > > > > resources? > > > > - Should we support adding a path to the mirror url to specify the > path > > > to > > > > the base common path? > > > > -- Example: lets say the binary is hosted on 'dl.acs.com' and there > > are > > > > three mirrors 'abc.com', 'pqr.com/files' and 'xyx.com/downloads'. > > > > -- If the path being requested is ' > > > > dl.acs.com/templates/systemvm-4.6.xen.vhd.bz2', it would result in > the > > > > following potential paths for the mirrors: > > > > -- 'abc.com/templates/systemvm-4.6.xen.vhd.bz2' > > > > -- 'pqr.com/files/templates/systemvm-4.6.xen.vhd.bz2' > > > > -- 'xyz.com/downloads/templates/systemvm-4.6.xen.vhd.bz2' > > > > > > > > Does this all make sense? > > > > > > > > *Will STEVENS* > > > > Lead Developer > > > > > > > > <https://goo.gl/NYZ8KK> > > > > > > > > On Mon, Feb 27, 2017 at 1:31 PM, Chiradeep Vittal < > > chirade...@gmail.com> > > > > wrote: > > > > > > > > > My bad. A few lines down, this has been added recently: > > > > > > > > > > this.request.setFollowRedirects(true); > > > > > > > > > > On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens < > > > williamstev...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > OK. Thanks for the heads up. > > > > > > > > > > > > On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" < > chirade...@gmail.com> > > > > > wrote: > > > > > > > > > > > > > Sounds workable. The downloader code in the SSVM won't follow > > > > > redirects I > > > > > > > think. > > > > > > > https://github.com/apache/cloudstack/blob/ > > > > > 5511065fc20787619d9cd0444a65a3 > > > > > > > 155fc9c921/core/src/com/cloud/storage/template/ > > > > > > > HttpTemplateDownloader.java#L93 > > > > > > > https://goo.gl/dSi0r5 > > > > > > > Might need to add > > > > > > > client.setRedirectStrategy(new LaxRedirectStrategy()); > > > > > > > > > > > > > > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens < > > > wstev...@cloudops.com > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > We haven't opened a ticket yet because we don't have a > strategy > > > > yet. > > > > > > > > > > > > > > > > What do you guys think of this: > > > > > > > > - We setup a new github repo in the 'apache' org which > consists > > > of > > > > a > > > > > > > single > > > > > > > > file with a list of active/supported mirrors. > > > > > > > > - I write a small web server, distributed as a binary, which > > can > > > be > > > > > > > hosted > > > > > > > > by ASF Infra. This web server will query the current list of > > > > mirrors > > > > > > and > > > > > > > > will select one and then do a 302 redirect to that mirror. > > > > > > > > > > > > > > > > The act of 'choosing' a mirror could be done in a number of > > ways. > > > > > > > > - If we want to define an order, then it could just try from > > the > > > > top > > > > > of > > > > > > > the > > > > > > > > list and work its way down. It would curl the target to make > > > sure > > > > it > > > > > > > gets > > > > > > > > a 200 and if it does, it would do a 302 redirect. > > > > > > > > - Or, if we want to distribute the load across the mirrors, > we > > > > could > > > > > > pick > > > > > > > > from the list randomly. Again, doing a curl to verify the > > mirror > > > > is > > > > > up > > > > > > > and > > > > > > > > then doing a redirect. > > > > > > > > - If we want to get fancy, we could do a reverse IP lookup > and > > > try > > > > to > > > > > > > match > > > > > > > > the requester with their closest geographical mirror. > > > > > > > > > > > > > > > > Thoughts? > > > > > > > > > > > > > > > > *Will STEVENS* > > > > > > > > Lead Developer > > > > > > > > > > > > > > > > <https://goo.gl/NYZ8KK> > > > > > > > > > > > > > > > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal < > > > > > > chirade...@gmail.com > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > What steps are needed to set up a mirror? What does Infra > > need > > > to > > > > > do? > > > > > > > Has > > > > > > > > > anybody filed a ticket with Infra? > > > > > > > > > > > > > > > > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela < > > > > > > > > > raja.pull...@accelerite.com> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi will, > > > > > > > > > > > > > > > > > > > > I believe, we didn’t get to close ‘getting a mirror on > > > Apache’ > > > > > > > because > > > > > > > > we > > > > > > > > > > needed someone on the Apache Infra side to close this. > BTW, > > > > > > > > > > cloudstack-apt.get.eu (I think Nux manages this?) has > > > all/most > > > > > of > > > > > > > the > > > > > > > > > > content. Once we can close on the Apache mirror for > > hosting > > > > the > > > > > > > > > content, I > > > > > > > > > > can help assist getting the content there. > > > > > > > > > > > > > > > > > > > > For now, we have replicated the download.cloud.com > content > > > to > > > > ‘ > > > > > > > > > > s3.download.accelerite.com’. > > > > > > > > > > Also, we are working on a set of steps/procedure to help > > with > > > > > this > > > > > > > > > > change. I will update everyone in about a week’s time on > > the > > > > > > > details. > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > Raja Pullela > > > > > > > > > > Engineering Team, > > > > > > > > > > Accelerite, 2055 Laurelwood Road, > > > > > > > > > > Santa Clara, CA, 95054 > > > > > > > > > > > > > > > > > > > > On 2/24/17, 11:23 PM, "williamstev...@gmail.com on > behalf > > of > > > > > Will > > > > > > > > > > Stevens" <williamstev...@gmail.com on behalf of > > > > > > > wstev...@cloudops.com> > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > unfortunately the template mirror conversation got caught > > up > > > in > > > > > > > details > > > > > > > > > and > > > > > > > > > > nobody took the lead on implementing a solution. > > > > > > > > > > > > > > > > > > > > citrix has been pinging me every couple months to say > > 'dude, > > > we > > > > > > need > > > > > > > to > > > > > > > > > > remove the dependency on download.citrix.com', but i > have > > > not > > > > > had > > > > > > > the > > > > > > > > > > cycles to get in and solve the problem. the shutdown of > > that > > > > is > > > > > > > > imminent > > > > > > > > > > right now, so we need to solve it asap. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Will STEVENS* > > > > > > > > > > Lead Developer > > > > > > > > > > > > > > > > > > > > <https://goo.gl/NYZ8KK> > > > > > > > > > > > > > > > > > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus < > > > > > > > paul.an...@shapeblue.com > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > Hi Nathan, > > > > > > > > > > > > > > > > > > > > > > Ideally, if you put the template location in (or use a > > > > template > > > > > > > > defined > > > > > > > > > > > in) test_data.py then the actual location can be > > > overridden > > > > by > > > > > > > > anyone > > > > > > > > > > > testing. > > > > > > > > > > > > > > > > > > > > > > For Trillian, we've copied all of the templates that > > people > > > > > have > > > > > > > > define > > > > > > > > > > to > > > > > > > > > > > a local repo and then replace the URLs in test_data.py > to > > > > > reduce > > > > > > > > > > bandwidth > > > > > > > > > > > use and download times. > > > > > > > > > > > > > > > > > > > > > > Ie: > > > > > > > > > > > > > > > > > > > > > > "bootableIso": > > > > > > > > > > > { > > > > > > > > > > > "displaytext": "Test Bootable ISO", > > > > > > > > > > > "name": "testISO", > > > > > > > > > > > "bootable": True, > > > > > > > > > > > "ispublic": False, > > > > > > > > > > > "url": "{{ marvin_images_location > > > > > > > > > > > }}/TinyCore-current.iso", > > > > > > > > > > > "ostype": 'Other Linux (64-bit)', > > > > > > > > > > > "mode": 'HTTP_DOWNLOAD' > > > > > > > > > > > }, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I thought that we had come up with a solution for > > > > > > > download.cloud.com > > > > > > > > , > > > > > > > > > by > > > > > > > > > > > having a mirrorlist hosted in Community Apache 'space' > > with > > > > > > anyone > > > > > > > > able > > > > > > > > > > to > > > > > > > > > > > out themselves forward as a mirror. > > > > > > > > > > > But I must admit I lost track of whether anyone made > the > > > > > > requisite > > > > > > > > > > changes > > > > > > > > > > > in code.... > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Kind regards, > > > > > > > > > > > > > > > > > > > > > > Paul Angus > > > > > > > > > > > > > > > > > > > > > > paul.an...@shapeblue.com > > > > > > > > > > > www.shapeblue.com > > > > > > > > > > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > > > > > > > > > > > @shapeblue > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > > > > From: williamstev...@gmail.com [mailto: > > > williamstevens@gmail. > > > > > com] > > > > > > > On > > > > > > > > > > > Behalf Of Will Stevens > > > > > > > > > > > Sent: 24 February 2017 16:30 > > > > > > > > > > > To: dev@cloudstack.apache.org > > > > > > > > > > > Subject: Re: Modern template hosting > > > > > > > > > > > > > > > > > > > > > > this is a hard questions. in general, we should be > > setting > > > > up > > > > > a > > > > > > > > mirror > > > > > > > > > > on > > > > > > > > > > > some cloudstack/apache domain and then mirror to other > > > > provided > > > > > > > > > > templates. > > > > > > > > > > > > > > > > > > > > > > we MUST come up with a solution to deprecate ' > > > > > download.cloud.com > > > > > > ', > > > > > > > > > that > > > > > > > > > > > is going to be going away any day now. > > > > > > > > > > > > > > > > > > > > > > i don't know the right way to solve this to be honest, > > but > > > if > > > > > you > > > > > > > > have > > > > > > > > > > > ideas, i am willing to help. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > *Will STEVENS* > > > > > > > > > > > Lead Developer > > > > > > > > > > > > > > > > > > > > > > <https://goo.gl/NYZ8KK> > > > > > > > > > > > > > > > > > > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson < > > > > > > njohn...@ena.com > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > So not to re-open a can of worms, but I’m in a > > situation > > > > > where > > > > > > I > > > > > > > > need > > > > > > > > > > > > to come up with a Marvin component test that depends > > on a > > > > > > > template > > > > > > > > > > > > based on a kernel that’s relatively new, i.e., newer > > than > > > > > > Centos > > > > > > > > 5.3 > > > > > > > > > / > > > > > > > > > > > Ubuntu 10.04 . > > > > > > > > > > > > I see openvm.eu has a suitable template (Ubuntu > 16.0.4 > > > for > > > > > > KVM), > > > > > > > > but > > > > > > > > > > > > from looking at the thread "Migrating CloudStack > > content > > > > from > > > > > > > > > > > > download.cloud.com” it looks like there is > resistance > > to > > > > > using > > > > > > > > this > > > > > > > > > at > > > > > > > > > > > > least for hosting system vm templates over concerns > of > > > > > > > neutrality. > > > > > > > > > > > > Would this be suitable for a component test? If not, > > > what > > > > > is a > > > > > > > > > > “blessed” > > > > > > > > > > > > template location? > > > > > > > > > > > > > > > > > > > > > > > > Thanks in advance! > > > > > > > > > > > > > > > > > > > > > > > > Nathan Johnson > > > > > > > > > > > > R&D Engineer > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 618 Grassmere Park Drive, Suite 12 > > > > > > > > > > > > Nashville, TN 37211 > > > > > > > > > > > > General Office: 615-312-6000 > > > > > > > > > > > > > > > > > > > > > > > > website | blog | support > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > DISCLAIMER > > > > > > > > > > ========== > > > > > > > > > > This e-mail may contain privileged and confidential > > > information > > > > > > which > > > > > > > > is > > > > > > > > > > the property of Accelerite, a Persistent Systems > business. > > It > > > > is > > > > > > > > intended > > > > > > > > > > only for the use of the individual or entity to which it > is > > > > > > > addressed. > > > > > > > > If > > > > > > > > > > you are not the intended recipient, you are not > authorized > > to > > > > > read, > > > > > > > > > retain, > > > > > > > > > > copy, print, distribute or use this message. If you have > > > > received > > > > > > > this > > > > > > > > > > communication in error, please notify the sender and > delete > > > all > > > > > > > copies > > > > > > > > of > > > > > > > > > > this message. Accelerite, a Persistent Systems business > > does > > > > not > > > > > > > accept > > > > > > > > > any > > > > > > > > > > liability for virus infected mails. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > Rafael Weingärtner > > > > > > -- Rafael Weingärtner
