Re: Modern template hosting

[Daan Hoogland]

on and on.
> >>>>
> >>>> If they can figure all that out and not figure out a web server, they
> > can
> >>>> always come to the mailing list.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Fri, Mar 3, 2017 at 1:45 PM, Paul Angus <paul.an...@shapeblue.com>
> >>>> wrote:
> >>>>
> >>>> > The issue is not with supporting highly experienced cloud 
operators.
> > It
> >>>> is
> >>>> > an issue for new users and other 'relatively' inexperienced
> operators.
> >>>> >
> >>>> > I have helped enough newbies and cloud operators who have been
> running
> >>>> > their cloud for a while to know that the barriers to entry are too
> > high
> >>>> as
    > >>>> > they are. And telling anyone that they need to create a web server
> so
> >>>> that
> >>>> > they can add their initial template to get started or in order to
> >> create
> >>>> a
> >>>> > new zone, just isn't going to fly.
> >>>> >
> >>>> > I'm a huge advocate of the 'download.cloudstack.org' endpoint which
> > the
> >>>> > community can add/remove mirrors to or from for system VMs or
> built-in
> >>>> > templates.  Can the same system be used for binary repos ? although
> >> there
> >>>> > is an added complication of redist vs no-redist there...
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > paul.an...@shapeblue.com
> >>>> > www.shapeblue.com
> >>>> > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> >>>> > @shapeblue
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > -Original Message-
> >>>> > From: Chiradeep Vittal [mailto:chirade...@gmail.com]
> >>>> > Sent: 03 March 2017 18:28
> >>>> > To: dev <dev@cloudstack.apache.org>
> >>>> > Subject: Re: Modern template hosting
> >>>> >
> >>>> > I do feel like this is early optimization. Mirrors rarely fail. I'd
> >>>> expect
> >>>> > a single web server hosted on Apache Infra without any monitors to
> > fail
> >>>> > more often than a mirror. We already expect Wido's systemvm
> repository
> >> to
> >>>> > be up all the time. And it has been. Similarly, I don't believe
> Nux's
> >>>> > repository has ever been down. And if Accelerite wants to host on
> S3,
> >>>> that
> >>>> > one is pretty solid as well.
> >>>> >
> >>>> > This is an infrequent operation in a cloud. After the cloud is
> >> installed,
> >>>> > the download servers are only needed for a new zone. If we trust 
the
> >> user
> >>>> > to run a cloud, surely he/she can run a web server to serve some
> >> built-in
> >>>> > templates. And if her cloud is successful, she needs to figure out
> how
> >> to
> >>>> > host her templates anyway and not rely on 3rd parties.
> >>>> >
> >>>> >
> >>>> > On Fri, Mar 3, 2017 at 9:31 AM, Will Stevens <wstev...@cloudops.com
> >
> >>>> > wrote:
> >>>> >
> >>>> > > 1) If the legacy implementations do not support redirects, that
> does
> >>>> > > cause a problem.  A potential solution in that case is to have 
the
> >> web
> >>>> > > server actually proxy the download, but that is not ideal and I
> > would
> >>>> > > like to avoid it if possible.  Thanks for bringing that up
> > Chiradeep.
> >>>> > >
> >>>> > > 2) I think we need to have a single URL which people can target.
> >> Once
> >>>> > > they make

Re: Modern template hosting

[Will Stevens]

>> > > > > client
> >>>> > > > > > > side is a risk.
> >>>> > > > > > >
> >>>> > > > > > > I may not be understanding the use of the 300 approach,
> so I
> >>>> > > > > > > will
> >>>> > > > > > research
> >>>> > > > > > > it to see if I can make it fit.
> >>>> > > > > > >
> >>>> > > > > > > On Mar 3, 2017 9:59 AM, "Rafael Weingärtner" <
> >>>> > > > > > rafaelweingart...@gmail.com>
> >>>> > > > > > > wrote:
> >>>> > > > > > >
> >>>> > > > > > > Will great job.
> >>>> > > > > > >
> >>>> > > > > > > I had the same doubt as Daan.
> >>>> > > > > > >
> >>>> > > > > > > Have you considered using HTTP 300 working mode? Then, we
> >>>> > > > > > > could let
> >>>> > > > the
> >>>> > > > > > > client decide which mirror is the best(closest?). Section
> >>>> > “10.3.1”
> >>>> > > of
> >>>> > > > > [1]
> >>>> > > > > > > talks about it; this would put the pressure on deciding
> upon
> >> a
> >>>> > > mirror
> >>>> > > > > on
> >>>> > > > > > > the client side, but I think it would be fair.
> >>>> > > > > > >
> >>>> > > > > > > [1] https://www.w3.org/Protocols/
> rfc2616/rfc2616-sec10.html
> >>>> > > > > > >
> >>>> > > > > > > PS: You really like very short variables names!
> >>>> > > > > > >
> >>>> > > > > > > On Fri, Mar 3, 2017 at 1:24 AM, Daan Hoogland <
> >>>> > > > > > daan.hoogl...@shapeblue.com
> >>>> > > > > > > >
> >>>> > > > > > > wrote:
> >>>> > > > > > >
> >>>> > > > > > > > Nice little thing Will,
> >>>> > > > > > > >
> >>>> > > > > > > > One question: if I read the code correctly it ‘go’es and
> >>>> > > > > > > > tries
> >>>> > > all
> >>>> > > > > > > mirrors
> >>>> > > > > > > > at once and whichever responses first is redirected to
> the
> >>>> > > client.
> >>>> > > > > This
> >>>> > > > > > > > might well be the same every time. This might be the one
> >>>> > > > > > > > closest
> >>>> > > to
> >>>> > > > > the
> >>>> > > > > > > > mirror302 and not to the client. Is that correct or did
> I
> >>>> > > > > > > > miss a
> >>>> > > > > > weighing
> >>>> > > > > > > > algorithm hidden in there?
> >>>> > > > > > > >
> >>>> > > > > > > > Good coding,
> >>>> > > > > > > >
> >>>> > > > > > > > On 03/03/17 00:23, "Will Stevens" <sw...@apache.org>
> > wrote:
> >>>> > > > > > > >
> >>>> > > > > > > > Hey All,
> >>>> > > > > > > > Please review this repo and tell me what you think:
> >>>> > > > > > > > https://github.com/swill/mirror302
> >>>> > > > > > > >
> >>>> > > > > > > > Let me know if you have questions or you would like
> me
> >>>> > > > > > > > to
> >>>> > > make
> >>>> > > > > > > changes
> >>>> > > > > > > > before I bring this topic up with ASF Infra and
> open a
> >>>> > > > > > > > ticket
> >>>> > > > to
> >>>> &

Re: Modern template hosting

[Erik Weber]

Feel free to do it, I wouldn't know where to begin to ask such a
question tbh :-)

-- 
Erik

On Sat, Mar 4, 2017 at 12:51 AM, Will Stevens <williamstev...@gmail.com> wrote:
> Erik, yes I think this is the right approach and covers the main problem
> case. I can start the conversation with infra unless you would like to. Let
> me know.
>
> We did get some statistics from the other mirrors last year, but I don't
> think we ever had visibility into the cloud.com repository traffic.
>
> On Mar 3, 2017 6:42 PM, "Erik Weber" <terbol...@gmail.com> wrote:
>
> It'll most likely only be an issue in the case where a mirror goes
> down without any immediate chance of getting back up, right?
> Could we check with ASF Infra beforehand if such case is OK to warrant
> an urgent Jira ticket to resolve - should it happen?
>
> We can document all the mirrors and provide the howtos for manually
> downloading and seeding the image as a backup.
>
> Do we have any traffic statistics for the current solution, or is that
> not available at S3?
>
> --
> Erik
>
> On Sat, Mar 4, 2017 at 12:28 AM, Will Stevens <williamstev...@gmail.com>
> wrote:
>> Yes Erik, I agree with you. The only thing that could add complexity in
>> this is the fact that we don't control the domain, the ASF does (from what
>> I understand).  Do we expect this to be managed by ASF infra or would
> there
>> be another way? Ideally the domain is community controlled so we can adapt
>> in case of change without long delays.
>>
>> I am 100% in agreement that this is the shortest path and likely to be
> good
>> enough.
>>
>> On Mar 3, 2017 6:19 PM, "Erik Weber" <terbol...@gmail.com> wrote:
>>
>> IMHO; implement it as simple as possible, use DNS RR and assume/expect
>> mirror hosts to use the same path.
>>
>>
>> Yes I know; DNS RR isn't bulletproof, but as already mentioned in this
>> thread - mirrors rarely go down and they aren't used all that often
>> (primarily seeding).
>> We can keep a low TTL on the record so that we're able to remove a
>> mirror that goes down for a significant period of time (more than
>> hours).
>>
>> And yes I know, some operators might find it a burden having to set up
>> a new vhost with another document root, but tbh if they don't care
>> enough to do it we'll manage without that mirror.
>>
>> --
>> Erik
>>
>> On Sat, Mar 4, 2017 at 12:09 AM, Will Stevens <williamstev...@gmail.com>
>> wrote:
>>> I agree with Paul. Look at the list of things they have to
>>> learn/master/care about. We don't want to add to that list, we want to
>>> remove from that list. Make the project/product more accessible. Reduce
>> the
>>> barrier to entry...
>>>
>>> On Mar 3, 2017 5:50 PM, "Chiradeep Vittal" <chirade...@gmail.com> wrote:
>>>
>>>> Seriously?
>>>> apt-get install apache2
>>>>
>>>> To install CloudStack, they need to know
>>>> - DB installation, perhaps some SQL
>>>> - VLAN configuration on their switches
>>>> - Ins-and-outs between port forwarding, static ips,
>>>> - NFS
>>>> - package management
>>>> - VHDs, qcow2. vmdk
>>>> - hyperviosrs
>>>> - and on and on.
>>>>
>>>> If they can figure all that out and not figure out a web server, they
> can
>>>> always come to the mailing list.
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, Mar 3, 2017 at 1:45 PM, Paul Angus <paul.an...@shapeblue.com>
>>>> wrote:
>>>>
>>>> > The issue is not with supporting highly experienced cloud operators.
> It
>>>> is
>>>> > an issue for new users and other 'relatively' inexperienced operators.
>>>> >
>>>> > I have helped enough newbies and cloud operators who have been running
>>>> > their cloud for a while to know that the barriers to entry are too
> high
>>>> as
>>>> > they are. And telling anyone that they need to create a web server so
>>>> that
>>>> > they can add their initial template to get started or in order to
>> create
>>>> a
>>>> > new zone, just isn't going to fly.
>>>> >
>>>> > I'm a huge advocate of the 'download.cloudstack.org' endpoint which
> the
>>>> > community can add/remove mirrors to or from for system VMs or built-in
>>>> > templates.  Can the same system be use

Re: Modern template hosting

[Will Stevens]

Erik, yes I think this is the right approach and covers the main problem
case. I can start the conversation with infra unless you would like to. Let
me know.

We did get some statistics from the other mirrors last year, but I don't
think we ever had visibility into the cloud.com repository traffic.

On Mar 3, 2017 6:42 PM, "Erik Weber" <terbol...@gmail.com> wrote:

It'll most likely only be an issue in the case where a mirror goes
down without any immediate chance of getting back up, right?
Could we check with ASF Infra beforehand if such case is OK to warrant
an urgent Jira ticket to resolve - should it happen?

We can document all the mirrors and provide the howtos for manually
downloading and seeding the image as a backup.

Do we have any traffic statistics for the current solution, or is that
not available at S3?

--
Erik

On Sat, Mar 4, 2017 at 12:28 AM, Will Stevens <williamstev...@gmail.com>
wrote:
> Yes Erik, I agree with you. The only thing that could add complexity in
> this is the fact that we don't control the domain, the ASF does (from what
> I understand).  Do we expect this to be managed by ASF infra or would
there
> be another way? Ideally the domain is community controlled so we can adapt
> in case of change without long delays.
>
> I am 100% in agreement that this is the shortest path and likely to be
good
> enough.
>
> On Mar 3, 2017 6:19 PM, "Erik Weber" <terbol...@gmail.com> wrote:
>
> IMHO; implement it as simple as possible, use DNS RR and assume/expect
> mirror hosts to use the same path.
>
>
> Yes I know; DNS RR isn't bulletproof, but as already mentioned in this
> thread - mirrors rarely go down and they aren't used all that often
> (primarily seeding).
> We can keep a low TTL on the record so that we're able to remove a
> mirror that goes down for a significant period of time (more than
> hours).
>
> And yes I know, some operators might find it a burden having to set up
> a new vhost with another document root, but tbh if they don't care
> enough to do it we'll manage without that mirror.
>
> --
> Erik
>
> On Sat, Mar 4, 2017 at 12:09 AM, Will Stevens <williamstev...@gmail.com>
> wrote:
>> I agree with Paul. Look at the list of things they have to
>> learn/master/care about. We don't want to add to that list, we want to
>> remove from that list. Make the project/product more accessible. Reduce
> the
>> barrier to entry...
>>
>> On Mar 3, 2017 5:50 PM, "Chiradeep Vittal" <chirade...@gmail.com> wrote:
>>
>>> Seriously?
>>> apt-get install apache2
>>>
>>> To install CloudStack, they need to know
>>> - DB installation, perhaps some SQL
>>> - VLAN configuration on their switches
>>> - Ins-and-outs between port forwarding, static ips,
>>> - NFS
>>> - package management
>>> - VHDs, qcow2. vmdk
>>> - hyperviosrs
>>> - and on and on.
>>>
>>> If they can figure all that out and not figure out a web server, they
can
>>> always come to the mailing list.
>>>
>>>
>>>
>>>
>>> On Fri, Mar 3, 2017 at 1:45 PM, Paul Angus <paul.an...@shapeblue.com>
>>> wrote:
>>>
>>> > The issue is not with supporting highly experienced cloud operators.
It
>>> is
>>> > an issue for new users and other 'relatively' inexperienced operators.
>>> >
>>> > I have helped enough newbies and cloud operators who have been running
>>> > their cloud for a while to know that the barriers to entry are too
high
>>> as
>>> > they are. And telling anyone that they need to create a web server so
>>> that
>>> > they can add their initial template to get started or in order to
> create
>>> a
>>> > new zone, just isn't going to fly.
>>> >
>>> > I'm a huge advocate of the 'download.cloudstack.org' endpoint which
the
>>> > community can add/remove mirrors to or from for system VMs or built-in
>>> > templates.  Can the same system be used for binary repos ? although
> there
>>> > is an added complication of redist vs no-redist there...
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > paul.an...@shapeblue.com
>>> > www.shapeblue.com
>>> > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>>> > @shapeblue
>>> >
>>> >
>>> >
>>> >
>>> > -Original Message-
>>> > From: Chiradeep Vittal [mailto:chirade...@gmail.com]
>>> > Sent: 03 March 2017 1

Re: Modern template hosting

[Will Stevens]

I don't think your analogy works. It is difficult to get acs up and
running. You may take it for granted, but I know that struggle has made
some people move on.

I think it is more analogous with not buying a car because the dealer won't
give you the keys until you can prove you can change the oil.

On Mar 3, 2017 6:36 PM, "Chiradeep Vittal" <chirade...@gmail.com> wrote:

> To me, that is like saying that a customer won't buy a car because he can't
> figure out how to roll down his window, so let's build a voice-controlled
> window controller. Can be done, sounds great, but what happens when that
> voice controller thing in the cloud goes down?
>
>
> On Fri, Mar 3, 2017 at 3:09 PM, Will Stevens <williamstev...@gmail.com>
> wrote:
>
> > I agree with Paul. Look at the list of things they have to
> > learn/master/care about. We don't want to add to that list, we want to
> > remove from that list. Make the project/product more accessible. Reduce
> the
> > barrier to entry...
> >
> > On Mar 3, 2017 5:50 PM, "Chiradeep Vittal" <chirade...@gmail.com> wrote:
> >
> > > Seriously?
> > > apt-get install apache2
> > >
> > > To install CloudStack, they need to know
> > > - DB installation, perhaps some SQL
> > > - VLAN configuration on their switches
> > > - Ins-and-outs between port forwarding, static ips,
> > > - NFS
> > > - package management
> > > - VHDs, qcow2. vmdk
> > > - hyperviosrs
> > > - and on and on.
> > >
> > > If they can figure all that out and not figure out a web server, they
> can
> > > always come to the mailing list.
> > >
> > >
> > >
> > >
> > > On Fri, Mar 3, 2017 at 1:45 PM, Paul Angus <paul.an...@shapeblue.com>
> > > wrote:
> > >
> > > > The issue is not with supporting highly experienced cloud operators.
> It
> > > is
> > > > an issue for new users and other 'relatively' inexperienced
> operators.
> > > >
> > > > I have helped enough newbies and cloud operators who have been
> running
> > > > their cloud for a while to know that the barriers to entry are too
> high
> > > as
> > > > they are. And telling anyone that they need to create a web server so
> > > that
> > > > they can add their initial template to get started or in order to
> > create
> > > a
> > > > new zone, just isn't going to fly.
> > > >
> > > > I'm a huge advocate of the 'download.cloudstack.org' endpoint which
> > the
> > > > community can add/remove mirrors to or from for system VMs or
> built-in
> > > > templates.  Can the same system be used for binary repos ? although
> > there
> > > > is an added complication of redist vs no-redist there...
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > paul.an...@shapeblue.com
> > > > www.shapeblue.com
> > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > @shapeblue
> > > >
> > > >
> > > >
> > > >
> > > > -Original Message-
> > > > From: Chiradeep Vittal [mailto:chirade...@gmail.com]
> > > > Sent: 03 March 2017 18:28
> > > > To: dev <dev@cloudstack.apache.org>
> > > > Subject: Re: Modern template hosting
> > > >
> > > > I do feel like this is early optimization. Mirrors rarely fail. I'd
> > > expect
> > > > a single web server hosted on Apache Infra without any monitors to
> fail
> > > > more often than a mirror. We already expect Wido's systemvm
> repository
> > to
> > > > be up all the time. And it has been. Similarly, I don't believe Nux's
> > > > repository has ever been down. And if Accelerite wants to host on S3,
> > > that
> > > > one is pretty solid as well.
> > > >
> > > > This is an infrequent operation in a cloud. After the cloud is
> > installed,
> > > > the download servers are only needed for a new zone. If we trust the
> > user
> > > > to run a cloud, surely he/she can run a web server to serve some
> > built-in
> > > > templates. And if her cloud is successful, she needs to figure out
> how
> > to
> > > > host her templates anyway and not rely on 3rd parties.
> > > >
> > > >
> > > > On Fri, Mar 3, 2017 at 9:31 AM, Will Stevens 

Re: Modern template hosting

[Erik Weber]

It'll most likely only be an issue in the case where a mirror goes
down without any immediate chance of getting back up, right?
Could we check with ASF Infra beforehand if such case is OK to warrant
an urgent Jira ticket to resolve - should it happen?

We can document all the mirrors and provide the howtos for manually
downloading and seeding the image as a backup.

Do we have any traffic statistics for the current solution, or is that
not available at S3?

-- 
Erik

On Sat, Mar 4, 2017 at 12:28 AM, Will Stevens <williamstev...@gmail.com> wrote:
> Yes Erik, I agree with you. The only thing that could add complexity in
> this is the fact that we don't control the domain, the ASF does (from what
> I understand).  Do we expect this to be managed by ASF infra or would there
> be another way? Ideally the domain is community controlled so we can adapt
> in case of change without long delays.
>
> I am 100% in agreement that this is the shortest path and likely to be good
> enough.
>
> On Mar 3, 2017 6:19 PM, "Erik Weber" <terbol...@gmail.com> wrote:
>
> IMHO; implement it as simple as possible, use DNS RR and assume/expect
> mirror hosts to use the same path.
>
>
> Yes I know; DNS RR isn't bulletproof, but as already mentioned in this
> thread - mirrors rarely go down and they aren't used all that often
> (primarily seeding).
> We can keep a low TTL on the record so that we're able to remove a
> mirror that goes down for a significant period of time (more than
> hours).
>
> And yes I know, some operators might find it a burden having to set up
> a new vhost with another document root, but tbh if they don't care
> enough to do it we'll manage without that mirror.
>
> --
> Erik
>
> On Sat, Mar 4, 2017 at 12:09 AM, Will Stevens <williamstev...@gmail.com>
> wrote:
>> I agree with Paul. Look at the list of things they have to
>> learn/master/care about. We don't want to add to that list, we want to
>> remove from that list. Make the project/product more accessible. Reduce
> the
>> barrier to entry...
>>
>> On Mar 3, 2017 5:50 PM, "Chiradeep Vittal" <chirade...@gmail.com> wrote:
>>
>>> Seriously?
>>> apt-get install apache2
>>>
>>> To install CloudStack, they need to know
>>> - DB installation, perhaps some SQL
>>> - VLAN configuration on their switches
>>> - Ins-and-outs between port forwarding, static ips,
>>> - NFS
>>> - package management
>>> - VHDs, qcow2. vmdk
>>> - hyperviosrs
>>> - and on and on.
>>>
>>> If they can figure all that out and not figure out a web server, they can
>>> always come to the mailing list.
>>>
>>>
>>>
>>>
>>> On Fri, Mar 3, 2017 at 1:45 PM, Paul Angus <paul.an...@shapeblue.com>
>>> wrote:
>>>
>>> > The issue is not with supporting highly experienced cloud operators. It
>>> is
>>> > an issue for new users and other 'relatively' inexperienced operators.
>>> >
>>> > I have helped enough newbies and cloud operators who have been running
>>> > their cloud for a while to know that the barriers to entry are too high
>>> as
>>> > they are. And telling anyone that they need to create a web server so
>>> that
>>> > they can add their initial template to get started or in order to
> create
>>> a
>>> > new zone, just isn't going to fly.
>>> >
>>> > I'm a huge advocate of the 'download.cloudstack.org' endpoint which the
>>> > community can add/remove mirrors to or from for system VMs or built-in
>>> > templates.  Can the same system be used for binary repos ? although
> there
>>> > is an added complication of redist vs no-redist there...
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > paul.an...@shapeblue.com
>>> > www.shapeblue.com
>>> > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>>> > @shapeblue
>>> >
>>> >
>>> >
>>> >
>>> > -Original Message-
>>> > From: Chiradeep Vittal [mailto:chirade...@gmail.com]
>>> > Sent: 03 March 2017 18:28
>>> > To: dev <dev@cloudstack.apache.org>
>>> > Subject: Re: Modern template hosting
>>> >
>>> > I do feel like this is early optimization. Mirrors rarely fail. I'd
>>> expect
>>> > a single web server hosted on Apache Infra without any monitors to fail
>>> > more often than a mirror. We alrea

Re: Modern template hosting

[Chiradeep Vittal]

To me, that is like saying that a customer won't buy a car because he can't
figure out how to roll down his window, so let's build a voice-controlled
window controller. Can be done, sounds great, but what happens when that
voice controller thing in the cloud goes down?


On Fri, Mar 3, 2017 at 3:09 PM, Will Stevens <williamstev...@gmail.com>
wrote:

> I agree with Paul. Look at the list of things they have to
> learn/master/care about. We don't want to add to that list, we want to
> remove from that list. Make the project/product more accessible. Reduce the
> barrier to entry...
>
> On Mar 3, 2017 5:50 PM, "Chiradeep Vittal" <chirade...@gmail.com> wrote:
>
> > Seriously?
> > apt-get install apache2
> >
> > To install CloudStack, they need to know
> > - DB installation, perhaps some SQL
> > - VLAN configuration on their switches
> > - Ins-and-outs between port forwarding, static ips,
> > - NFS
> > - package management
> > - VHDs, qcow2. vmdk
> > - hyperviosrs
> > - and on and on.
> >
> > If they can figure all that out and not figure out a web server, they can
> > always come to the mailing list.
> >
> >
> >
> >
> > On Fri, Mar 3, 2017 at 1:45 PM, Paul Angus <paul.an...@shapeblue.com>
> > wrote:
> >
> > > The issue is not with supporting highly experienced cloud operators. It
> > is
> > > an issue for new users and other 'relatively' inexperienced operators.
> > >
> > > I have helped enough newbies and cloud operators who have been running
> > > their cloud for a while to know that the barriers to entry are too high
> > as
> > > they are. And telling anyone that they need to create a web server so
> > that
> > > they can add their initial template to get started or in order to
> create
> > a
> > > new zone, just isn't going to fly.
> > >
> > > I'm a huge advocate of the 'download.cloudstack.org' endpoint which
> the
> > > community can add/remove mirrors to or from for system VMs or built-in
> > > templates.  Can the same system be used for binary repos ? although
> there
> > > is an added complication of redist vs no-redist there...
> > >
> > >
> > >
> > >
> > >
> > >
> > > paul.an...@shapeblue.com
> > > www.shapeblue.com
> > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > @shapeblue
> > >
> > >
> > >
> > >
> > > -Original Message-
> > > From: Chiradeep Vittal [mailto:chirade...@gmail.com]
> > > Sent: 03 March 2017 18:28
> > > To: dev <dev@cloudstack.apache.org>
> > > Subject: Re: Modern template hosting
> > >
> > > I do feel like this is early optimization. Mirrors rarely fail. I'd
> > expect
> > > a single web server hosted on Apache Infra without any monitors to fail
> > > more often than a mirror. We already expect Wido's systemvm repository
> to
> > > be up all the time. And it has been. Similarly, I don't believe Nux's
> > > repository has ever been down. And if Accelerite wants to host on S3,
> > that
> > > one is pretty solid as well.
> > >
> > > This is an infrequent operation in a cloud. After the cloud is
> installed,
> > > the download servers are only needed for a new zone. If we trust the
> user
> > > to run a cloud, surely he/she can run a web server to serve some
> built-in
> > > templates. And if her cloud is successful, she needs to figure out how
> to
> > > host her templates anyway and not rely on 3rd parties.
> > >
> > >
> > > On Fri, Mar 3, 2017 at 9:31 AM, Will Stevens <wstev...@cloudops.com>
> > > wrote:
> > >
> > > > 1) If the legacy implementations do not support redirects, that does
> > > > cause a problem.  A potential solution in that case is to have the
> web
> > > > server actually proxy the download, but that is not ideal and I would
> > > > like to avoid it if possible.  Thanks for bringing that up Chiradeep.
> > > >
> > > > 2) I think we need to have a single URL which people can target.
> Once
> > > > they make the switch to the new URL, we want the implementation to be
> > > > able to handle mirror failures without affecting the end client.  We
> > > > want to avoid the situation where an ACS user will ever have to
> change
> > > > this URL more than once.  Mirror failures SHOULD NOT affect the ACS
> > > > users assuming

Re: Modern template hosting

[Will Stevens]

Yes Erik, I agree with you. The only thing that could add complexity in
this is the fact that we don't control the domain, the ASF does (from what
I understand).  Do we expect this to be managed by ASF infra or would there
be another way? Ideally the domain is community controlled so we can adapt
in case of change without long delays.

I am 100% in agreement that this is the shortest path and likely to be good
enough.

On Mar 3, 2017 6:19 PM, "Erik Weber" <terbol...@gmail.com> wrote:

IMHO; implement it as simple as possible, use DNS RR and assume/expect
mirror hosts to use the same path.


Yes I know; DNS RR isn't bulletproof, but as already mentioned in this
thread - mirrors rarely go down and they aren't used all that often
(primarily seeding).
We can keep a low TTL on the record so that we're able to remove a
mirror that goes down for a significant period of time (more than
hours).

And yes I know, some operators might find it a burden having to set up
a new vhost with another document root, but tbh if they don't care
enough to do it we'll manage without that mirror.

--
Erik

On Sat, Mar 4, 2017 at 12:09 AM, Will Stevens <williamstev...@gmail.com>
wrote:
> I agree with Paul. Look at the list of things they have to
> learn/master/care about. We don't want to add to that list, we want to
> remove from that list. Make the project/product more accessible. Reduce
the
> barrier to entry...
>
> On Mar 3, 2017 5:50 PM, "Chiradeep Vittal" <chirade...@gmail.com> wrote:
>
>> Seriously?
>> apt-get install apache2
>>
>> To install CloudStack, they need to know
>> - DB installation, perhaps some SQL
>> - VLAN configuration on their switches
>> - Ins-and-outs between port forwarding, static ips,
>> - NFS
>> - package management
>> - VHDs, qcow2. vmdk
>> - hyperviosrs
>> - and on and on.
>>
>> If they can figure all that out and not figure out a web server, they can
>> always come to the mailing list.
>>
>>
>>
>>
>> On Fri, Mar 3, 2017 at 1:45 PM, Paul Angus <paul.an...@shapeblue.com>
>> wrote:
>>
>> > The issue is not with supporting highly experienced cloud operators. It
>> is
>> > an issue for new users and other 'relatively' inexperienced operators.
>> >
>> > I have helped enough newbies and cloud operators who have been running
>> > their cloud for a while to know that the barriers to entry are too high
>> as
>> > they are. And telling anyone that they need to create a web server so
>> that
>> > they can add their initial template to get started or in order to
create
>> a
>> > new zone, just isn't going to fly.
>> >
>> > I'm a huge advocate of the 'download.cloudstack.org' endpoint which the
>> > community can add/remove mirrors to or from for system VMs or built-in
>> > templates.  Can the same system be used for binary repos ? although
there
>> > is an added complication of redist vs no-redist there...
>> >
>> >
>> >
>> >
>> >
>> >
>> > paul.an...@shapeblue.com
>> > www.shapeblue.com
>> > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>> > @shapeblue
>> >
>> >
>> >
>> >
>> > -Original Message-
>> > From: Chiradeep Vittal [mailto:chirade...@gmail.com]
>> > Sent: 03 March 2017 18:28
>> > To: dev <dev@cloudstack.apache.org>
>> > Subject: Re: Modern template hosting
>> >
>> > I do feel like this is early optimization. Mirrors rarely fail. I'd
>> expect
>> > a single web server hosted on Apache Infra without any monitors to fail
>> > more often than a mirror. We already expect Wido's systemvm repository
to
>> > be up all the time. And it has been. Similarly, I don't believe Nux's
>> > repository has ever been down. And if Accelerite wants to host on S3,
>> that
>> > one is pretty solid as well.
>> >
>> > This is an infrequent operation in a cloud. After the cloud is
installed,
>> > the download servers are only needed for a new zone. If we trust the
user
>> > to run a cloud, surely he/she can run a web server to serve some
built-in
>> > templates. And if her cloud is successful, she needs to figure out how
to
>> > host her templates anyway and not rely on 3rd parties.
>> >
>> >
>> > On Fri, Mar 3, 2017 at 9:31 AM, Will Stevens <wstev...@cloudops.com>
>> > wrote:
>> >
>> > > 1) If the legacy implementations do not support redirects, that does
>> > > cause a problem.  A potential solution in

Re: Modern template hosting

[Erik Weber]

IMHO; implement it as simple as possible, use DNS RR and assume/expect
mirror hosts to use the same path.


Yes I know; DNS RR isn't bulletproof, but as already mentioned in this
thread - mirrors rarely go down and they aren't used all that often
(primarily seeding).
We can keep a low TTL on the record so that we're able to remove a
mirror that goes down for a significant period of time (more than
hours).

And yes I know, some operators might find it a burden having to set up
a new vhost with another document root, but tbh if they don't care
enough to do it we'll manage without that mirror.

-- 
Erik

On Sat, Mar 4, 2017 at 12:09 AM, Will Stevens <williamstev...@gmail.com> wrote:
> I agree with Paul. Look at the list of things they have to
> learn/master/care about. We don't want to add to that list, we want to
> remove from that list. Make the project/product more accessible. Reduce the
> barrier to entry...
>
> On Mar 3, 2017 5:50 PM, "Chiradeep Vittal" <chirade...@gmail.com> wrote:
>
>> Seriously?
>> apt-get install apache2
>>
>> To install CloudStack, they need to know
>> - DB installation, perhaps some SQL
>> - VLAN configuration on their switches
>> - Ins-and-outs between port forwarding, static ips,
>> - NFS
>> - package management
>> - VHDs, qcow2. vmdk
>> - hyperviosrs
>> - and on and on.
>>
>> If they can figure all that out and not figure out a web server, they can
>> always come to the mailing list.
>>
>>
>>
>>
>> On Fri, Mar 3, 2017 at 1:45 PM, Paul Angus <paul.an...@shapeblue.com>
>> wrote:
>>
>> > The issue is not with supporting highly experienced cloud operators. It
>> is
>> > an issue for new users and other 'relatively' inexperienced operators.
>> >
>> > I have helped enough newbies and cloud operators who have been running
>> > their cloud for a while to know that the barriers to entry are too high
>> as
>> > they are. And telling anyone that they need to create a web server so
>> that
>> > they can add their initial template to get started or in order to create
>> a
>> > new zone, just isn't going to fly.
>> >
>> > I'm a huge advocate of the 'download.cloudstack.org' endpoint which the
>> > community can add/remove mirrors to or from for system VMs or built-in
>> > templates.  Can the same system be used for binary repos ? although there
>> > is an added complication of redist vs no-redist there...
>> >
>> >
>> >
>> >
>> >
>> >
>> > paul.an...@shapeblue.com
>> > www.shapeblue.com
>> > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>> > @shapeblue
>> >
>> >
>> >
>> >
>> > -Original Message-
>> > From: Chiradeep Vittal [mailto:chirade...@gmail.com]
>> > Sent: 03 March 2017 18:28
>> > To: dev <dev@cloudstack.apache.org>
>> > Subject: Re: Modern template hosting
>> >
>> > I do feel like this is early optimization. Mirrors rarely fail. I'd
>> expect
>> > a single web server hosted on Apache Infra without any monitors to fail
>> > more often than a mirror. We already expect Wido's systemvm repository to
>> > be up all the time. And it has been. Similarly, I don't believe Nux's
>> > repository has ever been down. And if Accelerite wants to host on S3,
>> that
>> > one is pretty solid as well.
>> >
>> > This is an infrequent operation in a cloud. After the cloud is installed,
>> > the download servers are only needed for a new zone. If we trust the user
>> > to run a cloud, surely he/she can run a web server to serve some built-in
>> > templates. And if her cloud is successful, she needs to figure out how to
>> > host her templates anyway and not rely on 3rd parties.
>> >
>> >
>> > On Fri, Mar 3, 2017 at 9:31 AM, Will Stevens <wstev...@cloudops.com>
>> > wrote:
>> >
>> > > 1) If the legacy implementations do not support redirects, that does
>> > > cause a problem.  A potential solution in that case is to have the web
>> > > server actually proxy the download, but that is not ideal and I would
>> > > like to avoid it if possible.  Thanks for bringing that up Chiradeep.
>> > >
>> > > 2) I think we need to have a single URL which people can target.  Once
>> > > they make the switch to the new URL, we want the implementation to be
>> > > able to handle mirror failures without affecting the end client.  We
>> &g

Re: Modern template hosting

[Will Stevens]

I agree with Paul. Look at the list of things they have to
learn/master/care about. We don't want to add to that list, we want to
remove from that list. Make the project/product more accessible. Reduce the
barrier to entry...

On Mar 3, 2017 5:50 PM, "Chiradeep Vittal" <chirade...@gmail.com> wrote:

> Seriously?
> apt-get install apache2
>
> To install CloudStack, they need to know
> - DB installation, perhaps some SQL
> - VLAN configuration on their switches
> - Ins-and-outs between port forwarding, static ips,
> - NFS
> - package management
> - VHDs, qcow2. vmdk
> - hyperviosrs
> - and on and on.
>
> If they can figure all that out and not figure out a web server, they can
> always come to the mailing list.
>
>
>
>
> On Fri, Mar 3, 2017 at 1:45 PM, Paul Angus <paul.an...@shapeblue.com>
> wrote:
>
> > The issue is not with supporting highly experienced cloud operators. It
> is
> > an issue for new users and other 'relatively' inexperienced operators.
> >
> > I have helped enough newbies and cloud operators who have been running
> > their cloud for a while to know that the barriers to entry are too high
> as
> > they are. And telling anyone that they need to create a web server so
> that
> > they can add their initial template to get started or in order to create
> a
> > new zone, just isn't going to fly.
> >
> > I'm a huge advocate of the 'download.cloudstack.org' endpoint which the
> > community can add/remove mirrors to or from for system VMs or built-in
> > templates.  Can the same system be used for binary repos ? although there
> > is an added complication of redist vs no-redist there...
> >
> >
> >
> >
> >
> >
> > paul.an...@shapeblue.com
> > www.shapeblue.com
> > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > @shapeblue
> >
> >
> >
> >
> > -Original Message-
> > From: Chiradeep Vittal [mailto:chirade...@gmail.com]
> > Sent: 03 March 2017 18:28
> > To: dev <dev@cloudstack.apache.org>
> > Subject: Re: Modern template hosting
> >
> > I do feel like this is early optimization. Mirrors rarely fail. I'd
> expect
> > a single web server hosted on Apache Infra without any monitors to fail
> > more often than a mirror. We already expect Wido's systemvm repository to
> > be up all the time. And it has been. Similarly, I don't believe Nux's
> > repository has ever been down. And if Accelerite wants to host on S3,
> that
> > one is pretty solid as well.
> >
> > This is an infrequent operation in a cloud. After the cloud is installed,
> > the download servers are only needed for a new zone. If we trust the user
> > to run a cloud, surely he/she can run a web server to serve some built-in
> > templates. And if her cloud is successful, she needs to figure out how to
> > host her templates anyway and not rely on 3rd parties.
> >
> >
> > On Fri, Mar 3, 2017 at 9:31 AM, Will Stevens <wstev...@cloudops.com>
> > wrote:
> >
> > > 1) If the legacy implementations do not support redirects, that does
> > > cause a problem.  A potential solution in that case is to have the web
> > > server actually proxy the download, but that is not ideal and I would
> > > like to avoid it if possible.  Thanks for bringing that up Chiradeep.
> > >
> > > 2) I think we need to have a single URL which people can target.  Once
> > > they make the switch to the new URL, we want the implementation to be
> > > able to handle mirror failures without affecting the end client.  We
> > > want to avoid the situation where an ACS user will ever have to change
> > > this URL more than once.  Mirror failures SHOULD NOT affect the ACS
> > > users assuming there is still at least one mirror who can serve the
> > requested resource.
> > >
> > > These are obviously my personal opinions and others will probably have
> > > differing opinions.
> > >
> > > *Will STEVENS*
> > > Lead Developer
> > >
> > > <https://goo.gl/NYZ8KK>
> > >
> > > On Fri, Mar 3, 2017 at 12:23 PM, Chiradeep Vittal
> > > <chirade...@gmail.com>
> > > wrote:
> > >
> > > > 1. If you are targeting legacy installations, they are not able to
> > > > follow redirects. The line of code that added this capability was
> > > > added on
> > > 11/16.
> > > > 2. If you trust the users to edit the database to change the URL,
> > > > you can trust them to change it to

Re: Modern template hosting

[Chiradeep Vittal]

Seriously?
apt-get install apache2

To install CloudStack, they need to know
- DB installation, perhaps some SQL
- VLAN configuration on their switches
- Ins-and-outs between port forwarding, static ips,
- NFS
- package management
- VHDs, qcow2. vmdk
- hyperviosrs
- and on and on.

If they can figure all that out and not figure out a web server, they can
always come to the mailing list.




On Fri, Mar 3, 2017 at 1:45 PM, Paul Angus <paul.an...@shapeblue.com> wrote:

> The issue is not with supporting highly experienced cloud operators. It is
> an issue for new users and other 'relatively' inexperienced operators.
>
> I have helped enough newbies and cloud operators who have been running
> their cloud for a while to know that the barriers to entry are too high as
> they are. And telling anyone that they need to create a web server so that
> they can add their initial template to get started or in order to create a
> new zone, just isn't going to fly.
>
> I'm a huge advocate of the 'download.cloudstack.org' endpoint which the
> community can add/remove mirrors to or from for system VMs or built-in
> templates.  Can the same system be used for binary repos ? although there
> is an added complication of redist vs no-redist there...
>
>
>
>
>
>
> paul.an...@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>
>
>
>
> -Original Message-
> From: Chiradeep Vittal [mailto:chirade...@gmail.com]
> Sent: 03 March 2017 18:28
> To: dev <dev@cloudstack.apache.org>
> Subject: Re: Modern template hosting
>
> I do feel like this is early optimization. Mirrors rarely fail. I'd expect
> a single web server hosted on Apache Infra without any monitors to fail
> more often than a mirror. We already expect Wido's systemvm repository to
> be up all the time. And it has been. Similarly, I don't believe Nux's
> repository has ever been down. And if Accelerite wants to host on S3, that
> one is pretty solid as well.
>
> This is an infrequent operation in a cloud. After the cloud is installed,
> the download servers are only needed for a new zone. If we trust the user
> to run a cloud, surely he/she can run a web server to serve some built-in
> templates. And if her cloud is successful, she needs to figure out how to
> host her templates anyway and not rely on 3rd parties.
>
>
> On Fri, Mar 3, 2017 at 9:31 AM, Will Stevens <wstev...@cloudops.com>
> wrote:
>
> > 1) If the legacy implementations do not support redirects, that does
> > cause a problem.  A potential solution in that case is to have the web
> > server actually proxy the download, but that is not ideal and I would
> > like to avoid it if possible.  Thanks for bringing that up Chiradeep.
> >
> > 2) I think we need to have a single URL which people can target.  Once
> > they make the switch to the new URL, we want the implementation to be
> > able to handle mirror failures without affecting the end client.  We
> > want to avoid the situation where an ACS user will ever have to change
> > this URL more than once.  Mirror failures SHOULD NOT affect the ACS
> > users assuming there is still at least one mirror who can serve the
> requested resource.
> >
> > These are obviously my personal opinions and others will probably have
> > differing opinions.
> >
> > *Will STEVENS*
> > Lead Developer
> >
> > <https://goo.gl/NYZ8KK>
> >
> > On Fri, Mar 3, 2017 at 12:23 PM, Chiradeep Vittal
> > <chirade...@gmail.com>
> > wrote:
> >
> > > 1. If you are targeting legacy installations, they are not able to
> > > follow redirects. The line of code that added this capability was
> > > added on
> > 11/16.
> > > 2. If you trust the users to edit the database to change the URL,
> > > you can trust them to change it to anything. Just document a known
> > > list of good template locations.
> > >
> > > On Fri, Mar 3, 2017 at 8:13 AM, Will Stevens <wstev...@cloudops.com>
> > > wrote:
> > >
> > > > So the main issue I see with this is the following.
> > > >
> > > > This implementation is designed to target legacy installations
> > > > which
> > will
> > > > be affected when download.cloud.com disappears.  These people will
> > need
> > > to
> > > > make a DB change to replace the 'download.cloud.com' with some
> > > > other
> > url
> > > > (maybe 'download.cloudstack.org').  Once the DB has been updated,
> > > > we
> > can
> > > > not expect anything else of the client, th

RE: Modern template hosting

[Paul Angus]

The issue is not with supporting highly experienced cloud operators. It is an 
issue for new users and other 'relatively' inexperienced operators.

I have helped enough newbies and cloud operators who have been running their 
cloud for a while to know that the barriers to entry are too high as they are. 
And telling anyone that they need to create a web server so that they can add 
their initial template to get started or in order to create a new zone, just 
isn't going to fly.

I'm a huge advocate of the 'download.cloudstack.org' endpoint which the 
community can add/remove mirrors to or from for system VMs or built-in 
templates.  Can the same system be used for binary repos ? although there is an 
added complication of redist vs no-redist there...




 

paul.an...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 


-Original Message-
From: Chiradeep Vittal [mailto:chirade...@gmail.com] 
Sent: 03 March 2017 18:28
To: dev <dev@cloudstack.apache.org>
Subject: Re: Modern template hosting

I do feel like this is early optimization. Mirrors rarely fail. I'd expect a 
single web server hosted on Apache Infra without any monitors to fail more 
often than a mirror. We already expect Wido's systemvm repository to be up all 
the time. And it has been. Similarly, I don't believe Nux's repository has ever 
been down. And if Accelerite wants to host on S3, that one is pretty solid as 
well.

This is an infrequent operation in a cloud. After the cloud is installed, the 
download servers are only needed for a new zone. If we trust the user to run a 
cloud, surely he/she can run a web server to serve some built-in templates. And 
if her cloud is successful, she needs to figure out how to host her templates 
anyway and not rely on 3rd parties.


On Fri, Mar 3, 2017 at 9:31 AM, Will Stevens <wstev...@cloudops.com> wrote:

> 1) If the legacy implementations do not support redirects, that does 
> cause a problem.  A potential solution in that case is to have the web 
> server actually proxy the download, but that is not ideal and I would 
> like to avoid it if possible.  Thanks for bringing that up Chiradeep.
>
> 2) I think we need to have a single URL which people can target.  Once 
> they make the switch to the new URL, we want the implementation to be 
> able to handle mirror failures without affecting the end client.  We 
> want to avoid the situation where an ACS user will ever have to change 
> this URL more than once.  Mirror failures SHOULD NOT affect the ACS 
> users assuming there is still at least one mirror who can serve the requested 
> resource.
>
> These are obviously my personal opinions and others will probably have 
> differing opinions.
>
> *Will STEVENS*
> Lead Developer
>
> <https://goo.gl/NYZ8KK>
>
> On Fri, Mar 3, 2017 at 12:23 PM, Chiradeep Vittal 
> <chirade...@gmail.com>
> wrote:
>
> > 1. If you are targeting legacy installations, they are not able to 
> > follow redirects. The line of code that added this capability was 
> > added on
> 11/16.
> > 2. If you trust the users to edit the database to change the URL, 
> > you can trust them to change it to anything. Just document a known 
> > list of good template locations.
> >
> > On Fri, Mar 3, 2017 at 8:13 AM, Will Stevens <wstev...@cloudops.com>
> > wrote:
> >
> > > So the main issue I see with this is the following.
> > >
> > > This implementation is designed to target legacy installations 
> > > which
> will
> > > be affected when download.cloud.com disappears.  These people will
> need
> > to
> > > make a DB change to replace the 'download.cloud.com' with some 
> > > other
> url
> > > (maybe 'download.cloudstack.org').  Once the DB has been updated, 
> > > we
> can
> > > not expect anything else of the client, they need to be able to 
> > > just continue operation as they were without any need to rebuild 
> > > or upgrade their ACS.
> > >
> > > If we try to force the decision to the client, then we break the 
> > > legacy implementations.  Since the SSVM is likely going to be the 
> > > client in
> some
> > > cases, and since it does not already have the logic to handle the 
> > > 300 approach correctly, I feel like it is not a viable solution 
> > > for the
> > legacy
> > > deployments.
> > >
> > > In addition to that.  I don't think the client is in any better
> position
> > to
> > > make the mirror decision than I am.  I am likely able to make a 
> > > more 'educated' decision than the client would be able to because 
> > > I can do different tests on the endpoi

Re: Modern template hosting

[Will Stevens]

; > > > >
> > > > > > > > > We will have a web server which we can point something
> > like
> > > > > > > > > downloads.cloudstack.org (or whatever url) at.  Then
> we
> > > will
> > > > > > have
> > > > > > > > an ASF
> > > > > > > > > repo which tracks a mirror list and exposes a static
> site
> > > to
> > > > be
> > > > > > > able
> > > > > > > > to
> > > > > > > > > query the mirror list.
> > > > > > > > >
> > > > > > > > > The web server will receive a request and will do a 302
> > > > > redirect
> > > > > > to
> > > > > > > > the
> > > > > > > > > appropriate resource on one of the mirrors.
> > > > > > > > >
> > > > > > > > > This gets us started.  Legacy environments will have to
> > do
> > > a
> > > > DB
> > > > > > > > change to
> > > > > > > > > change from 'download.cloud.com' to '
> > > > downloads.cloudstack.org'
> > > > > > (or
> > > > > > > > > whatever), but otherwise it should be pretty simple.
> > > > > > > > >
> > > > > > > > > We can improve how we deliver templates going forward,
> > but
> > > > this
> > > > > > > > obviously
> > > > > > > > > requires some discussion still.  I will try to buy us
> > some
> > > > time
> > > > > > > with
> > > > > > > > an
> > > > > > > > > implementation which solves our problems today...
> > > > > > > > >
> > > > > > > > > *Will STEVENS*
> > > > > > > > > Lead Developer
> > > > > > > > >
> > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > >
> > > > > > > > > On Wed, Mar 1, 2017 at 7:52 AM, Abhinandan Prateek <
> > > > > > > > > abhinandan.prat...@shapeblue.com> wrote:
> > > > > > > > >
> > > > > > > > >>
> > > > > > > > >> Initial seeding is a manual step and that is not going
> > to
> > > > > change
> > > > > > > in
> > > > > > > > near
> > > > > > > > >> future. A handy list of official places from where
> these
> > > > > > templates
> > > > > > > > can be
> > > > > > > > >> downloaded won’t harm this part of the installation.
> > Just
> > > > > allow
> > > > > > > > anyone
> > > > > > > > >> credible who follows democratic process to publish
> their
> > > > > > templates
> > > > > > > > with
> > > > > > > > >> apache’s blessings.
> > > > > > > > >>
> > > > > > > > >>
> > > > > > > > >> Coming to urls hard coded in db, the popular
> suggestion
> > is
> > > > to
> > > > > > > point
> > > > > > > > them
> > > > > > > > >> to mirrors. This looks good but this is going to take
> > some
> > > > > > effort
> > > > > > > > that has
> > > > > > > > >> not materialised yet. Once this change is made it will
> > be
> > > > > going
> > > > > > in
> > > > > > > > some
> > > > > > > > >> future release like 4.11; the problem still remains
> for
> > > the
> > > > > > > > releases that
> > > > > > > > >> are already out there. In case access to
> > > download.cloud.com
> > > > > is
> > > > > > > > dropped
> > > > > > > > >> at some point we will have to document the procedure
> > such
> > > > that
> > > >

Re: Modern template hosting

[Rafael Weingärtner]

lly like very short variables names!
> > > > > >
> > > > > > On Fri, Mar 3, 2017 at 1:24 AM, Daan Hoogland <
> > > > > daan.hoogl...@shapeblue.com
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Nice little thing Will,
> > > > > > >
> > > > > > > One question: if I read the code correctly it ‘go’es and tries
> > all
> > > > > > mirrors
> > > > > > > at once and whichever responses first is redirected to the
> > client.
> > > > This
> > > > > > > might well be the same every time. This might be the one
> closest
> > to
> > > > the
> > > > > > > mirror302 and not to the client. Is that correct or did I miss
> a
> > > > > weighing
> > > > > > > algorithm hidden in there?
> > > > > > >
> > > > > > > Good coding,
> > > > > > >
> > > > > > > On 03/03/17 00:23, "Will Stevens" <sw...@apache.org> wrote:
> > > > > > >
> > > > > > > Hey All,
> > > > > > > Please review this repo and tell me what you think:
> > > > > > > https://github.com/swill/mirror302
> > > > > > >
> > > > > > > Let me know if you have questions or you would like me to
> > make
> > > > > > changes
> > > > > > > before I bring this topic up with ASF Infra and open a
> ticket
> > > to
> > > > > get
> > > > > > > this
> > > > > > > implemented.
> > > > > > >
> > > > > > > Cheers,
> > > > > > >
> > > > > > > *Will Stevens*
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Wed, Mar 1, 2017 at 11:37 AM, Will Stevens <
> > > > > wstev...@cloudops.com
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I am building a short term solution right now.  I
> hopefully
> > > > will
> > > > > > > have the
> > > > > > > > building blocks in place by the weekend so we can start
> > > working
> > > > > > with
> > > > > > > Infra
> > > > > > > > to get it in place.
> > > > > > > >
> > > > > > > > We will have a web server which we can point something
> like
> > > > > > > > downloads.cloudstack.org (or whatever url) at.  Then we
> > will
> > > > > have
> > > > > > > an ASF
> > > > > > > > repo which tracks a mirror list and exposes a static site
> > to
> > > be
> > > > > > able
> > > > > > > to
> > > > > > > > query the mirror list.
> > > > > > > >
> > > > > > > > The web server will receive a request and will do a 302
> > > > redirect
> > > > > to
> > > > > > > the
> > > > > > > > appropriate resource on one of the mirrors.
> > > > > > > >
> > > > > > > > This gets us started.  Legacy environments will have to
> do
> > a
> > > DB
> > > > > > > change to
> > > > > > > > change from 'download.cloud.com' to '
> > > downloads.cloudstack.org'
> > > > > (or
> > > > > > > > whatever), but otherwise it should be pretty simple.
> > > > > > > >
> > > > > > > > We can improve how we deliver templates going forward,
> but
> > > this
> > > > > > > obviously
> > > > > > > > requires some discussion still.  I will try to buy us
> some
> > > time
> > > > > > with
> > > > > > > an
> > > > > > > > implementation which solves our problems today...
> > > > > > > >
> > > > > > > > *Will STEVENS*
> > > > > > > > Lead Developer
> > > >

Re: Modern template hosting

[Rafael Weingärtner]

> > > I had the same doubt as Daan.
> >> > > >
> >> > > > Have you considered using HTTP 300 working mode? Then, we could
> let
> >> the
> >> > > > client decide which mirror is the best(closest?). Section “10.3.1”
> >> of
> >> > [1]
> >> > > > talks about it; this would put the pressure on deciding upon a
> >> mirror
> >> > on
> >> > > > the client side, but I think it would be fair.
> >> > > >
> >> > > > [1] https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html
> >> > > >
> >> > > > PS: You really like very short variables names!
> >> > > >
> >> > > > On Fri, Mar 3, 2017 at 1:24 AM, Daan Hoogland <
> >> > > daan.hoogl...@shapeblue.com
> >> > > > >
> >> > > > wrote:
> >> > > >
> >> > > > > Nice little thing Will,
> >> > > > >
> >> > > > > One question: if I read the code correctly it ‘go’es and tries
> all
> >> > > > mirrors
> >> > > > > at once and whichever responses first is redirected to the
> client.
> >> > This
> >> > > > > might well be the same every time. This might be the one closest
> >> to
> >> > the
> >> > > > > mirror302 and not to the client. Is that correct or did I miss a
> >> > > weighing
> >> > > > > algorithm hidden in there?
> >> > > > >
> >> > > > > Good coding,
> >> > > > >
> >> > > > > On 03/03/17 00:23, "Will Stevens" <sw...@apache.org> wrote:
> >> > > > >
> >> > > > > Hey All,
> >> > > > > Please review this repo and tell me what you think:
> >> > > > > https://github.com/swill/mirror302
> >> > > > >
> >> > > > > Let me know if you have questions or you would like me to
> make
> >> > > > changes
> >> > > > > before I bring this topic up with ASF Infra and open a
> ticket
> >> to
> >> > > get
> >> > > > > this
> >> > > > > implemented.
> >> > > > >
> >> > > > > Cheers,
> >> > > > >
> >> > > > > *Will Stevens*
> >> > > > >
> >> > > > >
> >> > > > >
> >> > > > > On Wed, Mar 1, 2017 at 11:37 AM, Will Stevens <
> >> > > wstev...@cloudops.com
> >> > > > >
> >> > > > > wrote:
> >> > > > >
> >> > > > > > I am building a short term solution right now.  I
> hopefully
> >> > will
> >> > > > > have the
> >> > > > > > building blocks in place by the weekend so we can start
> >> working
> >> > > > with
> >> > > > > Infra
> >> > > > > > to get it in place.
> >> > > > > >
> >> > > > > > We will have a web server which we can point something
> like
> >> > > > > > downloads.cloudstack.org (or whatever url) at.  Then we
> >> will
> >> > > have
> >> > > > > an ASF
> >> > > > > > repo which tracks a mirror list and exposes a static site
> >> to be
> >> > > > able
> >> > > > > to
> >> > > > > > query the mirror list.
> >> > > > > >
> >> > > > > > The web server will receive a request and will do a 302
> >> > redirect
> >> > > to
> >> > > > > the
> >> > > > > > appropriate resource on one of the mirrors.
> >> > > > > >
> >> > > > > > This gets us started.  Legacy environments will have to do
> >> a DB
> >> > > > > change to
> >> > > > > > change from 'download.cloud.com' to '
> >> downloads.cloudstack.org'
> >> > > (or
> >> > > > > > whatever), but otherwise it should be pretty simple.
> >> > > > > >
> >> > > > > > We can i

Re: Modern template hosting

[Chiradeep Vittal]

 and tell me what you think:
> > > > > > https://github.com/swill/mirror302
> > > > > >
> > > > > > Let me know if you have questions or you would like me to
> make
> > > > > changes
> > > > > > before I bring this topic up with ASF Infra and open a ticket
> > to
> > > > get
> > > > > > this
> > > > > > implemented.
> > > > > >
> > > > > > Cheers,
> > > > > >
> > > > > > *Will Stevens*
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Wed, Mar 1, 2017 at 11:37 AM, Will Stevens <
> > > > wstev...@cloudops.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > I am building a short term solution right now.  I hopefully
> > > will
> > > > > > have the
> > > > > > > building blocks in place by the weekend so we can start
> > working
> > > > > with
> > > > > > Infra
> > > > > > > to get it in place.
> > > > > > >
> > > > > > > We will have a web server which we can point something like
> > > > > > > downloads.cloudstack.org (or whatever url) at.  Then we
> will
> > > > have
> > > > > > an ASF
> > > > > > > repo which tracks a mirror list and exposes a static site
> to
> > be
> > > > > able
> > > > > > to
> > > > > > > query the mirror list.
> > > > > > >
> > > > > > > The web server will receive a request and will do a 302
> > > redirect
> > > > to
> > > > > > the
> > > > > > > appropriate resource on one of the mirrors.
> > > > > > >
> > > > > > > This gets us started.  Legacy environments will have to do
> a
> > DB
> > > > > > change to
> > > > > > > change from 'download.cloud.com' to '
> > downloads.cloudstack.org'
> > > > (or
> > > > > > > whatever), but otherwise it should be pretty simple.
> > > > > > >
> > > > > > > We can improve how we deliver templates going forward, but
> > this
> > > > > > obviously
> > > > > > > requires some discussion still.  I will try to buy us some
> > time
> > > > > with
> > > > > > an
> > > > > > > implementation which solves our problems today...
> > > > > > >
> > > > > > > *Will STEVENS*
> > > > > > > Lead Developer
> > > > > > >
> > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > >
> > > > > > > On Wed, Mar 1, 2017 at 7:52 AM, Abhinandan Prateek <
> > > > > > > abhinandan.prat...@shapeblue.com> wrote:
> > > > > > >
> > > > > > >>
> > > > > > >> Initial seeding is a manual step and that is not going to
> > > change
> > > > > in
> > > > > > near
> > > > > > >> future. A handy list of official places from where these
> > > > templates
> > > > > > can be
> > > > > > >> downloaded won’t harm this part of the installation. Just
> > > allow
> > > > > > anyone
> > > > > > >> credible who follows democratic process to publish their
> > > > templates
> > > > > > with
> > > > > > >> apache’s blessings.
> > > > > > >>
> > > > > > >>
> > > > > > >> Coming to urls hard coded in db, the popular suggestion is
> > to
> > > > > point
> > > > > > them
> > > > > > >> to mirrors. This looks good but this is going to take some
> > > > effort
> > > > > > that has
> > > > > > >> not materialised yet. Once this change is made it will be
> > > going
> > > > in
> > > > > > some
> > > > > >   

Re: Modern template hosting

[Will Stevens]

; >
>> > > > > Good coding,
>> > > > >
>> > > > > On 03/03/17 00:23, "Will Stevens" <sw...@apache.org> wrote:
>> > > > >
>> > > > > Hey All,
>> > > > > Please review this repo and tell me what you think:
>> > > > > https://github.com/swill/mirror302
>> > > > >
>> > > > > Let me know if you have questions or you would like me to make
>> > > > changes
>> > > > > before I bring this topic up with ASF Infra and open a ticket
>> to
>> > > get
>> > > > > this
>> > > > > implemented.
>> > > > >
>> > > > > Cheers,
>> > > > >
>> > > > > *Will Stevens*
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Wed, Mar 1, 2017 at 11:37 AM, Will Stevens <
>> > > wstev...@cloudops.com
>> > > > >
>> > > > > wrote:
>> > > > >
>> > > > > > I am building a short term solution right now.  I hopefully
>> > will
>> > > > > have the
>> > > > > > building blocks in place by the weekend so we can start
>> working
>> > > > with
>> > > > > Infra
>> > > > > > to get it in place.
>> > > > > >
>> > > > > > We will have a web server which we can point something like
>> > > > > > downloads.cloudstack.org (or whatever url) at.  Then we
>> will
>> > > have
>> > > > > an ASF
>> > > > > > repo which tracks a mirror list and exposes a static site
>> to be
>> > > > able
>> > > > > to
>> > > > > > query the mirror list.
>> > > > > >
>> > > > > > The web server will receive a request and will do a 302
>> > redirect
>> > > to
>> > > > > the
>> > > > > > appropriate resource on one of the mirrors.
>> > > > > >
>> > > > > > This gets us started.  Legacy environments will have to do
>> a DB
>> > > > > change to
>> > > > > > change from 'download.cloud.com' to '
>> downloads.cloudstack.org'
>> > > (or
>> > > > > > whatever), but otherwise it should be pretty simple.
>> > > > > >
>> > > > > > We can improve how we deliver templates going forward, but
>> this
>> > > > > obviously
>> > > > > > requires some discussion still.  I will try to buy us some
>> time
>> > > > with
>> > > > > an
>> > > > > > implementation which solves our problems today...
>> > > > > >
>> > > > > > *Will STEVENS*
>> > > > > > Lead Developer
>> > > > > >
>> > > > > > <https://goo.gl/NYZ8KK>
>> > > > > >
>> > > > > > On Wed, Mar 1, 2017 at 7:52 AM, Abhinandan Prateek <
>> > > > > > abhinandan.prat...@shapeblue.com> wrote:
>> > > > > >
>> > > > > >>
>> > > > > >> Initial seeding is a manual step and that is not going to
>> > change
>> > > > in
>> > > > > near
>> > > > > >> future. A handy list of official places from where these
>> > > templates
>> > > > > can be
>> > > > > >> downloaded won’t harm this part of the installation. Just
>> > allow
>> > > > > anyone
>> > > > > >> credible who follows democratic process to publish their
>> > > templates
>> > > > > with
>> > > > > >> apache’s blessings.
>> > > > > >>
>> > > > > >>
>> > > > > >> Coming to urls hard coded in db, the popular suggestion is
>> to
>> > > > point
>> > > > > them
>> > > > > >> to mirrors. This looks good but this is going to take some
>> > > effort
>> > > > > that has

Re: Modern template hosting

[Will Stevens]

o
> > > > > the
> > > > > > appropriate resource on one of the mirrors.
> > > > > >
> > > > > > This gets us started.  Legacy environments will have to do a
> DB
> > > > > change to
> > > > > > change from 'download.cloud.com' to '
> downloads.cloudstack.org'
> > > (or
> > > > > > whatever), but otherwise it should be pretty simple.
> > > > > >
> > > > > > We can improve how we deliver templates going forward, but
> this
> > > > > obviously
> > > > > > requires some discussion still.  I will try to buy us some
> time
> > > > with
> > > > > an
> > > > > > implementation which solves our problems today...
> > > > > >
> > > > > > *Will STEVENS*
> > > > > > Lead Developer
> > > > > >
> > > > > > <https://goo.gl/NYZ8KK>
> > > > > >
> > > > > > On Wed, Mar 1, 2017 at 7:52 AM, Abhinandan Prateek <
> > > > > > abhinandan.prat...@shapeblue.com> wrote:
> > > > > >
> > > > > >>
> > > > > >> Initial seeding is a manual step and that is not going to
> > change
> > > > in
> > > > > near
> > > > > >> future. A handy list of official places from where these
> > > templates
> > > > > can be
> > > > >     >> downloaded won’t harm this part of the installation. Just
> > allow
> > > > > anyone
> > > > > >> credible who follows democratic process to publish their
> > > templates
> > > > > with
> > > > > >> apache’s blessings.
> > > > > >>
> > > > > >>
> > > > > >> Coming to urls hard coded in db, the popular suggestion is
> to
> > > > point
> > > > > them
> > > > > >> to mirrors. This looks good but this is going to take some
> > > effort
> > > > > that has
> > > > > >> not materialised yet. Once this change is made it will be
> > going
> > > in
> > > > > some
> > > > > >> future release like 4.11; the problem still remains for the
> > > > > releases that
> > > > > >> are already out there. In case access to download.cloud.com
> > is
> > > > > dropped
> > > > > >> at some point we will have to document the procedure such
> that
> > > > > people know
> > > > > >> how to make these stable releases work. That is where the
> > > > procedure
> > > > > >> documented here https://shankerbalan.net/blog/
> > > > > >> seed-cloudstack-templates-offline/ should be officially
> > adapted
> > > > > with a
> > > > > >> marketplace to pick up templates of choosing. Maybe we can
> > have
> > > a
> > > > > pre-setup
> > > > > >> script to make choices instead of written wiki. Note that
> this
> > > > will
> > > > > be
> > > > > >> anyway needed by the older releases.
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> On 01/03/17, 4:24 PM, "Paul Angus" <
> paul.an...@shapeblue.com>
> > > > > wrote:
> > > > > >>
> > > > > >> >-1
> > > > > >> >
> > > > > >> >We are trying to get MORE people to use CloudStack.  And
> > > raising
> > > > > the
> > > > > >> barrier to entry does nothing to help that.
> > > > > >> >
> > > > > >> >
> > > > > >> >Kind regards,
> > > > > >> >
> > > > > >> >Paul Angus
> > > > > >> >
> > > > > >> >paul.an...@shapeblue.com
> > > > > >> >www.shapeblue.com
> > > > > >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > >

Re: Modern template hosting

[Will Stevens]

icket
> to
> > > get
> > > > > this
> > > > > implemented.
> > > > >
> > > > > Cheers,
> > > > >
> > > > > *Will Stevens*
> > > > >
> > > > >
> > > > >
> > > > > On Wed, Mar 1, 2017 at 11:37 AM, Will Stevens <
> > > wstev...@cloudops.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > I am building a short term solution right now.  I hopefully
> > will
> > > > > have the
> > > > > > building blocks in place by the weekend so we can start
> working
> > > > with
> > > > > Infra
> > > > > > to get it in place.
> > > > > >
> > > > > > We will have a web server which we can point something like
> > > > > > downloads.cloudstack.org (or whatever url) at.  Then we will
> > > have
> > > > > an ASF
> > > > > > repo which tracks a mirror list and exposes a static site to
> be
> > > > able
> > > > > to
> > > > > > query the mirror list.
> > > > > >
> > > > > > The web server will receive a request and will do a 302
> > redirect
> > > to
> > > > > the
> > > > > > appropriate resource on one of the mirrors.
> > > > > >
> > > > > > This gets us started.  Legacy environments will have to do a
> DB
> > > > > change to
> > > > > > change from 'download.cloud.com' to '
> downloads.cloudstack.org'
> > > (or
> > > > > > whatever), but otherwise it should be pretty simple.
> > > > > >
> > > > > > We can improve how we deliver templates going forward, but
> this
> > > > > obviously
> > > > > > requires some discussion still.  I will try to buy us some
> time
> > > > with
> > > > > an
> > > > > > implementation which solves our problems today...
> > > > > >
> > > > > > *Will STEVENS*
> > > > > > Lead Developer
> > > > > >
> > > > > > <https://goo.gl/NYZ8KK>
> > > > > >
> > > > > > On Wed, Mar 1, 2017 at 7:52 AM, Abhinandan Prateek <
> > > > > > abhinandan.prat...@shapeblue.com> wrote:
> > > > > >
> > > > > >>
> > > > > >> Initial seeding is a manual step and that is not going to
> > change
> > > > in
> > > > > near
> > > > > >> future. A handy list of official places from where these
> > > templates
> > > > > can be
> > > > > >> downloaded won’t harm this part of the installation. Just
> > allow
> > > > > anyone
> > > > > >> credible who follows democratic process to publish their
> > > templates
> > > > > with
> > > > > >> apache’s blessings.
> > > > > >>
> > > > > >>
> > > > > >> Coming to urls hard coded in db, the popular suggestion is
> to
> > > > point
> > > > > them
> > > > > >> to mirrors. This looks good but this is going to take some
> > > effort
> > > > > that has
> > > > > >> not materialised yet. Once this change is made it will be
> > going
> > > in
> > > > > some
> > > > > >> future release like 4.11; the problem still remains for the
> > > > > releases that
> > > > > >> are already out there. In case access to download.cloud.com
> > is
> > > > > dropped
> > > > > >> at some point we will have to document the procedure such
> that
> > > > > people know
> > > > > >> how to make these stable releases work. That is where the
> > > > procedure
> > > > > >> documented here https://shankerbalan.net/blog/
> > > > > >> seed-cloudstack-templates-offline/ should be officially
> > adapted
> > > > > with a
> > > > > >> marketplace to pick up templates of

Re: Modern template hosting

[Chiradeep Vittal]

t; > > >> Coming to urls hard coded in db, the popular suggestion is to
> > > point
> > > > them
> > > > >> to mirrors. This looks good but this is going to take some
> > effort
> > > > that has
> > > > >> not materialised yet. Once this change is made it will be
> going
> > in
> > > > some
> > > > >> future release like 4.11; the problem still remains for the
> > > > releases that
> > > > >> are already out there. In case access to download.cloud.com
> is
> > > > dropped
> > > > >> at some point we will have to document the procedure such that
> > > > people know
> > > > >> how to make these stable releases work. That is where the
> > > procedure
> > > > >> documented here https://shankerbalan.net/blog/
> > > > >> seed-cloudstack-templates-offline/ should be officially
> adapted
> > > > with a
> > > > >> marketplace to pick up templates of choosing. Maybe we can
> have
> > a
> > > > pre-setup
> > > > >> script to make choices instead of written wiki. Note that this
> > > will
> > > > be
> > > > >> anyway needed by the older releases.
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> On 01/03/17, 4:24 PM, "Paul Angus" <paul.an...@shapeblue.com>
> > > > wrote:
> > > > >>
> > > > >> >-1
> > > > >> >
> > > > >> >We are trying to get MORE people to use CloudStack.  And
> > raising
> > > > the
> > > > >> barrier to entry does nothing to help that.
> > > > >> >
> > > > >> >
> > > > >> >Kind regards,
> > > > >> >
> > > > >> >Paul Angus
> > > > >> >
> > > > >> >paul.an...@shapeblue.com
> > > > >> >www.shapeblue.com
> > > > >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > >> >@shapeblue
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >-Original Message-
> > > > >> >From: Abhinandan Prateek [mailto:abhinandan.prateek@
> > > shapeblue.com]
> > > > >> >Sent: 01 March 2017 09:39
> > > > >> >To: dev@cloudstack.apache.org
> > > > >> >Subject: Re: Modern template hosting
> > > > >> >
> > > > >> >+1
> > > > >> >
> > > > >> >After seeing this discussion running in circles several
> times,
> > I
> > > > think
> > > > >> we should at least get started with the simplest option.
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >On 27/02/17, 11:54 PM, "Chiradeep Vittal" <
> > chirade...@gmail.com>
> > > > wrote:
> > > > >> >
> > > > >> >>The alternative foolproof way is to simply not provide the
> > > > automatic
> > > > >> >>download.
> > > > >> >>
> > > > >> >>Just document the mirror list URL where the templates can be
> > > > found. I
> > > > >> >>don't know why people reject this option.
> > > > >> >>
> > > > >> >>https://shankerbalan.net/blog/seed-cloudstack-
> > > templates-offline/
> > > > >> >>
> > > > >> >>On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens
> > > > >> >><williamstev...@gmail.com>
> > > > >> >>wrote:
> > > > >> >>
> > > > >> >>> OK. Thanks for the heads up.
> > > > >> >>>
> > > > >> >>> On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <
> > > > chirade...@gmail.com>
> > > >

Re: Modern template hosting

[Rafael Weingärtner]

 >
> > > > >>
> > > > >> Initial seeding is a manual step and that is not going to
> change
> > > in
> > > > near
> > > > >> future. A handy list of official places from where these
> > templates
> > > > can be
> > > > >> downloaded won’t harm this part of the installation. Just
> allow
> > > > anyone
> > > > >> credible who follows democratic process to publish their
> > templates
> > > > with
> > > > >> apache’s blessings.
> > > > >>
> > > > >>
> > > > >> Coming to urls hard coded in db, the popular suggestion is to
> > > point
> > > > them
> > > > >> to mirrors. This looks good but this is going to take some
> > effort
> > > > that has
> > > > >> not materialised yet. Once this change is made it will be
> going
> > in
> > > > some
> > > > >> future release like 4.11; the problem still remains for the
> > > > releases that
> > > > >> are already out there. In case access to download.cloud.com
> is
> > > > dropped
> > > > >> at some point we will have to document the procedure such that
> > > > people know
> > > > >> how to make these stable releases work. That is where the
> > > procedure
> > > > >> documented here https://shankerbalan.net/blog/
> > > > >> seed-cloudstack-templates-offline/ should be officially
> adapted
> > > > with a
> > > > >> marketplace to pick up templates of choosing. Maybe we can
> have
> > a
> > > > pre-setup
> > > > >> script to make choices instead of written wiki. Note that this
> > > will
> > > > be
> > > > >> anyway needed by the older releases.
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> On 01/03/17, 4:24 PM, "Paul Angus" <paul.an...@shapeblue.com>
> > > > wrote:
> > > > >>
> > > > >> >-1
> > > > >> >
> > > > >> >We are trying to get MORE people to use CloudStack.  And
> > raising
> > > > the
> > > > >> barrier to entry does nothing to help that.
> > > > >> >
> > > > >> >
> > > > >> >Kind regards,
> > > > >> >
> > > > >> >Paul Angus
> > > > >> >
> > > > >> >paul.an...@shapeblue.com
> > > > >> >www.shapeblue.com
> > > > >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > >> >@shapeblue
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >-Original Message-
> > > > >> >From: Abhinandan Prateek [mailto:abhinandan.prateek@
> > > shapeblue.com]
> > > > >> >Sent: 01 March 2017 09:39
> > > > >> >To: dev@cloudstack.apache.org
> > > > >> >Subject: Re: Modern template hosting
> > > > >> >
> > > > >> >+1
> > > > >> >
> > > > >> >After seeing this discussion running in circles several
> times,
> > I
> > > > think
> > > > >> we should at least get started with the simplest option.
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> >On 27/02/17, 11:54 PM, "Chiradeep Vittal" <
> > chirade...@gmail.com>
> > > > wrote:
> > > > >> >
> > > > >> >>The alternative foolproof way is to simply not provide the
> > > > automatic
> > > > >> >>download.
> > > > >> >>
> > > > >> >>Just document the mirror list URL where the templates can be
> > > > found. I
> > > > >> >>don't know why people reject this option.
> > > > >> >&

Re: Modern template hosting

[Will Stevens]

me every time. This might be the one closest to the
> > > mirror302 and not to the client. Is that correct or did I miss a
> weighing
> > > algorithm hidden in there?
> > >
> > > Good coding,
> > >
> > > On 03/03/17 00:23, "Will Stevens" <sw...@apache.org> wrote:
> > >
> > > Hey All,
> > > Please review this repo and tell me what you think:
> > > https://github.com/swill/mirror302
> > >
> > > Let me know if you have questions or you would like me to make
> > changes
> > > before I bring this topic up with ASF Infra and open a ticket to
> get
> > > this
> > > implemented.
> > >
> > > Cheers,
> > >
> > > *Will Stevens*
> > >
> > >
> > >
> > > On Wed, Mar 1, 2017 at 11:37 AM, Will Stevens <
> wstev...@cloudops.com
> > >
> > > wrote:
> > >
> > > > I am building a short term solution right now.  I hopefully will
> > > have the
> > > > building blocks in place by the weekend so we can start working
> > with
> > > Infra
> > > > to get it in place.
> > > >
> > > > We will have a web server which we can point something like
> > > > downloads.cloudstack.org (or whatever url) at.  Then we will
> have
> > > an ASF
> > > > repo which tracks a mirror list and exposes a static site to be
> > able
> > > to
> > > > query the mirror list.
> > > >
> > > > The web server will receive a request and will do a 302 redirect
> to
> > > the
> > > > appropriate resource on one of the mirrors.
> > > >
> > > > This gets us started.  Legacy environments will have to do a DB
> > > change to
> > > > change from 'download.cloud.com' to 'downloads.cloudstack.org'
> (or
> > > > whatever), but otherwise it should be pretty simple.
> > > >
> > > > We can improve how we deliver templates going forward, but this
> > > obviously
> > > > requires some discussion still.  I will try to buy us some time
> > with
> > > an
> > > > implementation which solves our problems today...
> > > >
> > > > *Will STEVENS*
> > > > Lead Developer
> > > >
> > > > <https://goo.gl/NYZ8KK>
> > > >
> > > > On Wed, Mar 1, 2017 at 7:52 AM, Abhinandan Prateek <
> > > > abhinandan.prat...@shapeblue.com> wrote:
> > > >
> > > >>
> > > >> Initial seeding is a manual step and that is not going to change
> > in
> > > near
> > > >> future. A handy list of official places from where these
> templates
> > > can be
> > > >> downloaded won’t harm this part of the installation. Just allow
> > > anyone
> > > >> credible who follows democratic process to publish their
> templates
> > > with
> > > >> apache’s blessings.
> > > >>
> > > >>
> > > >> Coming to urls hard coded in db, the popular suggestion is to
> > point
> > > them
> > > >> to mirrors. This looks good but this is going to take some
> effort
> > > that has
> > > >> not materialised yet. Once this change is made it will be going
> in
> > > some
> > > >> future release like 4.11; the problem still remains for the
> > > releases that
> > > >> are already out there. In case access to download.cloud.com is
> > > dropped
> > > >> at some point we will have to document the procedure such that
> > > people know
> > > >> how to make these stable releases work. That is where the
> > procedure
> > > >> documented here https://shankerbalan.net/blog/
> > > >> seed-cloudstack-templates-offline/ should be officially adapted
> > > with a
> > > >> marketplace to pick up templates of choosing. Maybe we can have
> a
> > > pre-setup
> > > >> script to make choices instead of written wiki. Note that this
> > will
> > > be
> > > >> anyway needed by the older releases.
> > > >>
> > > >>
> > > >>
> > > >&g

Re: Modern template hosting

[Rafael Weingärtner]

rs.
> > >
> > > This gets us started.  Legacy environments will have to do a DB
> > change to
> > > change from 'download.cloud.com' to 'downloads.cloudstack.org' (or
> > > whatever), but otherwise it should be pretty simple.
> > >
> > > We can improve how we deliver templates going forward, but this
> > obviously
> > > requires some discussion still.  I will try to buy us some time
> with
> > an
> > > implementation which solves our problems today...
> > >
> > > *Will STEVENS*
> > > Lead Developer
> > >
> > > <https://goo.gl/NYZ8KK>
> > >
> > > On Wed, Mar 1, 2017 at 7:52 AM, Abhinandan Prateek <
> > > abhinandan.prat...@shapeblue.com> wrote:
> > >
> > >>
> > >> Initial seeding is a manual step and that is not going to change
> in
> > near
> > >> future. A handy list of official places from where these templates
> > can be
> > >> downloaded won’t harm this part of the installation. Just allow
> > anyone
> > >> credible who follows democratic process to publish their templates
> > with
> > >> apache’s blessings.
> > >>
> > >>
> > >> Coming to urls hard coded in db, the popular suggestion is to
> point
> > them
> > >> to mirrors. This looks good but this is going to take some effort
> > that has
> > >> not materialised yet. Once this change is made it will be going in
> > some
> > >> future release like 4.11; the problem still remains for the
> > releases that
> > >> are already out there. In case access to download.cloud.com is
> > dropped
> > >> at some point we will have to document the procedure such that
> > people know
> > >> how to make these stable releases work. That is where the
> procedure
> > >> documented here https://shankerbalan.net/blog/
> > >> seed-cloudstack-templates-offline/ should be officially adapted
> > with a
> > >> marketplace to pick up templates of choosing. Maybe we can have a
> > pre-setup
> > >> script to make choices instead of written wiki. Note that this
> will
> > be
> > >> anyway needed by the older releases.
> > >>
> > >>
> > >>
> > >>
> > >> On 01/03/17, 4:24 PM, "Paul Angus" <paul.an...@shapeblue.com>
> > wrote:
> > >>
> > >> >-1
> > >> >
> > >> >We are trying to get MORE people to use CloudStack.  And raising
> > the
> > >> barrier to entry does nothing to help that.
> > >> >
> > >> >
> > >> >Kind regards,
> > >> >
> > >> >Paul Angus
> > >> >
> > >> >paul.an...@shapeblue.com
> > >> >www.shapeblue.com
> > >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > >> >@shapeblue
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >-Original Message-
> > >> >From: Abhinandan Prateek [mailto:abhinandan.prateek@
> shapeblue.com]
> > >> >Sent: 01 March 2017 09:39
> > >> >To: dev@cloudstack.apache.org
> > >> >Subject: Re: Modern template hosting
> > >> >
> > >> >+1
> > >> >
> > >> >After seeing this discussion running in circles several times, I
> > think
> > >> we should at least get started with the simplest option.
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >On 27/02/17, 11:54 PM, "Chiradeep Vittal" <chirade...@gmail.com>
> > wrote:
> > >> >
> > >> >>The alternative foolproof way is to simply not provide the
> > automatic
> > >> >>download.
> > >> >>
> > >> >>Just document the mirror list URL where the templates can be
> > found. I
> > >> >>don't know why people reject this option.
> > >> >>
> > >> >>https://shankerbalan.net/blog/seed-cloudstack-
> templates-offline/
> &

Re: Modern template hosting

[Will Stevens]

t; anyway needed by the older releases.
> >>
> >>
> >>
> >>
> >> On 01/03/17, 4:24 PM, "Paul Angus" <paul.an...@shapeblue.com>
> wrote:
> >>
> >> >-1
> >> >
> >> >We are trying to get MORE people to use CloudStack.  And raising
> the
> >> barrier to entry does nothing to help that.
> >> >
> >> >
> >> >Kind regards,
> >> >
> >> >Paul Angus
> >> >
> >> >paul.an...@shapeblue.com
> >> >www.shapeblue.com
> >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> >> >@shapeblue
> >> >
> >> >
> >> >
> >> >
> >> >-Original Message-
> >> >From: Abhinandan Prateek [mailto:abhinandan.prat...@shapeblue.com]
> >> >Sent: 01 March 2017 09:39
> >> >To: dev@cloudstack.apache.org
> >> >Subject: Re: Modern template hosting
> >> >
> >> >+1
> >> >
> >> >After seeing this discussion running in circles several times, I
> think
> >> we should at least get started with the simplest option.
> >> >
> >> >
> >> >
> >> >
> >> >On 27/02/17, 11:54 PM, "Chiradeep Vittal" <chirade...@gmail.com>
> wrote:
> >> >
> >> >>The alternative foolproof way is to simply not provide the
> automatic
> >> >>download.
> >> >>
> >> >>Just document the mirror list URL where the templates can be
> found. I
> >> >>don't know why people reject this option.
> >> >>
> >> >>https://shankerbalan.net/blog/seed-cloudstack-templates-offline/
> >> >>
> >> >>On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens
> >> >><williamstev...@gmail.com>
> >> >>wrote:
> >> >>
> >> >>> OK. Thanks for the heads up.
> >> >>>
> >> >>> On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <
> chirade...@gmail.com>
> >> wrote:
> >> >>>
> >> >>> > Sounds workable. The downloader code in the SSVM won't follow
> >> >>> > redirects I think.
> >> >>> > https://github.com/apache/cloudstack/blob/5511065fc20787619d
> 9cd0444
> >> >>> > a65a3 155fc9c921/core/src/com/cloud/storage/template/
> >> >>> > HttpTemplateDownloader.java#L93
> >> >>> > https://goo.gl/dSi0r5
> >> >>> >  Might need to add
> >> >>> > client.setRedirectStrategy(new LaxRedirectStrategy());
> >> >>> >
> >> >>> > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens
> >> >>> > <wstev...@cloudops.com>
> >> >>> > wrote:
> >> >>> >
> >> >>> > > We haven't opened a ticket yet because we don't have a
> strategy
> >> yet.
> >> >>> > >
> >> >>> > > What do you guys think of this:
> >> >>> > > - We setup a new github repo in the 'apache' org which
> consists
> >> >>> > > of a
> >> >>> > single
> >> >>> > > file with a list of active/supported mirrors.
> >> >>> > > - I write a small web server, distributed as a binary,
> which can
> >> >>> > > be
> >> >>> > hosted
> >> >>> > > by ASF Infra.  This web server will query the current list
> of
> >> >>> > > mirrors
> >> >>> and
> >> >>> > > will select one and then do a 302 redirect to that mirror.
> >> >>> > >
> >> >>> > > The act of 'choosing' a mirror could be done in a number of
> ways.
> >> >>> > > - If we want to define an order, then it could just try
> from the
> >> >>> > > top of
> >> >>> > the
> >> >>> > > list and work its way down.  It would curl the target to
> make
> >> >>> > > sure it
> >> >>> > gets
&

Re: Modern template hosting

[Will Stevens]

7, 4:24 PM, "Paul Angus" <paul.an...@shapeblue.com>
> wrote:
> >>
> >> >-1
> >> >
> >> >We are trying to get MORE people to use CloudStack.  And raising
> the
> >> barrier to entry does nothing to help that.
> >> >
> >> >
> >> >Kind regards,
> >> >
> >> >Paul Angus
> >> >
> >> >paul.an...@shapeblue.com
> >> >www.shapeblue.com
> >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> >> >@shapeblue
> >> >
> >> >
> >> >
> >> >
> >> >-Original Message-
> >> >From: Abhinandan Prateek [mailto:abhinandan.prat...@shapeblue.com]
> >> >Sent: 01 March 2017 09:39
> >> >To: dev@cloudstack.apache.org
> >> >Subject: Re: Modern template hosting
> >> >
> >> >+1
> >> >
> >> >After seeing this discussion running in circles several times, I
> think
> >> we should at least get started with the simplest option.
> >> >
> >> >
> >> >
> >> >
> >> >On 27/02/17, 11:54 PM, "Chiradeep Vittal" <chirade...@gmail.com>
> wrote:
> >> >
> >> >>The alternative foolproof way is to simply not provide the
> automatic
> >> >>download.
> >> >>
> >> >>Just document the mirror list URL where the templates can be
> found. I
> >> >>don't know why people reject this option.
> >> >>
> >> >>https://shankerbalan.net/blog/seed-cloudstack-templates-offline/
> >> >>
> >> >>On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens
> >> >><williamstev...@gmail.com>
> >> >>wrote:
> >> >>
> >> >>> OK. Thanks for the heads up.
> >> >>>
> >> >>> On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <
> chirade...@gmail.com>
> >> wrote:
> >> >>>
> >> >>> > Sounds workable. The downloader code in the SSVM won't follow
> >> >>> > redirects I think.
> >> >>> > https://github.com/apache/cloudstack/blob/5511065fc20787619d
> 9cd0444
> >> >>> > a65a3 155fc9c921/core/src/com/cloud/storage/template/
> >> >>> > HttpTemplateDownloader.java#L93
> >> >>> > https://goo.gl/dSi0r5
> >> >>> >  Might need to add
> >> >>> > client.setRedirectStrategy(new LaxRedirectStrategy());
> >> >>> >
> >> >>> > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens
> >> >>> > <wstev...@cloudops.com>
> >> >>> > wrote:
> >> >>> >
> >> >>> > > We haven't opened a ticket yet because we don't have a
> strategy
> >> yet.
> >> >>> > >
> >> >>> > > What do you guys think of this:
> >> >>> > > - We setup a new github repo in the 'apache' org which
> consists
> >> >>> > > of a
> >> >>> > single
> >> >>> > > file with a list of active/supported mirrors.
> >> >>> > > - I write a small web server, distributed as a binary,
> which can
> >> >>> > > be
> >> >>> > hosted
> >> >>> > > by ASF Infra.  This web server will query the current list
> of
> >> >>> > > mirrors
> >> >>> and
> >> >>> > > will select one and then do a 302 redirect to that mirror.
> >> >>> > >
> >> >>> > > The act of 'choosing' a mirror could be done in a number of
> ways.
> >> >>> > > - If we want to define an order, then it could just try
> from the
> >> >>> > > top of
> >> >>> > the
> >> >>> > > list and work its way down.  It would curl the target to
> make
> >> >>> > > sure it
> >> >>> > gets
> >> >>> > > a 200 and if it does, it wo

Re: Modern template hosting

[Rafael Weingärtner]

Will great job.

I had the same doubt as Daan.

Have you considered using HTTP 300 working mode? Then, we could let the
client decide which mirror is the best(closest?). Section “10.3.1” of [1]
talks about it; this would put the pressure on deciding upon a mirror on
the client side, but I think it would be fair.

[1] https://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html

PS: You really like very short variables names!

On Fri, Mar 3, 2017 at 1:24 AM, Daan Hoogland <daan.hoogl...@shapeblue.com>
wrote:

> Nice little thing Will,
>
> One question: if I read the code correctly it ‘go’es and tries all mirrors
> at once and whichever responses first is redirected to the client. This
> might well be the same every time. This might be the one closest to the
> mirror302 and not to the client. Is that correct or did I miss a weighing
> algorithm hidden in there?
>
> Good coding,
>
> On 03/03/17 00:23, "Will Stevens" <sw...@apache.org> wrote:
>
> Hey All,
> Please review this repo and tell me what you think:
> https://github.com/swill/mirror302
>
> Let me know if you have questions or you would like me to make changes
> before I bring this topic up with ASF Infra and open a ticket to get
> this
> implemented.
>
> Cheers,
>
> *Will Stevens*
>
>
>
> On Wed, Mar 1, 2017 at 11:37 AM, Will Stevens <wstev...@cloudops.com>
> wrote:
>
> > I am building a short term solution right now.  I hopefully will
> have the
> > building blocks in place by the weekend so we can start working with
> Infra
> > to get it in place.
> >
> > We will have a web server which we can point something like
> > downloads.cloudstack.org (or whatever url) at.  Then we will have
> an ASF
> > repo which tracks a mirror list and exposes a static site to be able
> to
> > query the mirror list.
> >
> > The web server will receive a request and will do a 302 redirect to
> the
> > appropriate resource on one of the mirrors.
> >
> > This gets us started.  Legacy environments will have to do a DB
> change to
> > change from 'download.cloud.com' to 'downloads.cloudstack.org' (or
> > whatever), but otherwise it should be pretty simple.
> >
> > We can improve how we deliver templates going forward, but this
> obviously
> > requires some discussion still.  I will try to buy us some time with
> an
> > implementation which solves our problems today...
> >
> > *Will STEVENS*
> > Lead Developer
> >
> > <https://goo.gl/NYZ8KK>
> >
> > On Wed, Mar 1, 2017 at 7:52 AM, Abhinandan Prateek <
> > abhinandan.prat...@shapeblue.com> wrote:
> >
> >>
> >> Initial seeding is a manual step and that is not going to change in
> near
> >> future. A handy list of official places from where these templates
> can be
> >> downloaded won’t harm this part of the installation. Just allow
> anyone
> >> credible who follows democratic process to publish their templates
> with
> >> apache’s blessings.
> >>
> >>
> >> Coming to urls hard coded in db, the popular suggestion is to point
> them
> >> to mirrors. This looks good but this is going to take some effort
> that has
> >> not materialised yet. Once this change is made it will be going in
> some
> >> future release like 4.11; the problem still remains for the
> releases that
> >> are already out there. In case access to download.cloud.com is
> dropped
> >> at some point we will have to document the procedure such that
> people know
> >> how to make these stable releases work. That is where the procedure
> >> documented here https://shankerbalan.net/blog/
> >> seed-cloudstack-templates-offline/ should be officially adapted
> with a
> >> marketplace to pick up templates of choosing. Maybe we can have a
> pre-setup
> >> script to make choices instead of written wiki. Note that this will
> be
> >> anyway needed by the older releases.
> >>
> >>
> >>
> >>
> >> On 01/03/17, 4:24 PM, "Paul Angus" <paul.an...@shapeblue.com>
> wrote:
> >>
> >> >-1
>     >> >
> >> >We are trying to get MORE people to use CloudStack.  And raising
> the
> >> barrier to entry does nothing to help that.
> >> >
> >

Re: Modern template hosting

[Will Stevens]

sh their templates
> with
> >> apache’s blessings.
> >>
> >>
> >> Coming to urls hard coded in db, the popular suggestion is to point
> them
> >> to mirrors. This looks good but this is going to take some effort
> that has
> >> not materialised yet. Once this change is made it will be going in
> some
> >> future release like 4.11; the problem still remains for the
> releases that
> >> are already out there. In case access to download.cloud.com is
> dropped
> >> at some point we will have to document the procedure such that
> people know
> >> how to make these stable releases work. That is where the procedure
> >> documented here https://shankerbalan.net/blog/
> >> seed-cloudstack-templates-offline/ should be officially adapted
> with a
> >> marketplace to pick up templates of choosing. Maybe we can have a
> pre-setup
> >> script to make choices instead of written wiki. Note that this will
> be
> >> anyway needed by the older releases.
> >>
> >>
> >>
> >>
> >> On 01/03/17, 4:24 PM, "Paul Angus" <paul.an...@shapeblue.com>
> wrote:
> >>
> >> >-1
>     >> >
> >> >We are trying to get MORE people to use CloudStack.  And raising
> the
> >> barrier to entry does nothing to help that.
> >> >
> >> >
> >> >Kind regards,
> >> >
> >> >Paul Angus
> >> >
> >> >paul.an...@shapeblue.com
> >> >www.shapeblue.com
> >> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> >> >@shapeblue
> >> >
> >> >
> >> >
> >> >
> >> >-Original Message-
> >> >From: Abhinandan Prateek [mailto:abhinandan.prat...@shapeblue.com]
> >> >Sent: 01 March 2017 09:39
> >> >To: dev@cloudstack.apache.org
> >> >Subject: Re: Modern template hosting
> >> >
> >> >+1
> >> >
> >> >After seeing this discussion running in circles several times, I
> think
> >> we should at least get started with the simplest option.
> >> >
> >> >
> >> >
> >> >
> >> >On 27/02/17, 11:54 PM, "Chiradeep Vittal" <chirade...@gmail.com>
> wrote:
> >> >
> >> >>The alternative foolproof way is to simply not provide the
> automatic
> >> >>download.
> >> >>
> >> >>Just document the mirror list URL where the templates can be
> found. I
> >> >>don't know why people reject this option.
> >> >>
> >> >>https://shankerbalan.net/blog/seed-cloudstack-templates-offline/
> >> >>
> >> >>On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens
> >> >><williamstev...@gmail.com>
> >> >>wrote:
> >> >>
> >> >>> OK. Thanks for the heads up.
> >> >>>
> >> >>> On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <
> chirade...@gmail.com>
> >> wrote:
> >> >>>
> >> >>> > Sounds workable. The downloader code in the SSVM won't follow
> >> >>> > redirects I think.
> >> >>> > https://github.com/apache/cloudstack/blob/
> 5511065fc20787619d9cd0444
> >> >>> > a65a3 155fc9c921/core/src/com/cloud/storage/template/
> >> >>> > HttpTemplateDownloader.java#L93
> >> >>> > https://goo.gl/dSi0r5
> >> >>> >  Might need to add
> >> >>> > client.setRedirectStrategy(new LaxRedirectStrategy());
> >> >>> >
> >> >>> > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens
> >> >>> > <wstev...@cloudops.com>
> >> >>> > wrote:
> >> >>> >
> >> >>> > > We haven't opened a ticket yet because we don't have a
> strategy
> >> yet.
> >> >>> > >
> >> >>> > > What do you guys think of this:
> >> >>> > > - We setup a new github repo in the 'apache' org which
> consists
> >> >>> > &g

Re: Modern template hosting

[Daan Hoogland]

Nice little thing Will,

One question: if I read the code correctly it ‘go’es and tries all mirrors at 
once and whichever responses first is redirected to the client. This might well 
be the same every time. This might be the one closest to the mirror302 and not 
to the client. Is that correct or did I miss a weighing algorithm hidden in 
there?

Good coding,

On 03/03/17 00:23, "Will Stevens" <sw...@apache.org> wrote:

Hey All,
Please review this repo and tell me what you think:
https://github.com/swill/mirror302

Let me know if you have questions or you would like me to make changes
before I bring this topic up with ASF Infra and open a ticket to get this
implemented.

Cheers,

*Will Stevens*



On Wed, Mar 1, 2017 at 11:37 AM, Will Stevens <wstev...@cloudops.com> wrote:

> I am building a short term solution right now.  I hopefully will have the
> building blocks in place by the weekend so we can start working with Infra
> to get it in place.
>
> We will have a web server which we can point something like
> downloads.cloudstack.org (or whatever url) at.  Then we will have an ASF
> repo which tracks a mirror list and exposes a static site to be able to
> query the mirror list.
>
> The web server will receive a request and will do a 302 redirect to the
> appropriate resource on one of the mirrors.
>
> This gets us started.  Legacy environments will have to do a DB change to
> change from 'download.cloud.com' to 'downloads.cloudstack.org' (or
> whatever), but otherwise it should be pretty simple.
>
> We can improve how we deliver templates going forward, but this obviously
> requires some discussion still.  I will try to buy us some time with an
> implementation which solves our problems today...
>
> *Will STEVENS*
> Lead Developer
>
> <https://goo.gl/NYZ8KK>
>
> On Wed, Mar 1, 2017 at 7:52 AM, Abhinandan Prateek <
> abhinandan.prat...@shapeblue.com> wrote:
>
>>
>> Initial seeding is a manual step and that is not going to change in near
>> future. A handy list of official places from where these templates can be
>> downloaded won’t harm this part of the installation. Just allow anyone
>> credible who follows democratic process to publish their templates with
>> apache’s blessings.
>>
>>
>> Coming to urls hard coded in db, the popular suggestion is to point them
>> to mirrors. This looks good but this is going to take some effort that 
has
>> not materialised yet. Once this change is made it will be going in some
>> future release like 4.11; the problem still remains for the releases that
>> are already out there. In case access to download.cloud.com is dropped
>> at some point we will have to document the procedure such that people 
know
>> how to make these stable releases work. That is where the procedure
>> documented here https://shankerbalan.net/blog/
>> seed-cloudstack-templates-offline/ should be officially adapted with a
>> marketplace to pick up templates of choosing. Maybe we can have a 
pre-setup
>> script to make choices instead of written wiki. Note that this will be
>> anyway needed by the older releases.
>>
>>
>>
>>
>> On 01/03/17, 4:24 PM, "Paul Angus" <paul.an...@shapeblue.com> wrote:
>>
>> >-1
>> >
>> >We are trying to get MORE people to use CloudStack.  And raising the
>> barrier to entry does nothing to help that.
>> >
>> >
>> >Kind regards,
>> >
>> >Paul Angus
>> >
>> >paul.an...@shapeblue.com
>> >www.shapeblue.com
>> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>> >@shapeblue
>> >
>> >
>> >
>> >
>> >-Original Message-
>> >From: Abhinandan Prateek [mailto:abhinandan.prat...@shapeblue.com]
>> >Sent: 01 March 2017 09:39
>> >To: dev@cloudstack.apache.org
>> >Subject: Re: Modern template hosting
>> >
>> >+1
>> >
>> >After seeing this discussion running in circles several times, I think
>> we should at least get started with the simplest option.
>> >
>> >
>> >
>> >
>> >On 27/02/17, 11:54 PM, "Chiradeep Vittal" <chirade...@gmail.com> wrote:
>> >
>> >>The 

Re: Modern template hosting

[Will Stevens]

Hey All,
Please review this repo and tell me what you think:
https://github.com/swill/mirror302

Let me know if you have questions or you would like me to make changes
before I bring this topic up with ASF Infra and open a ticket to get this
implemented.

Cheers,

*Will Stevens*



On Wed, Mar 1, 2017 at 11:37 AM, Will Stevens <wstev...@cloudops.com> wrote:

> I am building a short term solution right now.  I hopefully will have the
> building blocks in place by the weekend so we can start working with Infra
> to get it in place.
>
> We will have a web server which we can point something like
> downloads.cloudstack.org (or whatever url) at.  Then we will have an ASF
> repo which tracks a mirror list and exposes a static site to be able to
> query the mirror list.
>
> The web server will receive a request and will do a 302 redirect to the
> appropriate resource on one of the mirrors.
>
> This gets us started.  Legacy environments will have to do a DB change to
> change from 'download.cloud.com' to 'downloads.cloudstack.org' (or
> whatever), but otherwise it should be pretty simple.
>
> We can improve how we deliver templates going forward, but this obviously
> requires some discussion still.  I will try to buy us some time with an
> implementation which solves our problems today...
>
> *Will STEVENS*
> Lead Developer
>
> <https://goo.gl/NYZ8KK>
>
> On Wed, Mar 1, 2017 at 7:52 AM, Abhinandan Prateek <
> abhinandan.prat...@shapeblue.com> wrote:
>
>>
>> Initial seeding is a manual step and that is not going to change in near
>> future. A handy list of official places from where these templates can be
>> downloaded won’t harm this part of the installation. Just allow anyone
>> credible who follows democratic process to publish their templates with
>> apache’s blessings.
>>
>>
>> Coming to urls hard coded in db, the popular suggestion is to point them
>> to mirrors. This looks good but this is going to take some effort that has
>> not materialised yet. Once this change is made it will be going in some
>> future release like 4.11; the problem still remains for the releases that
>> are already out there. In case access to download.cloud.com is dropped
>> at some point we will have to document the procedure such that people know
>> how to make these stable releases work. That is where the procedure
>> documented here https://shankerbalan.net/blog/
>> seed-cloudstack-templates-offline/ should be officially adapted with a
>> marketplace to pick up templates of choosing. Maybe we can have a pre-setup
>> script to make choices instead of written wiki. Note that this will be
>> anyway needed by the older releases.
>>
>>
>>
>>
>> On 01/03/17, 4:24 PM, "Paul Angus" <paul.an...@shapeblue.com> wrote:
>>
>> >-1
>> >
>> >We are trying to get MORE people to use CloudStack.  And raising the
>> barrier to entry does nothing to help that.
>> >
>> >
>> >Kind regards,
>> >
>> >Paul Angus
>> >
>> >paul.an...@shapeblue.com
>> >www.shapeblue.com
>> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>> >@shapeblue
>> >
>> >
>> >
>> >
>> >-Original Message-
>> >From: Abhinandan Prateek [mailto:abhinandan.prat...@shapeblue.com]
>> >Sent: 01 March 2017 09:39
>> >To: dev@cloudstack.apache.org
>> >Subject: Re: Modern template hosting
>> >
>> >+1
>> >
>> >After seeing this discussion running in circles several times, I think
>> we should at least get started with the simplest option.
>> >
>> >
>> >
>> >
>> >On 27/02/17, 11:54 PM, "Chiradeep Vittal" <chirade...@gmail.com> wrote:
>> >
>> >>The alternative foolproof way is to simply not provide the automatic
>> >>download.
>> >>
>> >>Just document the mirror list URL where the templates can be found. I
>> >>don't know why people reject this option.
>> >>
>> >>https://shankerbalan.net/blog/seed-cloudstack-templates-offline/
>> >>
>> >>On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens
>> >><williamstev...@gmail.com>
>> >>wrote:
>> >>
>> >>> OK. Thanks for the heads up.
>> >>>
>> >>> On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <chirade...@gmail.com>
>> wrote:
>> >>>
>> >>> > Sounds workable. The downloader code in the SSVM won't follow
>> >>>

Re: Modern template hosting

[Will Stevens]

I am building a short term solution right now.  I hopefully will have the
building blocks in place by the weekend so we can start working with Infra
to get it in place.

We will have a web server which we can point something like
downloads.cloudstack.org (or whatever url) at.  Then we will have an ASF
repo which tracks a mirror list and exposes a static site to be able to
query the mirror list.

The web server will receive a request and will do a 302 redirect to the
appropriate resource on one of the mirrors.

This gets us started.  Legacy environments will have to do a DB change to
change from 'download.cloud.com' to 'downloads.cloudstack.org' (or
whatever), but otherwise it should be pretty simple.

We can improve how we deliver templates going forward, but this obviously
requires some discussion still.  I will try to buy us some time with an
implementation which solves our problems today...

*Will STEVENS*
Lead Developer

<https://goo.gl/NYZ8KK>

On Wed, Mar 1, 2017 at 7:52 AM, Abhinandan Prateek <
abhinandan.prat...@shapeblue.com> wrote:

>
> Initial seeding is a manual step and that is not going to change in near
> future. A handy list of official places from where these templates can be
> downloaded won’t harm this part of the installation. Just allow anyone
> credible who follows democratic process to publish their templates with
> apache’s blessings.
>
>
> Coming to urls hard coded in db, the popular suggestion is to point them
> to mirrors. This looks good but this is going to take some effort that has
> not materialised yet. Once this change is made it will be going in some
> future release like 4.11; the problem still remains for the releases that
> are already out there. In case access to download.cloud.com is dropped at
> some point we will have to document the procedure such that people know how
> to make these stable releases work. That is where the procedure documented
> here https://shankerbalan.net/blog/seed-cloudstack-templates-offline/
> should be officially adapted with a marketplace to pick up templates of
> choosing. Maybe we can have a pre-setup script to make choices instead of
> written wiki. Note that this will be anyway needed by the older releases.
>
>
>
>
> On 01/03/17, 4:24 PM, "Paul Angus" <paul.an...@shapeblue.com> wrote:
>
> >-1
> >
> >We are trying to get MORE people to use CloudStack.  And raising the
> barrier to entry does nothing to help that.
> >
> >
> >Kind regards,
> >
> >Paul Angus
> >
> >paul.an...@shapeblue.com
> >www.shapeblue.com
> >53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> >@shapeblue
> >
> >
> >
> >
> >-Original Message-
> >From: Abhinandan Prateek [mailto:abhinandan.prat...@shapeblue.com]
> >Sent: 01 March 2017 09:39
> >To: dev@cloudstack.apache.org
> >Subject: Re: Modern template hosting
> >
> >+1
> >
> >After seeing this discussion running in circles several times, I think we
> should at least get started with the simplest option.
> >
> >
> >
> >
> >On 27/02/17, 11:54 PM, "Chiradeep Vittal" <chirade...@gmail.com> wrote:
> >
> >>The alternative foolproof way is to simply not provide the automatic
> >>download.
> >>
> >>Just document the mirror list URL where the templates can be found. I
> >>don't know why people reject this option.
> >>
> >>https://shankerbalan.net/blog/seed-cloudstack-templates-offline/
> >>
> >>On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens
> >><williamstev...@gmail.com>
> >>wrote:
> >>
> >>> OK. Thanks for the heads up.
> >>>
> >>> On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <chirade...@gmail.com>
> wrote:
> >>>
> >>> > Sounds workable. The downloader code in the SSVM won't follow
> >>> > redirects I think.
> >>> > https://github.com/apache/cloudstack/blob/5511065fc20787619d9cd0444
> >>> > a65a3 155fc9c921/core/src/com/cloud/storage/template/
> >>> > HttpTemplateDownloader.java#L93
> >>> > https://goo.gl/dSi0r5
> >>> >  Might need to add
> >>> > client.setRedirectStrategy(new LaxRedirectStrategy());
> >>> >
> >>> > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens
> >>> > <wstev...@cloudops.com>
> >>> > wrote:
> >>> >
> >>> > > We haven't opened a ticket yet because we don't have a strategy
> yet.
> >>> > >
> >>> > > What do you guys think of this:
> >>> > >

Re: Modern template hosting

[Abhinandan Prateek]

Initial seeding is a manual step and that is not going to change in near 
future. A handy list of official places from where these templates can be 
downloaded won’t harm this part of the installation. Just allow anyone credible 
who follows democratic process to publish their templates with apache’s 
blessings.


Coming to urls hard coded in db, the popular suggestion is to point them to 
mirrors. This looks good but this is going to take some effort that has not 
materialised yet. Once this change is made it will be going in some future 
release like 4.11; the problem still remains for the releases that are already 
out there. In case access to download.cloud.com is dropped at some point we 
will have to document the procedure such that people know how to make these 
stable releases work. That is where the procedure documented here 
https://shankerbalan.net/blog/seed-cloudstack-templates-offline/ should be 
officially adapted with a marketplace to pick up templates of choosing. Maybe 
we can have a pre-setup script to make choices instead of written wiki. Note 
that this will be anyway needed by the older releases.




On 01/03/17, 4:24 PM, "Paul Angus" <paul.an...@shapeblue.com> wrote:

>-1
>
>We are trying to get MORE people to use CloudStack.  And raising the barrier 
>to entry does nothing to help that.
>
>
>Kind regards,
>
>Paul Angus
>
>paul.an...@shapeblue.com 
>www.shapeblue.com
>53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>@shapeblue
>  
> 
>
>
>-Original Message-
>From: Abhinandan Prateek [mailto:abhinandan.prat...@shapeblue.com] 
>Sent: 01 March 2017 09:39
>To: dev@cloudstack.apache.org
>Subject: Re: Modern template hosting
>
>+1
>
>After seeing this discussion running in circles several times, I think we 
>should at least get started with the simplest option.
>
>
>
>
>On 27/02/17, 11:54 PM, "Chiradeep Vittal" <chirade...@gmail.com> wrote:
>
>>The alternative foolproof way is to simply not provide the automatic 
>>download.
>>
>>Just document the mirror list URL where the templates can be found. I 
>>don't know why people reject this option.
>>
>>https://shankerbalan.net/blog/seed-cloudstack-templates-offline/
>>
>>On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens 
>><williamstev...@gmail.com>
>>wrote:
>>
>>> OK. Thanks for the heads up.
>>>
>>> On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <chirade...@gmail.com> wrote:
>>>
>>> > Sounds workable. The downloader code in the SSVM won't follow 
>>> > redirects I think.
>>> > https://github.com/apache/cloudstack/blob/5511065fc20787619d9cd0444
>>> > a65a3 155fc9c921/core/src/com/cloud/storage/template/
>>> > HttpTemplateDownloader.java#L93
>>> > https://goo.gl/dSi0r5
>>> >  Might need to add
>>> > client.setRedirectStrategy(new LaxRedirectStrategy());
>>> >
>>> > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens 
>>> > <wstev...@cloudops.com>
>>> > wrote:
>>> >
>>> > > We haven't opened a ticket yet because we don't have a strategy yet.
>>> > >
>>> > > What do you guys think of this:
>>> > > - We setup a new github repo in the 'apache' org which consists 
>>> > > of a
>>> > single
>>> > > file with a list of active/supported mirrors.
>>> > > - I write a small web server, distributed as a binary, which can 
>>> > > be
>>> > hosted
>>> > > by ASF Infra.  This web server will query the current list of 
>>> > > mirrors
>>> and
>>> > > will select one and then do a 302 redirect to that mirror.
>>> > >
>>> > > The act of 'choosing' a mirror could be done in a number of ways.
>>> > > - If we want to define an order, then it could just try from the 
>>> > > top of
>>> > the
>>> > > list and work its way down.  It would curl the target to make 
>>> > > sure it
>>> > gets
>>> > > a 200 and if it does, it would do a 302 redirect.
>>> > > - Or, if we want to distribute the load across the mirrors, we 
>>> > > could
>>> pick
>>> > > from the list randomly.  Again, doing a curl to verify the mirror 
>>> > > is up
>>> > and
>>> > > then doing a redirect.
>>> > > - If we want to get fancy, we could do a reverse IP lookup and 
>>> > > try to
>>> > match
>>> > > the 

RE: Modern template hosting

[Paul Angus]

-1

We are trying to get MORE people to use CloudStack.  And raising the barrier to 
entry does nothing to help that.


Kind regards,

Paul Angus

paul.an...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 


-Original Message-
From: Abhinandan Prateek [mailto:abhinandan.prat...@shapeblue.com] 
Sent: 01 March 2017 09:39
To: dev@cloudstack.apache.org
Subject: Re: Modern template hosting

+1

After seeing this discussion running in circles several times, I think we 
should at least get started with the simplest option.




On 27/02/17, 11:54 PM, "Chiradeep Vittal" <chirade...@gmail.com> wrote:

>The alternative foolproof way is to simply not provide the automatic 
>download.
>
>Just document the mirror list URL where the templates can be found. I 
>don't know why people reject this option.
>
>https://shankerbalan.net/blog/seed-cloudstack-templates-offline/
>
>On Mon, Feb 27, 2017 at 10:15 AM, Will Stevens 
><williamstev...@gmail.com>
>wrote:
>
>> OK. Thanks for the heads up.
>>
>> On Feb 27, 2017 1:08 PM, "Chiradeep Vittal" <chirade...@gmail.com> wrote:
>>
>> > Sounds workable. The downloader code in the SSVM won't follow 
>> > redirects I think.
>> > https://github.com/apache/cloudstack/blob/5511065fc20787619d9cd0444
>> > a65a3 155fc9c921/core/src/com/cloud/storage/template/
>> > HttpTemplateDownloader.java#L93
>> > https://goo.gl/dSi0r5
>> >  Might need to add
>> > client.setRedirectStrategy(new LaxRedirectStrategy());
>> >
>> > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens 
>> > <wstev...@cloudops.com>
>> > wrote:
>> >
>> > > We haven't opened a ticket yet because we don't have a strategy yet.
>> > >
>> > > What do you guys think of this:
>> > > - We setup a new github repo in the 'apache' org which consists 
>> > > of a
>> > single
>> > > file with a list of active/supported mirrors.
>> > > - I write a small web server, distributed as a binary, which can 
>> > > be
>> > hosted
>> > > by ASF Infra.  This web server will query the current list of 
>> > > mirrors
>> and
>> > > will select one and then do a 302 redirect to that mirror.
>> > >
>> > > The act of 'choosing' a mirror could be done in a number of ways.
>> > > - If we want to define an order, then it could just try from the 
>> > > top of
>> > the
>> > > list and work its way down.  It would curl the target to make 
>> > > sure it
>> > gets
>> > > a 200 and if it does, it would do a 302 redirect.
>> > > - Or, if we want to distribute the load across the mirrors, we 
>> > > could
>> pick
>> > > from the list randomly.  Again, doing a curl to verify the mirror 
>> > > is up
>> > and
>> > > then doing a redirect.
>> > > - If we want to get fancy, we could do a reverse IP lookup and 
>> > > try to
>> > match
>> > > the requester with their closest geographical mirror.
>> > >
>> > > Thoughts?
>> > >
>> > > *Will STEVENS*
>> > > Lead Developer
>> > >
>> > > <https://goo.gl/NYZ8KK>
>> > >
>> > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal <
>> chirade...@gmail.com
>> > >
>> > > wrote:
>> > >
>> > > > What steps are needed to set up a mirror? What does Infra need to do?
>> > Has
>> > > > anybody filed a ticket with Infra?
>> > > >
>> > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela < 
>> > > > raja.pull...@accelerite.com>
>> > > > wrote:
>> > > >
>> > > > > Hi will,
>> > > > >
>> > > > > I believe, we didn’t get to close ‘getting a mirror on Apache’
>> > because
>> > > we
>> > > > > needed someone on the Apache Infra side to close this. BTW, 
>> > > > > cloudstack-apt.get.eu (I think Nux manages this?) has 
>> > > > > all/most of
>> > the
>> > > > > content.  Once we can close on the Apache mirror for hosting 
>> > > > > the
>> > > > content, I
>> > > > > can help assist getting the content there.
>> > > > >
>> > > > > For now, we have replicated the download.cloud.com con

Re: Modern template hosting

[Abhinandan Prateek]

ens" <williamstev...@gmail.com on behalf of
>> > wstev...@cloudops.com>
>> > > > > wrote:
>> > > > >
>> > > > > unfortunately the template mirror conversation got caught up in
>> > details
>> > > > and
>> > > > > nobody took the lead on implementing a solution.
>> > > > >
>> > > > > citrix has been pinging me every couple months to say 'dude, we
>> need
>> > to
>> > > > > remove the dependency on download.citrix.com', but i have not had
>> > the
>> > > > > cycles to get in and solve the problem.  the shutdown of that is
>> > > imminent
>> > > > > right now, so we need to solve it asap.
>> > > > >
>> > > > >
>> > > > >
>> > > > > *Will STEVENS*
>> > > > > Lead Developer
>> > > > >
>> > > > > <https://goo.gl/NYZ8KK>
>> > > > >
>> > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <
>> > paul.an...@shapeblue.com
>> > > >
>> > > > > wrote:
>> > > > >
>> > > > > > Hi Nathan,
>> > > > > >
>> > > > > > Ideally, if you put the template location in (or use a template
>> > > defined
>> > > > > > in)  test_data.py then the actual location can be overridden by
>> > > anyone
>> > > > > > testing.
>> > > > > >
>> > > > > > For Trillian, we've copied all of the templates that people have
>> > > define
>> > > > > to
>> > > > > > a local repo and then replace the URLs in test_data.py to reduce
>> > > > > bandwidth
>> > > > > > use and download times.
>> > > > > >
>> > > > > > Ie:
>> > > > > >
>> > > > > > "bootableIso":
>> > > > > > {
>> > > > > > "displaytext": "Test Bootable ISO",
>> > > > > > "name": "testISO",
>> > > > > > "bootable": True,
>> > > > > > "ispublic": False,
>> > > > > > "url": "{{ marvin_images_location
>> > > > > > }}/TinyCore-current.iso",
>> > > > > > "ostype": 'Other Linux (64-bit)',
>> > > > > > "mode": 'HTTP_DOWNLOAD'
>> > > > > > },
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > I thought that we had come up with a solution for
>> > download.cloud.com
>> > > ,
>> > > > by
>> > > > > > having a mirrorlist hosted in Community Apache 'space' with
>> anyone
>> > > able
>> > > > > to
>> > > > > > out themselves forward as a mirror.
>> > > > > > But I must admit I lost track of whether anyone made the
>> requisite
>> > > > > changes
>> > > > > > in code
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > Kind regards,
>> > > > > >
>> > > > > > Paul Angus
>> > > > > >
>> > > > > > paul.an...@shapeblue.com
>> > > > > > www.shapeblue.com
>> > > > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>> > > > > > @shapeblue
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > -Original Message-
>> > > > > > From: williamstev...@gmail.com [mailto:williamstev...@gmail.com]
>> > On
>> > > > > > Behalf Of Will Stevens
>> > > > > > Sent: 24 February 2017 16:30
>> > > > > > To: dev@cloudstack.apache.org
>> > > > > > Subject: Re: Modern template hosting
>> > > > > >
>> > > > > > this is a hard questions.  in general, we should be setting up a
>> > > mirror
>> > > > > on
>> > > > > > some cloudstack/apache domain and then mirror to other provided
>> > > > > templates.
>> > > > > >
>> > > > > > we MUST come up with a solution to deprecate 'download.cloud.com
>> ',
>> > > > that
>> > > > > > is going to be going away any day now.
>> > > > > >
>> > > > > > i don't know the right way to solve this to be honest, but if you
>> > > have
>> > > > > > ideas, i am willing to help.
>> > > > > >
>> > > > > >
>> > > > > >
>> > > > > > *Will STEVENS*
>> > > > > > Lead Developer
>> > > > > >
>> > > > > > <https://goo.gl/NYZ8KK>
>> > > > > >
>> > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <
>> njohn...@ena.com
>> > >
>> > > > > wrote:
>> > > > > >
>> > > > > > > So not to re-open a can of worms, but I’m in a situation where
>> I
>> > > need
>> > > > > > > to come up with a Marvin component test that depends on a
>> > template
>> > > > > > > based on a kernel that’s relatively new, i.e., newer than
>> Centos
>> > > 5.3
>> > > > /
>> > > > > > Ubuntu 10.04 .
>> > > > > > > I see openvm.eu has a suitable template (Ubuntu 16.0.4 for
>> KVM),
>> > > but
>> > > > > > > from looking at the thread "Migrating CloudStack content from
>> > > > > > > download.cloud.com” it looks like there is resistance to using
>> > > this
>> > > > at
>> > > > > > > least for hosting system vm templates over concerns of
>> > neutrality.
>> > > > > > > Would this be suitable for a component test?  If not, what is a
>> > > > > “blessed”
>> > > > > > > template location?
>> > > > > > >
>> > > > > > > Thanks in advance!
>> > > > > > >
>> > > > > > > Nathan Johnson
>> > > > > > > R Engineer
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > 618 Grassmere Park Drive, Suite 12
>> > > > > > > Nashville, TN 37211
>> > > > > > > General Office: 615-312-6000
>> > > > > > >
>> > > > > > > website | blog | support
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > DISCLAIMER
>> > > > > ==
>> > > > > This e-mail may contain privileged and confidential information
>> which
>> > > is
>> > > > > the property of Accelerite, a Persistent Systems business. It is
>> > > intended
>> > > > > only for the use of the individual or entity to which it is
>> > addressed.
>> > > If
>> > > > > you are not the intended recipient, you are not authorized to read,
>> > > > retain,
>> > > > > copy, print, distribute or use this message. If you have received
>> > this
>> > > > > communication in error, please notify the sender and delete all
>> > copies
>> > > of
>> > > > > this message. Accelerite, a Persistent Systems business does not
>> > accept
>> > > > any
>> > > > > liability for virus infected mails.
>> > > > >
>> > > >
>> > >
>> >
>>

abhinandan.prat...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 

Re: Modern template hosting

[Wido den Hollander]

gt; > > number
> > > > > > > of
> > > > > > > > > > ways.
> > > > > > > > > > > > > > > > - If we want to define an order, then it could
> > > > just try
> > > > > > > > from
> > > > > > > > > > the
> > > > > > > > > > > > top
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > list and work its way down.  It would curl the
> > > > target
> > > > > > to
> > > > > > > > make
> > > > > > > > > > > sure
> > > > > > > > > > > > it
> > > > > > > > > > > > > > > gets
> > > > > > > > > > > > > > > > a 200 and if it does, it would do a 302
> > redirect.
> > > > > > > > > > > > > > > > - Or, if we want to distribute the load across
> > the
> > > > > > > mirrors,
> > > > > > > > > we
> > > > > > > > > > > > could
> > > > > > > > > > > > > > pick
> > > > > > > > > > > > > > > > from the list randomly.  Again, doing a curl to
> > > > verify
> > > > > > > the
> > > > > > > > > > mirror
> > > > > > > > > > > > is
> > > > > > > > > > > > > up
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > then doing a redirect.
> > > > > > > > > > > > > > > > - If we want to get fancy, we could do a
> > reverse IP
> > > > > > > lookup
> > > > > > > > > and
> > > > > > > > > > > try
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > match
> > > > > > > > > > > > > > > > the requester with their closest geographical
> > > > mirror.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thoughts?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > *Will STEVENS*
> > > > > > > > > > > > > > > > Lead Developer
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep
> > Vittal
> > > > <
> > > > > > > > > > > > > > chirade...@gmail.com
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > What steps are needed to set up a mirror?
> > What
> > > > does
> > > > > > > Infra
> > > > > > > > > > need
> > > > > > > > > > > to
> > > > > > > > > > > > > do?
> > > > > > > > > > > > > > > Has
> > > > > > > > > > > > > > > > > anybody filed a ticket with Infra?
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja
> > Pullela <
> > > > > > > > > > > > > > > > > raja.pull...@accelerite.com>
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > 

Re: Modern template hosting

[Chiradeep Vittal]

t; > > > > > > > > > > >  Might need to add
> > > > > > > > > > > > > > client.setRedirectStrategy(new
> > > LaxRedirectStrategy());
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Mon, Feb 27, 2017 at 9:57 AM, Will Stevens <
> > > > > > > > > > wstev...@cloudops.com
> > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > We haven't opened a ticket yet because we don't
> > > have a
> > > > > > > > strategy
> > > > > > > > > > > yet.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > What do you guys think of this:
> > > > > > > > > > > > > > > - We setup a new github repo in the 'apache'
> org
> > > which
> > > > > > > > consists
> > > > > > > > > > of
> > > > > > > > > > > a
> > > > > > > > > > > > > > single
> > > > > > > > > > > > > > > file with a list of active/supported mirrors.
> > > > > > > > > > > > > > > - I write a small web server, distributed as a
> > > binary,
> > > > > > > which
> > > > > > > > > can
> > > > > > > > > > be
> > > > > > > > > > > > > > hosted
> > > > > > > > > > > > > > > by ASF Infra.  This web server will query the
> > > current
> > > > > > list
> > > > > > > of
> > > > > > > > > > > mirrors
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > will select one and then do a 302 redirect to
> that
> > > > > > mirror.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > The act of 'choosing' a mirror could be done
> in a
> > > > > number
> > > > > > of
> > > > > > > > > ways.
> > > > > > > > > > > > > > > - If we want to define an order, then it could
> > > just try
> > > > > > > from
> > > > > > > > > the
> > > > > > > > > > > top
> > > > > > > > > > > > of
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > list and work its way down.  It would curl the
> > > target
> > > > > to
> > > > > > > make
> > > > > > > > > > sure
> > > > > > > > > > > it
> > > > > > > > > > > > > > gets
> > > > > > > > > > > > > > > a 200 and if it does, it would do a 302
> redirect.
> > > > > > > > > > > > > > > - Or, if we want to distribute the load across
> the
> > > > > > mirrors,
> > > > > > > > we
> > > > > > > > > > > could
> > > > > > > > > > > > > pick
> > > > > > > > > > > > > > > from the list randomly.  Again, doing a curl to
> > > verify
> > > > > > the
> > > > > > > > > mirror
> > > > > > > > > > > is
> > > > > > > > > > > > up
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > then doing a redirect.
> > > > > > > > > > > > > > > - If we want to get fancy, we could do a
> reverse IP
> > > > > > lookup
> > > > > > > > and
> > > > > > > > > > try
> > > > > > > > > > > to
> > > > > > > > > > &

Re: Modern template hosting

[Wido den Hollander]

 > > > > > > >
> > > > > > > > > > > > > > The act of 'choosing' a mirror could be done in a
> > > > number
> > > > > of
> > > > > > > > ways.
> > > > > > > > > > > > > > - If we want to define an order, then it could
> > just try
> > > > > > from
> > > > > > > > the
> > > > > > > > > > top
> > > > > > > > > > > of
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > list and work its way down.  It would curl the
> > target
> > > > to
> > > > > > make
> > > > > > > > > sure
> > > > > > > > > > it
> > > > > > > > > > > > > gets
> > > > > > > > > > > > > > a 200 and if it does, it would do a 302 redirect.
> > > > > > > > > > > > > > - Or, if we want to distribute the load across the
> > > > > mirrors,
> > > > > > > we
> > > > > > > > > > could
> > > > > > > > > > > > pick
> > > > > > > > > > > > > > from the list randomly.  Again, doing a curl to
> > verify
> > > > > the
> > > > > > > > mirror
> > > > > > > > > > is
> > > > > > > > > > > up
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > then doing a redirect.
> > > > > > > > > > > > > > - If we want to get fancy, we could do a reverse IP
> > > > > lookup
> > > > > > > and
> > > > > > > > > try
> > > > > > > > > > to
> > > > > > > > > > > > > match
> > > > > > > > > > > > > > the requester with their closest geographical
> > mirror.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thoughts?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > *Will STEVENS*
> > > > > > > > > > > > > > Lead Developer
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal
> > <
> > > > > > > > > > > > chirade...@gmail.com
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > What steps are needed to set up a mirror? What
> > does
> > > > > Infra
> > > > > > > > need
> > > > > > > > > to
> > > > > > > > > > > do?
> > > > > > > > > > > > > Has
> > > > > > > > > > > > > > > anybody filed a ticket with Infra?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
> > > > > > > > > > > > > > > raja.pull...@accelerite.com>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi will,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > I believe, we didn’t get to close ‘getting a
> > mirror
> > > > > on
> > > > > > > > > Apache’
> > > > > > > > > > > > > because
> > > > > > > > > > > > > > we

Re: Modern template hosting

[Chiradeep Vittal]

gt; > > > > try
> > > > > > > > > to
> > > > > > > > > > > > match
> > > > > > > > > > > > > the requester with their closest geographical
> mirror.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thoughts?
> > > > > > > > > > > > >
> > > > > > > > > > > > > *Will STEVENS*
> > > > > > > > > > > > > Lead Developer
> > > > > > > > > > > > >
> > > > > > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal
> <
> > > > > > > > > > > chirade...@gmail.com
> > > > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > What steps are needed to set up a mirror? What
> does
> > > > Infra
> > > > > > > need
> > > > > > > > to
> > > > > > > > > > do?
> > > > > > > > > > > > Has
> > > > > > > > > > > > > > anybody filed a ticket with Infra?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
> > > > > > > > > > > > > > raja.pull...@accelerite.com>
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi will,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > I believe, we didn’t get to close ‘getting a
> mirror
> > > > on
> > > > > > > > Apache’
> > > > > > > > > > > > because
> > > > > > > > > > > > > we
> > > > > > > > > > > > > > > needed someone on the Apache Infra side to
> close
> > > > this.
> > > > > > BTW,
> > > > > > > > > > > > > > > cloudstack-apt.get.eu (I think Nux manages
> this?)
> > > > has
> > > > > > > > all/most
> > > > > > > > > > of
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > content.  Once we can close on the Apache
> mirror
> > > for
> > > > > > > hosting
> > > > > > > > > the
> > > > > > > > > > > > > > content, I
> > > > > > > > > > > > > > > can help assist getting the content there.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > For now, we have replicated the
> download.cloud.com
> > > > > > content
> > > > > > > > to
> > > > > > > > > ‘
> > > > > > > > > > > > > > > s3.download.accelerite.com’.
> > > > > > > > > > > > > > > Also, we are working on a set of
> steps/procedure to
> > > > > help
> > > > > > > with
> > > > > > > > > > this
> > > > > > > > > > > > > > > change.  I will update everyone in about a
> week’s
> > > > time
> > > > > on
> > > > > > > the
> > > > > > > > > > > > details.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Best,
> > > > > > > > > > > > > > > Raja Pullela
> > > > > > > > > > > > > > > Engineering Team,
> > > > > > > > > > > > > > 

Re: Modern template hosting

[Wido den Hollander]

gt; > > > > consists
> > > > > > > of
> > > > > > > > a
> > > > > > > > > > > single
> > > > > > > > > > > > file with a list of active/supported mirrors.
> > > > > > > > > > > > - I write a small web server, distributed as a binary,
> > > > which
> > > > > > can
> > > > > > > be
> > > > > > > > > > > hosted
> > > > > > > > > > > > by ASF Infra.  This web server will query the current
> > > list
> > > > of
> > > > > > > > mirrors
> > > > > > > > > > and
> > > > > > > > > > > > will select one and then do a 302 redirect to that
> > > mirror.
> > > > > > > > > > > >
> > > > > > > > > > > > The act of 'choosing' a mirror could be done in a
> > number
> > > of
> > > > > > ways.
> > > > > > > > > > > > - If we want to define an order, then it could just try
> > > > from
> > > > > > the
> > > > > > > > top
> > > > > > > > > of
> > > > > > > > > > > the
> > > > > > > > > > > > list and work its way down.  It would curl the target
> > to
> > > > make
> > > > > > > sure
> > > > > > > > it
> > > > > > > > > > > gets
> > > > > > > > > > > > a 200 and if it does, it would do a 302 redirect.
> > > > > > > > > > > > - Or, if we want to distribute the load across the
> > > mirrors,
> > > > > we
> > > > > > > > could
> > > > > > > > > > pick
> > > > > > > > > > > > from the list randomly.  Again, doing a curl to verify
> > > the
> > > > > > mirror
> > > > > > > > is
> > > > > > > > > up
> > > > > > > > > > > and
> > > > > > > > > > > > then doing a redirect.
> > > > > > > > > > > > - If we want to get fancy, we could do a reverse IP
> > > lookup
> > > > > and
> > > > > > > try
> > > > > > > > to
> > > > > > > > > > > match
> > > > > > > > > > > > the requester with their closest geographical mirror.
> > > > > > > > > > > >
> > > > > > > > > > > > Thoughts?
> > > > > > > > > > > >
> > > > > > > > > > > > *Will STEVENS*
> > > > > > > > > > > > Lead Developer
> > > > > > > > > > > >
> > > > > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal <
> > > > > > > > > > chirade...@gmail.com
> > > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > What steps are needed to set up a mirror? What does
> > > Infra
> > > > > > need
> > > > > > > to
> > > > > > > > > do?
> > > > > > > > > > > Has
> > > > > > > > > > > > > anybody filed a ticket with Infra?
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
> > > > > > > > > > > > > raja.pull...@accelerite.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi will,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I believe, we didn’t get to close ‘getting a mirror

Re: Modern template hosting

[Rajani Karuturi]

Pierre-Luc did an amazing job in getting
https://builds.cloudstack.org/ working [1].

We can work on similar lines and probably get
packages.cloudstack.org working with mirrored entries form
apt-get.eu or openvm.eu or s3.downloads.accelerite or
packages.shapeblue etc.

[1] https://issues.apache.org/jira/browse/INFRA-11903

Thanks,

~ Rajani

http://cloudplatform.accelerite.com/

On February 28, 2017 at 2:13 PM, Erik Weber (terbol...@gmail.com)
wrote:

On Tue, Feb 28, 2017 at 8:59 AM, Daan Hoogland
 wrote:

On Tue, Feb 28, 2017 at 2:40 AM, Pierre-Luc Dion
 wrote:

For our systemvm templates, can we just change the URL for
somthing like
cloudstack.apache.org/systemvm/... ?

I agree,
is there any change we can host a domain for this at apache, if
we
can't keep downloads.cloud.com then maybe
downloads.cloudstack.com. To
many people are used to this this way of working and a patch for
a
name change is the easiest. Devising a new mechanism is nice but
a
different job, is it?

packages.apache.org exists.

Has been discussed before in a thread called 'Package
Repositories'.

One thing that should be focused on is that if we go the ASF
hosted
route, we should be able to update the content without involving
ASF
Infra team.

--
Erik

Re: Modern template hosting

[Erik Weber]

On Tue, Feb 28, 2017 at 8:59 AM, Daan Hoogland  wrote:
> On Tue, Feb 28, 2017 at 2:40 AM, Pierre-Luc Dion  wrote:
>> For our systemvm templates, can we just change the URL for somthing like
>> cloudstack.apache.org/systemvm/... ?
>
> I agree,
> is there any change we can host a domain for this at apache, if we
> can't keep downloads.cloud.com then maybe downloads.cloudstack.com. To
> many people are used to this this way of working and a patch for a
> name change is the easiest. Devising a new mechanism is nice but a
> different job, is it?

packages.apache.org exists.

Has been discussed before in a thread called 'Package Repositories'.

One thing that should be focused on is that if we go the ASF hosted
route, we should be able to update the content without involving ASF
Infra team.

-- 
Erik

Re: Modern template hosting

[Daan Hoogland]

On Tue, Feb 28, 2017 at 2:40 AM, Pierre-Luc Dion  wrote:
> For our systemvm templates, can we just change the URL for somthing like
> cloudstack.apache.org/systemvm/... ?

I agree,
is there any change we can host a domain for this at apache, if we
can't keep downloads.cloud.com then maybe downloads.cloudstack.com. To
many people are used to this this way of working and a patch for a
name change is the easiest. Devising a new mechanism is nice but a
different job, is it?

-- 
Daan

Re: Modern template hosting

[Chiradeep Vittal]

; > > > > - We setup a new github repo in the 'apache' org
> which
> > > > > consists
> > > > > > > of
> > > > > > > > a
> > > > > > > > > > > single
> > > > > > > > > > > > file with a list of active/supported mirrors.
> > > > > > > > > > > > - I write a small web server, distributed as a
> binary,
> > > > which
> > > > > > can
> > > > > > > be
> > > > > > > > > > > hosted
> > > > > > > > > > > > by ASF Infra.  This web server will query the current
> > > list
> > > > of
> > > > > > > > mirrors
> > > > > > > > > > and
> > > > > > > > > > > > will select one and then do a 302 redirect to that
> > > mirror.
> > > > > > > > > > > >
> > > > > > > > > > > > The act of 'choosing' a mirror could be done in a
> > number
> > > of
> > > > > > ways.
> > > > > > > > > > > > - If we want to define an order, then it could just
> try
> > > > from
> > > > > > the
> > > > > > > > top
> > > > > > > > > of
> > > > > > > > > > > the
> > > > > > > > > > > > list and work its way down.  It would curl the target
> > to
> > > > make
> > > > > > > sure
> > > > > > > > it
> > > > > > > > > > > gets
> > > > > > > > > > > > a 200 and if it does, it would do a 302 redirect.
> > > > > > > > > > > > - Or, if we want to distribute the load across the
> > > mirrors,
> > > > > we
> > > > > > > > could
> > > > > > > > > > pick
> > > > > > > > > > > > from the list randomly.  Again, doing a curl to
> verify
> > > the
> > > > > > mirror
> > > > > > > > is
> > > > > > > > > up
> > > > > > > > > > > and
> > > > > > > > > > > > then doing a redirect.
> > > > > > > > > > > > - If we want to get fancy, we could do a reverse IP
> > > lookup
> > > > > and
> > > > > > > try
> > > > > > > > to
> > > > > > > > > > > match
> > > > > > > > > > > > the requester with their closest geographical mirror.
> > > > > > > > > > > >
> > > > > > > > > > > > Thoughts?
> > > > > > > > > > > >
> > > > > > > > > > > > *Will STEVENS*
> > > > > > > > > > > > Lead Developer
> > > > > > > > > > > >
> > > > > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > > > > >
> > > > > > > > > > > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal <
> > > > > > > > > > chirade...@gmail.com
> > > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > What steps are needed to set up a mirror? What does
> > > Infra
> > > > > > need
> > > > > > > to
> > > > > > > > > do?
> > > > > > > > > > > Has
> > > > > > > > > > > > > anybody filed a ticket with Infra?
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
> > > > > > > > > > > > > raja.pull...@accelerite.com>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi will,
> > > >

Re: Modern template hosting

[Pierre-Luc Dion]

; > > > > > > > > - Or, if we want to distribute the load across the
> > mirrors,
> > > > we
> > > > > > > could
> > > > > > > > > pick
> > > > > > > > > > > from the list randomly.  Again, doing a curl to verify
> > the
> > > > > mirror
> > > > > > > is
> > > > > > > > up
> > > > > > > > > > and
> > > > > > > > > > > then doing a redirect.
> > > > > > > > > > > - If we want to get fancy, we could do a reverse IP
> > lookup
> > > > and
> > > > > > try
> > > > > > > to
> > > > > > > > > > match
> > > > > > > > > > > the requester with their closest geographical mirror.
> > > > > > > > > > >
> > > > > > > > > > > Thoughts?
> > > > > > > > > > >
> > > > > > > > > > > *Will STEVENS*
> > > > > > > > > > > Lead Developer
> > > > > > > > > > >
> > > > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > > > >
> > > > > > > > > > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal <
> > > > > > > > > chirade...@gmail.com
> > > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > What steps are needed to set up a mirror? What does
> > Infra
> > > > > need
> > > > > > to
> > > > > > > > do?
> > > > > > > > > > Has
> > > > > > > > > > > > anybody filed a ticket with Infra?
> > > > > > > > > > > >
> > > > > > > > > > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
> > > > > > > > > > > > raja.pull...@accelerite.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi will,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I believe, we didn’t get to close ‘getting a mirror
> > on
> > > > > > Apache’
> > > > > > > > > > because
> > > > > > > > > > > we
> > > > > > > > > > > > > needed someone on the Apache Infra side to close
> > this.
> > > > BTW,
> > > > > > > > > > > > > cloudstack-apt.get.eu (I think Nux manages this?)
> > has
> > > > > > all/most
> > > > > > > > of
> > > > > > > > > > the
> > > > > > > > > > > > > content.  Once we can close on the Apache mirror
> for
> > > > > hosting
> > > > > > > the
> > > > > > > > > > > > content, I
> > > > > > > > > > > > > can help assist getting the content there.
> > > > > > > > > > > > >
> > > > > > > > > > > > > For now, we have replicated the download.cloud.com
> > > > content
> > > > > > to
> > > > > > > ‘
> > > > > > > > > > > > > s3.download.accelerite.com’.
> > > > > > > > > > > > > Also, we are working on a set of steps/procedure to
> > > help
> > > > > with
> > > > > > > > this
> > > > > > > > > > > > > change.  I will update everyone in about a week’s
> > time
> > > on
> > > > > the
> > > > > > > > > > details.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best,
> > > > > > > > > > > > > Raja Pullela
> > > > > > > > > > > > > Engineering Team,
> > > > > > > > > > > > > Acc

Re: Modern template hosting

[Chiradeep Vittal]

 ticket with Infra?
> > > > > > > > > > >
> > > > > > > > > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
> > > > > > > > > > > raja.pull...@accelerite.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi will,
> > > > > > > > > > > >
> > > > > > > > > > > > I believe, we didn’t get to close ‘getting a mirror
> on
> > > > > Apache’
> > > > > > > > > because
> > > > > > > > > > we
> > > > > > > > > > > > needed someone on the Apache Infra side to close
> this.
> > > BTW,
> > > > > > > > > > > > cloudstack-apt.get.eu (I think Nux manages this?)
> has
> > > > > all/most
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > > > content.  Once we can close on the Apache mirror for
> > > > hosting
> > > > > > the
> > > > > > > > > > > content, I
> > > > > > > > > > > > can help assist getting the content there.
> > > > > > > > > > > >
> > > > > > > > > > > > For now, we have replicated the download.cloud.com
> > > content
> > > > > to
> > > > > > ‘
> > > > > > > > > > > > s3.download.accelerite.com’.
> > > > > > > > > > > > Also, we are working on a set of steps/procedure to
> > help
> > > > with
> > > > > > > this
> > > > > > > > > > > > change.  I will update everyone in about a week’s
> time
> > on
> > > > the
> > > > > > > > > details.
> > > > > > > > > > > >
> > > > > > > > > > > > Best,
> > > > > > > > > > > > Raja Pullela
> > > > > > > > > > > > Engineering Team,
> > > > > > > > > > > > Accelerite, 2055 Laurelwood Road,
> > > > > > > > > > > > Santa Clara, CA, 95054
> > > > > > > > > > > >
> > > > > > > > > > > > On 2/24/17, 11:23 PM, "williamstev...@gmail.com on
> > > behalf
> > > > of
> > > > > > > Will
> > > > > > > > > > > > Stevens" <williamstev...@gmail.com on behalf of
> > > > > > > > > wstev...@cloudops.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > unfortunately the template mirror conversation got
> > caught
> > > > up
> > > > > in
> > > > > > > > > details
> > > > > > > > > > > and
> > > > > > > > > > > > nobody took the lead on implementing a solution.
> > > > > > > > > > > >
> > > > > > > > > > > > citrix has been pinging me every couple months to say
> > > > 'dude,
> > > > > we
> > > > > > > > need
> > > > > > > > > to
> > > > > > > > > > > > remove the dependency on download.citrix.com', but i
> > > have
> > > > > not
> > > > > > > had
> > > > > > > > > the
> > > > > > > > > > > > cycles to get in and solve the problem.  the shutdown
> > of
> > > > that
> > > > > > is
> > > > > > > > > > imminent
> > > > > > > > > > > > right now, so we need to solve it asap.
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > *Will STEVENS*

Re: Modern template hosting

[Rafael Weingärtner]

> > > > > > > and
> > > > > > > > > will select one and then do a 302 redirect to that mirror.
> > > > > > > > >
> > > > > > > > > The act of 'choosing' a mirror could be done in a number of
> > > ways.
> > > > > > > > > - If we want to define an order, then it could just try
> from
> > > the
> > > > > top
> > > > > > of
> > > > > > > > the
> > > > > > > > > list and work its way down.  It would curl the target to
> make
> > > > sure
> > > > > it
> > > > > > > > gets
> > > > > > > > > a 200 and if it does, it would do a 302 redirect.
> > > > > > > > > - Or, if we want to distribute the load across the mirrors,
> > we
> > > > > could
> > > > > > > pick
> > > > > > > > > from the list randomly.  Again, doing a curl to verify the
> > > mirror
> > > > > is
> > > > > > up
> > > > > > > > and
> > > > > > > > > then doing a redirect.
> > > > > > > > > - If we want to get fancy, we could do a reverse IP lookup
> > and
> > > > try
> > > > > to
> > > > > > > > match
> > > > > > > > > the requester with their closest geographical mirror.
> > > > > > > > >
> > > > > > > > > Thoughts?
> > > > > > > > >
> > > > > > > > > *Will STEVENS*
> > > > > > > > > Lead Developer
> > > > > > > > >
> > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > >
> > > > > > > > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal <
> > > > > > > chirade...@gmail.com
> > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > What steps are needed to set up a mirror? What does Infra
> > > need
> > > > to
> > > > > > do?
> > > > > > > > Has
> > > > > > > > > > anybody filed a ticket with Infra?
> > > > > > > > > >
> > > > > > > > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
> > > > > > > > > > raja.pull...@accelerite.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi will,
> > > > > > > > > > >
> > > > > > > > > > > I believe, we didn’t get to close ‘getting a mirror on
> > > > Apache’
> > > > > > > > because
> > > > > > > > > we
> > > > > > > > > > > needed someone on the Apache Infra side to close this.
> > BTW,
> > > > > > > > > > > cloudstack-apt.get.eu (I think Nux manages this?) has
> > > > all/most
> > > > > > of
> > > > > > > > the
> > > > > > > > > > > content.  Once we can close on the Apache mirror for
> > > hosting
> > > > > the
> > > > > > > > > > content, I
> > > > > > > > > > > can help assist getting the content there.
> > > > > > > > > > >
> > > > > > > > > > > For now, we have replicated the download.cloud.com
> > content
> > > > to
> > > > > ‘
> > > > > > > > > > > s3.download.accelerite.com’.
> > > > > > > > > > > Also, we are working on a set of steps/procedure to
> help
> > > with
> > > > > > this
> > > > > > > > > > > change.  I will update everyone in about a week’s time
> on
> > > the
> > > > > > > > details.
> > > > > > > > > > >
> > > > > > > > > > > Best,
> > > > > > > > > > > Raja Pullela
> > > > > > > > > > > Engineering Te

Re: Modern template hosting

[Will Stevens]

t; > > > > > > > - If we want to get fancy, we could do a reverse IP lookup
> and
> > > try
> > > > to
> > > > > > > match
> > > > > > > > the requester with their closest geographical mirror.
> > > > > > > >
> > > > > > > > Thoughts?
> > > > > > > >
> > > > > > > > *Will STEVENS*
> > > > > > > > Lead Developer
> > > > > > > >
> > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > >
> > > > > > > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal <
> > > > > > chirade...@gmail.com
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > What steps are needed to set up a mirror? What does Infra
> > need
> > > to
> > > > > do?
> > > > > > > Has
> > > > > > > > > anybody filed a ticket with Infra?
> > > > > > > > >
> > > > > > > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
> > > > > > > > > raja.pull...@accelerite.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi will,
> > > > > > > > > >
> > > > > > > > > > I believe, we didn’t get to close ‘getting a mirror on
> > > Apache’
> > > > > > > because
> > > > > > > > we
> > > > > > > > > > needed someone on the Apache Infra side to close this.
> BTW,
> > > > > > > > > > cloudstack-apt.get.eu (I think Nux manages this?) has
> > > all/most
> > > > > of
> > > > > > > the
> > > > > > > > > > content.  Once we can close on the Apache mirror for
> > hosting
> > > > the
> > > > > > > > > content, I
> > > > > > > > > > can help assist getting the content there.
> > > > > > > > > >
> > > > > > > > > > For now, we have replicated the download.cloud.com
> content
> > > to
> > > > ‘
> > > > > > > > > > s3.download.accelerite.com’.
> > > > > > > > > > Also, we are working on a set of steps/procedure to help
> > with
> > > > > this
> > > > > > > > > > change.  I will update everyone in about a week’s time on
> > the
> > > > > > > details.
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Raja Pullela
> > > > > > > > > > Engineering Team,
> > > > > > > > > > Accelerite, 2055 Laurelwood Road,
> > > > > > > > > > Santa Clara, CA, 95054
> > > > > > > > > >
> > > > > > > > > > On 2/24/17, 11:23 PM, "williamstev...@gmail.com on
> behalf
> > of
> > > > > Will
> > > > > > > > > > Stevens" <williamstev...@gmail.com on behalf of
> > > > > > > wstev...@cloudops.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > unfortunately the template mirror conversation got caught
> > up
> > > in
> > > > > > > details
> > > > > > > > > and
> > > > > > > > > > nobody took the lead on implementing a solution.
> > > > > > > > > >
> > > > > > > > > > citrix has been pinging me every couple months to say
> > 'dude,
> > > we
> > > > > > need
> > > > > > > to
> > > > > > > > > > remove the dependency on download.citrix.com', but i
> have
> > > not
> > > > > had
> > > > > > > the
> > > > > > > > > > cycles to get in and solve the problem.  the shutdown of
> > that
> > > > is
> > > > > > > > imminent
> > > > > > > >

Re: Modern template hosting

[Rafael Weingärtner]

t; > > > > > > >
> > > > > > > > *Will STEVENS*
> > > > > > > > Lead Developer
> > > > > > > >
> > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > >
> > > > > > > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal <
> > > > > > chirade...@gmail.com
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > What steps are needed to set up a mirror? What does Infra
> > need
> > > to
> > > > > do?
> > > > > > > Has
> > > > > > > > > anybody filed a ticket with Infra?
> > > > > > > > >
> > > > > > > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
> > > > > > > > > raja.pull...@accelerite.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi will,
> > > > > > > > > >
> > > > > > > > > > I believe, we didn’t get to close ‘getting a mirror on
> > > Apache’
> > > > > > > because
> > > > > > > > we
> > > > > > > > > > needed someone on the Apache Infra side to close this.
> BTW,
> > > > > > > > > > cloudstack-apt.get.eu (I think Nux manages this?) has
> > > all/most
> > > > > of
> > > > > > > the
> > > > > > > > > > content.  Once we can close on the Apache mirror for
> > hosting
> > > > the
> > > > > > > > > content, I
> > > > > > > > > > can help assist getting the content there.
> > > > > > > > > >
> > > > > > > > > > For now, we have replicated the download.cloud.com
> content
> > > to
> > > > ‘
> > > > > > > > > > s3.download.accelerite.com’.
> > > > > > > > > > Also, we are working on a set of steps/procedure to help
> > with
> > > > > this
> > > > > > > > > > change.  I will update everyone in about a week’s time on
> > the
> > > > > > > details.
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Raja Pullela
> > > > > > > > > > Engineering Team,
> > > > > > > > > > Accelerite, 2055 Laurelwood Road,
> > > > > > > > > > Santa Clara, CA, 95054
> > > > > > > > > >
> > > > > > > > > > On 2/24/17, 11:23 PM, "williamstev...@gmail.com on
> behalf
> > of
> > > > > Will
> > > > > > > > > > Stevens" <williamstev...@gmail.com on behalf of
> > > > > > > wstev...@cloudops.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > unfortunately the template mirror conversation got caught
> > up
> > > in
> > > > > > > details
> > > > > > > > > and
> > > > > > > > > > nobody took the lead on implementing a solution.
> > > > > > > > > >
> > > > > > > > > > citrix has been pinging me every couple months to say
> > 'dude,
> > > we
> > > > > > need
> > > > > > > to
> > > > > > > > > > remove the dependency on download.citrix.com', but i
> have
> > > not
> > > > > had
> > > > > > > the
> > > > > > > > > > cycles to get in and solve the problem.  the shutdown of
> > that
> > > > is
> > > > > > > > imminent
> > > > > > > > > > right now, so we need to solve it asap.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > *Will STEVENS*
> > > > > > > > > > Lead Developer
> > > > > > > > > >
> > &g

Re: Modern template hosting

[Will Stevens]

gt; > > we
> > > > > > > > > needed someone on the Apache Infra side to close this. BTW,
> > > > > > > > > cloudstack-apt.get.eu (I think Nux manages this?) has
> > all/most
> > > > of
> > > > > > the
> > > > > > > > > content.  Once we can close on the Apache mirror for
> hosting
> > > the
> > > > > > > > content, I
> > > > > > > > > can help assist getting the content there.
> > > > > > > > >
> > > > > > > > > For now, we have replicated the download.cloud.com content
> > to
> > > ‘
> > > > > > > > > s3.download.accelerite.com’.
> > > > > > > > > Also, we are working on a set of steps/procedure to help
> with
> > > > this
> > > > > > > > > change.  I will update everyone in about a week’s time on
> the
> > > > > > details.
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Raja Pullela
> > > > > > > > > Engineering Team,
> > > > > > > > > Accelerite, 2055 Laurelwood Road,
> > > > > > > > > Santa Clara, CA, 95054
> > > > > > > > >
> > > > > > > > > On 2/24/17, 11:23 PM, "williamstev...@gmail.com on behalf
> of
> > > > Will
> > > > > > > > > Stevens" <williamstev...@gmail.com on behalf of
> > > > > > wstev...@cloudops.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > unfortunately the template mirror conversation got caught
> up
> > in
> > > > > > details
> > > > > > > > and
> > > > > > > > > nobody took the lead on implementing a solution.
> > > > > > > > >
> > > > > > > > > citrix has been pinging me every couple months to say
> 'dude,
> > we
> > > > > need
> > > > > > to
> > > > > > > > > remove the dependency on download.citrix.com', but i have
> > not
> > > > had
> > > > > > the
> > > > > > > > > cycles to get in and solve the problem.  the shutdown of
> that
> > > is
> > > > > > > imminent
> > > > > > > > > right now, so we need to solve it asap.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > *Will STEVENS*
> > > > > > > > > Lead Developer
> > > > > > > > >
> > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > >
> > > > > > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <
> > > > > > paul.an...@shapeblue.com
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Nathan,
> > > > > > > > > >
> > > > > > > > > > Ideally, if you put the template location in (or use a
> > > template
> > > > > > > defined
> > > > > > > > > > in)  test_data.py then the actual location can be
> > overridden
> > > by
> > > > > > > anyone
> > > > > > > > > > testing.
> > > > > > > > > >
> > > > > > > > > > For Trillian, we've copied all of the templates that
> people
> > > > have
> > > > > > > define
> > > > > > > > > to
> > > > > > > > > > a local repo and then replace the URLs in test_data.py to
> > > > reduce
> > > > > > > > > bandwidth
> > > > > > > > > > use and download times.
> > > > > > > > > >
> > > > > > > > > > Ie:
> > > > > > > > > >
> > > > > > > > > > "bootableIso":
> > > > > > > > > > 

Re: Modern template hosting

[Rafael Weingärtner]

gt;
> > > > > > > > > I believe, we didn’t get to close ‘getting a mirror on
> > Apache’
> > > > > > because
> > > > > > > we
> > > > > > > > > needed someone on the Apache Infra side to close this. BTW,
> > > > > > > > > cloudstack-apt.get.eu (I think Nux manages this?) has
> > all/most
> > > > of
> > > > > > the
> > > > > > > > > content.  Once we can close on the Apache mirror for
> hosting
> > > the
> > > > > > > > content, I
> > > > > > > > > can help assist getting the content there.
> > > > > > > > >
> > > > > > > > > For now, we have replicated the download.cloud.com content
> > to
> > > ‘
> > > > > > > > > s3.download.accelerite.com’.
> > > > > > > > > Also, we are working on a set of steps/procedure to help
> with
> > > > this
> > > > > > > > > change.  I will update everyone in about a week’s time on
> the
> > > > > > details.
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Raja Pullela
> > > > > > > > > Engineering Team,
> > > > > > > > > Accelerite, 2055 Laurelwood Road,
> > > > > > > > > Santa Clara, CA, 95054
> > > > > > > > >
> > > > > > > > > On 2/24/17, 11:23 PM, "williamstev...@gmail.com on behalf
> of
> > > > Will
> > > > > > > > > Stevens" <williamstev...@gmail.com on behalf of
> > > > > > wstev...@cloudops.com>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > unfortunately the template mirror conversation got caught
> up
> > in
> > > > > > details
> > > > > > > > and
> > > > > > > > > nobody took the lead on implementing a solution.
> > > > > > > > >
> > > > > > > > > citrix has been pinging me every couple months to say
> 'dude,
> > we
> > > > > need
> > > > > > to
> > > > > > > > > remove the dependency on download.citrix.com', but i have
> > not
> > > > had
> > > > > > the
> > > > > > > > > cycles to get in and solve the problem.  the shutdown of
> that
> > > is
> > > > > > > imminent
> > > > > > > > > right now, so we need to solve it asap.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > *Will STEVENS*
> > > > > > > > > Lead Developer
> > > > > > > > >
> > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > >
> > > > > > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <
> > > > > > paul.an...@shapeblue.com
> > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Nathan,
> > > > > > > > > >
> > > > > > > > > > Ideally, if you put the template location in (or use a
> > > template
> > > > > > > defined
> > > > > > > > > > in)  test_data.py then the actual location can be
> > overridden
> > > by
> > > > > > > anyone
> > > > > > > > > > testing.
> > > > > > > > > >
> > > > > > > > > > For Trillian, we've copied all of the templates that
> people
> > > > have
> > > > > > > define
> > > > > > > > > to
> > > > > > > > > > a local repo and then replace the URLs in test_data.py to
> > > > reduce
> > > > > > > > > bandwidth
> > > > > > > > > > use and download times.
> > > > > > > > > >
> > > > > > > > > > Ie:
> > > > > > > > > >
> > > > > > > >

Re: Modern template hosting

[Chiradeep Vittal]

 > > >
> > > > > > > > unfortunately the template mirror conversation got caught up
> in
> > > > > details
> > > > > > > and
> > > > > > > > nobody took the lead on implementing a solution.
> > > > > > > >
> > > > > > > > citrix has been pinging me every couple months to say 'dude,
> we
> > > > need
> > > > > to
> > > > > > > > remove the dependency on download.citrix.com', but i have
> not
> > > had
> > > > > the
> > > > > > > > cycles to get in and solve the problem.  the shutdown of that
> > is
> > > > > > imminent
> > > > > > > > right now, so we need to solve it asap.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > *Will STEVENS*
> > > > > > > > Lead Developer
> > > > > > > >
> > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > >
> > > > > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <
> > > > > paul.an...@shapeblue.com
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Nathan,
> > > > > > > > >
> > > > > > > > > Ideally, if you put the template location in (or use a
> > template
> > > > > > defined
> > > > > > > > > in)  test_data.py then the actual location can be
> overridden
> > by
> > > > > > anyone
> > > > > > > > > testing.
> > > > > > > > >
> > > > > > > > > For Trillian, we've copied all of the templates that people
> > > have
> > > > > > define
> > > > > > > > to
> > > > > > > > > a local repo and then replace the URLs in test_data.py to
> > > reduce
> > > > > > > > bandwidth
> > > > > > > > > use and download times.
> > > > > > > > >
> > > > > > > > > Ie:
> > > > > > > > >
> > > > > > > > > "bootableIso":
> > > > > > > > > {
> > > > > > > > > "displaytext": "Test Bootable ISO",
> > > > > > > > > "name": "testISO",
> > > > > > > > > "bootable": True,
> > > > > > > > > "ispublic": False,
> > > > > > > > > "url": "{{ marvin_images_location
> > > > > > > > > }}/TinyCore-current.iso",
> > > > > > > > > "ostype": 'Other Linux (64-bit)',
> > > > > > > > > "mode": 'HTTP_DOWNLOAD'
> > > > > > > > > },
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > I thought that we had come up with a solution for
> > > > > download.cloud.com
> > > > > > ,
> > > > > > > by
> > > > > > > > > having a mirrorlist hosted in Community Apache 'space' with
> > > > anyone
> > > > > > able
> > > > > > > > to
> > > > > > > > > out themselves forward as a mirror.
> > > > > > > > > But I must admit I lost track of whether anyone made the
> > > > requisite
> > > > > > > > changes
> > > > > > > > > in code
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Kind regards,
> > > > > > > > >
> > > > > > > > > Paul Angus
> > > > > > > > >
> > > > > > > > > paul.an...@shapeblue.com
> > > > > > > > > www.shapeblue.com
> > > > > > > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > > > > > > @shapeblue
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > -Original Message-
> > > > > > > > > From: williamstev...@gmail.com [mailto:
> williamstevens@gmail.
> > > com]
> > > > > On
> > > > > > > > > Behalf Of Will Stevens
> > > > > > > > > Sent: 24 February 2017 16:30
> > > > > > > > > To: dev@cloudstack.apache.org
> > > > > > > > > Subject: Re: Modern template hosting
> > > > > > > > >
> > > > > > > > > this is a hard questions.  in general, we should be setting
> > up
> > > a
> > > > > > mirror
> > > > > > > > on
> > > > > > > > > some cloudstack/apache domain and then mirror to other
> > provided
> > > > > > > > templates.
> > > > > > > > >
> > > > > > > > > we MUST come up with a solution to deprecate '
> > > download.cloud.com
> > > > ',
> > > > > > > that
> > > > > > > > > is going to be going away any day now.
> > > > > > > > >
> > > > > > > > > i don't know the right way to solve this to be honest, but
> if
> > > you
> > > > > > have
> > > > > > > > > ideas, i am willing to help.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > *Will STEVENS*
> > > > > > > > > Lead Developer
> > > > > > > > >
> > > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > > >
> > > > > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <
> > > > njohn...@ena.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > So not to re-open a can of worms, but I’m in a situation
> > > where
> > > > I
> > > > > > need
> > > > > > > > > > to come up with a Marvin component test that depends on a
> > > > > template
> > > > > > > > > > based on a kernel that’s relatively new, i.e., newer than
> > > > Centos
> > > > > > 5.3
> > > > > > > /
> > > > > > > > > Ubuntu 10.04 .
> > > > > > > > > > I see openvm.eu has a suitable template (Ubuntu 16.0.4
> for
> > > > KVM),
> > > > > > but
> > > > > > > > > > from looking at the thread "Migrating CloudStack content
> > from
> > > > > > > > > > download.cloud.com” it looks like there is resistance to
> > > using
> > > > > > this
> > > > > > > at
> > > > > > > > > > least for hosting system vm templates over concerns of
> > > > > neutrality.
> > > > > > > > > > Would this be suitable for a component test?  If not,
> what
> > > is a
> > > > > > > > “blessed”
> > > > > > > > > > template location?
> > > > > > > > > >
> > > > > > > > > > Thanks in advance!
> > > > > > > > > >
> > > > > > > > > > Nathan Johnson
> > > > > > > > > > R Engineer
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 618 Grassmere Park Drive, Suite 12
> > > > > > > > > > Nashville, TN 37211
> > > > > > > > > > General Office: 615-312-6000
> > > > > > > > > >
> > > > > > > > > > website | blog | support
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > DISCLAIMER
> > > > > > > > ==
> > > > > > > > This e-mail may contain privileged and confidential
> information
> > > > which
> > > > > > is
> > > > > > > > the property of Accelerite, a Persistent Systems business. It
> > is
> > > > > > intended
> > > > > > > > only for the use of the individual or entity to which it is
> > > > > addressed.
> > > > > > If
> > > > > > > > you are not the intended recipient, you are not authorized to
> > > read,
> > > > > > > retain,
> > > > > > > > copy, print, distribute or use this message. If you have
> > received
> > > > > this
> > > > > > > > communication in error, please notify the sender and delete
> all
> > > > > copies
> > > > > > of
> > > > > > > > this message. Accelerite, a Persistent Systems business does
> > not
> > > > > accept
> > > > > > > any
> > > > > > > > liability for virus infected mails.
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
>
> --
> Rafael Weingärtner
>

Re: Modern template hosting

[Chiradeep Vittal]

to verify the mirror
> is
> > up
> > > > and
> > > > > then doing a redirect.
> > > > > - If we want to get fancy, we could do a reverse IP lookup and try
> to
> > > > match
> > > > > the requester with their closest geographical mirror.
> > > > >
> > > > > Thoughts?
> > > > >
> > > > > *Will STEVENS*
> > > > > Lead Developer
> > > > >
> > > > > <https://goo.gl/NYZ8KK>
> > > > >
> > > > > On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal <
> > > chirade...@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > What steps are needed to set up a mirror? What does Infra need to
> > do?
> > > > Has
> > > > > > anybody filed a ticket with Infra?
> > > > > >
> > > > > > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
> > > > > > raja.pull...@accelerite.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi will,
> > > > > > >
> > > > > > > I believe, we didn’t get to close ‘getting a mirror on Apache’
> > > > because
> > > > > we
> > > > > > > needed someone on the Apache Infra side to close this. BTW,
> > > > > > > cloudstack-apt.get.eu (I think Nux manages this?) has all/most
> > of
> > > > the
> > > > > > > content.  Once we can close on the Apache mirror for hosting
> the
> > > > > > content, I
> > > > > > > can help assist getting the content there.
> > > > > > >
> > > > > > > For now, we have replicated the download.cloud.com content to
> ‘
> > > > > > > s3.download.accelerite.com’.
> > > > > > > Also, we are working on a set of steps/procedure to help with
> > this
> > > > > > > change.  I will update everyone in about a week’s time on the
> > > > details.
> > > > > > >
> > > > > > > Best,
> > > > > > > Raja Pullela
> > > > > > > Engineering Team,
> > > > > > > Accelerite, 2055 Laurelwood Road,
> > > > > > > Santa Clara, CA, 95054
> > > > > > >
> > > > > > > On 2/24/17, 11:23 PM, "williamstev...@gmail.com on behalf of
> > Will
> > > > > > > Stevens" <williamstev...@gmail.com on behalf of
> > > > wstev...@cloudops.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > unfortunately the template mirror conversation got caught up in
> > > > details
> > > > > > and
> > > > > > > nobody took the lead on implementing a solution.
> > > > > > >
> > > > > > > citrix has been pinging me every couple months to say 'dude, we
> > > need
> > > > to
> > > > > > > remove the dependency on download.citrix.com', but i have not
> > had
> > > > the
> > > > > > > cycles to get in and solve the problem.  the shutdown of that
> is
> > > > > imminent
> > > > > > > right now, so we need to solve it asap.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > *Will STEVENS*
> > > > > > > Lead Developer
> > > > > > >
> > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > >
> > > > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <
> > > > paul.an...@shapeblue.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Nathan,
> > > > > > > >
> > > > > > > > Ideally, if you put the template location in (or use a
> template
> > > > > defined
> > > > > > > > in)  test_data.py then the actual location can be overridden
> by
> > > > > anyone
> > > > > > > > testing.
> > > > > > > >
> > > > > > > > For Trillian, we've copied all of the templates that people
> > have
> > > > > define
> > > > >

Re: Modern template hosting

[Rafael Weingärtner]

gt; > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Nathan,
> > > > > > > >
> > > > > > > > Ideally, if you put the template location in (or use a
> template
> > > > > defined
> > > > > > > > in)  test_data.py then the actual location can be overridden
> by
> > > > > anyone
> > > > > > > > testing.
> > > > > > > >
> > > > > > > > For Trillian, we've copied all of the templates that people
> > have
> > > > > define
> > > > > > > to
> > > > > > > > a local repo and then replace the URLs in test_data.py to
> > reduce
> > > > > > > bandwidth
> > > > > > > > use and download times.
> > > > > > > >
> > > > > > > > Ie:
> > > > > > > >
> > > > > > > > "bootableIso":
> > > > > > > > {
> > > > > > > > "displaytext": "Test Bootable ISO",
> > > > > > > > "name": "testISO",
> > > > > > > > "bootable": True,
> > > > > > > > "ispublic": False,
> > > > > > > > "url": "{{ marvin_images_location
> > > > > > > > }}/TinyCore-current.iso",
> > > > > > > > "ostype": 'Other Linux (64-bit)',
> > > > > > > > "mode": 'HTTP_DOWNLOAD'
> > > > > > > > },
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > I thought that we had come up with a solution for
> > > > download.cloud.com
> > > > > ,
> > > > > > by
> > > > > > > > having a mirrorlist hosted in Community Apache 'space' with
> > > anyone
> > > > > able
> > > > > > > to
> > > > > > > > out themselves forward as a mirror.
> > > > > > > > But I must admit I lost track of whether anyone made the
> > > requisite
> > > > > > > changes
> > > > > > > > in code
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > Kind regards,
> > > > > > > >
> > > > > > > > Paul Angus
> > > > > > > >
> > > > > > > > paul.an...@shapeblue.com
> > > > > > > > www.shapeblue.com
> > > > > > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > > > > > @shapeblue
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > -Original Message-
> > > > > > > > From: williamstev...@gmail.com [mailto:williamstevens@gmail.
> > com]
> > > > On
> > > > > > > > Behalf Of Will Stevens
> > > > > > > > Sent: 24 February 2017 16:30
> > > > > > > > To: dev@cloudstack.apache.org
> > > > > > > > Subject: Re: Modern template hosting
> > > > > > > >
> > > > > > > > this is a hard questions.  in general, we should be setting
> up
> > a
> > > > > mirror
> > > > > > > on
> > > > > > > > some cloudstack/apache domain and then mirror to other
> provided
> > > > > > > templates.
> > > > > > > >
> > > > > > > > we MUST come up with a solution to deprecate '
> > download.cloud.com
> > > ',
> > > > > > that
> > > > > > > > is going to be going away any day now.
> > > > > > > >
> > > > > > > > i don't know the right way to solve this to be honest, but if
> > you
> > > > > have
> > > > > > > > ideas, i am willing to help.
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > *Will STEVENS*
> > > > > > > > Lead Developer
> > > > > > > >
> > > > > > > > <https://goo.gl/NYZ8KK>
> > > > > > > >
> > > > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <
> > > njohn...@ena.com
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > So not to re-open a can of worms, but I’m in a situation
> > where
> > > I
> > > > > need
> > > > > > > > > to come up with a Marvin component test that depends on a
> > > > template
> > > > > > > > > based on a kernel that’s relatively new, i.e., newer than
> > > Centos
> > > > > 5.3
> > > > > > /
> > > > > > > > Ubuntu 10.04 .
> > > > > > > > > I see openvm.eu has a suitable template (Ubuntu 16.0.4 for
> > > KVM),
> > > > > but
> > > > > > > > > from looking at the thread "Migrating CloudStack content
> from
> > > > > > > > > download.cloud.com” it looks like there is resistance to
> > using
> > > > > this
> > > > > > at
> > > > > > > > > least for hosting system vm templates over concerns of
> > > > neutrality.
> > > > > > > > > Would this be suitable for a component test?  If not, what
> > is a
> > > > > > > “blessed”
> > > > > > > > > template location?
> > > > > > > > >
> > > > > > > > > Thanks in advance!
> > > > > > > > >
> > > > > > > > > Nathan Johnson
> > > > > > > > > R Engineer
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > 618 Grassmere Park Drive, Suite 12
> > > > > > > > > Nashville, TN 37211
> > > > > > > > > General Office: 615-312-6000
> > > > > > > > >
> > > > > > > > > website | blog | support
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > DISCLAIMER
> > > > > > > ==
> > > > > > > This e-mail may contain privileged and confidential information
> > > which
> > > > > is
> > > > > > > the property of Accelerite, a Persistent Systems business. It
> is
> > > > > intended
> > > > > > > only for the use of the individual or entity to which it is
> > > > addressed.
> > > > > If
> > > > > > > you are not the intended recipient, you are not authorized to
> > read,
> > > > > > retain,
> > > > > > > copy, print, distribute or use this message. If you have
> received
> > > > this
> > > > > > > communication in error, please notify the sender and delete all
> > > > copies
> > > > > of
> > > > > > > this message. Accelerite, a Persistent Systems business does
> not
> > > > accept
> > > > > > any
> > > > > > > liability for virus infected mails.
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>



-- 
Rafael Weingärtner

Re: Modern template hosting

[Will Stevens]

gt; > On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
> > > > > raja.pull...@accelerite.com>
> > > > > wrote:
> > > > >
> > > > > > Hi will,
> > > > > >
> > > > > > I believe, we didn’t get to close ‘getting a mirror on Apache’
> > > because
> > > > we
> > > > > > needed someone on the Apache Infra side to close this. BTW,
> > > > > > cloudstack-apt.get.eu (I think Nux manages this?) has all/most
> of
> > > the
> > > > > > content.  Once we can close on the Apache mirror for hosting the
> > > > > content, I
> > > > > > can help assist getting the content there.
> > > > > >
> > > > > > For now, we have replicated the download.cloud.com content to ‘
> > > > > > s3.download.accelerite.com’.
> > > > > > Also, we are working on a set of steps/procedure to help with
> this
> > > > > > change.  I will update everyone in about a week’s time on the
> > > details.
> > > > > >
> > > > > > Best,
> > > > > > Raja Pullela
> > > > > > Engineering Team,
> > > > > > Accelerite, 2055 Laurelwood Road,
> > > > > > Santa Clara, CA, 95054
> > > > > >
> > > > > > On 2/24/17, 11:23 PM, "williamstev...@gmail.com on behalf of
> Will
> > > > > > Stevens" <williamstev...@gmail.com on behalf of
> > > wstev...@cloudops.com>
> > > > > > wrote:
> > > > > >
> > > > > > unfortunately the template mirror conversation got caught up in
> > > details
> > > > > and
> > > > > > nobody took the lead on implementing a solution.
> > > > > >
> > > > > > citrix has been pinging me every couple months to say 'dude, we
> > need
> > > to
> > > > > > remove the dependency on download.citrix.com', but i have not
> had
> > > the
> > > > > > cycles to get in and solve the problem.  the shutdown of that is
> > > > imminent
> > > > > > right now, so we need to solve it asap.
> > > > > >
> > > > > >
> > > > > >
> > > > > > *Will STEVENS*
> > > > > > Lead Developer
> > > > > >
> > > > > > <https://goo.gl/NYZ8KK>
> > > > > >
> > > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <
> > > paul.an...@shapeblue.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Nathan,
> > > > > > >
> > > > > > > Ideally, if you put the template location in (or use a template
> > > > defined
> > > > > > > in)  test_data.py then the actual location can be overridden by
> > > > anyone
> > > > > > > testing.
> > > > > > >
> > > > > > > For Trillian, we've copied all of the templates that people
> have
> > > > define
> > > > > > to
> > > > > > > a local repo and then replace the URLs in test_data.py to
> reduce
> > > > > > bandwidth
> > > > > > > use and download times.
> > > > > > >
> > > > > > > Ie:
> > > > > > >
> > > > > > > "bootableIso":
> > > > > > > {
> > > > > > > "displaytext": "Test Bootable ISO",
> > > > > > > "name": "testISO",
> > > > > > > "bootable": True,
> > > > > > > "ispublic": False,
> > > > > > > "url": "{{ marvin_images_location
> > > > > > > }}/TinyCore-current.iso",
> > > > > > > "ostype": 'Other Linux (64-bit)',
> > > > > > > "mode": 'HTTP_DOWNLOAD'
> > > > > > > },
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > I thought that we had come up

Re: Modern template hosting

[Chiradeep Vittal]

; >
> > > > >
> > > > >
> > > > > *Will STEVENS*
> > > > > Lead Developer
> > > > >
> > > > > <https://goo.gl/NYZ8KK>
> > > > >
> > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <
> > paul.an...@shapeblue.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Nathan,
> > > > > >
> > > > > > Ideally, if you put the template location in (or use a template
> > > defined
> > > > > > in)  test_data.py then the actual location can be overridden by
> > > anyone
> > > > > > testing.
> > > > > >
> > > > > > For Trillian, we've copied all of the templates that people have
> > > define
> > > > > to
> > > > > > a local repo and then replace the URLs in test_data.py to reduce
> > > > > bandwidth
> > > > > > use and download times.
> > > > > >
> > > > > > Ie:
> > > > > >
> > > > > > "bootableIso":
> > > > > > {
> > > > > > "displaytext": "Test Bootable ISO",
> > > > > > "name": "testISO",
> > > > > > "bootable": True,
> > > > > > "ispublic": False,
> > > > > > "url": "{{ marvin_images_location
> > > > > > }}/TinyCore-current.iso",
> > > > > > "ostype": 'Other Linux (64-bit)',
> > > > > > "mode": 'HTTP_DOWNLOAD'
> > > > > > },
> > > > > >
> > > > > >
> > > > > >
> > > > > > I thought that we had come up with a solution for
> > download.cloud.com
> > > ,
> > > > by
> > > > > > having a mirrorlist hosted in Community Apache 'space' with
> anyone
> > > able
> > > > > to
> > > > > > out themselves forward as a mirror.
> > > > > > But I must admit I lost track of whether anyone made the
> requisite
> > > > > changes
> > > > > > in code
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > Kind regards,
> > > > > >
> > > > > > Paul Angus
> > > > > >
> > > > > > paul.an...@shapeblue.com
> > > > > > www.shapeblue.com
> > > > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > > > @shapeblue
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > -Original Message-
> > > > > > From: williamstev...@gmail.com [mailto:williamstev...@gmail.com]
> > On
> > > > > > Behalf Of Will Stevens
> > > > > > Sent: 24 February 2017 16:30
> > > > > > To: dev@cloudstack.apache.org
> > > > > > Subject: Re: Modern template hosting
> > > > > >
> > > > > > this is a hard questions.  in general, we should be setting up a
> > > mirror
> > > > > on
> > > > > > some cloudstack/apache domain and then mirror to other provided
> > > > > templates.
> > > > > >
> > > > > > we MUST come up with a solution to deprecate 'download.cloud.com
> ',
> > > > that
> > > > > > is going to be going away any day now.
> > > > > >
> > > > > > i don't know the right way to solve this to be honest, but if you
> > > have
> > > > > > ideas, i am willing to help.
> > > > > >
> > > > > >
> > > > > >
> > > > > > *Will STEVENS*
> > > > > > Lead Developer
> > > > > >
> > > > > > <https://goo.gl/NYZ8KK>
> > > > > >
> > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <
> njohn...@ena.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > So not to re-open a can of worms, but I’m in a situation where
> I
> > > need
> > > > > > > to come up with a Marvin component test that depends on a
> > template
> > > > > > > based on a kernel that’s relatively new, i.e., newer than
> Centos
> > > 5.3
> > > > /
> > > > > > Ubuntu 10.04 .
> > > > > > > I see openvm.eu has a suitable template (Ubuntu 16.0.4 for
> KVM),
> > > but
> > > > > > > from looking at the thread "Migrating CloudStack content from
> > > > > > > download.cloud.com” it looks like there is resistance to using
> > > this
> > > > at
> > > > > > > least for hosting system vm templates over concerns of
> > neutrality.
> > > > > > > Would this be suitable for a component test?  If not, what is a
> > > > > “blessed”
> > > > > > > template location?
> > > > > > >
> > > > > > > Thanks in advance!
> > > > > > >
> > > > > > > Nathan Johnson
> > > > > > > R Engineer
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > 618 Grassmere Park Drive, Suite 12
> > > > > > > Nashville, TN 37211
> > > > > > > General Office: 615-312-6000
> > > > > > >
> > > > > > > website | blog | support
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > DISCLAIMER
> > > > > ==
> > > > > This e-mail may contain privileged and confidential information
> which
> > > is
> > > > > the property of Accelerite, a Persistent Systems business. It is
> > > intended
> > > > > only for the use of the individual or entity to which it is
> > addressed.
> > > If
> > > > > you are not the intended recipient, you are not authorized to read,
> > > > retain,
> > > > > copy, print, distribute or use this message. If you have received
> > this
> > > > > communication in error, please notify the sender and delete all
> > copies
> > > of
> > > > > this message. Accelerite, a Persistent Systems business does not
> > accept
> > > > any
> > > > > liability for virus infected mails.
> > > > >
> > > >
> > >
> >
>

Re: Modern template hosting

[Chiradeep Vittal]

gt; > > > cycles to get in and solve the problem.  the shutdown of that is
> > > imminent
> > > > > right now, so we need to solve it asap.
> > > > >
> > > > >
> > > > >
> > > > > *Will STEVENS*
> > > > > Lead Developer
> > > > >
> > > > > <https://goo.gl/NYZ8KK>
> > > > >
> > > > > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <
> > paul.an...@shapeblue.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Hi Nathan,
> > > > > >
> > > > > > Ideally, if you put the template location in (or use a template
> > > defined
> > > > > > in)  test_data.py then the actual location can be overridden by
> > > anyone
> > > > > > testing.
> > > > > >
> > > > > > For Trillian, we've copied all of the templates that people have
> > > define
> > > > > to
> > > > > > a local repo and then replace the URLs in test_data.py to reduce
> > > > > bandwidth
> > > > > > use and download times.
> > > > > >
> > > > > > Ie:
> > > > > >
> > > > > > "bootableIso":
> > > > > > {
> > > > > > "displaytext": "Test Bootable ISO",
> > > > > > "name": "testISO",
> > > > > > "bootable": True,
> > > > > > "ispublic": False,
> > > > > > "url": "{{ marvin_images_location
> > > > > > }}/TinyCore-current.iso",
> > > > > > "ostype": 'Other Linux (64-bit)',
> > > > > > "mode": 'HTTP_DOWNLOAD'
> > > > > > },
> > > > > >
> > > > > >
> > > > > >
> > > > > > I thought that we had come up with a solution for
> > download.cloud.com
> > > ,
> > > > by
> > > > > > having a mirrorlist hosted in Community Apache 'space' with
> anyone
> > > able
> > > > > to
> > > > > > out themselves forward as a mirror.
> > > > > > But I must admit I lost track of whether anyone made the
> requisite
> > > > > changes
> > > > > > in code
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > Kind regards,
> > > > > >
> > > > > > Paul Angus
> > > > > >
> > > > > > paul.an...@shapeblue.com
> > > > > > www.shapeblue.com
> > > > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > > > @shapeblue
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > -Original Message-
> > > > > > From: williamstev...@gmail.com [mailto:williamstev...@gmail.com]
> > On
> > > > > > Behalf Of Will Stevens
> > > > > > Sent: 24 February 2017 16:30
> > > > > > To: dev@cloudstack.apache.org
> > > > > > Subject: Re: Modern template hosting
> > > > > >
> > > > > > this is a hard questions.  in general, we should be setting up a
> > > mirror
> > > > > on
> > > > > > some cloudstack/apache domain and then mirror to other provided
> > > > > templates.
> > > > > >
> > > > > > we MUST come up with a solution to deprecate 'download.cloud.com
> ',
> > > > that
> > > > > > is going to be going away any day now.
> > > > > >
> > > > > > i don't know the right way to solve this to be honest, but if you
> > > have
> > > > > > ideas, i am willing to help.
> > > > > >
> > > > > >
> > > > > >
> > > > > > *Will STEVENS*
> > > > > > Lead Developer
> > > > > >
> > > > > > <https://goo.gl/NYZ8KK>
> > > > > >
> > > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <
> njohn...@ena.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > So not to re-open a can of worms, but I’m in a situation where
> I
> > > need
> > > > > > > to come up with a Marvin component test that depends on a
> > template
> > > > > > > based on a kernel that’s relatively new, i.e., newer than
> Centos
> > > 5.3
> > > > /
> > > > > > Ubuntu 10.04 .
> > > > > > > I see openvm.eu has a suitable template (Ubuntu 16.0.4 for
> KVM),
> > > but
> > > > > > > from looking at the thread "Migrating CloudStack content from
> > > > > > > download.cloud.com” it looks like there is resistance to using
> > > this
> > > > at
> > > > > > > least for hosting system vm templates over concerns of
> > neutrality.
> > > > > > > Would this be suitable for a component test?  If not, what is a
> > > > > “blessed”
> > > > > > > template location?
> > > > > > >
> > > > > > > Thanks in advance!
> > > > > > >
> > > > > > > Nathan Johnson
> > > > > > > R Engineer
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > 618 Grassmere Park Drive, Suite 12
> > > > > > > Nashville, TN 37211
> > > > > > > General Office: 615-312-6000
> > > > > > >
> > > > > > > website | blog | support
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > DISCLAIMER
> > > > > ==
> > > > > This e-mail may contain privileged and confidential information
> which
> > > is
> > > > > the property of Accelerite, a Persistent Systems business. It is
> > > intended
> > > > > only for the use of the individual or entity to which it is
> > addressed.
> > > If
> > > > > you are not the intended recipient, you are not authorized to read,
> > > > retain,
> > > > > copy, print, distribute or use this message. If you have received
> > this
> > > > > communication in error, please notify the sender and delete all
> > copies
> > > of
> > > > > this message. Accelerite, a Persistent Systems business does not
> > accept
> > > > any
> > > > > liability for virus infected mails.
> > > > >
> > > >
> > >
> >
>

Re: Modern template hosting

[Will Stevens]

> > > For Trillian, we've copied all of the templates that people have
> > define
> > > > to
> > > > > a local repo and then replace the URLs in test_data.py to reduce
> > > > bandwidth
> > > > > use and download times.
> > > > >
> > > > > Ie:
> > > > >
> > > > > "bootableIso":
> > > > > {
> > > > > "displaytext": "Test Bootable ISO",
> > > > > "name": "testISO",
> > > > >     "bootable": True,
> > > > > "ispublic": False,
> > > > > "url": "{{ marvin_images_location
> > > > > }}/TinyCore-current.iso",
> > > > > "ostype": 'Other Linux (64-bit)',
> > > > > "mode": 'HTTP_DOWNLOAD'
> > > > > },
> > > > >
> > > > >
> > > > >
> > > > > I thought that we had come up with a solution for
> download.cloud.com
> > ,
> > > by
> > > > > having a mirrorlist hosted in Community Apache 'space' with anyone
> > able
> > > > to
> > > > > out themselves forward as a mirror.
> > > > > But I must admit I lost track of whether anyone made the requisite
> > > > changes
> > > > > in code
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Kind regards,
> > > > >
> > > > > Paul Angus
> > > > >
> > > > > paul.an...@shapeblue.com
> > > > > www.shapeblue.com
> > > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > > @shapeblue
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > -Original Message-
> > > > > From: williamstev...@gmail.com [mailto:williamstev...@gmail.com]
> On
> > > > > Behalf Of Will Stevens
> > > > > Sent: 24 February 2017 16:30
> > > > > To: dev@cloudstack.apache.org
> > > > > Subject: Re: Modern template hosting
> > > > >
> > > > > this is a hard questions.  in general, we should be setting up a
> > mirror
> > > > on
> > > > > some cloudstack/apache domain and then mirror to other provided
> > > > templates.
> > > > >
> > > > > we MUST come up with a solution to deprecate 'download.cloud.com',
> > > that
> > > > > is going to be going away any day now.
> > > > >
> > > > > i don't know the right way to solve this to be honest, but if you
> > have
> > > > > ideas, i am willing to help.
> > > > >
> > > > >
> > > > >
> > > > > *Will STEVENS*
> > > > > Lead Developer
> > > > >
> > > > > <https://goo.gl/NYZ8KK>
> > > > >
> > > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <njohn...@ena.com
> >
> > > > wrote:
> > > > >
> > > > > > So not to re-open a can of worms, but I’m in a situation where I
> > need
> > > > > > to come up with a Marvin component test that depends on a
> template
> > > > > > based on a kernel that’s relatively new, i.e., newer than Centos
> > 5.3
> > > /
> > > > > Ubuntu 10.04 .
> > > > > > I see openvm.eu has a suitable template (Ubuntu 16.0.4 for KVM),
> > but
> > > > > > from looking at the thread "Migrating CloudStack content from
> > > > > > download.cloud.com” it looks like there is resistance to using
> > this
> > > at
> > > > > > least for hosting system vm templates over concerns of
> neutrality.
> > > > > > Would this be suitable for a component test?  If not, what is a
> > > > “blessed”
> > > > > > template location?
> > > > > >
> > > > > > Thanks in advance!
> > > > > >
> > > > > > Nathan Johnson
> > > > > > R Engineer
> > > > > >
> > > > > >
> > > > > >
> > > > > > 618 Grassmere Park Drive, Suite 12
> > > > > > Nashville, TN 37211
> > > > > > General Office: 615-312-6000
> > > > > >
> > > > > > website | blog | support
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > DISCLAIMER
> > > > ==
> > > > This e-mail may contain privileged and confidential information which
> > is
> > > > the property of Accelerite, a Persistent Systems business. It is
> > intended
> > > > only for the use of the individual or entity to which it is
> addressed.
> > If
> > > > you are not the intended recipient, you are not authorized to read,
> > > retain,
> > > > copy, print, distribute or use this message. If you have received
> this
> > > > communication in error, please notify the sender and delete all
> copies
> > of
> > > > this message. Accelerite, a Persistent Systems business does not
> accept
> > > any
> > > > liability for virus infected mails.
> > > >
> > >
> >
>

Re: Modern template hosting

[Chiradeep Vittal]

blic": False,
> > > > "url": "{{ marvin_images_location
> > > > }}/TinyCore-current.iso",
> > > > "ostype": 'Other Linux (64-bit)',
> > > > "mode": 'HTTP_DOWNLOAD'
> > > > },
> > > >
> > > >
> > > >
> > > > I thought that we had come up with a solution for download.cloud.com
> ,
> > by
> > > > having a mirrorlist hosted in Community Apache 'space' with anyone
> able
> > > to
> > > > out themselves forward as a mirror.
> > > > But I must admit I lost track of whether anyone made the requisite
> > > changes
> > > > in code
> > > >
> > > >
> > > >
> > > >
> > > > Kind regards,
> > > >
> > > > Paul Angus
> > > >
> > > > paul.an...@shapeblue.com
> > > > www.shapeblue.com
> > > > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > > > @shapeblue
> > > >
> > > >
> > > >
> > > >
> > > > -Original Message-
> > > > From: williamstev...@gmail.com [mailto:williamstev...@gmail.com] On
> > > > Behalf Of Will Stevens
> > > > Sent: 24 February 2017 16:30
> > > > To: dev@cloudstack.apache.org
> > > > Subject: Re: Modern template hosting
> > > >
> > > > this is a hard questions.  in general, we should be setting up a
> mirror
> > > on
> > > > some cloudstack/apache domain and then mirror to other provided
> > > templates.
> > > >
> > > > we MUST come up with a solution to deprecate 'download.cloud.com',
> > that
> > > > is going to be going away any day now.
> > > >
> > > > i don't know the right way to solve this to be honest, but if you
> have
> > > > ideas, i am willing to help.
> > > >
> > > >
> > > >
> > > > *Will STEVENS*
> > > > Lead Developer
> > > >
> > > > <https://goo.gl/NYZ8KK>
> > > >
> > > > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <njohn...@ena.com>
> > > wrote:
> > > >
> > > > > So not to re-open a can of worms, but I’m in a situation where I
> need
> > > > > to come up with a Marvin component test that depends on a template
> > > > > based on a kernel that’s relatively new, i.e., newer than Centos
> 5.3
> > /
> > > > Ubuntu 10.04 .
> > > > > I see openvm.eu has a suitable template (Ubuntu 16.0.4 for KVM),
> but
> > > > > from looking at the thread "Migrating CloudStack content from
> > > > > download.cloud.com” it looks like there is resistance to using
> this
> > at
> > > > > least for hosting system vm templates over concerns of neutrality.
> > > > > Would this be suitable for a component test?  If not, what is a
> > > “blessed”
> > > > > template location?
> > > > >
> > > > > Thanks in advance!
> > > > >
> > > > > Nathan Johnson
> > > > > R Engineer
> > > > >
> > > > >
> > > > >
> > > > > 618 Grassmere Park Drive, Suite 12
> > > > > Nashville, TN 37211
> > > > > General Office: 615-312-6000
> > > > >
> > > > > website | blog | support
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > DISCLAIMER
> > > ==
> > > This e-mail may contain privileged and confidential information which
> is
> > > the property of Accelerite, a Persistent Systems business. It is
> intended
> > > only for the use of the individual or entity to which it is addressed.
> If
> > > you are not the intended recipient, you are not authorized to read,
> > retain,
> > > copy, print, distribute or use this message. If you have received this
> > > communication in error, please notify the sender and delete all copies
> of
> > > this message. Accelerite, a Persistent Systems business does not accept
> > any
> > > liability for virus infected mails.
> > >
> >
>

Re: Modern template hosting

[Will Stevens]

We haven't opened a ticket yet because we don't have a strategy yet.

What do you guys think of this:
- We setup a new github repo in the 'apache' org which consists of a single
file with a list of active/supported mirrors.
- I write a small web server, distributed as a binary, which can be hosted
by ASF Infra.  This web server will query the current list of mirrors and
will select one and then do a 302 redirect to that mirror.

The act of 'choosing' a mirror could be done in a number of ways.
- If we want to define an order, then it could just try from the top of the
list and work its way down.  It would curl the target to make sure it gets
a 200 and if it does, it would do a 302 redirect.
- Or, if we want to distribute the load across the mirrors, we could pick
from the list randomly.  Again, doing a curl to verify the mirror is up and
then doing a redirect.
- If we want to get fancy, we could do a reverse IP lookup and try to match
the requester with their closest geographical mirror.

Thoughts?

*Will STEVENS*
Lead Developer

<https://goo.gl/NYZ8KK>

On Mon, Feb 27, 2017 at 12:46 PM, Chiradeep Vittal <chirade...@gmail.com>
wrote:

> What steps are needed to set up a mirror? What does Infra need to do? Has
> anybody filed a ticket with Infra?
>
> On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <
> raja.pull...@accelerite.com>
> wrote:
>
> > Hi will,
> >
> > I believe, we didn’t get to close ‘getting a mirror on Apache’ because we
> > needed someone on the Apache Infra side to close this. BTW,
> > cloudstack-apt.get.eu (I think Nux manages this?) has all/most of the
> > content.  Once we can close on the Apache mirror for hosting the
> content, I
> > can help assist getting the content there.
> >
> > For now, we have replicated the download.cloud.com content to ‘
> > s3.download.accelerite.com’.
> > Also, we are working on a set of steps/procedure to help with this
> > change.  I will update everyone in about a week’s time on the details.
> >
> > Best,
> > Raja Pullela
> > Engineering Team,
> > Accelerite, 2055 Laurelwood Road,
> > Santa Clara, CA, 95054
> >
> > On 2/24/17, 11:23 PM, "williamstev...@gmail.com on behalf of Will
> > Stevens" <williamstev...@gmail.com on behalf of wstev...@cloudops.com>
> > wrote:
> >
> > unfortunately the template mirror conversation got caught up in details
> and
> > nobody took the lead on implementing a solution.
> >
> > citrix has been pinging me every couple months to say 'dude, we need to
> > remove the dependency on download.citrix.com', but i have not had the
> > cycles to get in and solve the problem.  the shutdown of that is imminent
> > right now, so we need to solve it asap.
> >
> >
> >
> > *Will STEVENS*
> > Lead Developer
> >
> > <https://goo.gl/NYZ8KK>
> >
> > On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <paul.an...@shapeblue.com>
> > wrote:
> >
> > > Hi Nathan,
> > >
> > > Ideally, if you put the template location in (or use a template defined
> > > in)  test_data.py then the actual location can be overridden by anyone
> > > testing.
> > >
> > > For Trillian, we've copied all of the templates that people have define
> > to
> > > a local repo and then replace the URLs in test_data.py to reduce
> > bandwidth
> > > use and download times.
> > >
> > > Ie:
> > >
> > > "bootableIso":
> > > {
> > > "displaytext": "Test Bootable ISO",
> > > "name": "testISO",
> > > "bootable": True,
> > > "ispublic": False,
> > > "url": "{{ marvin_images_location
> > > }}/TinyCore-current.iso",
> > > "ostype": 'Other Linux (64-bit)',
> > > "mode": 'HTTP_DOWNLOAD'
> > > },
> > >
> > >
> > >
> > > I thought that we had come up with a solution for download.cloud.com,
> by
> > > having a mirrorlist hosted in Community Apache 'space' with anyone able
> > to
> > > out themselves forward as a mirror.
> > > But I must admit I lost track of whether anyone made the requisite
> > changes
> > > in code
> > >
> > >
> > >
> > >
> > > Kind regards,
> > >
> > > Paul Angus
> > >
> > > paul.

Re: Modern template hosting

[Chiradeep Vittal]

What steps are needed to set up a mirror? What does Infra need to do? Has
anybody filed a ticket with Infra?

On Sun, Feb 26, 2017 at 10:17 PM, Raja Pullela <raja.pull...@accelerite.com>
wrote:

> Hi will,
>
> I believe, we didn’t get to close ‘getting a mirror on Apache’ because we
> needed someone on the Apache Infra side to close this. BTW,
> cloudstack-apt.get.eu (I think Nux manages this?) has all/most of the
> content.  Once we can close on the Apache mirror for hosting the content, I
> can help assist getting the content there.
>
> For now, we have replicated the download.cloud.com content to ‘
> s3.download.accelerite.com’.
> Also, we are working on a set of steps/procedure to help with this
> change.  I will update everyone in about a week’s time on the details.
>
> Best,
> Raja Pullela
> Engineering Team,
> Accelerite, 2055 Laurelwood Road,
> Santa Clara, CA, 95054
>
> On 2/24/17, 11:23 PM, "williamstev...@gmail.com on behalf of Will
> Stevens" <williamstev...@gmail.com on behalf of wstev...@cloudops.com>
> wrote:
>
> unfortunately the template mirror conversation got caught up in details and
> nobody took the lead on implementing a solution.
>
> citrix has been pinging me every couple months to say 'dude, we need to
> remove the dependency on download.citrix.com', but i have not had the
> cycles to get in and solve the problem.  the shutdown of that is imminent
> right now, so we need to solve it asap.
>
>
>
> *Will STEVENS*
> Lead Developer
>
> <https://goo.gl/NYZ8KK>
>
> On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <paul.an...@shapeblue.com>
> wrote:
>
> > Hi Nathan,
> >
> > Ideally, if you put the template location in (or use a template defined
> > in)  test_data.py then the actual location can be overridden by anyone
> > testing.
> >
> > For Trillian, we've copied all of the templates that people have define
> to
> > a local repo and then replace the URLs in test_data.py to reduce
> bandwidth
> > use and download times.
> >
> > Ie:
> >
> > "bootableIso":
> > {
> > "displaytext": "Test Bootable ISO",
> > "name": "testISO",
> > "bootable": True,
> > "ispublic": False,
> > "url": "{{ marvin_images_location
> > }}/TinyCore-current.iso",
> > "ostype": 'Other Linux (64-bit)',
> > "mode": 'HTTP_DOWNLOAD'
> > },
> >
> >
> >
> > I thought that we had come up with a solution for download.cloud.com, by
> > having a mirrorlist hosted in Community Apache 'space' with anyone able
> to
> > out themselves forward as a mirror.
> > But I must admit I lost track of whether anyone made the requisite
> changes
> > in code
> >
> >
> >
> >
> > Kind regards,
> >
> > Paul Angus
> >
> > paul.an...@shapeblue.com
> > www.shapeblue.com
> > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > @shapeblue
> >
> >
> >
> >
> > -Original Message-
> > From: williamstev...@gmail.com [mailto:williamstev...@gmail.com] On
> > Behalf Of Will Stevens
> > Sent: 24 February 2017 16:30
> > To: dev@cloudstack.apache.org
> > Subject: Re: Modern template hosting
> >
> > this is a hard questions.  in general, we should be setting up a mirror
> on
> > some cloudstack/apache domain and then mirror to other provided
> templates.
> >
> > we MUST come up with a solution to deprecate 'download.cloud.com', that
> > is going to be going away any day now.
> >
> > i don't know the right way to solve this to be honest, but if you have
> > ideas, i am willing to help.
> >
> >
> >
> > *Will STEVENS*
> > Lead Developer
> >
> > <https://goo.gl/NYZ8KK>
> >
> > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <njohn...@ena.com>
> wrote:
> >
> > > So not to re-open a can of worms, but I’m in a situation where I need
> > > to come up with a Marvin component test that depends on a template
> > > based on a kernel that’s relatively new, i.e., newer than Centos 5.3 /
> > Ubuntu 10.04 .
> > > I see openvm.eu has a suitable template (Ubuntu 16.0.4 for KVM), but
> > > from looking at the thread "Migrating CloudStack content from
> > > download.cloud.com” it looks like there is resistance to 

Re: Modern template hosting

[Nathan Johnson]

Paul Angus  wrote:

> Hi Nathan,
>
> Ideally, if you put the template location in (or use a template defined  
> in)  test_data.py then the actual location can be overridden by anyone  
> testing.
>
> For Trillian, we've copied all of the templates that people have define  
> to a local repo and then replace the URLs in test_data.py to reduce  
> bandwidth use and download times.
>
> Ie:
>
> "bootableIso":
> {
> "displaytext": "Test Bootable ISO",
> "name": "testISO",
> "bootable": True,
> "ispublic": False,
> "url": "{{ marvin_images_location 
> }}/TinyCore-current.iso",
> "ostype": 'Other Linux (64-bit)',
> "mode": 'HTTP_DOWNLOAD'
> },
>


Thanks for the tip, this could work pretty well for us I think.

Re: Modern template hosting

[Raja Pullela]

sorry Wido… forgot… thank you for correcting!

On 2/27/17, 4:50 PM, "Wido den Hollander" <w...@widodh.nl> wrote:
> Op 27 februari 2017 om 7:17 schreef Raja Pullela 
> <raja.pull...@accelerite.com>:
> 
> 
> Hi will, 
> 
> I believe, we didn’t get to close ‘getting a mirror on Apache’ because we 
> needed someone on the Apache Infra side to close this. BTW, 
> cloudstack-apt.get.eu (I think Nux manages this?) has all/most of the 
> content.  Once we can close on the Apache mirror for hosting the content, I 
> can help assist getting the content there.
> 

cloudstack.apt-get.eu is managed by me.

> For now, we have replicated the download.cloud.com content to 
> ‘s3.download.accelerite.com’.
> Also, we are working on a set of steps/procedure to help with this change.  I 
> will update everyone in about a week’s time on the details.
> 

Thanks! Although we should probably refrain for using a non-generic domain name 
in any of the source.

Wido

> Best,
> Raja Pullela
> Engineering Team,
> Accelerite, 2055 Laurelwood Road,
> Santa Clara, CA, 95054
> 
> On 2/24/17, 11:23 PM, "williamstev...@gmail.com on behalf of Will Stevens" 
> <williamstev...@gmail.com on behalf of wstev...@cloudops.com> wrote:
> 
> unfortunately the template mirror conversation got caught up in details and
> nobody took the lead on implementing a solution.
> 
> citrix has been pinging me every couple months to say 'dude, we need to
> remove the dependency on download.citrix.com', but i have not had the
> cycles to get in and solve the problem.  the shutdown of that is imminent
> right now, so we need to solve it asap.
> 
> 
> 
> *Will STEVENS*
> Lead Developer
> 
> <https://goo.gl/NYZ8KK>
> 
> On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <paul.an...@shapeblue.com>
> wrote:
> 
> > Hi Nathan,
> >
> > Ideally, if you put the template location in (or use a template defined
> > in)  test_data.py then the actual location can be overridden by anyone
> > testing.
> >
> > For Trillian, we've copied all of the templates that people have define to
> > a local repo and then replace the URLs in test_data.py to reduce bandwidth
> > use and download times.
> >
> > Ie:
> >
> > "bootableIso":
> > {
> > "displaytext": "Test Bootable ISO",
> > "name": "testISO",
> > "bootable": True,
> > "ispublic": False,
> > "url": "{{ marvin_images_location
> > }}/TinyCore-current.iso",
> > "ostype": 'Other Linux (64-bit)',
> > "mode": 'HTTP_DOWNLOAD'
> > },
> >
> >
> >
> > I thought that we had come up with a solution for download.cloud.com, by
> > having a mirrorlist hosted in Community Apache 'space' with anyone able to
> > out themselves forward as a mirror.
> > But I must admit I lost track of whether anyone made the requisite changes
> > in code
> >
> >
> >
> >
> > Kind regards,
> >
> > Paul Angus
> >
> > paul.an...@shapeblue.com
> > www.shapeblue.com
> > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > @shapeblue
> >
> >
> >
> >
> > -Original Message-
> > From: williamstev...@gmail.com [mailto:williamstev...@gmail.com] On
> > Behalf Of Will Stevens
> > Sent: 24 February 2017 16:30
> > To: dev@cloudstack.apache.org
> > Subject: Re: Modern template hosting
> >
> > this is a hard questions.  in general, we should be setting up a mirror on
> > some cloudstack/apache domain and then mirror to other provided templates.
> >
> > we MUST come up with a solution to deprecate 'download.cloud.com', that
> > is going to be going away any day now.
> >
> > i don't know the right way to solve this to be honest, but if you have
> > ideas, i am willing to help.
> >
> >
> >
> > *Will STEVENS*
> > Lead Developer
> >
> > <https://goo.gl/NYZ8KK>
> >
> > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <njohn...@ena.com> wrote:
> >
> > > So not to re-open a can of worms, but I’m in a situation where I need
> > > to come up with a Marvin component test that depends on a template
> > > based on a kernel that’s relatively new, i.e., newer than Centos 5.3 /
> > Ubuntu 10.04 .
> > > I see openvm.eu has a suitable template (Ubuntu 16.0.4

Re: Modern template hosting

[Wido den Hollander]

> Op 27 februari 2017 om 7:17 schreef Raja Pullela 
> <raja.pull...@accelerite.com>:
> 
> 
> Hi will, 
> 
> I believe, we didn’t get to close ‘getting a mirror on Apache’ because we 
> needed someone on the Apache Infra side to close this. BTW, 
> cloudstack-apt.get.eu (I think Nux manages this?) has all/most of the 
> content.  Once we can close on the Apache mirror for hosting the content, I 
> can help assist getting the content there.
> 

cloudstack.apt-get.eu is managed by me.

> For now, we have replicated the download.cloud.com content to 
> ‘s3.download.accelerite.com’.
> Also, we are working on a set of steps/procedure to help with this change.  I 
> will update everyone in about a week’s time on the details.
> 

Thanks! Although we should probably refrain for using a non-generic domain name 
in any of the source.

Wido

> Best,
> Raja Pullela
> Engineering Team,
> Accelerite, 2055 Laurelwood Road,
> Santa Clara, CA, 95054
> 
> On 2/24/17, 11:23 PM, "williamstev...@gmail.com on behalf of Will Stevens" 
> <williamstev...@gmail.com on behalf of wstev...@cloudops.com> wrote:
> 
> unfortunately the template mirror conversation got caught up in details and
> nobody took the lead on implementing a solution.
> 
> citrix has been pinging me every couple months to say 'dude, we need to
> remove the dependency on download.citrix.com', but i have not had the
> cycles to get in and solve the problem.  the shutdown of that is imminent
> right now, so we need to solve it asap.
> 
> 
> 
> *Will STEVENS*
> Lead Developer
> 
> <https://goo.gl/NYZ8KK>
> 
> On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <paul.an...@shapeblue.com>
> wrote:
> 
> > Hi Nathan,
> >
> > Ideally, if you put the template location in (or use a template defined
> > in)  test_data.py then the actual location can be overridden by anyone
> > testing.
> >
> > For Trillian, we've copied all of the templates that people have define to
> > a local repo and then replace the URLs in test_data.py to reduce bandwidth
> > use and download times.
> >
> > Ie:
> >
> > "bootableIso":
> > {
> > "displaytext": "Test Bootable ISO",
> > "name": "testISO",
> > "bootable": True,
> > "ispublic": False,
> > "url": "{{ marvin_images_location
> > }}/TinyCore-current.iso",
> > "ostype": 'Other Linux (64-bit)',
> > "mode": 'HTTP_DOWNLOAD'
> > },
> >
> >
> >
> > I thought that we had come up with a solution for download.cloud.com, by
> > having a mirrorlist hosted in Community Apache 'space' with anyone able to
> > out themselves forward as a mirror.
> > But I must admit I lost track of whether anyone made the requisite changes
> > in code
> >
> >
> >
> >
> > Kind regards,
> >
> > Paul Angus
> >
> > paul.an...@shapeblue.com
> > www.shapeblue.com
> > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > @shapeblue
> >
> >
> >
> >
> > -Original Message-
> > From: williamstev...@gmail.com [mailto:williamstev...@gmail.com] On
> > Behalf Of Will Stevens
> > Sent: 24 February 2017 16:30
> > To: dev@cloudstack.apache.org
> > Subject: Re: Modern template hosting
> >
> > this is a hard questions.  in general, we should be setting up a mirror on
> > some cloudstack/apache domain and then mirror to other provided templates.
> >
> > we MUST come up with a solution to deprecate 'download.cloud.com', that
> > is going to be going away any day now.
> >
> > i don't know the right way to solve this to be honest, but if you have
> > ideas, i am willing to help.
> >
> >
> >
> > *Will STEVENS*
> > Lead Developer
> >
> > <https://goo.gl/NYZ8KK>
> >
> > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <njohn...@ena.com> wrote:
> >
> > > So not to re-open a can of worms, but I’m in a situation where I need
> > > to come up with a Marvin component test that depends on a template
> > > based on a kernel that’s relatively new, i.e., newer than Centos 5.3 /
> > Ubuntu 10.04 .
> > > I see openvm.eu has a suitable template (Ubuntu 16.0.4 for KVM), but
> > > from looking at the thread "Migrating CloudStack content from
> > > download.cloud.com

Re: Modern template hosting

[Nux!]

Chiradeep,

I wouldn't know, needs investigating. Could be that.

--
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro

- Original Message -
> From: "Chiradeep Vittal" <chirade...@gmail.com>
> To: "dev" <dev@cloudstack.apache.org>
> Sent: Sunday, 26 February, 2017 00:07:53
> Subject: Re: Modern template hosting

> Is the issue that ACS doesn't support 169.254.169.254 as the metadata URL?
> 
> For those hypervisors that use the Linux bridge and iptables, it should be 
> easy
> to add a DNAT rule to translate it to the VR IP? I'm talking about code change
> here.
> Or send down a static route in the DHCP response that points 169.254.269.254 
> to
> the VR IP?
> 
> Sent from my iPhone
> 
>> On Feb 25, 2017, at 3:18 AM, Nux! <n...@li.nux.ro> wrote:
>> 
>> Hello,
>> 
>> The upstream Ubuntu and CentOS images will not work out of the box with
>> Cloudstack. I wanted to reuse them for my OpenVM project and failed.
>> 
>> The default metadata source is EC2 or Openstack, don't remember, cloud-init 
>> will
>> take ages to time-out and there is zero support for the root password 
>> feature -
>> in fact the cloud world is moving away from direct root access and so should 
>> we
>> - at least for compatibility reasons if not something else.
>> 
>> I am happy to put some more elbow grease into openvm.eu and make some
>> more/better templates.
>> 
>> I can also host other templates such as the cloud.com ones.
>> 
>> Open to comments, questions etc.
>> 
>> Lucian
>> 
>> --
>> Sent from the Delta quadrant using Borg technology!
>> 
>> Nux!
>> www.nux.ro
>> 
>> - Original Message -
>>> From: "Chiradeep Vittal" <chirade...@gmail.com>
>>> To: "dev" <dev@cloudstack.apache.org>
>>> Sent: Saturday, 25 February, 2017 00:18:48
>>> Subject: Re: Modern template hosting
>> 
>>> Also CentOS KVM images can be obtained here:
>>> http://cloud.centos.org/centos/7/images/
>>> 
>>> If these work, then the only question is where to host the systemvm images.
>>> 
>>> On Fri, Feb 24, 2017 at 4:05 PM, Chiradeep Vittal <chirade...@gmail.com>
>>> wrote:
>>> 
>>>> I noticed that Ubuntu provides daily builds, including VHDs. Perhaps we
>>>> could point the default templates to their URL? Anyone willing to quickly
>>>> try out their images to see if they work on ACS?
>>>> 
>>>> https://cloud-images.ubuntu.com/xenial/current/
>>>> 
>>>> On Fri, Feb 24, 2017 at 9:53 AM, Will Stevens <wstev...@cloudops.com>
>>>> wrote:
>>>> 
>>>>> unfortunately the template mirror conversation got caught up in details
>>>>> and
>>>>> nobody took the lead on implementing a solution.
>>>>> 
>>>>> citrix has been pinging me every couple months to say 'dude, we need to
>>>>> remove the dependency on download.citrix.com', but i have not had the
>>>>> cycles to get in and solve the problem.  the shutdown of that is imminent
>>>>> right now, so we need to solve it asap.
>>>>> 
>>>>> 
>>>>> 
>>>>> *Will STEVENS*
>>>>> Lead Developer
>>>>> 
>>>>> <https://goo.gl/NYZ8KK>
>>>>> 
>>>>> On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <paul.an...@shapeblue.com>
>>>>> wrote:
>>>>> 
>>>>>> Hi Nathan,
>>>>>> 
>>>>>> Ideally, if you put the template location in (or use a template defined
>>>>>> in)  test_data.py then the actual location can be overridden by anyone
>>>>>> testing.
>>>>>> 
>>>>>> For Trillian, we've copied all of the templates that people have define
>>>>> to
>>>>>> a local repo and then replace the URLs in test_data.py to reduce
>>>>> bandwidth
>>>>>> use and download times.
>>>>>> 
>>>>>> Ie:
>>>>>> 
>>>>>>"bootableIso":
>>>>>>{
>>>>>>"displaytext": "Test Bootable ISO",
>>>>>>    "name": "testISO",
>>>>>>"bootable": True,
>>>>>>

Re: Modern template hosting

[Nux!]

Raja,

apt-get.eu is Wido's.

--
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro

- Original Message -
> From: "Raja Pullela" <raja.pull...@accelerite.com>
> To: "dev" <dev@cloudstack.apache.org>
> Sent: Monday, 27 February, 2017 06:17:36
> Subject: Re: Modern template hosting

> Hi will,
> 
> I believe, we didn’t get to close ‘getting a mirror on Apache’ because we 
> needed
> someone on the Apache Infra side to close this. BTW, cloudstack-apt.get.eu (I
> think Nux manages this?) has all/most of the content.  Once we can close on 
> the
> Apache mirror for hosting the content, I can help assist getting the content
> there.
> 
> For now, we have replicated the download.cloud.com content to
> ‘s3.download.accelerite.com’.
> Also, we are working on a set of steps/procedure to help with this change.  I
> will update everyone in about a week’s time on the details.
> 
> Best,
> Raja Pullela
> Engineering Team,
> Accelerite, 2055 Laurelwood Road,
> Santa Clara, CA, 95054
> 
> On 2/24/17, 11:23 PM, "williamstev...@gmail.com on behalf of Will Stevens"
> <williamstev...@gmail.com on behalf of wstev...@cloudops.com> wrote:
> 
> unfortunately the template mirror conversation got caught up in details and
> nobody took the lead on implementing a solution.
> 
> citrix has been pinging me every couple months to say 'dude, we need to
> remove the dependency on download.citrix.com', but i have not had the
> cycles to get in and solve the problem.  the shutdown of that is imminent
> right now, so we need to solve it asap.
> 
> 
> 
> *Will STEVENS*
> Lead Developer
> 
> <https://goo.gl/NYZ8KK>
> 
> On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <paul.an...@shapeblue.com>
> wrote:
> 
>> Hi Nathan,
>>
>> Ideally, if you put the template location in (or use a template defined
>> in)  test_data.py then the actual location can be overridden by anyone
>> testing.
>>
>> For Trillian, we've copied all of the templates that people have define to
>> a local repo and then replace the URLs in test_data.py to reduce bandwidth
>> use and download times.
>>
>> Ie:
>>
>> "bootableIso":
>> {
>> "displaytext": "Test Bootable ISO",
>> "name": "testISO",
>> "bootable": True,
>> "ispublic": False,
>> "url": "{{ marvin_images_location
>> }}/TinyCore-current.iso",
>> "ostype": 'Other Linux (64-bit)',
>> "mode": 'HTTP_DOWNLOAD'
>> },
>>
>>
>>
>> I thought that we had come up with a solution for download.cloud.com, by
>> having a mirrorlist hosted in Community Apache 'space' with anyone able to
>> out themselves forward as a mirror.
>> But I must admit I lost track of whether anyone made the requisite changes
>> in code
>>
>>
>>
>>
>> Kind regards,
>>
>> Paul Angus
>>
>> paul.an...@shapeblue.com
>> www.shapeblue.com
>> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>> @shapeblue
>>
>>
>>
>>
>> -Original Message-
>> From: williamstev...@gmail.com [mailto:williamstev...@gmail.com] On
>> Behalf Of Will Stevens
>> Sent: 24 February 2017 16:30
>> To: dev@cloudstack.apache.org
>> Subject: Re: Modern template hosting
>>
>> this is a hard questions.  in general, we should be setting up a mirror on
>> some cloudstack/apache domain and then mirror to other provided templates.
>>
>> we MUST come up with a solution to deprecate 'download.cloud.com', that
>> is going to be going away any day now.
>>
>> i don't know the right way to solve this to be honest, but if you have
>> ideas, i am willing to help.
>>
>>
>>
>> *Will STEVENS*
>> Lead Developer
>>
>> <https://goo.gl/NYZ8KK>
>>
>> On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <njohn...@ena.com> wrote:
>>
>> > So not to re-open a can of worms, but I’m in a situation where I need
>> > to come up with a Marvin component test that depends on a template
>> > based on a kernel that’s relatively new, i.e., newer than Centos 5.3 /
>> Ubuntu 10.04 .
>> > I see openvm.eu has a suitable template (Ubuntu 16.0.4 for KVM), but
>> > from looking at the thread "Migrating CloudStack content from
>> >

Re: Modern template hosting

[Raja Pullela]

Hi will, 

I believe, we didn’t get to close ‘getting a mirror on Apache’ because we 
needed someone on the Apache Infra side to close this. BTW, 
cloudstack-apt.get.eu (I think Nux manages this?) has all/most of the content.  
Once we can close on the Apache mirror for hosting the content, I can help 
assist getting the content there.

For now, we have replicated the download.cloud.com content to 
‘s3.download.accelerite.com’.
Also, we are working on a set of steps/procedure to help with this change.  I 
will update everyone in about a week’s time on the details.

Best,
Raja Pullela
Engineering Team,
Accelerite, 2055 Laurelwood Road,
Santa Clara, CA, 95054

On 2/24/17, 11:23 PM, "williamstev...@gmail.com on behalf of Will Stevens" 
<williamstev...@gmail.com on behalf of wstev...@cloudops.com> wrote:

unfortunately the template mirror conversation got caught up in details and
nobody took the lead on implementing a solution.

citrix has been pinging me every couple months to say 'dude, we need to
remove the dependency on download.citrix.com', but i have not had the
cycles to get in and solve the problem.  the shutdown of that is imminent
right now, so we need to solve it asap.



*Will STEVENS*
Lead Developer

<https://goo.gl/NYZ8KK>

On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <paul.an...@shapeblue.com>
wrote:

> Hi Nathan,
>
> Ideally, if you put the template location in (or use a template defined
> in)  test_data.py then the actual location can be overridden by anyone
> testing.
>
> For Trillian, we've copied all of the templates that people have define to
> a local repo and then replace the URLs in test_data.py to reduce bandwidth
> use and download times.
>
> Ie:
>
> "bootableIso":
> {
> "displaytext": "Test Bootable ISO",
> "name": "testISO",
> "bootable": True,
> "ispublic": False,
> "url": "{{ marvin_images_location
> }}/TinyCore-current.iso",
> "ostype": 'Other Linux (64-bit)',
> "mode": 'HTTP_DOWNLOAD'
> },
>
>
>
> I thought that we had come up with a solution for download.cloud.com, by
> having a mirrorlist hosted in Community Apache 'space' with anyone able to
> out themselves forward as a mirror.
> But I must admit I lost track of whether anyone made the requisite changes
> in code
>
>
>
>
> Kind regards,
>
> Paul Angus
>
> paul.an...@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>
>
>
>
> -Original Message-
> From: williamstev...@gmail.com [mailto:williamstev...@gmail.com] On
> Behalf Of Will Stevens
> Sent: 24 February 2017 16:30
> To: dev@cloudstack.apache.org
> Subject: Re: Modern template hosting
>
> this is a hard questions.  in general, we should be setting up a mirror on
> some cloudstack/apache domain and then mirror to other provided templates.
>
> we MUST come up with a solution to deprecate 'download.cloud.com', that
> is going to be going away any day now.
>
> i don't know the right way to solve this to be honest, but if you have
> ideas, i am willing to help.
>
>
>
> *Will STEVENS*
> Lead Developer
>
> <https://goo.gl/NYZ8KK>
>
> On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <njohn...@ena.com> wrote:
>
> > So not to re-open a can of worms, but I’m in a situation where I need
> > to come up with a Marvin component test that depends on a template
> > based on a kernel that’s relatively new, i.e., newer than Centos 5.3 /
> Ubuntu 10.04 .
> > I see openvm.eu has a suitable template (Ubuntu 16.0.4 for KVM), but
> > from looking at the thread "Migrating CloudStack content from
> > download.cloud.com” it looks like there is resistance to using this at
> > least for hosting system vm templates over concerns of neutrality.
> > Would this be suitable for a component test?  If not, what is a “blessed”
> > template location?
> >
> > Thanks in advance!
> >
> > Nathan Johnson
> > R Engineer
> >
> >
> >
> > 618 Grassmere Park Drive, Suite 12
> > Nashville, TN 37211
> > General Office: 615-312-6000
> >
> > website | blog | support
> >
> >
> >
> >
> >
> >
> >
>






DISCLAIMER
==
This e-mail may contain privileged and confidential information which is the 
property of Accelerite, a Persistent Systems business. It is intended only for 
the use of the individual or entity to which it is addressed. If you are not 
the intended recipient, you are not authorized to read, retain, copy, print, 
distribute or use this message. If you have received this communication in 
error, please notify the sender and delete all copies of this message. 
Accelerite, a Persistent Systems business does not accept any liability for 
virus infected mails.

Re: Modern template hosting

[Chiradeep Vittal]

Is the issue that ACS doesn't support 169.254.169.254 as the metadata URL?

For those hypervisors that use the Linux bridge and iptables, it should be easy 
to add a DNAT rule to translate it to the VR IP? I'm talking about code change 
here. 
Or send down a static route in the DHCP response that points 169.254.269.254 to 
the VR IP? 

Sent from my iPhone

> On Feb 25, 2017, at 3:18 AM, Nux! <n...@li.nux.ro> wrote:
> 
> Hello,
> 
> The upstream Ubuntu and CentOS images will not work out of the box with 
> Cloudstack. I wanted to reuse them for my OpenVM project and failed.
> 
> The default metadata source is EC2 or Openstack, don't remember, cloud-init 
> will take ages to time-out and there is zero support for the root password 
> feature - in fact the cloud world is moving away from direct root access and 
> so should we - at least for compatibility reasons if not something else.
> 
> I am happy to put some more elbow grease into openvm.eu and make some 
> more/better templates. 
> 
> I can also host other templates such as the cloud.com ones.
> 
> Open to comments, questions etc.
> 
> Lucian
> 
> --
> Sent from the Delta quadrant using Borg technology!
> 
> Nux!
> www.nux.ro
> 
> - Original Message -
>> From: "Chiradeep Vittal" <chirade...@gmail.com>
>> To: "dev" <dev@cloudstack.apache.org>
>> Sent: Saturday, 25 February, 2017 00:18:48
>> Subject: Re: Modern template hosting
> 
>> Also CentOS KVM images can be obtained here:
>> http://cloud.centos.org/centos/7/images/
>> 
>> If these work, then the only question is where to host the systemvm images.
>> 
>> On Fri, Feb 24, 2017 at 4:05 PM, Chiradeep Vittal <chirade...@gmail.com>
>> wrote:
>> 
>>> I noticed that Ubuntu provides daily builds, including VHDs. Perhaps we
>>> could point the default templates to their URL? Anyone willing to quickly
>>> try out their images to see if they work on ACS?
>>> 
>>> https://cloud-images.ubuntu.com/xenial/current/
>>> 
>>> On Fri, Feb 24, 2017 at 9:53 AM, Will Stevens <wstev...@cloudops.com>
>>> wrote:
>>> 
>>>> unfortunately the template mirror conversation got caught up in details
>>>> and
>>>> nobody took the lead on implementing a solution.
>>>> 
>>>> citrix has been pinging me every couple months to say 'dude, we need to
>>>> remove the dependency on download.citrix.com', but i have not had the
>>>> cycles to get in and solve the problem.  the shutdown of that is imminent
>>>> right now, so we need to solve it asap.
>>>> 
>>>> 
>>>> 
>>>> *Will STEVENS*
>>>> Lead Developer
>>>> 
>>>> <https://goo.gl/NYZ8KK>
>>>> 
>>>> On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <paul.an...@shapeblue.com>
>>>> wrote:
>>>> 
>>>>> Hi Nathan,
>>>>> 
>>>>> Ideally, if you put the template location in (or use a template defined
>>>>> in)  test_data.py then the actual location can be overridden by anyone
>>>>> testing.
>>>>> 
>>>>> For Trillian, we've copied all of the templates that people have define
>>>> to
>>>>> a local repo and then replace the URLs in test_data.py to reduce
>>>> bandwidth
>>>>> use and download times.
>>>>> 
>>>>> Ie:
>>>>> 
>>>>>"bootableIso":
>>>>>{
>>>>>"displaytext": "Test Bootable ISO",
>>>>>"name": "testISO",
>>>>>"bootable": True,
>>>>>"ispublic": False,
>>>>>"url": "{{ marvin_images_location
>>>>> }}/TinyCore-current.iso",
>>>>>"ostype": 'Other Linux (64-bit)',
>>>>>"mode": 'HTTP_DOWNLOAD'
>>>>>},
>>>>> 
>>>>> 
>>>>> 
>>>>> I thought that we had come up with a solution for download.cloud.com,
>>>> by
>>>>> having a mirrorlist hosted in Community Apache 'space' with anyone able
>>>> to
>>>>> out themselves forward as a mirror.
>>>>> But I must admit I lost track of whether anyone made the requisite

Re: Modern template hosting

[Nux!]

Hello,

The upstream Ubuntu and CentOS images will not work out of the box with 
Cloudstack. I wanted to reuse them for my OpenVM project and failed.

The default metadata source is EC2 or Openstack, don't remember, cloud-init 
will take ages to time-out and there is zero support for the root password 
feature - in fact the cloud world is moving away from direct root access and so 
should we - at least for compatibility reasons if not something else.

I am happy to put some more elbow grease into openvm.eu and make some 
more/better templates. 

I can also host other templates such as the cloud.com ones.

Open to comments, questions etc.

Lucian

--
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro

- Original Message -
> From: "Chiradeep Vittal" <chirade...@gmail.com>
> To: "dev" <dev@cloudstack.apache.org>
> Sent: Saturday, 25 February, 2017 00:18:48
> Subject: Re: Modern template hosting

> Also CentOS KVM images can be obtained here:
> http://cloud.centos.org/centos/7/images/
> 
> If these work, then the only question is where to host the systemvm images.
> 
> On Fri, Feb 24, 2017 at 4:05 PM, Chiradeep Vittal <chirade...@gmail.com>
> wrote:
> 
>> I noticed that Ubuntu provides daily builds, including VHDs. Perhaps we
>> could point the default templates to their URL? Anyone willing to quickly
>> try out their images to see if they work on ACS?
>>
>> https://cloud-images.ubuntu.com/xenial/current/
>>
>> On Fri, Feb 24, 2017 at 9:53 AM, Will Stevens <wstev...@cloudops.com>
>> wrote:
>>
>>> unfortunately the template mirror conversation got caught up in details
>>> and
>>> nobody took the lead on implementing a solution.
>>>
>>> citrix has been pinging me every couple months to say 'dude, we need to
>>> remove the dependency on download.citrix.com', but i have not had the
>>> cycles to get in and solve the problem.  the shutdown of that is imminent
>>> right now, so we need to solve it asap.
>>>
>>>
>>>
>>> *Will STEVENS*
>>> Lead Developer
>>>
>>> <https://goo.gl/NYZ8KK>
>>>
>>> On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <paul.an...@shapeblue.com>
>>> wrote:
>>>
>>> > Hi Nathan,
>>> >
>>> > Ideally, if you put the template location in (or use a template defined
>>> > in)  test_data.py then the actual location can be overridden by anyone
>>> > testing.
>>> >
>>> > For Trillian, we've copied all of the templates that people have define
>>> to
>>> > a local repo and then replace the URLs in test_data.py to reduce
>>> bandwidth
>>> > use and download times.
>>> >
>>> > Ie:
>>> >
>>> > "bootableIso":
>>> > {
>>> > "displaytext": "Test Bootable ISO",
>>> > "name": "testISO",
>>> > "bootable": True,
>>> > "ispublic": False,
>>> > "url": "{{ marvin_images_location
>>> > }}/TinyCore-current.iso",
>>> > "ostype": 'Other Linux (64-bit)',
>>> > "mode": 'HTTP_DOWNLOAD'
>>> > },
>>> >
>>> >
>>> >
>>> > I thought that we had come up with a solution for download.cloud.com,
>>> by
>>> > having a mirrorlist hosted in Community Apache 'space' with anyone able
>>> to
>>> > out themselves forward as a mirror.
>>> > But I must admit I lost track of whether anyone made the requisite
>>> changes
>>> > in code
>>> >
>>> >
>>> >
>>> >
>>> > Kind regards,
>>> >
>>> > Paul Angus
>>> >
>>> > paul.an...@shapeblue.com
>>> > www.shapeblue.com
>>> > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>>> > @shapeblue
>>> >
>>> >
>>> >
>>> >
>>> > -Original Message-
>>> > From: williamstev...@gmail.com [mailto:williamstev...@gmail.com] On
>>> > Behalf Of Will Stevens
>>> > Sent: 24 February 2017 16:30
>>> > To: dev@cloudstack.apache.org
>>> > Subject: Re: Modern template hosting
>>> >
>>> > this is a hard

Re: Modern template hosting

[Chiradeep Vittal]

Also CentOS KVM images can be obtained here:
http://cloud.centos.org/centos/7/images/

If these work, then the only question is where to host the systemvm images.

On Fri, Feb 24, 2017 at 4:05 PM, Chiradeep Vittal <chirade...@gmail.com>
wrote:

> I noticed that Ubuntu provides daily builds, including VHDs. Perhaps we
> could point the default templates to their URL? Anyone willing to quickly
> try out their images to see if they work on ACS?
>
> https://cloud-images.ubuntu.com/xenial/current/
>
> On Fri, Feb 24, 2017 at 9:53 AM, Will Stevens <wstev...@cloudops.com>
> wrote:
>
>> unfortunately the template mirror conversation got caught up in details
>> and
>> nobody took the lead on implementing a solution.
>>
>> citrix has been pinging me every couple months to say 'dude, we need to
>> remove the dependency on download.citrix.com', but i have not had the
>> cycles to get in and solve the problem.  the shutdown of that is imminent
>> right now, so we need to solve it asap.
>>
>>
>>
>> *Will STEVENS*
>> Lead Developer
>>
>> <https://goo.gl/NYZ8KK>
>>
>> On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <paul.an...@shapeblue.com>
>> wrote:
>>
>> > Hi Nathan,
>> >
>> > Ideally, if you put the template location in (or use a template defined
>> > in)  test_data.py then the actual location can be overridden by anyone
>> > testing.
>> >
>> > For Trillian, we've copied all of the templates that people have define
>> to
>> > a local repo and then replace the URLs in test_data.py to reduce
>> bandwidth
>> > use and download times.
>> >
>> > Ie:
>> >
>> > "bootableIso":
>> > {
>> > "displaytext": "Test Bootable ISO",
>> > "name": "testISO",
>> > "bootable": True,
>> > "ispublic": False,
>> > "url": "{{ marvin_images_location
>> > }}/TinyCore-current.iso",
>> > "ostype": 'Other Linux (64-bit)',
>> > "mode": 'HTTP_DOWNLOAD'
>> > },
>> >
>> >
>> >
>> > I thought that we had come up with a solution for download.cloud.com,
>> by
>> > having a mirrorlist hosted in Community Apache 'space' with anyone able
>> to
>> > out themselves forward as a mirror.
>> > But I must admit I lost track of whether anyone made the requisite
>> changes
>> > in code
>> >
>> >
>> >
>> >
>> > Kind regards,
>> >
>> > Paul Angus
>> >
>> > paul.an...@shapeblue.com
>> > www.shapeblue.com
>> > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
>> > @shapeblue
>> >
>> >
>> >
>> >
>> > -Original Message-
>> > From: williamstev...@gmail.com [mailto:williamstev...@gmail.com] On
>> > Behalf Of Will Stevens
>> > Sent: 24 February 2017 16:30
>> > To: dev@cloudstack.apache.org
>> > Subject: Re: Modern template hosting
>> >
>> > this is a hard questions.  in general, we should be setting up a mirror
>> on
>> > some cloudstack/apache domain and then mirror to other provided
>> templates.
>> >
>> > we MUST come up with a solution to deprecate 'download.cloud.com', that
>> > is going to be going away any day now.
>> >
>> > i don't know the right way to solve this to be honest, but if you have
>> > ideas, i am willing to help.
>> >
>> >
>> >
>> > *Will STEVENS*
>> > Lead Developer
>> >
>> > <https://goo.gl/NYZ8KK>
>> >
>> > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <njohn...@ena.com>
>> wrote:
>> >
>> > > So not to re-open a can of worms, but I’m in a situation where I need
>> > > to come up with a Marvin component test that depends on a template
>> > > based on a kernel that’s relatively new, i.e., newer than Centos 5.3 /
>> > Ubuntu 10.04 .
>> > > I see openvm.eu has a suitable template (Ubuntu 16.0.4 for KVM), but
>> > > from looking at the thread "Migrating CloudStack content from
>> > > download.cloud.com” it looks like there is resistance to using this
>> at
>> > > least for hosting system vm templates over concerns of neutrality.
>> > > Would this be suitable for a component test?  If not, what is a
>> “blessed”
>> > > template location?
>> > >
>> > > Thanks in advance!
>> > >
>> > > Nathan Johnson
>> > > R Engineer
>> > >
>> > >
>> > >
>> > > 618 Grassmere Park Drive, Suite 12
>> > > Nashville, TN 37211
>> > > General Office: 615-312-6000
>> > >
>> > > website | blog | support
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> >
>>
>
>

Re: Modern template hosting

[Chiradeep Vittal]

I noticed that Ubuntu provides daily builds, including VHDs. Perhaps we
could point the default templates to their URL? Anyone willing to quickly
try out their images to see if they work on ACS?

https://cloud-images.ubuntu.com/xenial/current/

On Fri, Feb 24, 2017 at 9:53 AM, Will Stevens <wstev...@cloudops.com> wrote:

> unfortunately the template mirror conversation got caught up in details and
> nobody took the lead on implementing a solution.
>
> citrix has been pinging me every couple months to say 'dude, we need to
> remove the dependency on download.citrix.com', but i have not had the
> cycles to get in and solve the problem.  the shutdown of that is imminent
> right now, so we need to solve it asap.
>
>
>
> *Will STEVENS*
> Lead Developer
>
> <https://goo.gl/NYZ8KK>
>
> On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <paul.an...@shapeblue.com>
> wrote:
>
> > Hi Nathan,
> >
> > Ideally, if you put the template location in (or use a template defined
> > in)  test_data.py then the actual location can be overridden by anyone
> > testing.
> >
> > For Trillian, we've copied all of the templates that people have define
> to
> > a local repo and then replace the URLs in test_data.py to reduce
> bandwidth
> > use and download times.
> >
> > Ie:
> >
> > "bootableIso":
> > {
> > "displaytext": "Test Bootable ISO",
> > "name": "testISO",
> > "bootable": True,
> > "ispublic": False,
> > "url": "{{ marvin_images_location
> > }}/TinyCore-current.iso",
> > "ostype": 'Other Linux (64-bit)',
> > "mode": 'HTTP_DOWNLOAD'
> > },
> >
> >
> >
> > I thought that we had come up with a solution for download.cloud.com, by
> > having a mirrorlist hosted in Community Apache 'space' with anyone able
> to
> > out themselves forward as a mirror.
> > But I must admit I lost track of whether anyone made the requisite
> changes
> > in code
> >
> >
> >
> >
> > Kind regards,
> >
> > Paul Angus
> >
> > paul.an...@shapeblue.com
> > www.shapeblue.com
> > 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> > @shapeblue
> >
> >
> >
> >
> > -Original Message-
> > From: williamstev...@gmail.com [mailto:williamstev...@gmail.com] On
> > Behalf Of Will Stevens
> > Sent: 24 February 2017 16:30
> > To: dev@cloudstack.apache.org
> > Subject: Re: Modern template hosting
> >
> > this is a hard questions.  in general, we should be setting up a mirror
> on
> > some cloudstack/apache domain and then mirror to other provided
> templates.
> >
> > we MUST come up with a solution to deprecate 'download.cloud.com', that
> > is going to be going away any day now.
> >
> > i don't know the right way to solve this to be honest, but if you have
> > ideas, i am willing to help.
> >
> >
> >
> > *Will STEVENS*
> > Lead Developer
> >
> > <https://goo.gl/NYZ8KK>
> >
> > On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <njohn...@ena.com>
> wrote:
> >
> > > So not to re-open a can of worms, but I’m in a situation where I need
> > > to come up with a Marvin component test that depends on a template
> > > based on a kernel that’s relatively new, i.e., newer than Centos 5.3 /
> > Ubuntu 10.04 .
> > > I see openvm.eu has a suitable template (Ubuntu 16.0.4 for KVM), but
> > > from looking at the thread "Migrating CloudStack content from
> > > download.cloud.com” it looks like there is resistance to using this at
> > > least for hosting system vm templates over concerns of neutrality.
> > > Would this be suitable for a component test?  If not, what is a
> “blessed”
> > > template location?
> > >
> > > Thanks in advance!
> > >
> > > Nathan Johnson
> > > R Engineer
> > >
> > >
> > >
> > > 618 Grassmere Park Drive, Suite 12
> > > Nashville, TN 37211
> > > General Office: 615-312-6000
> > >
> > > website | blog | support
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> >
>

Re: Modern template hosting

[Will Stevens]

unfortunately the template mirror conversation got caught up in details and
nobody took the lead on implementing a solution.

citrix has been pinging me every couple months to say 'dude, we need to
remove the dependency on download.citrix.com', but i have not had the
cycles to get in and solve the problem.  the shutdown of that is imminent
right now, so we need to solve it asap.



*Will STEVENS*
Lead Developer

<https://goo.gl/NYZ8KK>

On Fri, Feb 24, 2017 at 12:38 PM, Paul Angus <paul.an...@shapeblue.com>
wrote:

> Hi Nathan,
>
> Ideally, if you put the template location in (or use a template defined
> in)  test_data.py then the actual location can be overridden by anyone
> testing.
>
> For Trillian, we've copied all of the templates that people have define to
> a local repo and then replace the URLs in test_data.py to reduce bandwidth
> use and download times.
>
> Ie:
>
> "bootableIso":
> {
> "displaytext": "Test Bootable ISO",
> "name": "testISO",
> "bootable": True,
> "ispublic": False,
> "url": "{{ marvin_images_location
> }}/TinyCore-current.iso",
> "ostype": 'Other Linux (64-bit)',
> "mode": 'HTTP_DOWNLOAD'
> },
>
>
>
> I thought that we had come up with a solution for download.cloud.com, by
> having a mirrorlist hosted in Community Apache 'space' with anyone able to
> out themselves forward as a mirror.
> But I must admit I lost track of whether anyone made the requisite changes
> in code
>
>
>
>
> Kind regards,
>
> Paul Angus
>
> paul.an...@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK
> @shapeblue
>
>
>
>
> -Original Message-
> From: williamstev...@gmail.com [mailto:williamstev...@gmail.com] On
> Behalf Of Will Stevens
> Sent: 24 February 2017 16:30
> To: dev@cloudstack.apache.org
> Subject: Re: Modern template hosting
>
> this is a hard questions.  in general, we should be setting up a mirror on
> some cloudstack/apache domain and then mirror to other provided templates.
>
> we MUST come up with a solution to deprecate 'download.cloud.com', that
> is going to be going away any day now.
>
> i don't know the right way to solve this to be honest, but if you have
> ideas, i am willing to help.
>
>
>
> *Will STEVENS*
> Lead Developer
>
> <https://goo.gl/NYZ8KK>
>
> On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <njohn...@ena.com> wrote:
>
> > So not to re-open a can of worms, but I’m in a situation where I need
> > to come up with a Marvin component test that depends on a template
> > based on a kernel that’s relatively new, i.e., newer than Centos 5.3 /
> Ubuntu 10.04 .
> > I see openvm.eu has a suitable template (Ubuntu 16.0.4 for KVM), but
> > from looking at the thread "Migrating CloudStack content from
> > download.cloud.com” it looks like there is resistance to using this at
> > least for hosting system vm templates over concerns of neutrality.
> > Would this be suitable for a component test?  If not, what is a “blessed”
> > template location?
> >
> > Thanks in advance!
> >
> > Nathan Johnson
> > R Engineer
> >
> >
> >
> > 618 Grassmere Park Drive, Suite 12
> > Nashville, TN 37211
> > General Office: 615-312-6000
> >
> > website | blog | support
> >
> >
> >
> >
> >
> >
> >
>

RE: Modern template hosting

[Paul Angus]

Hi Nathan,

Ideally, if you put the template location in (or use a template defined in)  
test_data.py then the actual location can be overridden by anyone testing. 

For Trillian, we've copied all of the templates that people have define to a 
local repo and then replace the URLs in test_data.py to reduce bandwidth use 
and download times.

Ie:

"bootableIso":
{
"displaytext": "Test Bootable ISO",
"name": "testISO",
"bootable": True,
"ispublic": False,
"url": "{{ marvin_images_location }}/TinyCore-current.iso",
"ostype": 'Other Linux (64-bit)',
"mode": 'HTTP_DOWNLOAD'
},



I thought that we had come up with a solution for download.cloud.com, by having 
a mirrorlist hosted in Community Apache 'space' with anyone able to out 
themselves forward as a mirror.
But I must admit I lost track of whether anyone made the requisite changes in 
code




Kind regards,

Paul Angus

paul.an...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 


-Original Message-
From: williamstev...@gmail.com [mailto:williamstev...@gmail.com] On Behalf Of 
Will Stevens
Sent: 24 February 2017 16:30
To: dev@cloudstack.apache.org
Subject: Re: Modern template hosting

this is a hard questions.  in general, we should be setting up a mirror on some 
cloudstack/apache domain and then mirror to other provided templates.

we MUST come up with a solution to deprecate 'download.cloud.com', that is 
going to be going away any day now.

i don't know the right way to solve this to be honest, but if you have ideas, i 
am willing to help.



*Will STEVENS*
Lead Developer

<https://goo.gl/NYZ8KK>

On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson <njohn...@ena.com> wrote:

> So not to re-open a can of worms, but I’m in a situation where I need 
> to come up with a Marvin component test that depends on a template 
> based on a kernel that’s relatively new, i.e., newer than Centos 5.3 / Ubuntu 
> 10.04 .
> I see openvm.eu has a suitable template (Ubuntu 16.0.4 for KVM), but 
> from looking at the thread "Migrating CloudStack content from 
> download.cloud.com” it looks like there is resistance to using this at 
> least for hosting system vm templates over concerns of neutrality.  
> Would this be suitable for a component test?  If not, what is a “blessed”
> template location?
>
> Thanks in advance!
>
> Nathan Johnson
> R Engineer
>
>
>
> 618 Grassmere Park Drive, Suite 12
> Nashville, TN 37211
> General Office: 615-312-6000
>
> website | blog | support
>
>
>
>
>
>
>

Re: Modern template hosting

[Will Stevens]

this is a hard questions.  in general, we should be setting up a mirror on
some cloudstack/apache domain and then mirror to other provided templates.

we MUST come up with a solution to deprecate 'download.cloud.com', that is
going to be going away any day now.

i don't know the right way to solve this to be honest, but if you have
ideas, i am willing to help.



*Will STEVENS*
Lead Developer



On Fri, Feb 24, 2017 at 11:25 AM, Nathan Johnson  wrote:

> So not to re-open a can of worms, but I’m in a situation where I need to
> come up with a Marvin component test that depends on a template based on a
> kernel that’s relatively new, i.e., newer than Centos 5.3 / Ubuntu 10.04 .
> I see openvm.eu has a suitable template (Ubuntu 16.0.4 for KVM), but from
> looking at the thread "Migrating CloudStack content from
> download.cloud.com” it looks like there is resistance to using this at
> least for hosting system vm templates over concerns of neutrality.  Would
> this be suitable for a component test?  If not, what is a “blessed”
> template location?
>
> Thanks in advance!
>
> Nathan Johnson
> R Engineer
>
>
>
> 618 Grassmere Park Drive, Suite 12
> Nashville, TN 37211
> General Office: 615-312-6000
>
> website | blog | support
>
>
>
>
>
>
>