Re: Delivery Service Origin Refactor

[Rawlin Peters]

Location and Origin columns were listed out in my more detailed
initial proposal [1]. In there I proposed having either a
DeliveryService FK in the Origin table or a new deliveryservice_origin
table to support many-to-many. But I think keeping the DS FK in the
Origin table is the right thing to do, because we could temporarily
allow duplicate origins (as we do today), give everyone a few releases
to make sure they've fixed all their existing DSes with duplicate
origins, then slap a unique constraint on the Origin fqdn. That would
prevent the erroneous config that occurs from having duplicate origins
like you described.

- Rawlin

[1] 
https://lists.apache.org/thread.html/ed09765621271250893ae3ac1f5b1e984eae0071eab46057b8264c00@%3Cdev.trafficcontrol.apache.org%3E

On Thu, Mar 22, 2018 at 1:31 PM, Eric Friedrich (efriedri)
 wrote:
>
>> On Mar 22, 2018, at 12:27 PM, Rawlin Peters  wrote:
>>
>> This Origin Refactor proposal was probably too much to parse at once.
>> Here's a slightly shorter version:
>> 1. Split Locations out of the Cachegroup table into their own table
> EF> Location is latitude/longitude?
>
>
>> 2. Split Origins out of the Server and Delivery Service tables into
>> their own table (Origin would have a FK to DeliveryService, supporting
>> a one-to-many relationship between DSes and Origins)
> EF> What are the properties of an Origin?
>
> EF> Also, having a separate origin object might let us prevent for the 
> messiness that comes when adding the same origin as both a Live DS and a VOD 
> DS on the same cache (today hosting.config stores both DS’ on disk)
>
>>
>> In order to make the transition smoother, I'd like to try to keep the
>> existing Delivery Service and Cachegroup API request/response formats
>> in place, but their implementations would change to populate the new
>> DB tables. For instance, if I create a DS with the Origin
>> http://www.example.com, the DS API backend would now parse that URL
>> and create a new row in the Origin table with a foreign key to that
>> DS.
>>
>> For the Cachegroup API, if I create a new Cachegroup with the
>> coordinates [1, 2], the backend would then create a new row in the
>> Location table using those coordinates (probably name it generically
>> using the Cachegroup name), and the cachegroup row would contain a
>> foreign key to that Location.
>>
>> Of course there would also be new API endpoints to CRUD Locations and
>> Origins by themselves.
>>
>> Initially, the current MSO implementation would continue to use
>> origins created using the Server table, but we would transition the
>> MSO implementation over to using the new Origin table eventually.
>>
>> Questions/thoughts/concerns about any of this? All feedback is welcome.
>>
>> Thanks,
>> Rawlin
>>
>> On Mon, Mar 12, 2018 at 1:46 PM, Rawlin Peters  
>> wrote:
>>> Hey folks,
>>>
>>> As promised, this email thread will be to discuss how to best
>>> associate an Origin Latitude/Longitude with a Delivery Service,
>>> primarily so that steering targets can be ordered/sent to the client
>>> based upon the location of those targets (i.e. the Origin), a.k.a.
>>> Steering Target Geo-Ordering. This is potentially going to be a pretty
>>> large change, so all your feedback/questions/concerns are appreciated.
>>>
>>> Here were a handful of bad ideas I had in order to accomplish this DS
>>> Origin Lat/Long association (feel free to skip to PROPOSED SOLUTION
>>> below):
>>>
>>> 1. Reuse the current MSO (multisite origin) backend (i.e. add the
>>> origin into the servers table, give it a lat/long from its cachegroup,
>>> assign the origin server to the DS)
>>> Pros:
>>> - reuse of existing db schema, probably wouldn't have to add any new
>>> tables/columns
>>> Cons:
>>> - MSO configuration is already very complex
>>> - for the simple case of just wanting to give an Origin a lat/long you
>>> have to create a server (of which only a few fields make sense for an
>>> Origin), add it to a cachegroup (only name and lat/long make sense,
>>> won't use parent relationships, isn't really a "group" of origins),
>>> assign it to a server profile (have to create one first, no parameters
>>> are needed), and finally assign that Origin server to the delivery
>>> service (did I miss anything?)
>>>
>>> 2. Add Origin lat/long columns to the deliveryservice table
>>> Pros:
>>> - probably the most straightforward solution for Steering Target
>>> Geo-Ordering given that Origin FQDN is currently a DS field.
>>> Cons:
>>> - doesn't work well with MSO
>>> - could be confused with Default Miss Lat/Long
>>> - if two different delivery services use colocated origins, the same
>>> lat/long needs entered twice
>>> - adds yet another column to the crowded deliveryservice table
>>>
>>> 3. Add origin lat/long parameters to a Delivery Service Profile
>>> Pros:
>>> - Delivery Services using colocated origins could share the same profile
>>> - no DB schema updates needed
>>> 

Re: Delivery Service Origin Refactor

[Eric Friedrich (efriedri)]

> On Mar 22, 2018, at 12:27 PM, Rawlin Peters  wrote:
> 
> This Origin Refactor proposal was probably too much to parse at once.
> Here's a slightly shorter version:
> 1. Split Locations out of the Cachegroup table into their own table
EF> Location is latitude/longitude?


> 2. Split Origins out of the Server and Delivery Service tables into
> their own table (Origin would have a FK to DeliveryService, supporting
> a one-to-many relationship between DSes and Origins)
EF> What are the properties of an Origin?

EF> Also, having a separate origin object might let us prevent for the 
messiness that comes when adding the same origin as both a Live DS and a VOD DS 
on the same cache (today hosting.config stores both DS’ on disk)

> 
> In order to make the transition smoother, I'd like to try to keep the
> existing Delivery Service and Cachegroup API request/response formats
> in place, but their implementations would change to populate the new
> DB tables. For instance, if I create a DS with the Origin
> http://www.example.com, the DS API backend would now parse that URL
> and create a new row in the Origin table with a foreign key to that
> DS.
> 
> For the Cachegroup API, if I create a new Cachegroup with the
> coordinates [1, 2], the backend would then create a new row in the
> Location table using those coordinates (probably name it generically
> using the Cachegroup name), and the cachegroup row would contain a
> foreign key to that Location.
> 
> Of course there would also be new API endpoints to CRUD Locations and
> Origins by themselves.
> 
> Initially, the current MSO implementation would continue to use
> origins created using the Server table, but we would transition the
> MSO implementation over to using the new Origin table eventually.
> 
> Questions/thoughts/concerns about any of this? All feedback is welcome.
> 
> Thanks,
> Rawlin
> 
> On Mon, Mar 12, 2018 at 1:46 PM, Rawlin Peters  
> wrote:
>> Hey folks,
>> 
>> As promised, this email thread will be to discuss how to best
>> associate an Origin Latitude/Longitude with a Delivery Service,
>> primarily so that steering targets can be ordered/sent to the client
>> based upon the location of those targets (i.e. the Origin), a.k.a.
>> Steering Target Geo-Ordering. This is potentially going to be a pretty
>> large change, so all your feedback/questions/concerns are appreciated.
>> 
>> Here were a handful of bad ideas I had in order to accomplish this DS
>> Origin Lat/Long association (feel free to skip to PROPOSED SOLUTION
>> below):
>> 
>> 1. Reuse the current MSO (multisite origin) backend (i.e. add the
>> origin into the servers table, give it a lat/long from its cachegroup,
>> assign the origin server to the DS)
>> Pros:
>> - reuse of existing db schema, probably wouldn't have to add any new
>> tables/columns
>> Cons:
>> - MSO configuration is already very complex
>> - for the simple case of just wanting to give an Origin a lat/long you
>> have to create a server (of which only a few fields make sense for an
>> Origin), add it to a cachegroup (only name and lat/long make sense,
>> won't use parent relationships, isn't really a "group" of origins),
>> assign it to a server profile (have to create one first, no parameters
>> are needed), and finally assign that Origin server to the delivery
>> service (did I miss anything?)
>> 
>> 2. Add Origin lat/long columns to the deliveryservice table
>> Pros:
>> - probably the most straightforward solution for Steering Target
>> Geo-Ordering given that Origin FQDN is currently a DS field.
>> Cons:
>> - doesn't work well with MSO
>> - could be confused with Default Miss Lat/Long
>> - if two different delivery services use colocated origins, the same
>> lat/long needs entered twice
>> - adds yet another column to the crowded deliveryservice table
>> 
>> 3. Add origin lat/long parameters to a Delivery Service Profile
>> Pros:
>> - Delivery Services using colocated origins could share the same profile
>> - no DB schema updates needed
>> Cons:
>> - profile parameters lack validation
>> - still doesn't support lat/long for multiple origins associated with a DS
>> 
>> 4. Add the lat/long to the steering target itself (i.e. where you
>> choose weight/order, you'd also enter lat/long)
>> Pros:
>> - probably the easiest/quickest solution in terms of development
>> Cons:
>> - only applies lat/long to a steering target
>> - using the same target in multiple Steering DSes means having to keep
>> the lat/long synced between them all
>> - lat/long not easily reused by other areas that may need it in the future
>> 
>> 
>> 
>> PROPOSED SOLUTION:
>> 
>> All of those ideas were suboptimal, which is why I think we need to:
>> 1. Split Locations out of the cachegroup table into their own table
>> with the following columns (cachegroup would have a foreign key to
>> Location):
>> - name
>> - latitude
>> - longitude
>> 
>> 2. Split Origins out of the server and deliveryservice 

Re: [VOTE] Release Apache Traffic Control (incubating) 2.2.0-RC2

[Robert Butts]

Ah, thanks for the clarification.

Ok, the 2.0.0 RPMs have been removed from dist, and the webpage links
updated to archive.apache.org.

On Mon, Mar 19, 2018 at 5:48 PM, sebb  wrote:

> On 19 March 2018 at 19:01, Robert Butts  wrote:
> > MD5s have been removed.
> >
> > Henk, I'm not sure what you mean about a 2.x branch. We don't have a
> branch
> > named 2.x. We do our primary development in master, then create branches
> > for each release, e.g. 2.0.x, 2.1.x, 2.2.x. Those branches are only ever
> > updated with backported fixes. There's no separate branch for version 2
> > development, just master.
> >
> > Is that ok? Or do we still need to move 2.0.0?
>
> I think Henk meant that the mirrors should only hold the current release.
> At present that is 2.1.0, so 2.0.0 should be removed, and the download
> page updated to refer to the archive server at:
> http://archive.apache.org/dist/incubator/trafficcontrol/2.0.0-incubating/
>
> When 2.2.0 is released, the same should be done for 2.1.0.
> i.e. update the download page to add 2.2.0; change 2.1.0 and any
> earlier links to use the archive server.
> Once the new release has been announced, remove the older releases from
>
> https://dist.apache.org/repos/dist/release/incubator/trafficcontrol/
>
> which will remove them from the ASF mirror system.
>
> >
> > On Mon, Mar 19, 2018 at 11:57 AM, Henk P. Penning  wrote:
> >
> >> On Mon, 19 Mar 2018, Robert Butts wrote:
> >>
> >> Date: Mon, 19 Mar 2018 18:34:22 +0100
> >>> From: Robert Butts 
> >>> To: gene...@incubator.apache.org
> >>> Cc: dev@trafficcontrol.incubator.apache.org
> >>> Subject: [VOTE] Release Apache Traffic Control (incubating) 2.2.0-RC2
> >>>
> >>> Hello Incubator PMC,
> >>>
> >>> The Apache Traffic Control community has voted on and approved a
> >>> proposal to release Apache Traffic Control 2.2-incubating. We now
> >>> kindly request that the Incubator PMC members review and vote on this
> >>> incubator release.
> >>>
> >>
> >> The previous approved releases and release notes (along with links to
> >>> artifacts, signatures/checksums, and updated documentation) can be
> >>> found here:
> >>> http://trafficcontrol.incubator.apache.org/downloads/
> >>>
> >>
> >>   Regarding you download page :
> >>
> >>   -- It appears you have only one development branch (2.x) going ;
> >>  If so, 2.0.0 should not be in /dist/ (and on the mirrors) anymore.
> >>  Please refer to 2.0.0 files as :
> >>
> >>  http://archive.apache.org/dist/incubator/trafficcontrol/2.
> >> 0.0-incubating/
> >>
> >> The source distribution is here:
> >>> https://dist.apache.org/repos/dist/dev/incubator/trafficcont
> >>> rol/2.2.0/RC2/
> >>>
> >>
> >>   Please don't publish .md5's anymore ; remove before moving 2.2.0
> >>   from dist/dev/ to dist/release/.
> >>
> >>   Thanks ; regards,
> >>
> >>   Henk Penning
> >>
> >>    _
> >> Henk P. Penning, ICT-beta R Uithof MG-403_/ \_
> >> Faculty of Science, Utrecht UniversityT +31 30 253 4106 / \_/ \
> >> Leuvenlaan 4, 3584CE Utrecht, NL  F +31 30 253 4553 \_/ \_/
> >> http://www.staff.science.uu.nl/~penni101/ M penn...@uu.nl \_/
> >>
>

Re: Delivery Service Origin Refactor

[Rawlin Peters]

This Origin Refactor proposal was probably too much to parse at once.
Here's a slightly shorter version:
1. Split Locations out of the Cachegroup table into their own table
2. Split Origins out of the Server and Delivery Service tables into
their own table (Origin would have a FK to DeliveryService, supporting
a one-to-many relationship between DSes and Origins)

In order to make the transition smoother, I'd like to try to keep the
existing Delivery Service and Cachegroup API request/response formats
in place, but their implementations would change to populate the new
DB tables. For instance, if I create a DS with the Origin
http://www.example.com, the DS API backend would now parse that URL
and create a new row in the Origin table with a foreign key to that
DS.

For the Cachegroup API, if I create a new Cachegroup with the
coordinates [1, 2], the backend would then create a new row in the
Location table using those coordinates (probably name it generically
using the Cachegroup name), and the cachegroup row would contain a
foreign key to that Location.

Of course there would also be new API endpoints to CRUD Locations and
Origins by themselves.

Initially, the current MSO implementation would continue to use
origins created using the Server table, but we would transition the
MSO implementation over to using the new Origin table eventually.

Questions/thoughts/concerns about any of this? All feedback is welcome.

Thanks,
Rawlin

On Mon, Mar 12, 2018 at 1:46 PM, Rawlin Peters  wrote:
> Hey folks,
>
> As promised, this email thread will be to discuss how to best
> associate an Origin Latitude/Longitude with a Delivery Service,
> primarily so that steering targets can be ordered/sent to the client
> based upon the location of those targets (i.e. the Origin), a.k.a.
> Steering Target Geo-Ordering. This is potentially going to be a pretty
> large change, so all your feedback/questions/concerns are appreciated.
>
> Here were a handful of bad ideas I had in order to accomplish this DS
> Origin Lat/Long association (feel free to skip to PROPOSED SOLUTION
> below):
>
> 1. Reuse the current MSO (multisite origin) backend (i.e. add the
> origin into the servers table, give it a lat/long from its cachegroup,
> assign the origin server to the DS)
> Pros:
> - reuse of existing db schema, probably wouldn't have to add any new
> tables/columns
> Cons:
> - MSO configuration is already very complex
> - for the simple case of just wanting to give an Origin a lat/long you
> have to create a server (of which only a few fields make sense for an
> Origin), add it to a cachegroup (only name and lat/long make sense,
> won't use parent relationships, isn't really a "group" of origins),
> assign it to a server profile (have to create one first, no parameters
> are needed), and finally assign that Origin server to the delivery
> service (did I miss anything?)
>
> 2. Add Origin lat/long columns to the deliveryservice table
> Pros:
> - probably the most straightforward solution for Steering Target
> Geo-Ordering given that Origin FQDN is currently a DS field.
> Cons:
> - doesn't work well with MSO
> - could be confused with Default Miss Lat/Long
> - if two different delivery services use colocated origins, the same
> lat/long needs entered twice
> - adds yet another column to the crowded deliveryservice table
>
> 3. Add origin lat/long parameters to a Delivery Service Profile
> Pros:
> - Delivery Services using colocated origins could share the same profile
> - no DB schema updates needed
> Cons:
> - profile parameters lack validation
> - still doesn't support lat/long for multiple origins associated with a DS
>
> 4. Add the lat/long to the steering target itself (i.e. where you
> choose weight/order, you'd also enter lat/long)
> Pros:
> - probably the easiest/quickest solution in terms of development
> Cons:
> - only applies lat/long to a steering target
> - using the same target in multiple Steering DSes means having to keep
> the lat/long synced between them all
> - lat/long not easily reused by other areas that may need it in the future
>
>
>
> PROPOSED SOLUTION:
>
> All of those ideas were suboptimal, which is why I think we need to:
> 1. Split Locations out of the cachegroup table into their own table
> with the following columns (cachegroup would have a foreign key to
> Location):
> - name
> - latitude
> - longitude
>
> 2. Split Origins out of the server and deliveryservice tables into
> their own table with the following columns:
> - fqdn
> - protocol (http or https)
> - port (optional, can be inferred from protocol)
> - location (optional FK to Location table)
> - deliveryservice FK (if an Origin can only be associated with a
> single DS. Might need step 3 below for many-to-many)
> - ip_address (optional, necessary to support `use_ip_address` profile
> parameter for using the origin's IP address rather than fqdn in
> parent.config)
> - ip6_address (optional, necessary because we'd have an ip_address
>