Re: Backup Cache Group Selection

[Rawlin Peters]

If we start by putting this in the Cache Group API first, then later
we really only have to worry about adding the CIDRs to the API. The
backup config is really just relationships between cache groups, which
makes perfect sense to model in a relational DB rather than the CZF.
Why put something in the CZF to just tear it out later?

- Rawlin

On Mon, Mar 12, 2018 at 3:12 PM, Dave Neuman  wrote:
> Good point Rawlin, but I think it does answer your questions.  CZF for now,
> whatever the new CZF thing is after that.
>
> On Mon, Mar 12, 2018 at 1:44 PM, Rawlin Peters 
> wrote:
>
>> The original scope of this thread was determining where the Backup
>> Cache Group config should live (API vs CZF), not necessarily about
>> building the entire CZF in the database, although I'm +1 on that idea
>> as well. I think any decisions made about doing that should probably
>> be started in a separate thread.
>>
>> - Rawlin
>>
>> On Mon, Mar 12, 2018 at 1:11 PM, Dave Neuman  wrote:
>> > +1 on building the CZF in the database.  Jan tried to go down that rabbit
>> > hole but realized it was a pretty hard problem to solve.  I think this is
>> > something we might want to re-visit.  Maybe this is something we should
>> > discuss at our meetup and then update this thread with our decisions?
>> >
>> > On Mon, Mar 12, 2018 at 11:25 AM, Rawlin Peters > >
>> > wrote:
>> >
>> >> @VijayAnand:
>> >>
>> >> Right, a Coverage Zone that doesn't map to a Cache Group in TO won't
>> >> be chosen as a backup in case of failure, but you could have a
>> >> Coverage-Zone-not-in-TO that configures Coverage-Zones-in-TO as
>> >> backups. But, I think the general sentiment right now is that all
>> >> Coverage Zones in the CZF should map back to Cache Groups in TO, so
>> >> the backup config should also be done via the Cache Group API.
>> >>
>> >> So from the Traffic Router perspective, the process should become:
>> >> 1. Rather than parsing from the CZF into the NetworkNode class, parse
>> >> Cache Group backup config from the CRConfig into the existing
>> >> CacheLocation class
>> >> 2. in the DS request flow, the NetworkNode will map back to a
>> >> registered CacheLocation which would contain the backup CG config
>> >>
>> >> The rest of the PR's behavior should stay the same, it's just a matter
>> >> of the config being located in a different class. To give you an idea
>> >> of how I would format it in the CRConfig (so you know how to parse it
>> >> out), here is a snippet of "edgeLocations" from CRConfig.json:
>> >>
>> >> "edgeLocations": {
>> >> "edge-cg-1": {
>> >>   "latitude": 1.00,
>> >>   "longitude": 2.00,
>> >>   "backupLocations": {
>> >>   "list": ["edge-cg-2"],
>> >>   "fallbackToClosest": false
>> >>   }
>> >> },
>> >> "edge-cg-2": {
>> >>   "latitude": 3.00,
>> >>   "longitude": 4.00
>> >> },
>> >> }
>> >>
>> >> The "backupLocations" section would still remain optional (if missing,
>> >> follow existing behavior of falling back to next closest CG). Existing
>> >> defaults in the current PR should remain the same.
>> >>
>> >> How would you feel about making those changes in your PR? Feel free to
>> >> tackle the new TO API and Traffic Portal changes too if you want, but
>> >> I don't want to burden you with this unexpected work if you don't want
>> >> it. I (or another willing contributor) could work on the necessary TO
>> >> API and Traffic Portal changes sometime in the near future and
>> >> integrate them with your Traffic Router enhancement.
>> >>
>> >> - Rawlin
>> >>
>> >>
>> >> On Mon, Mar 12, 2018 at 7:39 AM, vijayanand.jayaman...@gmail.com
>> >>  wrote:
>> >> >
>> >> > Rawlin,
>> >> >
>> >> > I believe the following statement is not correct.
>> >> >
>> >> > 
>> >> > However, after reading your initial proposal below, it sounds like you
>> >> > might have Coverage Zones in your CZF that don't necessarily map back
>> >> > to Cache Groups in TO. Might that be the case?
>> >> > 
>> >> >
>> >> > We can have Coverage Zones in CZF which don’t necessarily map in to
>> TO’s
>> >> configured list of Cache Groups. But then , it won’t be chosen as a
>> valid
>> >> backup in case of failure.
>> >> >
>> >> > For example:
>> >> > GROUP1 and GROUP2 are Cache Groups configured in TO (and hence
>> >> cr-config) , where GROUP3 is not in TO. Even though GROUP3 is specified
>> as
>> >> a backup for GROUP1, it wont be  chosen in case of GROUP1 failure ,
>> since
>> >> it is not in TO.
>> >> > {
>> >> >   "coverageZones": {
>> >> >  "GROUP3": {
>> >> >   "network6": [
>> >> > "1234:567a::\/64",
>> >> > "1234:567b::\/64"
>> >> >   ],
>> >> >   "network": [
>> >> > "10.197.89.0\/24"
>> >> >   ]
>> >> > },
>> >> >
>> >> >  "GROUP2": {
>> >> >   "network6": [
>> >> > "1234:567a::\/64",
>> >> > 

Re: Delivery Service Origin Refactor

[Rawlin Peters]

Hey Nir,

I think part of the motivation for doing this in Traffic Router rather
than the Caching Proxy is separation of concerns. TR is already
concerned with routing a client to the best cache based upon the
client's location, so TR is already well-equipped to make the decision
of how Delivery Services (origins) should be prioritized based upon
the client's location. That way the Caching Proxy (e.g. ATS) doesn't
need to concern itself with its own location, the client's location,
and the location of origins; it just needs to know how to get the
origin's content and cache it. All the client needs to know is that
they have a prioritized list of URLs to choose from; they don't need
to be concerned about origin/edge locations because that
prioritization will be made for them by TR.

The target DSes will have different origins primarily because they
will be in different locations, and the origins should be
interchangeable in terms of the content they provide because a smart
client may fail over to any of the target DSes in a CLIENT_STEERING DS
for the same content.

- Rawlin

On Mon, Mar 12, 2018 at 2:37 PM, Nir Sopher  wrote:
> Hi Rawlin,
> Can you please add a few word for the motivation behind basing the steering
> target selection on the location of the client?
> As the content goes through the caches, isn't it the job of the cache to
> select the best origin for the cache?  Why the client should be the one to
> take the origin location into consideration?
> Why the target DSes have different origins in the first place? Are they
> have different characteristics additionally to their location?
> Thanks,
> Nir
>
> -- Forwarded message --
> From: Rawlin Peters 
> Date: Mon, Mar 12, 2018 at 9:46 PM
> Subject: Delivery Service Origin Refactor
> To: dev@trafficcontrol.incubator.apache.org
>
>
> Hey folks,
>
> As promised, this email thread will be to discuss how to best
> associate an Origin Latitude/Longitude with a Delivery Service,
> primarily so that steering targets can be ordered/sent to the client
> based upon the location of those targets (i.e. the Origin), a.k.a.
> Steering Target Geo-Ordering. This is potentially going to be a pretty
> large change, so all your feedback/questions/concerns are appreciated.
>
> Here were a handful of bad ideas I had in order to accomplish this DS
> Origin Lat/Long association (feel free to skip to PROPOSED SOLUTION
> below):
>
> 1. Reuse the current MSO (multisite origin) backend (i.e. add the
> origin into the servers table, give it a lat/long from its cachegroup,
> assign the origin server to the DS)
> Pros:
> - reuse of existing db schema, probably wouldn't have to add any new
> tables/columns
> Cons:
> - MSO configuration is already very complex
> - for the simple case of just wanting to give an Origin a lat/long you
> have to create a server (of which only a few fields make sense for an
> Origin), add it to a cachegroup (only name and lat/long make sense,
> won't use parent relationships, isn't really a "group" of origins),
> assign it to a server profile (have to create one first, no parameters
> are needed), and finally assign that Origin server to the delivery
> service (did I miss anything?)
>
> 2. Add Origin lat/long columns to the deliveryservice table
> Pros:
> - probably the most straightforward solution for Steering Target
> Geo-Ordering given that Origin FQDN is currently a DS field.
> Cons:
> - doesn't work well with MSO
> - could be confused with Default Miss Lat/Long
> - if two different delivery services use colocated origins, the same
> lat/long needs entered twice
> - adds yet another column to the crowded deliveryservice table
>
> 3. Add origin lat/long parameters to a Delivery Service Profile
> Pros:
> - Delivery Services using colocated origins could share the same profile
> - no DB schema updates needed
> Cons:
> - profile parameters lack validation
> - still doesn't support lat/long for multiple origins associated with a DS
>
> 4. Add the lat/long to the steering target itself (i.e. where you
> choose weight/order, you'd also enter lat/long)
> Pros:
> - probably the easiest/quickest solution in terms of development
> Cons:
> - only applies lat/long to a steering target
> - using the same target in multiple Steering DSes means having to keep
> the lat/long synced between them all
> - lat/long not easily reused by other areas that may need it in the future
>
>
>
> PROPOSED SOLUTION:
>
> All of those ideas were suboptimal, which is why I think we need to:
> 1. Split Locations out of the cachegroup table into their own table
> with the following columns (cachegroup would have a foreign key to
> Location):
> - name
> - latitude
> - longitude
>
> 2. Split Origins out of the server and deliveryservice tables into
> their own table with the following columns:
> - fqdn
> - protocol (http or https)
> - port (optional, can be inferred from protocol)
> - location (optional FK to Location table)
> - 

Re: Backup Cache Group Selection

[Dave Neuman]

Good point Rawlin, but I think it does answer your questions.  CZF for now,
whatever the new CZF thing is after that.

On Mon, Mar 12, 2018 at 1:44 PM, Rawlin Peters 
wrote:

> The original scope of this thread was determining where the Backup
> Cache Group config should live (API vs CZF), not necessarily about
> building the entire CZF in the database, although I'm +1 on that idea
> as well. I think any decisions made about doing that should probably
> be started in a separate thread.
>
> - Rawlin
>
> On Mon, Mar 12, 2018 at 1:11 PM, Dave Neuman  wrote:
> > +1 on building the CZF in the database.  Jan tried to go down that rabbit
> > hole but realized it was a pretty hard problem to solve.  I think this is
> > something we might want to re-visit.  Maybe this is something we should
> > discuss at our meetup and then update this thread with our decisions?
> >
> > On Mon, Mar 12, 2018 at 11:25 AM, Rawlin Peters  >
> > wrote:
> >
> >> @VijayAnand:
> >>
> >> Right, a Coverage Zone that doesn't map to a Cache Group in TO won't
> >> be chosen as a backup in case of failure, but you could have a
> >> Coverage-Zone-not-in-TO that configures Coverage-Zones-in-TO as
> >> backups. But, I think the general sentiment right now is that all
> >> Coverage Zones in the CZF should map back to Cache Groups in TO, so
> >> the backup config should also be done via the Cache Group API.
> >>
> >> So from the Traffic Router perspective, the process should become:
> >> 1. Rather than parsing from the CZF into the NetworkNode class, parse
> >> Cache Group backup config from the CRConfig into the existing
> >> CacheLocation class
> >> 2. in the DS request flow, the NetworkNode will map back to a
> >> registered CacheLocation which would contain the backup CG config
> >>
> >> The rest of the PR's behavior should stay the same, it's just a matter
> >> of the config being located in a different class. To give you an idea
> >> of how I would format it in the CRConfig (so you know how to parse it
> >> out), here is a snippet of "edgeLocations" from CRConfig.json:
> >>
> >> "edgeLocations": {
> >> "edge-cg-1": {
> >>   "latitude": 1.00,
> >>   "longitude": 2.00,
> >>   "backupLocations": {
> >>   "list": ["edge-cg-2"],
> >>   "fallbackToClosest": false
> >>   }
> >> },
> >> "edge-cg-2": {
> >>   "latitude": 3.00,
> >>   "longitude": 4.00
> >> },
> >> }
> >>
> >> The "backupLocations" section would still remain optional (if missing,
> >> follow existing behavior of falling back to next closest CG). Existing
> >> defaults in the current PR should remain the same.
> >>
> >> How would you feel about making those changes in your PR? Feel free to
> >> tackle the new TO API and Traffic Portal changes too if you want, but
> >> I don't want to burden you with this unexpected work if you don't want
> >> it. I (or another willing contributor) could work on the necessary TO
> >> API and Traffic Portal changes sometime in the near future and
> >> integrate them with your Traffic Router enhancement.
> >>
> >> - Rawlin
> >>
> >>
> >> On Mon, Mar 12, 2018 at 7:39 AM, vijayanand.jayaman...@gmail.com
> >>  wrote:
> >> >
> >> > Rawlin,
> >> >
> >> > I believe the following statement is not correct.
> >> >
> >> > 
> >> > However, after reading your initial proposal below, it sounds like you
> >> > might have Coverage Zones in your CZF that don't necessarily map back
> >> > to Cache Groups in TO. Might that be the case?
> >> > 
> >> >
> >> > We can have Coverage Zones in CZF which don’t necessarily map in to
> TO’s
> >> configured list of Cache Groups. But then , it won’t be chosen as a
> valid
> >> backup in case of failure.
> >> >
> >> > For example:
> >> > GROUP1 and GROUP2 are Cache Groups configured in TO (and hence
> >> cr-config) , where GROUP3 is not in TO. Even though GROUP3 is specified
> as
> >> a backup for GROUP1, it wont be  chosen in case of GROUP1 failure ,
> since
> >> it is not in TO.
> >> > {
> >> >   "coverageZones": {
> >> >  "GROUP3": {
> >> >   "network6": [
> >> > "1234:567a::\/64",
> >> > "1234:567b::\/64"
> >> >   ],
> >> >   "network": [
> >> > "10.197.89.0\/24"
> >> >   ]
> >> > },
> >> >
> >> >  "GROUP2": {
> >> >   "network6": [
> >> > "1234:567a::\/64",
> >> > "1234:567b::\/64"
> >> >   ],
> >> >   "network": [
> >> > "10.197.69.0\/24"
> >> >   ]
> >> > },
> >> > "GROUP1": {
> >> >"backupZones":{
> >> >   "list": ["GROUP3"],? This wont be chosen as backup Cache Group
> in
> >> case of failure , since it is not in crconfig.
> >> >   "fallbackToClosestGroup":false
> >> >},
> >> >   "network6": [
> >> > "1234:5677::\/64",
> >> > "1234:5676::\/64"
> >> >   ],
> >> >   "network": [
> >> > "10.126.250.0\/24"
> >> >   ]
> >> 

Fwd: Delivery Service Origin Refactor

[Nir Sopher]

Hi Rawlin,
Can you please add a few word for the motivation behind basing the steering
target selection on the location of the client?
As the content goes through the caches, isn't it the job of the cache to
select the best origin for the cache?  Why the client should be the one to
take the origin location into consideration?
Why the target DSes have different origins in the first place? Are they
have different characteristics additionally to their location?
Thanks,
Nir

-- Forwarded message --
From: Rawlin Peters 
Date: Mon, Mar 12, 2018 at 9:46 PM
Subject: Delivery Service Origin Refactor
To: dev@trafficcontrol.incubator.apache.org


Hey folks,

As promised, this email thread will be to discuss how to best
associate an Origin Latitude/Longitude with a Delivery Service,
primarily so that steering targets can be ordered/sent to the client
based upon the location of those targets (i.e. the Origin), a.k.a.
Steering Target Geo-Ordering. This is potentially going to be a pretty
large change, so all your feedback/questions/concerns are appreciated.

Here were a handful of bad ideas I had in order to accomplish this DS
Origin Lat/Long association (feel free to skip to PROPOSED SOLUTION
below):

1. Reuse the current MSO (multisite origin) backend (i.e. add the
origin into the servers table, give it a lat/long from its cachegroup,
assign the origin server to the DS)
Pros:
- reuse of existing db schema, probably wouldn't have to add any new
tables/columns
Cons:
- MSO configuration is already very complex
- for the simple case of just wanting to give an Origin a lat/long you
have to create a server (of which only a few fields make sense for an
Origin), add it to a cachegroup (only name and lat/long make sense,
won't use parent relationships, isn't really a "group" of origins),
assign it to a server profile (have to create one first, no parameters
are needed), and finally assign that Origin server to the delivery
service (did I miss anything?)

2. Add Origin lat/long columns to the deliveryservice table
Pros:
- probably the most straightforward solution for Steering Target
Geo-Ordering given that Origin FQDN is currently a DS field.
Cons:
- doesn't work well with MSO
- could be confused with Default Miss Lat/Long
- if two different delivery services use colocated origins, the same
lat/long needs entered twice
- adds yet another column to the crowded deliveryservice table

3. Add origin lat/long parameters to a Delivery Service Profile
Pros:
- Delivery Services using colocated origins could share the same profile
- no DB schema updates needed
Cons:
- profile parameters lack validation
- still doesn't support lat/long for multiple origins associated with a DS

4. Add the lat/long to the steering target itself (i.e. where you
choose weight/order, you'd also enter lat/long)
Pros:
- probably the easiest/quickest solution in terms of development
Cons:
- only applies lat/long to a steering target
- using the same target in multiple Steering DSes means having to keep
the lat/long synced between them all
- lat/long not easily reused by other areas that may need it in the future



PROPOSED SOLUTION:

All of those ideas were suboptimal, which is why I think we need to:
1. Split Locations out of the cachegroup table into their own table
with the following columns (cachegroup would have a foreign key to
Location):
- name
- latitude
- longitude

2. Split Origins out of the server and deliveryservice tables into
their own table with the following columns:
- fqdn
- protocol (http or https)
- port (optional, can be inferred from protocol)
- location (optional FK to Location table)
- deliveryservice FK (if an Origin can only be associated with a
single DS. Might need step 3 below for many-to-many)
- ip_address (optional, necessary to support `use_ip_address` profile
parameter for using the origin's IP address rather than fqdn in
parent.config)
- ip6_address (optional, necessary because we'd have an ip_address
column for the same reasons)
- profile (optional, primarily for MSO-specific parameters - rank and
weight - but I could be convinced that this is unnecessary)
- cachegroup (optional, necessary to maintain primary/secondary
relationship between MID_LOC and ORG_LOC cachegroups for MSO)

3. If many-to-many DSes to Origins will still be possible, create a
new deliveryservice_origin table to support a many-to-many
relationship between DSes and origins
- the rank/weight fields for MSO could be added here possibly, maybe
other things as well?

4. Consider constraints in the origin and deliveryservice_origin table
- must fqdn alone be unique? fqdn, protocol, and port combined?

The process for creating a Delivery Service would change in that
Origins would have to be created separately and added to the delivery
service. However, to aid migration to the new way of doing things, our
UIs could keep the "Origin FQDN" field but the API backend would then
create a new row in the Origin table and add it 

Delivery Service Origin Refactor

[Rawlin Peters]

Hey folks,

As promised, this email thread will be to discuss how to best
associate an Origin Latitude/Longitude with a Delivery Service,
primarily so that steering targets can be ordered/sent to the client
based upon the location of those targets (i.e. the Origin), a.k.a.
Steering Target Geo-Ordering. This is potentially going to be a pretty
large change, so all your feedback/questions/concerns are appreciated.

Here were a handful of bad ideas I had in order to accomplish this DS
Origin Lat/Long association (feel free to skip to PROPOSED SOLUTION
below):

1. Reuse the current MSO (multisite origin) backend (i.e. add the
origin into the servers table, give it a lat/long from its cachegroup,
assign the origin server to the DS)
Pros:
- reuse of existing db schema, probably wouldn't have to add any new
tables/columns
Cons:
- MSO configuration is already very complex
- for the simple case of just wanting to give an Origin a lat/long you
have to create a server (of which only a few fields make sense for an
Origin), add it to a cachegroup (only name and lat/long make sense,
won't use parent relationships, isn't really a "group" of origins),
assign it to a server profile (have to create one first, no parameters
are needed), and finally assign that Origin server to the delivery
service (did I miss anything?)

2. Add Origin lat/long columns to the deliveryservice table
Pros:
- probably the most straightforward solution for Steering Target
Geo-Ordering given that Origin FQDN is currently a DS field.
Cons:
- doesn't work well with MSO
- could be confused with Default Miss Lat/Long
- if two different delivery services use colocated origins, the same
lat/long needs entered twice
- adds yet another column to the crowded deliveryservice table

3. Add origin lat/long parameters to a Delivery Service Profile
Pros:
- Delivery Services using colocated origins could share the same profile
- no DB schema updates needed
Cons:
- profile parameters lack validation
- still doesn't support lat/long for multiple origins associated with a DS

4. Add the lat/long to the steering target itself (i.e. where you
choose weight/order, you'd also enter lat/long)
Pros:
- probably the easiest/quickest solution in terms of development
Cons:
- only applies lat/long to a steering target
- using the same target in multiple Steering DSes means having to keep
the lat/long synced between them all
- lat/long not easily reused by other areas that may need it in the future



PROPOSED SOLUTION:

All of those ideas were suboptimal, which is why I think we need to:
1. Split Locations out of the cachegroup table into their own table
with the following columns (cachegroup would have a foreign key to
Location):
- name
- latitude
- longitude

2. Split Origins out of the server and deliveryservice tables into
their own table with the following columns:
- fqdn
- protocol (http or https)
- port (optional, can be inferred from protocol)
- location (optional FK to Location table)
- deliveryservice FK (if an Origin can only be associated with a
single DS. Might need step 3 below for many-to-many)
- ip_address (optional, necessary to support `use_ip_address` profile
parameter for using the origin's IP address rather than fqdn in
parent.config)
- ip6_address (optional, necessary because we'd have an ip_address
column for the same reasons)
- profile (optional, primarily for MSO-specific parameters - rank and
weight - but I could be convinced that this is unnecessary)
- cachegroup (optional, necessary to maintain primary/secondary
relationship between MID_LOC and ORG_LOC cachegroups for MSO)

3. If many-to-many DSes to Origins will still be possible, create a
new deliveryservice_origin table to support a many-to-many
relationship between DSes and origins
- the rank/weight fields for MSO could be added here possibly, maybe
other things as well?

4. Consider constraints in the origin and deliveryservice_origin table
- must fqdn alone be unique? fqdn, protocol, and port combined?

The process for creating a Delivery Service would change in that
Origins would have to be created separately and added to the delivery
service. However, to aid migration to the new way of doing things, our
UIs could keep the "Origin FQDN" field but the API backend would then
create a new row in the Origin table and add it to the DS. More
Origins could then be added (for MSO purposes) to the DS via a new API
endpoint. MSO configuration would change at least in how Origins are
assigned to a DS ("server assignments" would then just be for
EDGE-type servers).

Cachegroup creation also changes in that Locations need to be created
before associating them to a Cachegroup. However, our UIs could also
stay the same with the backend API updated to create a Location from
the Cachegroup request and tie it to the Cachegroup.



I know there are a lot of backend and frontend implications with these
changes that would still need to be worked out, but in general does
this proposal sound good? 

Re: Backup Cache Group Selection

[Rawlin Peters]

The original scope of this thread was determining where the Backup
Cache Group config should live (API vs CZF), not necessarily about
building the entire CZF in the database, although I'm +1 on that idea
as well. I think any decisions made about doing that should probably
be started in a separate thread.

- Rawlin

On Mon, Mar 12, 2018 at 1:11 PM, Dave Neuman  wrote:
> +1 on building the CZF in the database.  Jan tried to go down that rabbit
> hole but realized it was a pretty hard problem to solve.  I think this is
> something we might want to re-visit.  Maybe this is something we should
> discuss at our meetup and then update this thread with our decisions?
>
> On Mon, Mar 12, 2018 at 11:25 AM, Rawlin Peters 
> wrote:
>
>> @VijayAnand:
>>
>> Right, a Coverage Zone that doesn't map to a Cache Group in TO won't
>> be chosen as a backup in case of failure, but you could have a
>> Coverage-Zone-not-in-TO that configures Coverage-Zones-in-TO as
>> backups. But, I think the general sentiment right now is that all
>> Coverage Zones in the CZF should map back to Cache Groups in TO, so
>> the backup config should also be done via the Cache Group API.
>>
>> So from the Traffic Router perspective, the process should become:
>> 1. Rather than parsing from the CZF into the NetworkNode class, parse
>> Cache Group backup config from the CRConfig into the existing
>> CacheLocation class
>> 2. in the DS request flow, the NetworkNode will map back to a
>> registered CacheLocation which would contain the backup CG config
>>
>> The rest of the PR's behavior should stay the same, it's just a matter
>> of the config being located in a different class. To give you an idea
>> of how I would format it in the CRConfig (so you know how to parse it
>> out), here is a snippet of "edgeLocations" from CRConfig.json:
>>
>> "edgeLocations": {
>> "edge-cg-1": {
>>   "latitude": 1.00,
>>   "longitude": 2.00,
>>   "backupLocations": {
>>   "list": ["edge-cg-2"],
>>   "fallbackToClosest": false
>>   }
>> },
>> "edge-cg-2": {
>>   "latitude": 3.00,
>>   "longitude": 4.00
>> },
>> }
>>
>> The "backupLocations" section would still remain optional (if missing,
>> follow existing behavior of falling back to next closest CG). Existing
>> defaults in the current PR should remain the same.
>>
>> How would you feel about making those changes in your PR? Feel free to
>> tackle the new TO API and Traffic Portal changes too if you want, but
>> I don't want to burden you with this unexpected work if you don't want
>> it. I (or another willing contributor) could work on the necessary TO
>> API and Traffic Portal changes sometime in the near future and
>> integrate them with your Traffic Router enhancement.
>>
>> - Rawlin
>>
>>
>> On Mon, Mar 12, 2018 at 7:39 AM, vijayanand.jayaman...@gmail.com
>>  wrote:
>> >
>> > Rawlin,
>> >
>> > I believe the following statement is not correct.
>> >
>> > 
>> > However, after reading your initial proposal below, it sounds like you
>> > might have Coverage Zones in your CZF that don't necessarily map back
>> > to Cache Groups in TO. Might that be the case?
>> > 
>> >
>> > We can have Coverage Zones in CZF which don’t necessarily map in to TO’s
>> configured list of Cache Groups. But then , it won’t be chosen as a valid
>> backup in case of failure.
>> >
>> > For example:
>> > GROUP1 and GROUP2 are Cache Groups configured in TO (and hence
>> cr-config) , where GROUP3 is not in TO. Even though GROUP3 is specified as
>> a backup for GROUP1, it wont be  chosen in case of GROUP1 failure , since
>> it is not in TO.
>> > {
>> >   "coverageZones": {
>> >  "GROUP3": {
>> >   "network6": [
>> > "1234:567a::\/64",
>> > "1234:567b::\/64"
>> >   ],
>> >   "network": [
>> > "10.197.89.0\/24"
>> >   ]
>> > },
>> >
>> >  "GROUP2": {
>> >   "network6": [
>> > "1234:567a::\/64",
>> > "1234:567b::\/64"
>> >   ],
>> >   "network": [
>> > "10.197.69.0\/24"
>> >   ]
>> > },
>> > "GROUP1": {
>> >"backupZones":{
>> >   "list": ["GROUP3"],? This wont be chosen as backup Cache Group in
>> case of failure , since it is not in crconfig.
>> >   "fallbackToClosestGroup":false
>> >},
>> >   "network6": [
>> > "1234:5677::\/64",
>> > "1234:5676::\/64"
>> >   ],
>> >   "network": [
>> > "10.126.250.0\/24"
>> >   ]
>> > }
>> >   }
>> > }
>> >
>> > So, i feel, the existing implementation of specifying backupZones
>> configuratioin in CZF should be fine.
>> >
>> > Thanks,
>> > Vijayanand S
>> >
>> > On 2018/03/09 18:31:56, Rawlin Peters  wrote:
>> >> Hey Eric (and others),
>> >>
>> >> I'm resurrecting this thread because the PR [1] implementing this
>> >> proposed functionality is just about ready to be merged. The full
>> >> mailing list 

Re: Traffic Ops Access Control v2

[Dave Neuman]

This sounds great Jeremy, looking forward to it getting implemented.  A few
things though:

1) The "proposed roles" are really just "default roles" right?  Meaning we
will provide a way to create new roles and assign capabilities to them?
2) We will provide a way to CRUD capabilities, correct?
3) Is it assumed that Admin gets everything?  What does Operations NOT get
that admin DOES get?  Trying to differentiate between the two.

Thanks,
Dave

On Thu, Mar 8, 2018 at 9:53 AM, Jeremy Mitchell 
wrote:

> There has been some discussion for quite some time regarding an overhaul of
> the TO access control model. I'd like to refresh eveyone's memory on that
> discussion.
>
>
> *Current system:*
>
> Since the beginning, resources (or routes (UI and API)) have been locked
> down by role, or more specifically, privilege level. For example, if a
> route requires a privilege level of 20, then only users with the operations
> role (priv level=20) or admin role (priv level=30) could access that route.
> Here are our current roles (and their priv levels):
>
> Admin (30)
> Operations (20)
> Portal (15)
> Federation (15)
> Steering (15)
> ORT (11)
> Read-Only (10)
> Disallowed (0)
>
> This method has served us well for quite a while but there are some
> drawbacks to this approach. Here are a few I can think of:
>
> - No clear understanding of which routes each role provides access to. For
> example, what is the difference between the Admin and Operations role? All
> I know is that the admin role has a priv level of 30 and operations has 20.
> I can't tell you which routes an admin has access to that operations does
> not without reading the code or going thru all the docs. Ain't nobody got
> time for that!
>
> - The "Additive" nature of the roles (via priv level) prevents the ability
> to create unrelated roles. You can't create 2 roles with unique access.
> Higher level roles always inherit from lower level roles. The Federation
> role is a good example. Federation users only need access to a couple
> routes yet since it has a priv-level=15, federation users look like they
> can do federation, steering, portal, ort and read-only things...
>
> - Not easy to alter the access level of a role. For example, if you wanted
> the Portal role to have access to a few more routes, what would you do?
> Raise priv level to 18? Not sure what that would do...if anything. You'd
> have to make code changes to ensure an 18 would actually do something.
>
> - Many API consumers have elevated permissions when they only need access
> to a few routes. I.e. traffic monitors, traffic routers, traffic stats all
> have to be given the admin role. so basically, they've been granted access
> to do EVERYTHING when they only access a few routes.
>
> - There is also inconsistency in how roles are enforced. Most routes use
> priv level to determine access but some routes simply check if the user has
> the role (i.e. steering).
>
>
> *New proposed system:*
>
> *Tenancy*
>
> Last summer tenancy was introduced (thanks Qwilt) giving us the ability to
> "scope" certain resources (delivery services, users and also tenants) to
> certain users. This was a big step towards self-service as we can now limit
> what certain users see. Access control is now role + tenancy (if tenancy is
> applicable and turned on via the use_tenancy parameter).
>
> *Roles/Capabilities*
>
> Actually, a lot of work has already been done (thanks again, Qwilt) in this
> area but of course, there is more to do. Let me explain a bit how it works
> conceptually.
>
> Proposed Roles:
>
> Admin
> Operations
> Content Provider (formerly known as Portal)
> Federation
> Cache (formerly known as ORT)
> Monitor (new)
> Router (new)
> Stats (new)
> Read-Only
> Disallowed
>
> Concept:
>
> - a user has one role
> - a role has N capabilities (i.e. ds-read, ds-write, etc)
> - a capability is mapped to N API endpoints (i.e. ds-read is mapped to GET
> /api/deliveryservices and GET /api/deliveryservices/:id)
>
> A user's capabilities (and not privilege level) determine whether a user
> has access to an API endpoint or not.
>
> Advantages:
>
> - By mapping roles to capabilities and capabilities to API endpoints, it's
> easy to see what level of API access each role provides. For example, easy
> to see the difference between the Admin and Operations role.
>
> - Roles are not "additive". Unrelated, unique roles can be created. For
> example, the Federation role and Content Provider role (formerly Portal
> role) can provide 2 completely different levels of access control.
> Currently, they provide the exact same level of access because of priv
> level.
>
> - Tightly defined roles with specific capabilities provides better
> security. I.e. you don't have to give a user an admin role so they can do
> only 2 things.
>
> - Can create custom roles on the fly to only include access to certain API
> endpoints. If you want to create a Bob role with just the ds-read
> capability, go for it. You 

Re: Backup Cache Group Selection

[Rawlin Peters]

@VijayAnand:

Right, a Coverage Zone that doesn't map to a Cache Group in TO won't
be chosen as a backup in case of failure, but you could have a
Coverage-Zone-not-in-TO that configures Coverage-Zones-in-TO as
backups. But, I think the general sentiment right now is that all
Coverage Zones in the CZF should map back to Cache Groups in TO, so
the backup config should also be done via the Cache Group API.

So from the Traffic Router perspective, the process should become:
1. Rather than parsing from the CZF into the NetworkNode class, parse
Cache Group backup config from the CRConfig into the existing
CacheLocation class
2. in the DS request flow, the NetworkNode will map back to a
registered CacheLocation which would contain the backup CG config

The rest of the PR's behavior should stay the same, it's just a matter
of the config being located in a different class. To give you an idea
of how I would format it in the CRConfig (so you know how to parse it
out), here is a snippet of "edgeLocations" from CRConfig.json:

"edgeLocations": {
"edge-cg-1": {
  "latitude": 1.00,
  "longitude": 2.00,
  "backupLocations": {
  "list": ["edge-cg-2"],
  "fallbackToClosest": false
  }
},
"edge-cg-2": {
  "latitude": 3.00,
  "longitude": 4.00
},
}

The "backupLocations" section would still remain optional (if missing,
follow existing behavior of falling back to next closest CG). Existing
defaults in the current PR should remain the same.

How would you feel about making those changes in your PR? Feel free to
tackle the new TO API and Traffic Portal changes too if you want, but
I don't want to burden you with this unexpected work if you don't want
it. I (or another willing contributor) could work on the necessary TO
API and Traffic Portal changes sometime in the near future and
integrate them with your Traffic Router enhancement.

- Rawlin


On Mon, Mar 12, 2018 at 7:39 AM, vijayanand.jayaman...@gmail.com
 wrote:
>
> Rawlin,
>
> I believe the following statement is not correct.
>
> 
> However, after reading your initial proposal below, it sounds like you
> might have Coverage Zones in your CZF that don't necessarily map back
> to Cache Groups in TO. Might that be the case?
> 
>
> We can have Coverage Zones in CZF which don’t necessarily map in to TO’s 
> configured list of Cache Groups. But then , it won’t be chosen as a valid 
> backup in case of failure.
>
> For example:
> GROUP1 and GROUP2 are Cache Groups configured in TO (and hence cr-config) , 
> where GROUP3 is not in TO. Even though GROUP3 is specified as a backup for 
> GROUP1, it wont be  chosen in case of GROUP1 failure , since it is not in TO.
> {
>   "coverageZones": {
>  "GROUP3": {
>   "network6": [
> "1234:567a::\/64",
> "1234:567b::\/64"
>   ],
>   "network": [
> "10.197.89.0\/24"
>   ]
> },
>
>  "GROUP2": {
>   "network6": [
> "1234:567a::\/64",
> "1234:567b::\/64"
>   ],
>   "network": [
> "10.197.69.0\/24"
>   ]
> },
> "GROUP1": {
>"backupZones":{
>   "list": ["GROUP3"],? This wont be chosen as backup Cache Group in case 
> of failure , since it is not in crconfig.
>   "fallbackToClosestGroup":false
>},
>   "network6": [
> "1234:5677::\/64",
> "1234:5676::\/64"
>   ],
>   "network": [
> "10.126.250.0\/24"
>   ]
> }
>   }
> }
>
> So, i feel, the existing implementation of specifying backupZones 
> configuratioin in CZF should be fine.
>
> Thanks,
> Vijayanand S
>
> On 2018/03/09 18:31:56, Rawlin Peters  wrote:
>> Hey Eric (and others),
>>
>> I'm resurrecting this thread because the PR [1] implementing this
>> proposed functionality is just about ready to be merged. The full
>> mailing list discussion can be read here [2] if interested.
>>
>> I've discussed this PR a bit more with my colleagues here at Comcast,
>> and while it provides the functionality we need, we think in the
>> long-term this configuration should live in the Cache Group API in
>> Traffic Ops rather than just the Coverage Zone File.
>>
>> However, after reading your initial proposal below, it sounds like you
>> might have Coverage Zones in your CZF that don't necessarily map back
>> to Cache Groups in TO. Might that be the case? That scenario seems to
>> be allowed by Traffic Router but might not necessarily be "supported"
>> given the CZF docs [3] that state:
>> > "The Coverage Zone File (CZF) should contain a cachegroup name to network 
>> > prefix mapping in the form:"
>>
>> If we do indeed "support" this scenario, that would mean that having
>> the backupZone config only in TO wouldn't solve all your use cases if
>> your CZF heavily uses Coverage Zones that don't directly map to a
>> Cache Group in TO.
>>
>> If we should officially support this scenario, then maybe we merge the
>> PR [1] as is, then later we can 

Re: Backup Cache Group Selection

[Dewayne Richardson]

Eric, I agree with this as well, maybe we build separate API's for
"loading" CZF formats (or any other types of external data) into the same
area of the data model (however that ultimately looks).  If we keep the CZF
data centralized it'll be easier to build relationships if needed.


-Dew

On Mon, Mar 12, 2018 at 7:14 AM, Eric Friedrich (efriedri) <
efrie...@cisco.com> wrote:

> Good point. I think it makes sense to move both the backupList and
> coordinates into the CG API. By move coordinates into the API, I’m implying
> that we consolidate into 1 set of coordinates per CG. The existing CG
> coordinates would now be used for both backup edge CG selection and initial
> edge cg selection when doing client geolocation. This can certainly happen
> in a different PR than the backupList getting committed.
>
> It looks as though the CZ file will give us more flexibility, but it is
> unneeded. Today that flexibility is only causing more complexity and more
> operational overhead.
>
> —Eric
>
>
> > On Mar 9, 2018, at 5:21 PM, Rawlin Peters 
> wrote:
> >
> > So in your CZF example, we can't actually have two CZs using the same
> > name ("a"), because when that JSON gets parsed one of the CZs will be
> > overwritten so that there is only one "a" key in the JSON. The names
> > would have to be "a1" and "a2" for instance and backupList [a, b, c]
> > and [a, c, b], but I see your point in how that might be useful. But
> > at that point maybe it's just better to create two empty cachegroups
> > "a1" and "a2" in the API that first fall back to cachegroup "a".
> >
> > If I'd been around at the time coordinates were added to the CZF, I
> > think I would've -1'd the idea. The coordinates for Client Source and
> > Cache Group Destination should be the same typically because you
> > should know based upon your network topology which cachegroup to route
> > clients to. By being able to specify backups, the coordinates in the
> > CZF would become even less relevant. And fwiw at Comcast we generate
> > the CZF using the coordinates from the Cache Group API anyways, so
> > they always match.
> >
> > Another major concern with this is the scalability of having the
> > config in the CZF alone. If we can change these backupZone relations
> > using the API, that means we can add it to Traffic Portal and have a
> > much better UX rather than hard-coding these relations in whatever
> > script we're using to generate the CZF. We'd get benefits like
> > validation and typo safety, and who knows maybe in the future we could
> > have a map in TP to visualize the relationships between cache groups
> > for troubleshooting.
> >
> > - Rawlin
> >
> > On Fri, Mar 9, 2018 at 2:22 PM, Eric Friedrich (efriedri)
> >  wrote:
> >> "I can't imagine why we'd ever want the two sets of coordinates to
> differ for the same Cache Group. “
> >> Maybe someone else can chime in about why coordinates were added to the
> CZF in the first place, but I’ve also thought of them like this:
> >> CG API Coordinates - Where the cache servers are. To be used as a
> destination location routing traffic towards
> >> CZF Coordinates - Where the clients are. To be used as the source
> location routing traffic from these coordinates to the CG API coordinates.
> >>
> >> I could see cases where:
> >> a) you might set the CG coordinates of DenCG to Denver’s lat long
> >> b) you might set the CZF Coordinates of that coverageZone to somewhere
> other than Denver (because the coverageZone is not a perfect circle
> centered on Denver, like the caches might be)
> >>
> >> So I think there are valid reason’s for two sets of coordinates to
> exist, but I’m not sure if people set them differently in practice? If they
> are always the same, for every CG it seems like they should get
> consolidated. (I don’t think we’re using CZF coordinates currently)
> >>
> >> Second Point:
> >>  By having the backupList in the CZF, we have more granularity on how
> clients are routed during failures. For example, could we create a CZF that
> looks like
> >> “coverageZones”: {
> >>  “a”: {
> >>“backupList”: [b,c]
> >>“networks": [1,2,3]
> >>  },
> >>
> >> “a”: {
> >>“backupList”: [c,b]
> >>“networks": [4,5,6]
> >>  }
> >> }
> >>
> >> Here all clients are part of the “a” CacheGroup/Coverage Zone, but
> depending on their specific subnet they have different backup policies.
> >>
> >> Our particular requirement for this feature is a backup at the
> CacheGroup level, not the CZ level as I’ve shown here- so perhaps we’re
> overbuilding it.
> >>
> >> —Eric
> >>
> >>
> >>
> >>
> >>
> >>> On Mar 9, 2018, at 4:05 PM, Rawlin Peters 
> wrote:
> >>>
> >>> Ok, so the coordinates in the CZF are only used when no available
> >>> cache is found in the matched cachegroup. Rather than using the
> >>> coordinates of the matched cachegroup that it already knows about from
> >>> the API, Traffic Router uses the coordinates from the CZF 

Re: Backup Cache Group Selection

[vijayanand . jayamanian]

Rawlin,

I believe the following statement is not correct.


However, after reading your initial proposal below, it sounds like you
might have Coverage Zones in your CZF that don't necessarily map back
to Cache Groups in TO. Might that be the case?

 
We can have Coverage Zones in CZF which don’t necessarily map in to TO’s 
configured list of Cache Groups. But then , it won’t be chosen as a valid 
backup in case of failure. 
 
For example:
GROUP1 and GROUP2 are Cache Groups configured in TO (and hence cr-config) , 
where GROUP3 is not in TO. Even though GROUP3 is specified as a backup for 
GROUP1, it wont be  chosen in case of GROUP1 failure , since it is not in TO.
{
  "coverageZones": {
 "GROUP3": {
  "network6": [
"1234:567a::\/64",
"1234:567b::\/64"
  ],
  "network": [
"10.197.89.0\/24"
  ]
},
 
 "GROUP2": {
  "network6": [
"1234:567a::\/64",
"1234:567b::\/64"
  ],
  "network": [
"10.197.69.0\/24"
  ]
},
"GROUP1": {
   "backupZones":{
  "list": ["GROUP3"],? This wont be chosen as backup Cache Group in case of 
failure , since it is not in crconfig.
  "fallbackToClosestGroup":false
   },
  "network6": [
"1234:5677::\/64",
"1234:5676::\/64"
  ],
  "network": [
"10.126.250.0\/24"
  ]
}
  }
}

So, i feel, the existing implementation of specifying backupZones 
configuratioin in CZF should be fine.

Thanks,
Vijayanand S

On 2018/03/09 18:31:56, Rawlin Peters  wrote: 
> Hey Eric (and others),
> 
> I'm resurrecting this thread because the PR [1] implementing this
> proposed functionality is just about ready to be merged. The full
> mailing list discussion can be read here [2] if interested.
> 
> I've discussed this PR a bit more with my colleagues here at Comcast,
> and while it provides the functionality we need, we think in the
> long-term this configuration should live in the Cache Group API in
> Traffic Ops rather than just the Coverage Zone File.
> 
> However, after reading your initial proposal below, it sounds like you
> might have Coverage Zones in your CZF that don't necessarily map back
> to Cache Groups in TO. Might that be the case? That scenario seems to
> be allowed by Traffic Router but might not necessarily be "supported"
> given the CZF docs [3] that state:
> > "The Coverage Zone File (CZF) should contain a cachegroup name to network 
> > prefix mapping in the form:"
> 
> If we do indeed "support" this scenario, that would mean that having
> the backupZone config only in TO wouldn't solve all your use cases if
> your CZF heavily uses Coverage Zones that don't directly map to a
> Cache Group in TO.
> 
> If we should officially support this scenario, then maybe we merge the
> PR [1] as is, then later we can augment the feature so that we can use
> the Cache Group API to provide the backupZone config as well as in the
> CZF. If the config was provided in both the API and the CZF, then the
> API would take precedent.
> 
> If this scenario should NOT officially be supported, then I think we
> should update the PR [1] to have Traffic Router parse the config from
> CRConfig.json rather than the CZF and augment the Cache Group API to
> support the backupZone config. I think this would be the most ideal
> solution, but I also don't want to sign up our contributors for extra
> work that they weren't planning on doing. I'd be happy to help augment
> this feature on the TO side.
> 
> What do you all think of this proposal? TO-only or both TO and CZF?
> 
> - Rawlin
> 
> [1] https://github.com/apache/incubator-trafficcontrol/pull/1908
> [2] 
> https://lists.apache.org/thread.html/b033b3943c22a606370ad3981fa05fb0e7039161b88bbc035bc49b25@%3Cdev.trafficcontrol.apache.org%3E
> [3] 
> http://traffic-control-cdn.readthedocs.io/en/latest/admin/traffic_ops/using.html#the-coverage-zone-file-and-asn-table
> 
> On 2016/12/22 19:28:17, Eric Friedrich (efriedri)  wrote:
> > The current behavior of cache group selection works as follows
> > 1) Look for a subnet match in CZF
> > 2) Use MaxMind/Neustar for GeoLocation based on client IP. Choose closest 
> > cache group.
> > 3) Use Delivery Service Geo-Miss Lat/Long. Choose closest cache group.
> >
> >
> > For deployments where IP addressing is primarily private (say RFC-1918 
> > addresses), client IP Geo Location (#2) is not useful.
> >
> >
> > We are considering adding another field to the Coverage Zone File that 
> > configures an ordered list of backup cache groups to try if the primary 
> > cache group does not have any available caches.
> >
> > Example:
> >
> > "coverageZones": {
> > "cache-group-01": {
> > “backupList”: [“cache-group-02”, “cache-group-03”],
> > "network6": [
> > "1234:5678::\/64”,
> > "1234:5679::\/64"],
> > "network": [
> > "192.168.8.0\/24",
> > "192.168.9.0\/24”]
> > }
> >
> > This configuration could also be part of the 

Re: Backup Cache Group Selection

[Eric Friedrich (efriedri)]

Good point. I think it makes sense to move both the backupList and coordinates 
into the CG API. By move coordinates into the API, I’m implying that we 
consolidate into 1 set of coordinates per CG. The existing CG coordinates would 
now be used for both backup edge CG selection and initial edge cg selection 
when doing client geolocation. This can certainly happen in a different PR than 
the backupList getting committed. 

It looks as though the CZ file will give us more flexibility, but it is 
unneeded. Today that flexibility is only causing more complexity and more 
operational overhead. 

—Eric


> On Mar 9, 2018, at 5:21 PM, Rawlin Peters  wrote:
> 
> So in your CZF example, we can't actually have two CZs using the same
> name ("a"), because when that JSON gets parsed one of the CZs will be
> overwritten so that there is only one "a" key in the JSON. The names
> would have to be "a1" and "a2" for instance and backupList [a, b, c]
> and [a, c, b], but I see your point in how that might be useful. But
> at that point maybe it's just better to create two empty cachegroups
> "a1" and "a2" in the API that first fall back to cachegroup "a".
> 
> If I'd been around at the time coordinates were added to the CZF, I
> think I would've -1'd the idea. The coordinates for Client Source and
> Cache Group Destination should be the same typically because you
> should know based upon your network topology which cachegroup to route
> clients to. By being able to specify backups, the coordinates in the
> CZF would become even less relevant. And fwiw at Comcast we generate
> the CZF using the coordinates from the Cache Group API anyways, so
> they always match.
> 
> Another major concern with this is the scalability of having the
> config in the CZF alone. If we can change these backupZone relations
> using the API, that means we can add it to Traffic Portal and have a
> much better UX rather than hard-coding these relations in whatever
> script we're using to generate the CZF. We'd get benefits like
> validation and typo safety, and who knows maybe in the future we could
> have a map in TP to visualize the relationships between cache groups
> for troubleshooting.
> 
> - Rawlin
> 
> On Fri, Mar 9, 2018 at 2:22 PM, Eric Friedrich (efriedri)
>  wrote:
>> "I can't imagine why we'd ever want the two sets of coordinates to differ 
>> for the same Cache Group. “
>> Maybe someone else can chime in about why coordinates were added to the CZF 
>> in the first place, but I’ve also thought of them like this:
>> CG API Coordinates - Where the cache servers are. To be used as a 
>> destination location routing traffic towards
>> CZF Coordinates - Where the clients are. To be used as the source location 
>> routing traffic from these coordinates to the CG API coordinates.
>> 
>> I could see cases where:
>> a) you might set the CG coordinates of DenCG to Denver’s lat long
>> b) you might set the CZF Coordinates of that coverageZone to somewhere other 
>> than Denver (because the coverageZone is not a perfect circle centered on 
>> Denver, like the caches might be)
>> 
>> So I think there are valid reason’s for two sets of coordinates to exist, 
>> but I’m not sure if people set them differently in practice? If they are 
>> always the same, for every CG it seems like they should get consolidated. (I 
>> don’t think we’re using CZF coordinates currently)
>> 
>> Second Point:
>>  By having the backupList in the CZF, we have more granularity on how 
>> clients are routed during failures. For example, could we create a CZF that 
>> looks like
>> “coverageZones”: {
>>  “a”: {
>>“backupList”: [b,c]
>>“networks": [1,2,3]
>>  },
>> 
>> “a”: {
>>“backupList”: [c,b]
>>“networks": [4,5,6]
>>  }
>> }
>> 
>> Here all clients are part of the “a” CacheGroup/Coverage Zone, but depending 
>> on their specific subnet they have different backup policies.
>> 
>> Our particular requirement for this feature is a backup at the CacheGroup 
>> level, not the CZ level as I’ve shown here- so perhaps we’re overbuilding it.
>> 
>> —Eric
>> 
>> 
>> 
>> 
>> 
>>> On Mar 9, 2018, at 4:05 PM, Rawlin Peters  wrote:
>>> 
>>> Ok, so the coordinates in the CZF are only used when no available
>>> cache is found in the matched cachegroup. Rather than using the
>>> coordinates of the matched cachegroup that it already knows about from
>>> the API, Traffic Router uses the coordinates from the CZF cachegroup
>>> instead. That seems...not great?
>>> 
>>> That means we basically have two sets of coordinates for the same
>>> cachegroup, one set in the API and one in the CZF. The API coordinates
>>> are used when caches are available, but the CZF coordinates are used
>>> when no caches are available in the matched CG. To me it seems like
>>> the CG coordinates from the API should always take precedent over the
>>> CZF. In fact the CZF coordinates are optional, but TR currently won't
>>> use the