Re: traffic_ops_ort doesn't update empty config file problem

[Jifeng Yang (jifyang)]

Thank you, Mark!

Thanks,
Jifeng

On 03/01/2017, 22:26, "Mark Torluemke"  wrote:

I think the initial thought was to protect against empty files being
generated from Traffic Ops. However, over time that code has potentially
been shifted some, and that is no longer the functionality. :)

So, yes, I think we could skip that check for files already on disk.

On Tue, Jan 3, 2017 at 12:57 AM, Jifeng Yang (jifyang) 
wrote:

> Hi,
>
> We met a problem about traffic_ops_ort:
>
> The ATS config file “remap.config” happened to be zero size. After that,
> the “remap.config” file couldn't be updated by the traffic_ops_ort.
>
> Checking the code in the file “/opt/ort/traffic_ops_ort.pl”, there is:
>
> sub can_read_write_file {
>my $filename = shift;
>..
>if ( -z $file ) {
> ( $log_level >> $ERROR ) && print "ERROR $file has
> size=0!\n";
> $cfg_file_tracker->{$filename}->{'audit_failed'}++;
> return 0;
> }
> ..
> return 1;
> }
>
> By the code, the zero size file isn't treated as can_read_write_file.
>
> Can we treat zero size file as can_read_write_file? Is there any
> consideration about this?
>
> The configuration file may become zero size file by some reason (such as
> by accident). If zero size config file isn't treated as
> can_read_write_file, it can't be updated any more if a configuration file
> becomes zero size.
>
> Thanks,
> Jifeng
>

Re: regex_revalidate.config

[Jeremy Mitchell]

derek gelinas is working on a PR that "scopes" these config files. i.e.
some config files are at the server level, profile level or cdn level.
regex_revalidate.config happens to be at the cdn level. Here is his PR that
I think will be targeted for 2.1.

https://github.com/apache/incubator-trafficcontrol/pull/141

what this means is that if a server needs to fetch regex_revalidate.config,
it would fetch it like this
to-domain.com/api/1.2/cdn/cdn1/configfiles/ats/regex_revalidate.config and
if this file is served from a cache of its own, the request by X number of
servers to get the same file would take advantage of caching to reduce the
delivery time. sooo.maybe it's a second the first time and a few ms
each subsequent time...

hope that made sense.

jeremy



On Tue, Jan 3, 2017 at 7:05 PM, Steve Malenfant 
wrote:

> Jeremy,
>
> I missed this email. Seems like close to a second is quite slow. Is the
> regex_revalidate.config per server? It seems like it was per CDN before.
>
> On Tue, Dec 13, 2016 at 1:02 PM, Jeremy Mitchell 
> wrote:
>
> > Obviously, it is going to be slower to pull regex_revalidate.config thru
> TO
> > as opposed to pulling it off the file system. The question is...is this
> > acceptable?
> >
> > curling to
> > https://to.domain.com/Trafficserver-Snapshots/cdn-
> > name/regex_revalidate.config
> >
> > time_namelookup: 0.005
> > time_connect: 0.058
> > time_appconnect: 0.000
> > time_pretransfer: 0.000
> > time_redirect: 0.000
> > time_starttransfer: 0.000
> > --
> > time_total: 0.063
> >
> > curling to
> > https://to.domain.com/genfiles/view/server-host-
> > name/regex_revalidate.config
> >
> > time_namelookup: 0.005
> > time_connect: 0.062
> > time_appconnect: 0.307
> > time_pretransfer: 0.307
> > time_redirect: 0.000
> > time_starttransfer: 0.781
> > --
> > time_total: 0.781
> >
> > On Mon, Dec 12, 2016 at 9:03 PM, Steve Malenfant 
> > wrote:
> >
> > > I'm OK with this. I think most of the revalidate functionality is not
> > well
> > > understood here... I guess I need to look into the API since we are
> using
> > > the UI to issue revalidate today.
> > >
> > > On Mon, Dec 12, 2016 at 5:43 PM, Jeremy Mitchell <
> mitchell...@apache.org
> > >
> > > wrote:
> > >
> > > > I've created an issue to solicit some feedback regarding the removal
> of
> > > > regex_revalidate.config from the file system. If you are not aware,
> > > > regex_revalidate.config is the ATS config file used to "invalidate
> > > > content". Storing regex_revalidate.config on the file system makes it
> > > > harder to set up a highly-available, redundant traffic ops.
> > > >
> > > > https://issues.apache.org/jira/browse/TC-68
> > > >
> > > > Thanks!
> > > >
> > >
> >
>

TC 2.0.x branch

[Jeremy Mitchell]

As discussed in the traffic-control-cdn#dev slack channel, the plan is to
merge the contents of the postgres branch (
https://github.com/apache/incubator-trafficcontrol/tree/postgres) to the
master branch by Friday, 1/6/16 end of day and subsequently cut a 2.0.x
branch.

Included in 2.0.x will be any commits applied to master after 1.8.x was cut
- https://github.com/apache/incubator-trafficcontrol/compare/1.8.x...master

plus, of course, changes made to support Traffic Ops on Postgres.

This also means if you have any outstanding PR's (
https://github.com/apache/incubator-trafficcontrol/pulls) that you want in
2.0.x, reach out to a committer to get them merged. Otherwise, they will
have to target the 2.1 release.

Questions? Concerns?

thanks.

Re: Backup Cache Group Selection

[Jeff Elsloo]

If we applied the proposed change, given your scenario we should fall
through to the return statement that calls getClosestCacheLocation().
That method will order all cache groups based on their lat/long and
the lat/long of the cache group we hit on in the CZF. Once the list is
ordered, we iterate through the list until we find a cache group that
has available caches for that DS.

BTW, the stuff on line 536 is likely to produce the exact same result
as the check that precedes it. networkNode.getLoc() will return the
string name of the cache group, so when we find the CacheLocation, it
will be the same as what we had just checked. We could probably get
away with removing that part of the method as it's redundant.
--
Thanks,
Jeff


On Wed, Jan 4, 2017 at 11:54 AM, Eric Friedrich (efriedri)
 wrote:
> Where would TR look outside the assigned cache group to find the next closest 
> cache group?
>
>> On Jan 4, 2017, at 11:25 AM, Eric Friedrich (efriedri)  
>> wrote:
>>
>>
>> On Jan 3, 2017, at 5:20 PM, Jeff Elsloo 
>> > wrote:
>>
>> Hey Eric,
>>
>> It sounds like the use case you're after is an RFC 1918 client
>> associated with a cache group whose caches are all unavailable for one
>> reason or another. Is that correct?
>> Yes, exactly.
>>
>>
>> I looked at the code a bit, and I think that we can make a minor
>> change to achieve the behavior you're looking for as long as you're
>> able to put your RFC 1918 ranges in the CZF.
>> Yes, we would want those ranges in the CZF. I can’t think of any other place 
>> they would go.
>>
>>
>> There's a small logic gap in the existing algorithm around cache
>> location selection and I think if we fix that (two line change), we
>> should be better off all around. I think the only time we'd ever want
>> to go to the geolocation provider is in the event of a miss on the
>> CZF, so as long as we have a hit there, we should find the cache group
>> closest to that hit location that has available caches. This would
>> automatically provide the "backup" cache group concept, and has the
>> added benefit of doing this selection dynamically based on the state
>> of the CDN.
>> Wow, thanks for picking up on this solution. Sounds like a strong 
>> possibility. I like that it can extend dynamically.
>>
>>
>>
>> See this to get an idea of what I mean: http://apaste.info/u3PQo
>> https://github.com/apache/incubator-trafficcontrol/blob/249bd7504eeb7cc43402126f3719017e2475ad33/traffic_router/core/src/main/java/com/comcast/cdn/traffic_control/traffic_router/core/router/TrafficRouter.java#L536
>> Does this line set cacheLocation to the closest cache group with active 
>> caches on that DS?
>>
>> What does networkNode.getLoc() actually return?
>>
>> —Eric
>>
>>
>>
>> Obviously we'd need to test this to ensure we don't break other 
>> functionality.
>> --
>> Thanks,
>> Jeff
>>
>>
>> On Tue, Jan 3, 2017 at 10:07 AM, Eric Friedrich (efriedri)
>> > wrote:
>> If all caches in the primary cache group are unavailable, our goal is to 
>> provide a backup routing policy for RFC1918 clients.
>>
>> When client IP is an public Internet IP, the current backup policy is to 
>> assign the client to the geographically closest cache (Distance = MaxMind 
>> Geo Lat/Long - configured CG lat/long).
>>
>> When client IP is an RFC1918 IP, the client would not have a maxmind 
>> geo-loc, so would fall back to the DS geo-miss lat long. We’d prefer some 
>> more granular control over where these clients are routed to, rather than a 
>> per-DS setting.
>>
>>
>> So with an RFC1918 client, the lookup process would be (step 3 is only 
>> addition)
>> 1) Check CZF for a subnet match (and find a match for existing cache group). 
>> Assign client to CG
>> 2) Check CG for available (online and associated w/ DS) servers. In this 
>> particular case, assume CG has no servers available to route the client to
>> 3) Walk the CZF's list of backup CGs and perform the check from #2 for each 
>> CG. Use first server that is found
>> 4) Assuming no server is found in #3, perform geo-location and find closest 
>> cache group. Use a server from the closest CG if one is found
>> 4a) If geo-location returns null, use the DS’ default geo-miss location as 
>> the client location.
>>
>> —Eric
>>
>>
>> On Dec 26, 2016, at 10:01 AM, Jan van Doorn 
>> > wrote:
>>
>> Hi Eric,
>>
>> How does the backup list relate to the RFC1918-is-not-in-geo problem?
>>
>> To get to a cachegroup you need to get a match in the coverage zone, I would 
>> think?
>>
>> Rgds,
>> JvD
>>
>> On Dec 22, 2016, at 12:28, Eric Friedrich (efriedri) 
>> > wrote:
>>
>> The current behavior of cache group selection works as follows
>> 1) Look for a subnet match in CZF
>> 2) Use MaxMind/Neustar for GeoLocation based on client IP. Choose closest 
>> cache