TLS Client Authentication in Traffic Control

[Eric Friedrich (efriedri)]

Someone else may find this useful, so I thought I would share. (Apologies for 
the earlier cross-post)


Configuring TLS Client Authentication in Traffic Control (Experimental Testing 
Procedure)
=
Note: Trafficserver does not currently allow per-Delivery Service (per-remap) 
configuration of client authentication. The below instructions will enable 
client authentication for all HTTPS services on a given profile/cache.

1) In TrafficOps, configure the Edge cache “Profile” to turn on client 
authentication. Set the following parameters:
  - name: CONFIG proxy.config.ssl.client.certification_level
  - file: records.config
  - value: INT 2
Screenshot: https://cisco.box.com/s/lxtlfbfrbpnaa17cnp4dddj2p0wwzril

  - name: CONFIG proxy.config.ssl.CA.cert.filename
  - file: records.config
  - value: STRING etc/trafficserver/ssl/ca.crt
Screenshot: https://cisco.box.com/s/hq7vubwd9z0k1g8705eaagbvdg0aokjc
See below for instructions on generating the Certificate Authority (CA), 
Certificate and private key.


  You can add the CA file via TrafficOps, but its a painful process. Please see 
the screenshot. If you wish to skip this step, you can scp the file directly to 
the cache (/opt/trafficserver/etc/trafficserver/ssl/client_ca.crt)
  Screenshot: https://cisco.box.com/s/849imlapxj1e30zi6y63a8fwd31swv21
 (Now that I know what a take and bake is, I think I was better off before. 
Configuring a whole SSL Cert in here is pretty painful, but thanks to Jeff for 
the help on this step)


2) Queue and run ORT On caches to get updated settings

3) Verify by making a curl request
$ curl -k --cert ~/client_auth/client.crt --key ~/client_auth/client.key -v 
 https://edge-cache-1.cdn.cisco.com/test.m3u8

On success, you will receive the content.

On failure, you will see something like:
[cloud-user trafficserver]$ curl -k -v  
https://edge-cache-1.cdn.cisco.com/test.m3u8
* About to connect() to localhost port 443 (#0)
*   Trying ::1...
* Connected to localhost (::1) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* NSS: client certificate not found (nickname not specified)
* NSS error -12227 (SSL_ERROR_HANDSHAKE_FAILURE_ALERT)
* SSL peer was unable to negotiate an acceptable set of security parameters.
* Closing connection 0
curl: (35) NSS: client certificate not found (nickname not specified)


Generating a Certificate Authority and Client Certificate (optional)
=
1) Create the Certificate Authority Key
$ openssl genrsa -out client_ca.key 2048

2) Generate the Certificate Authority Cert
$ openssl req -new -x509 -key ./client_ca.key -out client_ca.crt

2) Generate the Client Key and Certificate Signing Request
$ openssl req -newkey rsa:2048 -nodes -keyout client.key -out client.csr

3) Use the Certificate Authority to sign the client certificate signing request
   $ openssl x509 -req -in ./client.csr -CA ./client_ca.crt -CAkey 
./client_ca.key -CAcreateserial -out client.crt

4) The  client_ca.crt file is copied to the Trafficserver. The client (curl) is 
given client.crt and client.key

Re: ATC Spring Summit 2018 Registration and CFP

[Eric Friedrich (efriedri)]

Apologies for the late change, but we’ve moved into a larger room to give 
everyone more space.

The room is called Newburyport and will be immediately on your left as soon as 
you enter. Please still check in with the lobby ambassador for 
badge/wifi-credentials first.

—Eric

On Apr 23, 2018, at 1:40 PM, Dave Neuman 
> wrote:

Hey All,
I am looking forward to seeing everyone tomorrow! Eric has updated the
spring summit wiki page with information on checking in at Cisco
tomorrow[1]. Please make sure to take a look before you head in. Here is
the gist of it:

Directions to the meeting room: Park in front building 500 and enter
through the front doors. Check in with the lobby ambassador to receive a
temporary badge and Wi-Fi credentials. The meeting will be in the
Marblehead room. Proceed past the lobby desk and turn right into the
Customer Briefing Center. Follow the hallway to the left, turning right at
the end of the hallway.

Remember, breakfast is from 0800 - 0900 and we will start at 0900.

Thanks,
Dave


[1]
https://cwiki.apache.org/confluence/display/TC/ATC+Spring+Summit+-+April+2018%2C+Boxborough%2C+MA

​

On Thu, Apr 5, 2018 at 12:12 PM, Dave Neuman 
> wrote:

Hi everyone,
I have created a premliminary schedule for the spring summit!  Please take
a look and let me know if you have any questions or concerns.

https://cwiki.apache.org/confluence/display/TC/ATC+
Spring+Summit+-+April+2018%2C+Boxborough%2C+MA

Thanks!
Dave

On Mon, Apr 2, 2018 at 12:40 PM, Dave Neuman  wrote:

Hey All,
We have a few slots still available to speak at the summit. Let me know
if you are interested!

Thanks,
Dave

On Thu, Mar 29, 2018 at 2:24 PM, Dave Neuman  wrote:

Hey Everyone,
Just a reminder that our spring summit is less than a month away and our
CFP will be closing in just 3 more days!  If you haven't had the
opportunity to register yet, I encourage you to do so!  There is no
registration fee, you just need to get yourself to Boston.  If you are
planning on submitting a talk please do so by April 1st so that I can get a
schedule out.  Please be assured that we have plenty of speaking slots
still available.

Register here:  https://cwiki.apache.org/confluence/display/TC/ATC+Sp
ring+Summit+-+April+2018%2C+Boxborough%2C+MA.

Submit a talk by sending an email here:   summits@trafficcontrol
.incubator.apache.org 
.>

Thanks,
Dave

On Fri, Mar 2, 2018 at 2:51 PM, Dave Neuman 
> wrote:

I just wanted to let everyone know that I have updated updated the wiki
page with hotel and transportation information.

Thanks to everyone that has signed up and I look forward to more people
signing up.

Thanks,
Dave

On Sat, Feb 24, 2018 at 9:26 AM, Dave Neuman 
> wrote:

Hey All,
It's my pleasure to announce that Apache Traffic Control will be
having our spring summit on April 24 and 25 in Boxborough (Boston),
Massachusetts!
If you want to learn more about ATC, get some questions answered by
members of the community, or just hang out with us for a few days, please
see the following wiki page to RSVP:  https://cwiki.apache.org/conf
luence/display/TC/ATC+Spring+Summit+-+April+2018%2C+Boxborou
gh%2C+MA.


This email also servers as a CFP (Call for papers/presentations), if
you have a great idea for a presentation let us know at
summ...@trafficcontrol.incubator.apache.org
.>
 by April 1, 2018.  We
will work on publishing a schedule as soon as we start getting submissions.

Any questions just let me know.

Thanks and I look forward to seeing everyone there!

Dave

Re: Traffic Server Secondary Streaming IPs Design

[Eric Friedrich (efriedri)]

Hey Nir-
  For our particular use case, we are looking at making an IP the delivery 
unit. We would like to use a single interface with multiple IPs. DSs would be 
assigned to one of the IPs on that interface.  Interface (or IP) priority does 
not come into play here as there is no failover between IPs

—Eric


> On Apr 4, 2018, at 1:23 AM, Nir Sopher <n...@qwilt.com> wrote:
> 
> +1
> Note that beyond the DB, the change should also be reflected into the
> cr-config.
> As I see it, a flexible model may be built of the below items:
> 1 - Edge server.
> 2- Interface
> 3. IPs
> 
> The Interface (or should it be called "delivery unit") is the element we
> redirect the traffic to and which is monitored by the traffic-monitor:
> * Each server may have multiple interfaces
> * Each interface may have multiple IPs
> * Interfaces has priorities (abstraction for primary/secondary)
> * Each interface is given a seperate DNS name by the router. Single name
> for the multiple IPs.
> * Each interface is monitored and reported seperately by the traffic
> monitor, including health an stats.
> 
> 
> The router "redirect target decision" may look as follows
> 0. Select cache as we do today taking into account the consistent hash. A
> server is in the selection group only if one of its interfaces is found to
> be healthy
> 1. Once we have server selected, select an interface out of all interfaces
> of the server with max available priority.
> 
> An additional improvement, may assign DS to interfaces instead of servers.
> A server serves DS X iff one of its interfaces is assigned to the DS.
> 
> Nir
> 
> 
> On Apr 4, 2018 6:56 AM, "Zhilin Huang (zhilhuan)" <zhilh...@cisco.com>
> wrote:
> 
> Updated the DB schema in section 3.1.1.4
> 
> Thanks,
> Zhilin
> 
> 
> 
> On 04/04/2018, 11:02 AM, "Zhilin Huang (zhilhuan)" <zhilh...@cisco.com>
> wrote:
> 
>Good points. I am happy to make this change in the design doc.
> 
>Thanks,
>Zhilin
> 
> 
>On 03/04/2018, 8:17 PM, "Eric Friedrich (efriedri)" <efrie...@cisco.com>
> wrote:
> 
>I would prefer a consistent way to store all interface and IP
> address information. Its good database design practice to store similar
> information in similar tables (i.e. all IP info in 1 table) rather than
> keep some IPs in the server table and some IPs in another table.
> 
>I also think this refactoring will give us greater flexibility for
> more changes in the future. Outside of this particular use case, we might
> have additional features like sharing edges between public/private networks
> or having multiple (equal priority) streaming interfaces on a cache.
> 
>These future features would be easier if the interface data and IP
> data is all organized into separate tables.
> 
>I’d also like to see the delivery service to IP mapping be a many
> to many mapping in the DB. For this particular feature we will only assign
> a single IP (and we can restrict that in the API if we want), but I am near
> certain that in the future we would like the ability to assign a DS to
> multiple IPs on the same cache.
> 
> 
>—Eric
> 
> 
> 
>> On Apr 3, 2018, at 2:42 AM, Zhilin Huang (zhilhuan) <
> zhilh...@cisco.com> wrote:
>> 
>> Hi Mark,
>> 
>> Thanks for your comments. Please check my reply in another thread:
>> 
>> If we all agreed to use unified tables for all IPs and/or
> interfaces: primary, management, secondary, then there need to be two
> tables: IP and interface.
>> And in the server table, we need to replace the original
> "interface_xxx", "ip_xxx", "ip6_xxx" fields with a "primary_ip_id" field.
> And do similar things to management IP.
>> 
>> Thanks,
>> Zhilin
>> 
>> 
>> On 03/04/2018, 7:08 AM, "Mark Torluemke" <mtorlue...@apache.org>
> wrote:
>> 
>>   I would support an 'interfaces' table (adding some sort of a
> 'type' column)
>>   that would include moving the management and lights out
> management
>>   interfaces to that table as well.
>> 
>>   Cheers,
>>   Mark
>> 
>>   On Mon, Apr 2, 2018 at 2:39 PM, Nir Sopher <n...@qwilt.com>
> wrote:
>> 
>>> Hi Zhilin,
>>> 
>>> I took a quick look into the spec. Hope to have the opportunity
> to dive
>>> deeper into it soon so we can further discuss it.
>>> 
>>> For now I have a 2 questions.
>>> In the spec, you refer to "secondary interfaces", and you have a
> list of
>>> secondary

Re: Traffic Control Last-resort Routing

[Eric Friedrich (efriedri)]

Sounds good John. 

If you can provide a path, we shouldn’t call it Error Domain :-) We should call 
it “Error Redirect URL”.

—Eric

> On Apr 3, 2018, at 11:02 PM, John Shen (weifensh) <weife...@cisco.com> wrote:
> 
> Hey Eric,
> 
> Thanks a lot for your comments. Please refer to my reply inline.
> 
> Thanks,
> John
> 
> 
> On 2018/4/3, 8:36 PM, "Eric Friedrich (efriedri)" <efrie...@cisco.com> wrote:
> 
>Hey John-
>1)“Last Resort Alternate domain” is incredibly similar to the Bypass FQDN. 
> I’d rather see us enhance the Bypass FQDN with an optional scheme and port 
> number, rather than add something so close in functionality. 
>[John] Yeah, we had the same thought. The only disadvantage is that we are 
> planning to use "DS Profile" instead of changing the GUI and DB schema to 
> support the new parameters, thus we will have the new options (e.g. scheme 
> and port) configured by "DS profile" and Bypass FQDN configured through GUI 
> for each DS. If no one has objection with this mixed configuration solution, 
> we would implement the Alternate domain/Bypass FQDN in this way.
> 
>2) Is there a definite need for the “To content origin” checkbox? If this 
> is only used when Bypass FQDN/Alternate domain is not configured, then we 
> could just ask the user to configure the Origin FQDN in this field. 
>[John] Yes, “To content origin” is only used when alternate domain is not 
> configured. Asking user to configure OFQDN in the alternate domain would make 
> the implementation easier. It's just that a user has to manually fill the 
> field.
> 
>3) Alternate Domain and Error domain also seem incredibly similar in the 
> proposal. Would error domain be better described as “Redirect location for 
> unknown delivery service FQDNs or blocked clients. If not set, TR would 
> typically return a 503” 
>[John] One difference is that for Error domain, user can configure a file 
> to redirect to, e.g. to "http://error-domain.org/404.html;. Error domain 
> (with filename) is the redirect location when there is no available cache in 
> a DS.
> 
>—Eric
> 
>> On Apr 2, 2018, at 7:43 AM, John Shen (weifensh) <weife...@cisco.com> wrote:
>> 
>> Hi guys,
>> 
>> We are planning to add a Last-resort routing feature to Traffic Control. It 
>> will make TR redirect a request to an alternate domain, an error domain, or 
>> an origin server when there is not suitable cache available (e.g. all caches 
>> are busy). It is similar with current “Bypass FQDN” in TC, but it adds more 
>> parameters like redirecting to error domains or origin servers.
>> 
>> All the new parameters will be configured through “DS Profile” (to avoid GUI 
>> and DB schema changes), which is different with the current GUI 
>> configuration of “Bypass FQDN”. So the design will not change the current 
>> “Bypass FQDN” feature, and just add Last-resort as another extra feature. If 
>> Last-resort feature is not enabled (by default), the current behavior of 
>> Traffic Router will not change.
>> 
>> Please refer to following doc for more details: 
>> https://docs.google.com/document/d/1ttrZUGoGZTdCT6g78v62xBVoDCC_cWesPzxfLaHBcp0/edit
>> 
>> Any comments are welcome.
>> 
>> Thanks,
>> John
>> 
> 
> 
> 

Re: Traffic Control Last-resort Routing

[Eric Friedrich (efriedri)]

Hey John-
1)“Last Resort Alternate domain” is incredibly similar to the Bypass FQDN. I’d 
rather see us enhance the Bypass FQDN with an optional scheme and port number, 
rather than add something so close in functionality. 

2) Is there a definite need for the “To content origin” checkbox? If this is 
only used when Bypass FQDN/Alternate domain is not configured, then we could 
just ask the user to configure the Origin FQDN in this field. 

3) Alternate Domain and Error domain also seem incredibly similar in the 
proposal. Would error domain be better described as “Redirect location for 
unknown delivery service FQDNs or blocked clients. If not set, TR would 
typically return a 503” 

—Eric

> On Apr 2, 2018, at 7:43 AM, John Shen (weifensh)  wrote:
> 
> Hi guys,
> 
> We are planning to add a Last-resort routing feature to Traffic Control. It 
> will make TR redirect a request to an alternate domain, an error domain, or 
> an origin server when there is not suitable cache available (e.g. all caches 
> are busy). It is similar with current “Bypass FQDN” in TC, but it adds more 
> parameters like redirecting to error domains or origin servers.
> 
> All the new parameters will be configured through “DS Profile” (to avoid GUI 
> and DB schema changes), which is different with the current GUI configuration 
> of “Bypass FQDN”. So the design will not change the current “Bypass FQDN” 
> feature, and just add Last-resort as another extra feature. If Last-resort 
> feature is not enabled (by default), the current behavior of Traffic Router 
> will not change.
> 
> Please refer to following doc for more details: 
> https://docs.google.com/document/d/1ttrZUGoGZTdCT6g78v62xBVoDCC_cWesPzxfLaHBcp0/edit
> 
> Any comments are welcome.
> 
> Thanks,
> John
> 

Re: Traffic Server Secondary Streaming IPs Design

[Eric Friedrich (efriedri)]

I would prefer a consistent way to store all interface and IP address 
information. Its good database design practice to store similar information in 
similar tables (i.e. all IP info in 1 table) rather than keep some IPs in the 
server table and some IPs in another table. 

I also think this refactoring will give us greater flexibility for more changes 
in the future. Outside of this particular use case, we might have additional 
features like sharing edges between public/private networks or having multiple 
(equal priority) streaming interfaces on a cache.

These future features would be easier if the interface data and IP data is all 
organized into separate tables. 

I’d also like to see the delivery service to IP mapping be a many to many 
mapping in the DB. For this particular feature we will only assign a single IP 
(and we can restrict that in the API if we want), but I am near certain that in 
the future we would like the ability to assign a DS to multiple IPs on the same 
cache. 


—Eric



> On Apr 3, 2018, at 2:42 AM, Zhilin Huang (zhilhuan)  
> wrote:
> 
> Hi Mark,
> 
> Thanks for your comments. Please check my reply in another thread:
> 
> If we all agreed to use unified tables for all IPs and/or interfaces: 
> primary, management, secondary, then there need to be two tables: IP and 
> interface.
> And in the server table, we need to replace the original "interface_xxx", 
> "ip_xxx", "ip6_xxx" fields with a "primary_ip_id" field. And do similar 
> things to management IP.
> 
> Thanks,
> Zhilin
> 
> 
> On 03/04/2018, 7:08 AM, "Mark Torluemke"  wrote:
> 
>I would support an 'interfaces' table (adding some sort of a 'type' column)
>that would include moving the management and lights out management
>interfaces to that table as well.
> 
>Cheers,
>Mark
> 
>On Mon, Apr 2, 2018 at 2:39 PM, Nir Sopher  wrote:
> 
>> Hi Zhilin,
>> 
>> I took a quick look into the spec. Hope to have the opportunity to dive
>> deeper into it soon so we can further discuss it.
>> 
>> For now I have a 2 questions.
>> In the spec, you refer to "secondary interfaces", and you have a list of
>> secondary interfaces added.
>> IIUC the secondary interfaces are used as long as they are available, and
>> when down, you move to the primary interface.
>> 
>> Why not, instead of holding a secondary interfaces table, move all
>> interfaces to a separate table? Primary and secondary.
>> For each interface you can hold:
>> 
>>   - Server id
>>   - name (e.g. eth0)
>>   - IPv6
>>   - IPv4
>>   - Priority (Integer as flexible as you wish: e.g. "1" for "secondary",
>>   "2" for "primary" in your example,)
>> 
>> 
>> Additionally, it is not clear to me what happens if one of the interfaces
>> fails?
>> Does every interface has a unique DNS name? If an interface fails, are
>> redirects
>> sent only to the available (secondary) interfaces?
>> 
>> Thanks,
>> Nir
>> 
>> 
>> On Mon, Apr 2, 2018 at 10:21 AM, Zhilin Huang (zhilhuan) <
>> zhilh...@cisco.com
>>> wrote:
>> 
>>> Hi Guys,
>>> 
>>> This was originally posted in another discussion. Resend this in a
>>> standalone topic to catch more awareness. The link for the design doc:
>>> https://docs.google.com/document/d/1vgq-pGNoLLYf7Y3cu5hWu67TUKpN5hucrp
>>> -ZS9nSsd4/edit?usp=sharing
>>> 
>>> 
>>> Short summary for the feature design:
>>> ---
>>> There is feature request from market to add secondary IPs support on edge
>>> cache servers, and the functionality to assign a delivery service to a
>>> secondary IP of an edge cache.
>>> 
>>> This feature requires Traffic Ops implementation to support secondary IP
>>> configuration for edge cache, and delivery service assignment to
>> secondary
>>> IP.
>>> 
>>> Traffic Monitor should also monitor connectivity of secondary IPs
>>> configured. And Traffic Router needs support to resolve streamer FQDN to
>>> secondary IP assigned in a delivery service.
>>> 
>>> Traffic Server should record the IP serving client request. And should
>>> reject request to an unassigned IP for a delivery service.
>>> 
>>> This design has taken compatibility into consideration: if no secondary
>> IP
>>> configured, or some parts of the system has not been upgraded to the
>>> version supports this feature, the traffic will be served by primary IPs
>> as
>>> before.
>>> ---
>>> 
>>> Much appreciated and welcome to any comments. If no major objections, we
>>> planned to start coding this week.
>>> 
>>> Thanks,
>>> Zhilin
>>> 
>>> 
>> 
> 
> 

Re: Traffic Server Secondary Streaming IPs Design

[Eric Friedrich (efriedri)]

For our particular use case, it is important that if a secondary IP is down, 
the traffic does not move to the primary IP.

This use case relates to use of the source IP address to differentiate traffic 
for billing purposes. We don’t want traffic to move to the primary IP as this 
would miscategorize the traffic. 

—Eric

> On Apr 2, 2018, at 4:39 PM, Nir Sopher  wrote:
> 
> Hi Zhilin,
> 
> I took a quick look into the spec. Hope to have the opportunity to dive
> deeper into it soon so we can further discuss it.
> 
> For now I have a 2 questions.
> In the spec, you refer to "secondary interfaces", and you have a list of
> secondary interfaces added.
> IIUC the secondary interfaces are used as long as they are available, and
> when down, you move to the primary interface.
> 
> Why not, instead of holding a secondary interfaces table, move all
> interfaces to a separate table? Primary and secondary.
> For each interface you can hold:
> 
>   - Server id
>   - name (e.g. eth0)
>   - IPv6
>   - IPv4
>   - Priority (Integer as flexible as you wish: e.g. "1" for "secondary",
>   "2" for "primary" in your example,)
> 
> 
> Additionally, it is not clear to me what happens if one of the interfaces
> fails?
> Does every interface has a unique DNS name? If an interface fails, are
> redirects
> sent only to the available (secondary) interfaces?
> 
> Thanks,
> Nir
> 
> 
> On Mon, Apr 2, 2018 at 10:21 AM, Zhilin Huang (zhilhuan) > wrote:
> 
>> Hi Guys,
>> 
>> This was originally posted in another discussion. Resend this in a
>> standalone topic to catch more awareness. The link for the design doc:
>> https://docs.google.com/document/d/1vgq-pGNoLLYf7Y3cu5hWu67TUKpN5hucrp
>> -ZS9nSsd4/edit?usp=sharing
>> 
>> 
>> Short summary for the feature design:
>> ---
>> There is feature request from market to add secondary IPs support on edge
>> cache servers, and the functionality to assign a delivery service to a
>> secondary IP of an edge cache.
>> 
>> This feature requires Traffic Ops implementation to support secondary IP
>> configuration for edge cache, and delivery service assignment to secondary
>> IP.
>> 
>> Traffic Monitor should also monitor connectivity of secondary IPs
>> configured. And Traffic Router needs support to resolve streamer FQDN to
>> secondary IP assigned in a delivery service.
>> 
>> Traffic Server should record the IP serving client request. And should
>> reject request to an unassigned IP for a delivery service.
>> 
>> This design has taken compatibility into consideration: if no secondary IP
>> configured, or some parts of the system has not been upgraded to the
>> version supports this feature, the traffic will be served by primary IPs as
>> before.
>> ---
>> 
>> Much appreciated and welcome to any comments. If no major objections, we
>> planned to start coding this week.
>> 
>> Thanks,
>> Zhilin
>> 
>> 

Re: [VOTE] Resolution for Traffic Control graduation to TLP

[Eric Friedrich (efriedri)]

+1
> On Apr 2, 2018, at 4:11 PM, David Neuman  wrote:
> 
> Dear Traffic Control community members:
> 
> I would like to call a vote on the resolution for Traffic Control to
> graduate from to an Apache TLP.  We have already voted on whether or not we
> should start the graduation process [1] and this is the next step.  Please
> see the resolution below and vote as follows:
> 
> [ ] +1 Graduate Traffic Control from the incubator
> [ ] +0 No Opinion
> [ ] -1 Do not graduate Traffic Control from the incubator (please provide a
> reason)
> 
> The vote is open for a minimum of 72 hours and will need at least 3 more +1
> votes than -1 votes from PMC members to succeed.
> 
> If this VOTE succeeds, a similar VOTE will be started on the
> gene...@incubator.apache.org mailing list. If that VOTE succeeds, a
> resolution will be included in the next Apache Board Meeting.
> 
> Please feel free to reach out with any questions.
> 
> Thanks,
> Dave
> 
> [1]
> https://lists.apache.org/thread.html/fb1fae0785feb6568cef6deb6fa20723eba54ed63a445462d44564d3@%3Cdev.trafficcontrol.apache.org%3E
> 
> -
> 
> Establish the Apache Traffic Control Project
> 
> WHEREAS, the Board of Directors deems it to be in the best interests of
> the Foundation and consistent with the Foundation's purpose to establish
> a Project Management Committee charged with the creation and maintenance
> of open-source software, for distribution at no charge to the public,
> related to building, monitoring, configuring, and provisioning a large
> scale content delivery network (CDN)..
> 
> NOW, THEREFORE, BE IT RESOLVED, that a Project Management Committee
> (PMC), to be known as the "Apache Traffic Control Project", be and
> hereby is established pursuant to Bylaws of the Foundation; and be it
> further
> 
> RESOLVED, that the Apache Traffic Control Project be and hereby is
> responsible for the creation and maintenance of software related to
> building, monitoring, configuring, and provisioning a large scale
> content delivery network (CDN).; and be it further
> 
> RESOLVED, that the office of "Vice President, Apache Traffic Control" be
> and hereby is created, the person holding such office to serve at the
> direction of the Board of Directors as the chair of the Apache Traffic
> Control Project, and to have primary responsibility for management of
> the projects within the scope of responsibility of the Apache Traffic
> Control Project; and be it further
> 
> RESOLVED, that the persons listed immediately below be and hereby are
> appointed to serve as the initial members of the Apache Traffic Control
> Project:
> 
> * Dan Kirkwood   
> * David Neuman   
> * Dewayne Richardson 
> * Eric Covener   
> * Eric Friedrich 
> * Hank Beatty
> * Jan van Doorn  
> * Jeff Elsloo
> * Jeremy Mitchell
> * Leif Hedstrom  
> * Mark Torluemke 
> * Phil Sorber
> * Steve Malenfant
> 
> NOW, THEREFORE, BE IT FURTHER RESOLVED, that David Neuman be appointed
> to the office of Vice President, Apache Traffic Control, to serve in
> accordance with and subject to the direction of the Board of Directors
> and the Bylaws of the Foundation until death, resignation, retirement,
> removal or disqualification, or until a successor is appointed; and be
> it further
> 
> RESOLVED, that the initial Apache Traffic Control PMC be and hereby is
> tasked with the creation of a set of bylaws intended to encourage open
> development and increased participation in the Apache Traffic Control
> Project; and be it further
> 
> RESOLVED, that the Apache Traffic Control Project be and hereby is
> tasked with the migration and rationalization of the Apache Incubator
> Traffic Control podling; and be it further
> 
> RESOLVED, that all responsibilities pertaining to the Apache Incubator
> Traffic Control podling encumbered upon the Apache Incubator PMC are
> hereafter discharged.

Re: TM: Question about the poll model of the Traffic Monitor

[Eric Friedrich (efriedri)]

The use case behind this question probably deserves a longer dev@ email.

I will oversimplify: we are extending TC to support multiple IPv4 (or multiple 
IPv6) addresses per edge cache (across 1 or more NICs). 

Assume all addresses are reachable from the TM. 

—Eric


> On Mar 28, 2018, at 11:37 AM, Robert Butts  wrote:
> 
> When you say different interfaces, do you mean IPv4 versus IPv6? Or
> something else?
> 
> If you mean IPv4 vs IPv6, we have a PR for that from Dylan Volz
> https://github.com/apache/incubator-trafficcontrol/pull/1627
> 
> I'm hoping to get to it early next week, just haven't found the time to
> review and test it yet.
> 
> Or did you mean something else by "interface"? Linux network interfaces?
> Ports?
> 
> 
> On Wed, Mar 28, 2018 at 12:02 AM, Neil Hao (nbaoping) 
> wrote:
> 
>> Hi,
>> 
>> Currently, we poll exact one URL request to each cache server for one
>> interface, but now we’d like to add multiple interfaces support, therefore,
>> we need multiple requests to query each interface of the cache server, I
>> check the code of Traffic Monitor, it seems we don’t support this kind of
>> polling, right?
>> 
>> I figure out different ways to support this:
>> 1) The first way: change the ‘Urls’ field in the HttpPollerConfig from
>> ‘map[string]PollConfig’ to ‘map[string][]PollConfig’, so that we can have
>> multiple polling config to query the multiple interfaces info.
>> 
>> 2) The second way: Change the ‘URL’ field in the PollConfig from ‘string’
>> to ‘[]string’.
>> 
>> No matter which way, it seems it will bring a little big change to the
>> current polling model. I’m not sure if I’m on the right direction, would
>> you guys have suggestions for this?
>> 
>> Thanks,
>> Neil
>> 

Re: Delivery Service Origin Refactor

[Eric Friedrich (efriedri)]

> On Mar 22, 2018, at 12:27 PM, Rawlin Peters  wrote:
> 
> This Origin Refactor proposal was probably too much to parse at once.
> Here's a slightly shorter version:
> 1. Split Locations out of the Cachegroup table into their own table
EF> Location is latitude/longitude?


> 2. Split Origins out of the Server and Delivery Service tables into
> their own table (Origin would have a FK to DeliveryService, supporting
> a one-to-many relationship between DSes and Origins)
EF> What are the properties of an Origin?

EF> Also, having a separate origin object might let us prevent for the 
messiness that comes when adding the same origin as both a Live DS and a VOD DS 
on the same cache (today hosting.config stores both DS’ on disk)

> 
> In order to make the transition smoother, I'd like to try to keep the
> existing Delivery Service and Cachegroup API request/response formats
> in place, but their implementations would change to populate the new
> DB tables. For instance, if I create a DS with the Origin
> http://www.example.com, the DS API backend would now parse that URL
> and create a new row in the Origin table with a foreign key to that
> DS.
> 
> For the Cachegroup API, if I create a new Cachegroup with the
> coordinates [1, 2], the backend would then create a new row in the
> Location table using those coordinates (probably name it generically
> using the Cachegroup name), and the cachegroup row would contain a
> foreign key to that Location.
> 
> Of course there would also be new API endpoints to CRUD Locations and
> Origins by themselves.
> 
> Initially, the current MSO implementation would continue to use
> origins created using the Server table, but we would transition the
> MSO implementation over to using the new Origin table eventually.
> 
> Questions/thoughts/concerns about any of this? All feedback is welcome.
> 
> Thanks,
> Rawlin
> 
> On Mon, Mar 12, 2018 at 1:46 PM, Rawlin Peters  
> wrote:
>> Hey folks,
>> 
>> As promised, this email thread will be to discuss how to best
>> associate an Origin Latitude/Longitude with a Delivery Service,
>> primarily so that steering targets can be ordered/sent to the client
>> based upon the location of those targets (i.e. the Origin), a.k.a.
>> Steering Target Geo-Ordering. This is potentially going to be a pretty
>> large change, so all your feedback/questions/concerns are appreciated.
>> 
>> Here were a handful of bad ideas I had in order to accomplish this DS
>> Origin Lat/Long association (feel free to skip to PROPOSED SOLUTION
>> below):
>> 
>> 1. Reuse the current MSO (multisite origin) backend (i.e. add the
>> origin into the servers table, give it a lat/long from its cachegroup,
>> assign the origin server to the DS)
>> Pros:
>> - reuse of existing db schema, probably wouldn't have to add any new
>> tables/columns
>> Cons:
>> - MSO configuration is already very complex
>> - for the simple case of just wanting to give an Origin a lat/long you
>> have to create a server (of which only a few fields make sense for an
>> Origin), add it to a cachegroup (only name and lat/long make sense,
>> won't use parent relationships, isn't really a "group" of origins),
>> assign it to a server profile (have to create one first, no parameters
>> are needed), and finally assign that Origin server to the delivery
>> service (did I miss anything?)
>> 
>> 2. Add Origin lat/long columns to the deliveryservice table
>> Pros:
>> - probably the most straightforward solution for Steering Target
>> Geo-Ordering given that Origin FQDN is currently a DS field.
>> Cons:
>> - doesn't work well with MSO
>> - could be confused with Default Miss Lat/Long
>> - if two different delivery services use colocated origins, the same
>> lat/long needs entered twice
>> - adds yet another column to the crowded deliveryservice table
>> 
>> 3. Add origin lat/long parameters to a Delivery Service Profile
>> Pros:
>> - Delivery Services using colocated origins could share the same profile
>> - no DB schema updates needed
>> Cons:
>> - profile parameters lack validation
>> - still doesn't support lat/long for multiple origins associated with a DS
>> 
>> 4. Add the lat/long to the steering target itself (i.e. where you
>> choose weight/order, you'd also enter lat/long)
>> Pros:
>> - probably the easiest/quickest solution in terms of development
>> Cons:
>> - only applies lat/long to a steering target
>> - using the same target in multiple Steering DSes means having to keep
>> the lat/long synced between them all
>> - lat/long not easily reused by other areas that may need it in the future
>> 
>> 
>> 
>> PROPOSED SOLUTION:
>> 
>> All of those ideas were suboptimal, which is why I think we need to:
>> 1. Split Locations out of the cachegroup table into their own table
>> with the following columns (cachegroup would have a foreign key to
>> Location):
>> - name
>> - latitude
>> - longitude
>> 
>> 2. Split Origins out of the server and deliveryservice 

Re: Delivery Service Origin Refactor

[Eric Friedrich (efriedri)]

How much does distance to origin actually impact a Live delivery service? 

  Its only the first client for each segment that has to go back to the origin- 
the rest of the responses are cached, so they don’t touch the origin at all. 


Whats the use case for HTTP_NO_CACHE with Client Steering? Is there a reason 
plain-old MultiSiteOrigin wouldn’t work for you on the NO_CACHE delivery 
services?

—Eric





> On Mar 14, 2018, at 11:45 AM, Rawlin Peters <rawlin.pet...@gmail.com> wrote:
> 
> Yes, I'd say that's essentially the main goal - prioritizing redundant
> HTTP_LIVE/HTTP_NO_CACHE deliveryservices to have the shortest distance
> between the edge and the origin. HTTP-type deliveryservices that use
> the MID tier, though, don't really make sense for this feature, so we
> might want to limit geo-steering targets to just
> HTTP_LIVE/HTTP_NO_CACHE.
> 
> On Wed, Mar 14, 2018 at 8:56 AM, Eric Friedrich (efriedri)
> <efrie...@cisco.com> wrote:
>> I understand the goals behind Client Steering Delivery Services, but I don’t 
>> fully understand the motivation behind these changes.
>> 
>> We want redundancy in our live delivery services.  Take a national channel 
>> and make two copies of it on two origins. Clients can choose DiscoveryA or 
>> DiscoveryB based on which one is working better (or at all) for them. All 
>> CGs would need both Delivery Services assigned to account for failures where 
>> caches holding either Just A or Just B might go offline.
>> 
>> OriginA and OriginB would typically be in different locations for the most 
>> redundancy.
>> 
>> If I’m a client in Boston and I ask for Discovery channel, TR should give me 
>> redirects to DiscoveryA and DiscoveryB  both in the Boston cache group.
>> 
>> Is the goal behind this feature for TR to prioritize the DS list given to 
>> the client based on how far origin A is from Boston vs. how far B is from 
>> Boston?
>> 
>> —Eric
>> 
>> 
>> 
>> 
>> 
>> 
>>> On Mar 13, 2018, at 1:42 PM, Rawlin Peters <rawlin.pet...@gmail.com> wrote:
>>> 
>>> replies inline
>>> 
>>> On Mon, Mar 12, 2018 at 5:21 PM, Nir Sopher <n...@qwilt.com> wrote:
>>>> Thank you Rawlin for the clarification:)
>>> 
>>> You're welcome. Anything I can do to help :)
>>> 
>>>> 
>>>> Still, I feel like I'm missing a piece of the puzzle here.
>>>> Maybe I do no understand the relations of "origin" and "steering target"
>>>> 
>>>> As I see it the router job is to send end users to the optimal cache. It
>>>> has 2 tools for doing so: CZF and Geo
>>>> Using the CZF is preferable, as it is based on the real network topology.
>>>> Geo is a best effort solution, used when we cannot do better. It is not
>>>> necessarily optimal, and has GEO misses, but we must use it since we cannot
>>>> map all IPs.
>>> 
>>> 
>>> Yes, the client's location will be found from the CZF first, falling
>>> back to GEO upon a CZF-miss. Then the most optimal edge cachegroup is
>>> chosen for each steering target deliveryservice. Then, the resulting
>>> list of target deliveryservices will be sorted by total distance
>>> following the path from client -> edge -> origin.
>>> 
>>>> 
>>>> The cache job is to fetch the content and serve the user.
>>>> It can be optimized to bring the content from the optimal Origin. It can be
>>>> configured to do so by specifying the best origin per cache group (in ops
>>>> DB).
>>> 
>>> 
>>> This is intentionally done as a CLIENT_STEERING deliveryservice so
>>> that a smart client can make the decision to use a different
>>> deliveryservice upon failure. If this decision was made at the caching
>>> proxy level, it would end up being like an optimized version of MSO
>>> (multi-site origin) where the client only has a single URL to request
>>> and the most optimal origin of multiple origins is chosen by the
>>> caching proxy. I don't think that's a bad idea; it's just not the
>>> architecture we want for this. By doing it as client steering we can
>>> also assign weights/ordering between colocated origins and update
>>> those steering assignments at any time. We can form the steering
>>> target list very flexibly this way.
>>> 
>>> 
>>>> I might be naive here, but as the amount of cache groups is reasonable, and
>>>> their network location is much clearer the the end user 

Re: Delivery Service Origin Refactor

[Eric Friedrich (efriedri)]

I understand the goals behind Client Steering Delivery Services, but I don’t 
fully understand the motivation behind these changes.

We want redundancy in our live delivery services.  Take a national channel and 
make two copies of it on two origins. Clients can choose DiscoveryA or 
DiscoveryB based on which one is working better (or at all) for them. All CGs 
would need both Delivery Services assigned to account for failures where caches 
holding either Just A or Just B might go offline.

OriginA and OriginB would typically be in different locations for the most 
redundancy. 

If I’m a client in Boston and I ask for Discovery channel, TR should give me 
redirects to DiscoveryA and DiscoveryB  both in the Boston cache group.

Is the goal behind this feature for TR to prioritize the DS list given to the 
client based on how far origin A is from Boston vs. how far B is from Boston?

—Eric
   

 



> On Mar 13, 2018, at 1:42 PM, Rawlin Peters  wrote:
> 
> replies inline
> 
> On Mon, Mar 12, 2018 at 5:21 PM, Nir Sopher  wrote:
>> Thank you Rawlin for the clarification:)
> 
> You're welcome. Anything I can do to help :)
> 
>> 
>> Still, I feel like I'm missing a piece of the puzzle here.
>> Maybe I do no understand the relations of "origin" and "steering target"
>> 
>> As I see it the router job is to send end users to the optimal cache. It
>> has 2 tools for doing so: CZF and Geo
>> Using the CZF is preferable, as it is based on the real network topology.
>> Geo is a best effort solution, used when we cannot do better. It is not
>> necessarily optimal, and has GEO misses, but we must use it since we cannot
>> map all IPs.
> 
> 
> Yes, the client's location will be found from the CZF first, falling
> back to GEO upon a CZF-miss. Then the most optimal edge cachegroup is
> chosen for each steering target deliveryservice. Then, the resulting
> list of target deliveryservices will be sorted by total distance
> following the path from client -> edge -> origin.
> 
>> 
>> The cache job is to fetch the content and serve the user.
>> It can be optimized to bring the content from the optimal Origin. It can be
>> configured to do so by specifying the best origin per cache group (in ops
>> DB).
> 
> 
> This is intentionally done as a CLIENT_STEERING deliveryservice so
> that a smart client can make the decision to use a different
> deliveryservice upon failure. If this decision was made at the caching
> proxy level, it would end up being like an optimized version of MSO
> (multi-site origin) where the client only has a single URL to request
> and the most optimal origin of multiple origins is chosen by the
> caching proxy. I don't think that's a bad idea; it's just not the
> architecture we want for this. By doing it as client steering we can
> also assign weights/ordering between colocated origins and update
> those steering assignments at any time. We can form the steering
> target list very flexibly this way.
> 
> 
>> I might be naive here, but as the amount of cache groups is reasonable, and
>> their network location is much clearer the the end user location, the
>> mapping and configuration would be reasonable. Therefore, using sub-optimal
>> Geo as a tool for choosing the Origin can be avoided.
> 
> 
> In practice, you could set the coordinates of the Origin to that of
> the most optimal cachegroup, rather than assigning the Origin directly
> to said cachegroup. The effect would be the same I believe.
> 
>> 
>> I also did not understand if the suggestion is to use the client location
>> for choosing the origin, or the cache group location for choosing the
>> origin.
>> Using the client location for choosing the origin practically ignores the
>> accurate information provided by the CZF.
> 
> 
> It's a combination of the client location, the edge location, and the
> origin location (total distance from client -> edge -> origin).
> 
>> 
>> What am I missing?
>> 10x
>> Nir
>> 
>> On Mon, Mar 12, 2018 at 11:19 PM, Rawlin Peters 
>> wrote:
>> 
>>> Hey Nir,
>>> 
>>> I think part of the motivation for doing this in Traffic Router rather
>>> than the Caching Proxy is separation of concerns. TR is already
>>> concerned with routing a client to the best cache based upon the
>>> client's location, so TR is already well-equipped to make the decision
>>> of how Delivery Services (origins) should be prioritized based upon
>>> the client's location. That way the Caching Proxy (e.g. ATS) doesn't
>>> need to concern itself with its own location, the client's location,
>>> and the location of origins; it just needs to know how to get the
>>> origin's content and cache it. All the client needs to know is that
>>> they have a prioritized list of URLs to choose from; they don't need
>>> to be concerned about origin/edge locations because that
>>> prioritization will be made for them by TR.
>>> 
>>> The target DSes will have different origins primarily because they
>>> 

Re: Backup Cache Group Selection

[Eric Friedrich (efriedri)]

e the new TO API and Traffic Portal changes too if you want,
>>> but
>>>>>>>> I don't want to burden you with this unexpected work if you don't
>>>> want
>>>>>>>> it. I (or another willing contributor) could work on the necessary
>>> TO
>>>>>>>> API and Traffic Portal changes sometime in the near future and
>>>>>>>> integrate them with your Traffic Router enhancement.
>>>>>>>> 
>>>>>>>> - Rawlin
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Mon, Mar 12, 2018 at 7:39 AM, vijayanand.jayaman...@gmail.com
>>>>>>>> <vijayanand.jayaman...@gmail.com> wrote:
>>>>>>>>> 
>>>>>>>>> Rawlin,
>>>>>>>>> 
>>>>>>>>> I believe the following statement is not correct.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> However, after reading your initial proposal below, it sounds
>>> like
>>>> you
>>>>>>>>> might have Coverage Zones in your CZF that don't necessarily map
>>>> back
>>>>>>>>> to Cache Groups in TO. Might that be the case?
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> We can have Coverage Zones in CZF which don’t necessarily map in
>>> to
>>>>>> TO’s
>>>>>>>> configured list of Cache Groups. But then , it won’t be chosen as a
>>>>>> valid
>>>>>>>> backup in case of failure.
>>>>>>>>> 
>>>>>>>>> For example:
>>>>>>>>> GROUP1 and GROUP2 are Cache Groups configured in TO (and hence
>>>>>>>> cr-config) , where GROUP3 is not in TO. Even though GROUP3 is
>>>> specified
>>>>>> as
>>>>>>>> a backup for GROUP1, it wont be  chosen in case of GROUP1 failure ,
>>>>>> since
>>>>>>>> it is not in TO.
>>>>>>>>> {
>>>>>>>>>  "coverageZones": {
>>>>>>>>> "GROUP3": {
>>>>>>>>>  "network6": [
>>>>>>>>>"1234:567a::\/64",
>>>>>>>>>"1234:567b::\/64"
>>>>>>>>>  ],
>>>>>>>>>  "network": [
>>>>>>>>>"10.197.89.0\/24"
>>>>>>>>>  ]
>>>>>>>>>},
>>>>>>>>> 
>>>>>>>>> "GROUP2": {
>>>>>>>>>  "network6": [
>>>>>>>>>"1234:567a::\/64",
>>>>>>>>>"1234:567b::\/64"
>>>>>>>>>  ],
>>>>>>>>>  "network": [
>>>>>>>>>"10.197.69.0\/24"
>>>>>>>>>  ]
>>>>>>>>>},
>>>>>>>>>"GROUP1": {
>>>>>>>>>   "backupZones":{
>>>>>>>>>  "list": ["GROUP3"],? This wont be chosen as backup Cache
>>>> Group
>>>>>> in
>>>>>>>> case of failure , since it is not in crconfig.
>>>>>>>>>  "fallbackToClosestGroup":false
>>>>>>>>>   },
>>>>>>>>>  "network6": [
>>>>>>>>>"1234:5677::\/64",
>>>>>>>>>"1234:5676::\/64"
>>>>>>>>>  ],
>>>>>>>>>  "network": [
>>>>>>>>>"10.126.250.0\/24"
>>>>>>>>>  ]
>>>>>>>>>}
>>>>>>>>>  }
>>>>>>>>> }
>>>>>>>>> 
>>>>>>>>> So, i feel, the existing implementation of specifying backupZones
>>>>>>>> configuratioin in CZF should be fine.
>>>>>>>>> 
>>>>>>>>> Thanks,
>>>>>>>>> Vijayanand S
>>>>>>>>

Re: Backup Cache Group Selection

[Eric Friedrich (efriedri)]

Good point. I think it makes sense to move both the backupList and coordinates 
into the CG API. By move coordinates into the API, I’m implying that we 
consolidate into 1 set of coordinates per CG. The existing CG coordinates would 
now be used for both backup edge CG selection and initial edge cg selection 
when doing client geolocation. This can certainly happen in a different PR than 
the backupList getting committed. 

It looks as though the CZ file will give us more flexibility, but it is 
unneeded. Today that flexibility is only causing more complexity and more 
operational overhead. 

—Eric


> On Mar 9, 2018, at 5:21 PM, Rawlin Peters <rawlin.pet...@gmail.com> wrote:
> 
> So in your CZF example, we can't actually have two CZs using the same
> name ("a"), because when that JSON gets parsed one of the CZs will be
> overwritten so that there is only one "a" key in the JSON. The names
> would have to be "a1" and "a2" for instance and backupList [a, b, c]
> and [a, c, b], but I see your point in how that might be useful. But
> at that point maybe it's just better to create two empty cachegroups
> "a1" and "a2" in the API that first fall back to cachegroup "a".
> 
> If I'd been around at the time coordinates were added to the CZF, I
> think I would've -1'd the idea. The coordinates for Client Source and
> Cache Group Destination should be the same typically because you
> should know based upon your network topology which cachegroup to route
> clients to. By being able to specify backups, the coordinates in the
> CZF would become even less relevant. And fwiw at Comcast we generate
> the CZF using the coordinates from the Cache Group API anyways, so
> they always match.
> 
> Another major concern with this is the scalability of having the
> config in the CZF alone. If we can change these backupZone relations
> using the API, that means we can add it to Traffic Portal and have a
> much better UX rather than hard-coding these relations in whatever
> script we're using to generate the CZF. We'd get benefits like
> validation and typo safety, and who knows maybe in the future we could
> have a map in TP to visualize the relationships between cache groups
> for troubleshooting.
> 
> - Rawlin
> 
> On Fri, Mar 9, 2018 at 2:22 PM, Eric Friedrich (efriedri)
> <efrie...@cisco.com> wrote:
>> "I can't imagine why we'd ever want the two sets of coordinates to differ 
>> for the same Cache Group. “
>> Maybe someone else can chime in about why coordinates were added to the CZF 
>> in the first place, but I’ve also thought of them like this:
>> CG API Coordinates - Where the cache servers are. To be used as a 
>> destination location routing traffic towards
>> CZF Coordinates - Where the clients are. To be used as the source location 
>> routing traffic from these coordinates to the CG API coordinates.
>> 
>> I could see cases where:
>> a) you might set the CG coordinates of DenCG to Denver’s lat long
>> b) you might set the CZF Coordinates of that coverageZone to somewhere other 
>> than Denver (because the coverageZone is not a perfect circle centered on 
>> Denver, like the caches might be)
>> 
>> So I think there are valid reason’s for two sets of coordinates to exist, 
>> but I’m not sure if people set them differently in practice? If they are 
>> always the same, for every CG it seems like they should get consolidated. (I 
>> don’t think we’re using CZF coordinates currently)
>> 
>> Second Point:
>>  By having the backupList in the CZF, we have more granularity on how 
>> clients are routed during failures. For example, could we create a CZF that 
>> looks like
>> “coverageZones”: {
>>  “a”: {
>>“backupList”: [b,c]
>>“networks": [1,2,3]
>>  },
>> 
>> “a”: {
>>“backupList”: [c,b]
>>“networks": [4,5,6]
>>  }
>> }
>> 
>> Here all clients are part of the “a” CacheGroup/Coverage Zone, but depending 
>> on their specific subnet they have different backup policies.
>> 
>> Our particular requirement for this feature is a backup at the CacheGroup 
>> level, not the CZ level as I’ve shown here- so perhaps we’re overbuilding it.
>> 
>> —Eric
>> 
>> 
>> 
>> 
>> 
>>> On Mar 9, 2018, at 4:05 PM, Rawlin Peters <rawlin.pet...@gmail.com> wrote:
>>> 
>>> Ok, so the coordinates in the CZF are only used when no available
>>> cache is found in the matched cachegroup. Rather than using the
>>> coordinates of the matched cachegroup that it already knows about from
>>> the API, Traffi

Re: Backup Cache Group Selection

[Eric Friedrich (efriedri)]

"I can't imagine why we'd ever want the two sets of coordinates to differ for 
the same Cache Group. “
Maybe someone else can chime in about why coordinates were added to the CZF in 
the first place, but I’ve also thought of them like this:
CG API Coordinates - Where the cache servers are. To be used as a destination 
location routing traffic towards
CZF Coordinates - Where the clients are. To be used as the source location 
routing traffic from these coordinates to the CG API coordinates. 

I could see cases where:
a) you might set the CG coordinates of DenCG to Denver’s lat long
b) you might set the CZF Coordinates of that coverageZone to somewhere other 
than Denver (because the coverageZone is not a perfect circle centered on 
Denver, like the caches might be)

So I think there are valid reason’s for two sets of coordinates to exist, but 
I’m not sure if people set them differently in practice? If they are always the 
same, for every CG it seems like they should get consolidated. (I don’t think 
we’re using CZF coordinates currently)

Second Point:
  By having the backupList in the CZF, we have more granularity on how clients 
are routed during failures. For example, could we create a CZF that looks like
“coverageZones”: {
  “a”: {
“backupList”: [b,c]
“networks": [1,2,3]
  },

 “a”: {
“backupList”: [c,b]
“networks": [4,5,6]
  }
}

Here all clients are part of the “a” CacheGroup/Coverage Zone, but depending on 
their specific subnet they have different backup policies. 

Our particular requirement for this feature is a backup at the CacheGroup 
level, not the CZ level as I’ve shown here- so perhaps we’re overbuilding it.

—Eric





> On Mar 9, 2018, at 4:05 PM, Rawlin Peters <rawlin.pet...@gmail.com> wrote:
> 
> Ok, so the coordinates in the CZF are only used when no available
> cache is found in the matched cachegroup. Rather than using the
> coordinates of the matched cachegroup that it already knows about from
> the API, Traffic Router uses the coordinates from the CZF cachegroup
> instead. That seems...not great?
> 
> That means we basically have two sets of coordinates for the same
> cachegroup, one set in the API and one in the CZF. The API coordinates
> are used when caches are available, but the CZF coordinates are used
> when no caches are available in the matched CG. To me it seems like
> the CG coordinates from the API should always take precedent over the
> CZF. In fact the CZF coordinates are optional, but TR currently won't
> use the coordinates from the API even if the CZF has no coordinates.
> That sounds like a bug to me.
> 
> Should we update TR to let the API coordinates take precedence over
> the CZF coordinates when available? I can't imagine why we'd ever want
> the two sets of coordinates to differ for the same Cache Group. Then
> the coordinates backup logic will truly live in the Cache Group API,
> and we could put the new backup config there as well. After that,
> coordinates would only be needed in the CZF for coverage zones that
> don't map to cache groups.
> 
> - Rawlin
> 
> On Fri, Mar 9, 2018 at 12:46 PM, Eric Friedrich (efriedri)
> <efrie...@cisco.com> wrote:
>> I think the original reason for putting it in the CZF was to stay consistent 
>> with the coordinates backup logic which is also in the CZF.
>> 
>> Unless you have multiple “coordinates” for different networks in the same 
>> zone, would it also make sense to add the coordinates to the Cache Group API 
>> as well?
>> 
>> I think all of our zones are also Cachegroups.
>> 
>> —Eric
>> 
>>> On Mar 9, 2018, at 1:31 PM, Rawlin Peters <rawlin.pet...@gmail.com> wrote:
>>> 
>>> Hey Eric (and others),
>>> 
>>> I'm resurrecting this thread because the PR [1] implementing this
>>> proposed functionality is just about ready to be merged. The full
>>> mailing list discussion can be read here [2] if interested.
>>> 
>>> I've discussed this PR a bit more with my colleagues here at Comcast,
>>> and while it provides the functionality we need, we think in the
>>> long-term this configuration should live in the Cache Group API in
>>> Traffic Ops rather than just the Coverage Zone File.
>>> 
>>> However, after reading your initial proposal below, it sounds like you
>>> might have Coverage Zones in your CZF that don't necessarily map back
>>> to Cache Groups in TO. Might that be the case? That scenario seems to
>>> be allowed by Traffic Router but might not necessarily be "supported"
>>> given the CZF docs [3] that state:
>>>> "The Coverage Zone File (CZF) should contain a cachegroup name to network 
>>>> prefix mapping in the for

Re: Backup Cache Group Selection

[Eric Friedrich (efriedri)]

I think the original reason for putting it in the CZF was to stay consistent 
with the coordinates backup logic which is also in the CZF. 

Unless you have multiple “coordinates” for different networks in the same zone, 
would it also make sense to add the coordinates to the Cache Group API as well? 

I think all of our zones are also Cachegroups. 

—Eric

> On Mar 9, 2018, at 1:31 PM, Rawlin Peters <rawlin.pet...@gmail.com> wrote:
> 
> Hey Eric (and others),
> 
> I'm resurrecting this thread because the PR [1] implementing this
> proposed functionality is just about ready to be merged. The full
> mailing list discussion can be read here [2] if interested.
> 
> I've discussed this PR a bit more with my colleagues here at Comcast,
> and while it provides the functionality we need, we think in the
> long-term this configuration should live in the Cache Group API in
> Traffic Ops rather than just the Coverage Zone File.
> 
> However, after reading your initial proposal below, it sounds like you
> might have Coverage Zones in your CZF that don't necessarily map back
> to Cache Groups in TO. Might that be the case? That scenario seems to
> be allowed by Traffic Router but might not necessarily be "supported"
> given the CZF docs [3] that state:
>> "The Coverage Zone File (CZF) should contain a cachegroup name to network 
>> prefix mapping in the form:"
> 
> If we do indeed "support" this scenario, that would mean that having
> the backupZone config only in TO wouldn't solve all your use cases if
> your CZF heavily uses Coverage Zones that don't directly map to a
> Cache Group in TO.
> 
> If we should officially support this scenario, then maybe we merge the
> PR [1] as is, then later we can augment the feature so that we can use
> the Cache Group API to provide the backupZone config as well as in the
> CZF. If the config was provided in both the API and the CZF, then the
> API would take precedent.
> 
> If this scenario should NOT officially be supported, then I think we
> should update the PR [1] to have Traffic Router parse the config from
> CRConfig.json rather than the CZF and augment the Cache Group API to
> support the backupZone config. I think this would be the most ideal
> solution, but I also don't want to sign up our contributors for extra
> work that they weren't planning on doing. I'd be happy to help augment
> this feature on the TO side.
> 
> What do you all think of this proposal? TO-only or both TO and CZF?
> 
> - Rawlin
> 
> [1] https://github.com/apache/incubator-trafficcontrol/pull/1908
> [2] 
> https://lists.apache.org/thread.html/b033b3943c22a606370ad3981fa05fb0e7039161b88bbc035bc49b25@%3Cdev.trafficcontrol.apache.org%3E
> [3] 
> http://traffic-control-cdn.readthedocs.io/en/latest/admin/traffic_ops/using.html#the-coverage-zone-file-and-asn-table
> 
> On 2016/12/22 19:28:17, Eric Friedrich (efriedri) <efrie...@cisco.com> wrote:
>> The current behavior of cache group selection works as follows
>> 1) Look for a subnet match in CZF
>> 2) Use MaxMind/Neustar for GeoLocation based on client IP. Choose closest 
>> cache group.
>> 3) Use Delivery Service Geo-Miss Lat/Long. Choose closest cache group.
>> 
>> 
>> For deployments where IP addressing is primarily private (say RFC-1918 
>> addresses), client IP Geo Location (#2) is not useful.
>> 
>> 
>> We are considering adding another field to the Coverage Zone File that 
>> configures an ordered list of backup cache groups to try if the primary 
>> cache group does not have any available caches.
>> 
>> Example:
>> 
>> "coverageZones": {
>> "cache-group-01": {
>> “backupList”: [“cache-group-02”, “cache-group-03”],
>> "network6": [
>> "1234:5678::\/64”,
>> "1234:5679::\/64"],
>> "network": [
>> "192.168.8.0\/24",
>> "192.168.9.0\/24”]
>> }
>> 
>> This configuration could also be part of the per-cache group configuration, 
>> but that would give less control over which clients preferred which cache 
>> groups. For example, you may have cache groups in LA, Chicago and NY. If the 
>> Chicago Cache group fails, you may want some of the Chicago clients to go to 
>> LA and some to go to NY. If the backup CG configuration is per-cg, we would 
>> not be able to control where clients are allocated.
>> 
>> Looking for opinions and comments on the above proposal, this is still in 
>> idea stage.
>> 
>> Thanks All!
>> Eric

Re: [VOTE] Release Apache Traffic Control (incubating) 2.2.0-RC2

[Eric Friedrich (efriedri)]

+1 

I checked the following:
- Release hashes and signatures good
- Release builds via pkg
- TO, TR, TM fresh install and start/retrieve config
- Looked through RAT

Nitpick in the below email: the SHA checksum is a SHA-512 not a SHA-1

—Eric



> On Mar 5, 2018, at 1:25 PM, Robert Butts  wrote:
> 
> Hello All,
> 
> I've prepared a release for v2.2.0-RC2
> 
> The vote is open for at least 72 hours and passes if a majority of at least
> 3 +1 PPMC votes are cast.
> 
> [ ] +1 Approve the release
> 
> [ ] -1 Do not release this package because ...
> 
> Changes since 2.1.0:
> https://github.com/apache/incubator-trafficcontrol/compare/RELEASE-2.1.0...RELEASE-2.2.0-RC2
> 
> This corresponds to git:
> Hash: 6ea85056a1a69973be4a74b82d602df29f21f42d
> Tag: RELEASE-2.2.0-RC2
> 
> Which can be verified with the following: git tag -v RELEASE-2.2.0-RC2
> 
> My code signing key is available here:
> http://keys.gnupg.net/pks/lookup?search=0xFDD04F7F=vindex
> 
> Make sure you refresh from a key server to get all relevant signatures.
> 
> The source .tar.gz file, pgp signature (.asc signed with my key from
> above), md5 and sha1 checksums are provided here:
> 
> https://dist.apache.org/repos/dist/dev/incubator/trafficcontrol/2.2.0/RC2
> 
> 
> Thanks!

Re: [VOTE] Traffic Control graduation to TLP

[Eric Friedrich (efriedri)]

+1 for graduation
> On Mar 1, 2018, at 12:13 PM, Jan van Doorn  wrote:
> 
> +1
> 
> On Thu, Mar 1, 2018 at 9:38 AM Chris Lemmons  wrote:
> 
>> The Traffic Control project has grown so much over the last year, it's
>> incredible.
>> 
>> +1
>> 
>> On Thu, Mar 1, 2018 at 9:33 AM, Gelinas, Derek
>>  wrote:
>>> +1
>>> 
 On Mar 1, 2018, at 10:41 AM, Dave Neuman  wrote:
 
 Hey All,
 
 After a great discussion amongst the Apache Traffic Control PPMC,
>> reviewing
 the graduation checklist[1], updating the podling status page[2], and
 updating the project website to ensure the whimsy podling website checks
 pass[3], I would like to call a vote for Apache Traffic Control
>> graduating
 to a top level project.
 
 Apache Traffic Control entered the incubator on July 12, 2016.  Since
>> then
 we have announced 4 releases, nominated 4 new committers, organized 3
 summits, had almost 8,000 commits from 63 different contributors, and --
 most importantly -- we have grown and diversified our community.  Apache
 Traffic Control is a healthy project that is already acting like an
>> Apache
 top level project, so we should take the next step.
 
 If we agree that we should graduate to a top level project, the next
>> steps
 will be to pick the initial PMC chair for the TLP and draft a Resolution
 for the PPMC and IPMC to vote upon.
 
 Please take a minute to vote on wheter or not Traffic Control should
 graduate to a Top Level Project by responding with one of the following:
 
 [ ] +1 Apache Traffic Control should graduate.
 [ ] +0 No opinion
 [ ] -1 Apache Traffic Control should not graduate (please provide the
 reason)
 
 The VOTE will be opened for at least the next 72 hours.  Per Apache
 guidelines[4] I will also be notifying the incubator mailing list that a
 community vote is under way.
 
 Thanks,
 Dave
 
 
 [1]
 
>> https://incubator.apache.org/guides/graduation.html#graduation_check_list
 [2] http://incubator.apache.org/projects/trafficcontrol.html
 [3] https://whimsy.apache.org/pods/project/trafficcontrol
 [4]
 
>> https://incubator.apache.org/guides/graduation.html#community_graduation_vote
>>> 
>> 

Re: Google Summer of Code 2018 Mentor Registration

[Eric Friedrich (efriedri)]

I think API porting is certainly one option. 

What about features that aren’t on the roadmap- something that could be a good 
3 month project but isn’t likely to happen on its own?

- Perhaps a prototype feature like updates needed for HTTP/2 or a new 
implementation of the NetworkNode class?
- Maybe something UI related- drag and drop graphs on Traffic Portal or viewing 
Traffic Monitor “health” inside of Traffic Portal
- Not sure where things stand with Traffic Logs but maybe some type of data 
sciencey project?

Any other ideas? 

—Eric




> On Feb 27, 2018, at 12:26 PM, Mark Torluemke <mark.torlue...@gmail.com> wrote:
> 
> +1 on the API migration suggestion.
> 
> On Mon, Feb 26, 2018 at 10:50 AM, Dave Neuman <neu...@apache.org> wrote:
> 
>> I think any of the perl -> go API stuff would be great.
>> 
>> On Mon, Feb 26, 2018 at 10:19 AM, Eric Friedrich (efriedri) <
>> efrie...@cisco.com> wrote:
>> 
>>> I agree, this would be a great way to grow the community.
>>> 
>>> Do we have a list of marked issues in Github that would be good
>> candidates?
>>> 
>>> Or perhaps we can brainstorm some issues here on the mailer?
>>> 
>>> —Eric
>>> 
>>>> On Feb 24, 2018, at 5:18 PM, Phil Sorber <sor...@apache.org> wrote:
>>>> 
>>>> I think we should do this.
>>>> 
>>>> -- Forwarded message -
>>>> From: Ulrich Stärk <u...@apache.org>
>>>> Date: Sat, Feb 24, 2018 at 2:19 PM
>>>> Subject: Google Summer of Code 2018 Mentor Registration
>>>> To: <ment...@community.apache.org>
>>>> Cc: d...@community.apache.org <d...@community.apache.org>
>>>> 
>>>> 
>>>> Dear PMCs,
>>>> 
>>>> I'm happy to announce that the ASF has made it onto the list of
>> accepted
>>>> organizations for
>>>> Google Summer of Code 2018! [1,2]
>>>> 
>>>> It is now time for mentors to sign up, so please pass this email on to
>>> your
>>>> community and
>>>> podlings. If you aren’t already subscribed to
>>> ment...@community.apache.org
>>>> you should do so now else
>>>> you might miss important information.
>>>> 
>>>> Mentor signup requires two steps: mentor signup in Google's system [3]
>>> and
>>>> PMC acknowledgement.
>>>> 
>>>> If you want to mentor a project in this year's SoC you will have to
>>>> 
>>>> 1. Be an Apache committer.
>>>> 2. Request an acknowledgement from the PMC for which you want to mentor
>>>> projects. Use the below
>>>> template and *do not forget to copy ment...@community.apache.org*. We
>>> will
>>>> use the email adress you
>>>> indicate to send the invite to be a mentor for Apache.
>>>> 
>>>> PMCs, read carefully please.
>>>> 
>>>> We request that each mentor is acknowledged by a PMC member. This is to
>>>> ensure the mentor is in good
>>>> standing with the community. When you receive a request for
>>>> acknowledgement, please ACK it and cc
>>>> ment...@community.apache.org
>>>> 
>>>> Lastly, it is not yet too late to record your ideas in Jira (see my
>>>> previous emails for details).
>>>> Students will now begin to explore ideas so if you haven’t already done
>>> so,
>>>> record your ideas
>>>> immediately!
>>>> 
>>>> Cheers,
>>>> 
>>>> Uli
>>>> 
>>>> mentor request email template:
>>>> 
>>>> to: private@.apache.org
>>>> cc: ment...@community.apache.org
>>>> subject: GSoC 2018 mentor request for 
>>>> 
>>>>  PMC,
>>>> 
>>>> please acknowledge my request to become a mentor for Google Summer of
>>> Code
>>>> 2018 projects for Apache
>>>> .
>>>> 
>>>> I would like to receive the mentor invite to 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> [1] https://summerofcode.withgoogle.com/organizations/
>>>> [2] https://summerofcode.withgoogle.com/organizations/
>> 5718432427802624/
>>>> [3] https://summerofcode.withgoogle.com/
>>>> 
>>>> -
>>>> To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
>>>> For additional commands, e-mail: dev-h...@community.apache.org
>>> 
>>> 
>> 

Re: Steering Target Geo-Ordering

[Eric Friedrich (efriedri)]

Thanks Rawlin-
  Thanks for bearing with me. I’m thinking through this and wanted to 
brainstorm a little.

What’s the benefit of going to the Boston edge cache to get the Seattle or 
Denver DeliveryServices?
  - If you already have redundant origins in Boston, you’re protected against 
failure of a single origin.
  - If both Boston origins fail, the client would be routed to Denver and hit 
the local Denver origins.
  - If both Boston origins are down, then you wouldn’t be able to get Boston 
content in Denver anyway
 - But clients in Boston would now need to hit the Denver edges, perhaps 
overloading that CG. If this is a failure case you want to address, maybe we 
could do something like the following:

Would it be possible to create multiple levels of steering delivery services?
- Top Level: Client Steering DS 
(tr.nbc.cdn.customer.com<http://tr.nbc.cdn.com>) - assigned to all CG. Does 
GeoOrdering based on client proximity to assigned cache groups of the target DS'
- Target 1: Boston Client Steering DS 
(tr.bos-nbc.cdn.customer.com<http://tr.boston-nbc.cdn.customer.com>) - assigned 
only to Boston CG. Ranked steering policy
  - Target 1.1: Boston Live DS 
(tr.bos-bos-nbc.cdn.customer.com<http://tr.bos-bos-nbc.cdn.customer.com>), 
rank=1
  - Target 1.2: Denver Live DS 
(tr.den-bos-nbc.cdn.customer.com<http://tr.den-bos-nbc.cdn.customer.com>), 
rank=2
  - Target 1.3: Seattle Live DS 
(tr.sea-bos-nbc.cdn.customer.com<http://tr.sea-bos-nbc.cdn.customer.com>), 
rank=3

- Target 2: Denver Client Steering DS 
(tr.den-nbc.cdn.customer.com<http://tr.denver-nbc.cdn.customer.com>) - assigned 
only to Denver CG
- Target 3: Seattle Client Steering DS 
(tr.seat-nbc.cdn.customer.com<http://tr.seattle-nbc.cdn.customer.com>) - 
assigned only to Seattle CG

The goal would be this order of DS’s returned to the client
Location: [tr.bos-bos-nbc, tr.den-bos-nbc, tr.sea-bos-nbc, tr.den-den-nbc, 
tr.sea-den-nbc, r.bos-den-nbc,…. ]

Is this what are you ultimately looking for?

This lets us choose CG and DS based on the location of the client relative to 
the edge cache, rather than proximity of client to origin. Since the client is 
talking to the edge cache and not the origin, this seems like a better metric. 
Being able to compose Steering DS would also give us more flexibility for other 
use cases in the future as well




On Feb 27, 2018, at 4:40 PM, Rawlin Peters 
<rawlin.pet...@gmail.com<mailto:rawlin.pet...@gmail.com>> wrote:

Hey Eric,

In this example I'd think that all the target DSes would need to be
assigned to all 3 Cache Groups. That way the client could possibly use
any of the target DSes from the cachegroup they're routed to. But I
suppose that could change on a case-by-case basis where maybe the
target DSes are only assigned to cache groups that are close to their
origins. In that case I'd think a client in Seattle would possibly be
routed to an edge cache in Boston, which would optimize the connection
between the Boston edge cache and the Boston origin. But it might be
better to have the client connect to an edge cache in Seattle which
then retrieves the content from a Boston origin (although these are
both worst-case scenarios).

As far as the content on the origins themselves go, they'd have to be
interchangeable from the client's perspective, but I'm not sure if it
would have to be identical or not. I imagine that would really depend
on what the steering DS is providing the client.

- Rawlin

On Tue, Feb 27, 2018 at 10:37 AM, Eric Friedrich (efriedri)
<efrie...@cisco.com<mailto:efrie...@cisco.com>> wrote:
In this example, what would be the assignments of delivery services to edge 
Cache Groups? Are all 3DS’ assigned to all 3 Cache Groups?

I’ll also assume that the content on the origins, while interchangeable from a 
clients perspective, is not identical? (i.e. might contain regionalized 
content?)

—Eric




On Feb 23, 2018, at 5:40 PM, Rawlin Peters 
<rawlin.pet...@gmail.com<mailto:rawlin.pet...@gmail.com>> wrote:

Hey folks,

At Comcast we have a need to augment CLIENT_STEERING (also regular
STEERING while we're at it) to allow targets to be ordered/sorted
based upon the client's proximity to the origin of the target delivery
services. I'd like to get your feedback on my proposed design and
address any of your concerns.

For HTTP_LIVE targets for instance, we'd like edge caches to ideally
retrieve/serve data from an Origin that is close to the client and
fall back to Origins that are farther and farther away. This allows
for increased redundancy while ordering optimal Origins (Delivery
Services) for a client to choose from.

For example, I have 3 Origins in different locations: Seattle, Denver,
and Boston. I would create an HTTP_LIVE delivery service for each
origin and a CLIENT_STEERING delivery service with those delivery
services as targets. I would then like to have those target

Re: Steering Target Geo-Ordering

[Eric Friedrich (efriedri)]

In this example, what would be the assignments of delivery services to edge 
Cache Groups? Are all 3DS’ assigned to all 3 Cache Groups?

I’ll also assume that the content on the origins, while interchangeable from a 
clients perspective, is not identical? (i.e. might contain regionalized 
content?)

—Eric




> On Feb 23, 2018, at 5:40 PM, Rawlin Peters  wrote:
> 
> Hey folks,
> 
> At Comcast we have a need to augment CLIENT_STEERING (also regular
> STEERING while we're at it) to allow targets to be ordered/sorted
> based upon the client's proximity to the origin of the target delivery
> services. I'd like to get your feedback on my proposed design and
> address any of your concerns.
> 
> For HTTP_LIVE targets for instance, we'd like edge caches to ideally
> retrieve/serve data from an Origin that is close to the client and
> fall back to Origins that are farther and farther away. This allows
> for increased redundancy while ordering optimal Origins (Delivery
> Services) for a client to choose from.
> 
> For example, I have 3 Origins in different locations: Seattle, Denver,
> and Boston. I would create an HTTP_LIVE delivery service for each
> origin and a CLIENT_STEERING delivery service with those delivery
> services as targets. I would then like to have those targets ordered
> based upon proximity to the client. So a client in Seattle would get
> the list [Seattle, Denver, Boston], while a client in Boston would get
> the list [Boston, Denver, Seattle].
> 
> To make things more complicated, I want to add a redundant origin in
> each location and split traffic between them (like STEERING_WEIGHT
> today) while taking into account the geo-ordering. I also want to be
> able to force an ordering (like STEERING_ORDER today) among co-located
> targets.
> 
> In order to accomplish this I propose to:
> 1. add two new steering types: GEO_ORDER and GEO_WEIGHT (by adding a
> target of type GEO_*, a steering DS would then enable geo-ordering)
> 2. associate a Delivery Service Origin with a latitude/longitude,
> thereby associating a lat/long to a GEO_* target
> 
> Item 1 is pretty straightforward and will also play nicely with the
> current steering types (STEERING_ORDER and STEERING_WEIGHT). I've
> completed a POC within Traffic Router that basically provides the
> following ordering when mixing all 4 types in a single steering
> delivery service:
> - Negative STEERING_ORDER targets
> - GEO_WEIGHT and GEO_ORDER targets, grouped by proximity to the
> client, ordered by geo-order and the consistent-hashing from the
> weightings
> - STEERING_WEIGHT targets (consistent-hashed)
> - Positive STEERING_ORDER targets
> 
> Item 2 is not as straightforward because the simple thing would be to
> just add an Origin Lat/Long field to the Delivery Service and call it
> a day. However I don't think we should do that, and I'll dive more
> into that in a separate thread (coming soon).
> 
> Does anyone have questions/concerns about adding these new GEO_ORDER
> and GEO_STEERING target types and geo-sorting them based upon
> proximity to the client? Are you okay with the proposed ordering when
> all the steering types are mixed together?
> 
> - Rawlin

Re: Traffic Ops API Swagger Doc

[Eric Friedrich (efriedri)]

Is it possible to take the swagger generated documentation and have that 
automatically included in the read-the-docs site? 

Asked another way: Can swagger generate docs in ReStructed Text (.rst) format?

—Eric


> On Feb 20, 2018, at 11:38 AM, Dave Neuman  wrote:
> 
> Sounds good.  I look forward to seeing it merged into our repo.
> I guess this means there will need to be a PR to remove our current API
> docs as they get moved to swagger.
> 
> On Tue, Feb 20, 2018 at 8:40 AM, Jeremy Mitchell 
> wrote:
> 
>> I think this all sounds very promising. Some advantages that I see are:
>> 
>> - docs never drift from API implementation (currently our docs get out of
>> sync real fast)
>> - this provides yet another interface -
>> https://app.swaggerhub.com/apis/dewrich/traffic-ops_api/1.3 -  (in
>> addition
>> to TP) to interact with the API
>> - a great way to demonstrate / educate developers on the capabilities of
>> the api
>> 
>> plus, i heard some bonus features that could be really interesting:
>> 
>> - auto generation of a TO golang client. can we deprecate the old TO client
>> in favor of an auto-generated one? The current TO client never seems to
>> stay in sync with the api
>> - generating server stubs
>> 
>> imo our api can never be fully auto-generated because of the business logic
>> that needs to be accounted forbut it would be pretty neat if all we had
>> to do was "define" the api in a yaml file and that yaml file would spit out
>> documentation, tests, clients (be it golang or java or whatever), and
>> server side code (stubs) and then all we could focus on writing code
>> specific to business logic.
>> 
>> my guess is to really do this right, we'd have to rev the api version to
>> 2.0 to make the api more swagger friendly...tools (swagger) like
>> consistency and our api is far from consistent...
>> 
>> jeremy
>> 
>> On Wed, Feb 14, 2018 at 10:08 AM, Durfey, Ryan 
>> wrote:
>> 
>>> I am +1 on the swagger concept.  This makes working with APIs much easier
>>> for non-developer staff and makes it easier to educate customers as well.
>>> 
>>> Ryan DurfeyM | 303-524-5099
>>> CDN Support (24x7): 866-405-2993 or cdn_supp...@comcast.com>> cdn_supp...@comcast.com>
>>> 
>>> From: Dewayne Richardson 
>>> Reply-To: "dev@trafficcontrol.incubator.apache.org" <
>>> dev@trafficcontrol.incubator.apache.org>
>>> Date: Wednesday, February 14, 2018 at 9:34 AM
>>> To: "dev@trafficcontrol.incubator.apache.org" <
>>> dev@trafficcontrol.incubator.apache.org>
>>> Subject: Traffic Ops API Swagger Doc
>>> 
>>> We've been working diligently on the TO Golang Rewrite
>>>  project
>> to
>>> obviously rewrite the Perl into Go, but also to improve our Testing and
>>> Documentation efforts.  I presented the idea of using Swagger several
>>> summits (years) ago about using Swagger to help drive our Traffic Ops API
>>> documentation.  Since then Swagger has evolved and is becoming the de
>> facto
>>> standard for building (the potential for generating TO Golang Client and
>>> Server stubs is now available) and documenting REST APIs.
>>> 
>>> I would like to propose going forward that we use Swagger for future API
>>> level documentation (because it can be generated out of our Golang
>>> code/structs).  The below resources point out a TO API version 1.3 (the
>>> version we decided to rev for the rewritten Golang endpoints).  The
>> intent
>>> behind 1.3 is obviously an improved version of the API (entirely backward
>>> compatible to 1.2), but also to give us a starting point for building the
>>> API doc in Swagger.
>>> 
>>> The following resources are my examples:
>>> 
>>> Swagger has several implementations and I chose go-swagger
>>>  because it has more Golang
>>> features.
>>> 
>>> *Sample TO API doc *
>>> 
>>> https://app.swaggerhub.com/apis/dewrich/traffic-ops_api/1.3
>>> 
>>> 
>>> *Sample TO Golang code with embedded doc*
>>> 
>>> Generated from the combination of these Golang documentation "hooks"
>>> (there's current a go-swagger bug that prevents the doc from being tied
>>> directly into the handlers)
>>> https://github.com/dewrich/incubator-trafficcontrol/tree/
>>> swagger-demo/traffic_ops/traffic_ops_golang/docs
>>> 
>>> And the *asns.go*, *cdns.go*, *divisions.go*, *regions.go* and
>>> *statuses.go*
>>> structs in my branch here:
>>> https://github.com/dewrich/incubator-trafficcontrol/tree/
>>> swagger-demo/lib/go-tc
>>> 
>>> 
>>> *TO Client Generated from Swagger*
>>> 
>>> This new Golang package is only a sample of a TO client generated (based
>>> upon the the code generated swagger.json
>>> >> trafficcontrol/blob/swagger-demo/traffic_ops/traffic_ops_
>>> golang/docs/swagger.json>> dewrich/incubator-trafficcontrol/blob/swagger-

Re: Traffic Router Enhancement - Default Maxmind Geolocation Override

[Eric Friedrich (efriedri)]

Sounds great, thanks!

> On Feb 15, 2018, at 10:33 AM, Rivas, Jesse <jesse_ri...@comcast.com> wrote:
> 
> Eric,
> 
> We can determine a response from MaxMind is a default location when the 
> following conditions are met: the country code is populated, the city is 
> null, the postal code is null, and the subdivisions list is empty. If these 
> conditions are met, we check for an instance of maxmind.default.override with 
> the same country code. This allows users to have one MaxMind override per 
> country, per CDN.
> 
> -Jesse
> 
> On 2/15/18, 6:04 AM, "Eric Friedrich (efriedri)" <efrie...@cisco.com> wrote:
> 
>How does the suggested fix know when maxmind is returning a “default 
> location” versus an actual location?
> 
>Hopefully the solution is applicable to CDNs which are spread across 
> multiple countries and geographies?
> 
>—Eric
> 
>> On Feb 13, 2018, at 1:34 PM, Rawlin Peters <rawlin.pet...@gmail.com> wrote:
>> 
>> Yeah, this basically solves the problem where MaxMind knows a client
>> is in the US (or another country) but doesn't know the state, city,
>> zip, etc., so it's not a "true" miss. In that case MaxMind returns the
>> geographic center of that country as the client's location, but we
>> don't want to route those clients to the cache group closest to that
>> location because it might not be the ideal cachegroup. By using this
>> parameter we can shift this high volume of "US" traffic that is
>> essentially being localized to a lake in Kansas to a cachegroup more
>> capable of handling that load. And we can do this on a per-country
>> basis because we can create multiple of these parameters (which we
>> wouldn't be able to do if we just used the Default Miss Lat/Lon of a
>> DeliveryService).
>> 
>> -Rawlin
>> 
>> On Tue, Feb 13, 2018 at 11:10 AM, Rivas, Jesse <jesse_ri...@comcast.com> 
>> wrote:
>>> Steve,
>>> 
>>> Using the miss location for the DS was a potential solution that we talked 
>>> about. However, the miss location is intended for use when the client IP 
>>> falls through MaxMind without any data. Since the default location doesn't 
>>> fit this criteria, it was decided to use a profile parameter to preserve 
>>> granularity.
>>> 
>>> Jesse
>>> 
>>> On 2/13/18, 11:06 AM, "Steve Malenfant" <smalenf...@gmail.com> wrote:
>>> 
>>>   Jesse,
>>> 
>>>   I'm not exactly sure how MaxMind return this default value but would there
>>>   be a way to use the MISS location specified in the DS? Seems like that is
>>>   what it was intended for.
>>> 
>>>   Steve
>>> 
>>>   On Tue, Feb 13, 2018 at 12:42 PM, Rivas, Jesse <jesse_ri...@comcast.com>
>>>   wrote:
>>> 
>>>> Hi all,
>>>> 
>>>> 
>>>> 
>>>> At Comcast, we have been seeing a pattern of the same cache group being
>>>> overloaded nightly as traffic increases on the CDN. The cause was
>>>> determined to be a default location that the geolocation provider MaxMind
>>>> returns for client IPs that it does not have additional data for. For the
>>>> US, MaxMind returns a geolocation with the coordinates: 37.751,-97.822;
>>>> this is a substantial amount of traffic that is all directed to the nearest
>>>> cache group.
>>>> 
>>>> 
>>>> 
>>>> The fix I have introduced is a new profile parameter for CRConfig.json
>>>> named 'maxmind.default.override' in the format:
>>>> ';,'. When MaxMind returns a default location, the
>>>> code checks for a parameter entry with the same country code. If an entry
>>>> exists, the default location will be overwritten with the coordinates of
>>>> the parameter. This allows users to determine where this traffic should be
>>>> sent rather than using the cache group closest to the MaxMind default
>>>> location. The new parameter supports multiple entries so that there can be
>>>> override coordinates for more than one country.
>>>> 
>>>> 
>>>> 
>>>> Here is the PR: https://github.com/apache/incubator-trafficcontrol/pull/
>>>> 1866
>>>> 
>>>> 
>>>> 
>>>> Thanks,
>>>> 
>>>> 
>>>> 
>>>> Jesse
>>>> 
>>> 
>>> 
> 
> 
> 

Re: Traffic Router Enhancement - Default Maxmind Geolocation Override

[Eric Friedrich (efriedri)]

How does the suggested fix know when maxmind is returning a “default location” 
versus an actual location?

Hopefully the solution is applicable to CDNs which are spread across multiple 
countries and geographies?

—Eric

> On Feb 13, 2018, at 1:34 PM, Rawlin Peters  wrote:
> 
> Yeah, this basically solves the problem where MaxMind knows a client
> is in the US (or another country) but doesn't know the state, city,
> zip, etc., so it's not a "true" miss. In that case MaxMind returns the
> geographic center of that country as the client's location, but we
> don't want to route those clients to the cache group closest to that
> location because it might not be the ideal cachegroup. By using this
> parameter we can shift this high volume of "US" traffic that is
> essentially being localized to a lake in Kansas to a cachegroup more
> capable of handling that load. And we can do this on a per-country
> basis because we can create multiple of these parameters (which we
> wouldn't be able to do if we just used the Default Miss Lat/Lon of a
> DeliveryService).
> 
> -Rawlin
> 
> On Tue, Feb 13, 2018 at 11:10 AM, Rivas, Jesse  
> wrote:
>> Steve,
>> 
>> Using the miss location for the DS was a potential solution that we talked 
>> about. However, the miss location is intended for use when the client IP 
>> falls through MaxMind without any data. Since the default location doesn't 
>> fit this criteria, it was decided to use a profile parameter to preserve 
>> granularity.
>> 
>> Jesse
>> 
>> On 2/13/18, 11:06 AM, "Steve Malenfant"  wrote:
>> 
>>Jesse,
>> 
>>I'm not exactly sure how MaxMind return this default value but would there
>>be a way to use the MISS location specified in the DS? Seems like that is
>>what it was intended for.
>> 
>>Steve
>> 
>>On Tue, Feb 13, 2018 at 12:42 PM, Rivas, Jesse 
>>wrote:
>> 
>>> Hi all,
>>> 
>>> 
>>> 
>>> At Comcast, we have been seeing a pattern of the same cache group being
>>> overloaded nightly as traffic increases on the CDN. The cause was
>>> determined to be a default location that the geolocation provider MaxMind
>>> returns for client IPs that it does not have additional data for. For the
>>> US, MaxMind returns a geolocation with the coordinates: 37.751,-97.822;
>>> this is a substantial amount of traffic that is all directed to the nearest
>>> cache group.
>>> 
>>> 
>>> 
>>> The fix I have introduced is a new profile parameter for CRConfig.json
>>> named 'maxmind.default.override' in the format:
>>> ';,'. When MaxMind returns a default location, the
>>> code checks for a parameter entry with the same country code. If an entry
>>> exists, the default location will be overwritten with the coordinates of
>>> the parameter. This allows users to determine where this traffic should be
>>> sent rather than using the cache group closest to the MaxMind default
>>> location. The new parameter supports multiple entries so that there can be
>>> override coordinates for more than one country.
>>> 
>>> 
>>> 
>>> Here is the PR: https://github.com/apache/incubator-trafficcontrol/pull/
>>> 1866
>>> 
>>> 
>>> 
>>> Thanks,
>>> 
>>> 
>>> 
>>> Jesse
>>> 
>> 
>> 

Re: Traffic Server DB Issue

[Eric Friedrich (efriedri)]

Traffic Control alone cannot do this. Try looking at tools like Nagio, Sensu, 
Prometheus, or the InfluxData TICK stack for alerting.

—Eric
> On Feb 1, 2018, at 4:35 AM, Satheeshkumar  wrote:
> 
> One more doubt how to set email alert if any Mid and Edge server down
> 
> 
> 
> On Thu, Feb 1, 2018 at 10:15 AM, Satheeshkumar 
> wrote:
> 
>> Dear Team,
>> 
>> I had doubt if possible can use MySQL DB inside of PostgreSQL Traffic OPS
>> DB?
>> if yes mean request you please share me document.
>> 
>> Thanks in advance
>> 
>> Thanks,
>> 
>> Satheesh R
>> 
>> 
>> 
> 
> 
> -- 
> 
> 
> *Regards,*
> 
> *Satheesh.R*

Re: TC 2.2 Release - Outstanding Issues

[Eric Friedrich (efriedri)]

His PR will update server.xml back to BIO

—Eric

> On Jan 30, 2018, at 12:03 PM, David Neuman <david.neuma...@gmail.com> wrote:
> 
> Jeff is planning to open a PR to actually fix the leak or to update the
> server.xml to use the BIO connector?
> 
> On Tue, Jan 30, 2018 at 9:39 AM, Eric Friedrich (efriedri) <
> efrie...@cisco.com> wrote:
> 
>> Thanks Dave-
>>  I’d like to keep it as a must-have in 2.2. We had a severe issue in
>> production with the NIO connection when deploying 2.1. Jeff M is planning
>> to open a PR to fix it in the 2.2 release.
>> 
>> I understand the issue will be moot after upgrading to Tomcat, but the
>> current 2.1 release has crippling TR scale problems out of the box and I’d
>> like to avoid that with 2.2 given how small and low risk the fix is
>> 
>> —Eric
>> 
>>> On Jan 29, 2018, at 5:36 PM, Dave Neuman <neu...@apache.org> wrote:
>>> 
>>> Thanks Rob,
>>> I think `916 [TC-197] File descriptor leak caused by NIO protocol
>>> connector` should be moved to 2.3.  This issue will be resolved with the
>>> upgrade to Tomcat 8.5.x.
>>> I don't think we should hold up the release for `1356 Step by Step
>>> installation guide for new users`.  I think it can be moved to 2.3 as
>>> well.
>>> I took a look at the issues in 2.3 and I am fine with what's there
>> staying
>>> there.
>>> 
>>> Thanks,
>>> Dave
>>> 
>>> On Mon, Jan 29, 2018 at 11:47 AM, Robert Butts <r...@apache.org> wrote:
>>> 
>>>> There are no `major` bugs without milestones, now.
>>>> 
>>>> Many issues without milestones were moved from 2.1 to 2.2, and I moved
>> them
>>>> to 2.3 indiscriminately. We should verify they're ok to not be in 2.2.
>>>> 
>>>> Traffic Ops
>>>> --
>>>> 1026 [TC-191] Steering DS API should use standard response codes as
>> defined
>>>> in API guidelines
>>>> 913 ort does not restart ATS when it should
>>>> 911 Invalid DS regex causes nasty exception and unexpected results
>> Github
>>>> Issue #514
>>>> 905 Inactive delivery services are added to remap.config
>>>> 902 api/1.2/deliveryservices/:id/routing.json hangs when Traffic
>> Router is
>>>> unreachable
>>>> 899 IPv6 Origin Configuration Not Allowed
>>>> 898 Clone DS assignments button on Server times out
>>>> 896 Traffic Router statistics are not aggregated
>>>> 892 GenIso can't handle concurrent requests
>>>> 891 error_url required in the url_sig file
>>>> 890 Should use "dest_ip" instead of "dest_domain" in "parent.config" and
>>>> "cache.config" when ofqdn is ip
>>>> 888 cacheurl_qstring.conf file is not generated when qstringIgnore=1 and
>>>> location profile parameter is missing for edge/mid cache
>>>> 885 Fix regression issues caused by TC-187
>>>> 882 Deleting a DS thru the TO API should also delete all SSL keys (if
>>>> applicable)
>>>> 
>>>> Traffic Router
>>>> --
>>>> 914 Failed to restart tomcat in Traffic Router when failed to get SSL
>> keys
>>>> 907 Traffic Router requires at least one cache assigned to at least one
>> DS
>>>> for TLD zone generation
>>>> 
>>>> 
>>>> There are 11 open issues in the 2.2 Milestone:
>>>> 
>>>> Code
>>>> --
>>>> 1777 TO golang -- api/1.3/parameters?orderby=id produces an error
>>>> 1620 API: Server API sets xmpp_id to null
>>>> 1397 ORT tries to get ats_uid before installing trafficserver
>>>> 1042 [TC-242] tm_user.username and role should be NOT NULL in the db
>>>> 986 [TC-544] TR should de-dupe Reponse Headers when sending a Steering
>>>> response.
>>>> 916 [TC-197] File descriptor leak caused by NIO protocol connector
>>>> 
>>>> Documentation
>>>> --
>>>> 1356 Step by Step installation guide for new users
>>>> 1173 'Multi Site Origin Algorithm' is removed in delivery service UI in
>>>> traffic_ops in TC-2.1
>>>> 1135 Traffic Server administration docs are out of date
>>>> 1130 cacheurl is deprecated in ATS 7.x
>>>> 
>>>> Licensing
>>>> --
>>>> 1750 /traffic_ops/app/public/images/ contains non-free images
>>>> 
>>>> 
>>>> Do we still feel like we can have all these fixed for a 2.2 Release in 2
>>>> weeks? Say, Monday, February 12?
>>>> 
>>>> Are there any cases moved to 2.3 that need to be moved back and done in
>>>> 2.2?
>>>> 
>>>> Thanks,
>>>> 
>> 
>> 

Re: TC 2.2 Release - Outstanding Issues

[Eric Friedrich (efriedri)]

Thanks Dave-
  I’d like to keep it as a must-have in 2.2. We had a severe issue in 
production with the NIO connection when deploying 2.1. Jeff M is planning to 
open a PR to fix it in the 2.2 release. 

I understand the issue will be moot after upgrading to Tomcat, but the current 
2.1 release has crippling TR scale problems out of the box and I’d like to 
avoid that with 2.2 given how small and low risk the fix is

—Eric

> On Jan 29, 2018, at 5:36 PM, Dave Neuman  wrote:
> 
> Thanks Rob,
> I think `916 [TC-197] File descriptor leak caused by NIO protocol
> connector` should be moved to 2.3.  This issue will be resolved with the
> upgrade to Tomcat 8.5.x.
> I don't think we should hold up the release for `1356 Step by Step
> installation guide for new users`.  I think it can be moved to 2.3 as
> well.
> I took a look at the issues in 2.3 and I am fine with what's there staying
> there.
> 
> Thanks,
> Dave
> 
> On Mon, Jan 29, 2018 at 11:47 AM, Robert Butts  wrote:
> 
>> There are no `major` bugs without milestones, now.
>> 
>> Many issues without milestones were moved from 2.1 to 2.2, and I moved them
>> to 2.3 indiscriminately. We should verify they're ok to not be in 2.2.
>> 
>> Traffic Ops
>> --
>> 1026 [TC-191] Steering DS API should use standard response codes as defined
>> in API guidelines
>> 913 ort does not restart ATS when it should
>> 911 Invalid DS regex causes nasty exception and unexpected results Github
>> Issue #514
>> 905 Inactive delivery services are added to remap.config
>> 902 api/1.2/deliveryservices/:id/routing.json hangs when Traffic Router is
>> unreachable
>> 899 IPv6 Origin Configuration Not Allowed
>> 898 Clone DS assignments button on Server times out
>> 896 Traffic Router statistics are not aggregated
>> 892 GenIso can't handle concurrent requests
>> 891 error_url required in the url_sig file
>> 890 Should use "dest_ip" instead of "dest_domain" in "parent.config" and
>> "cache.config" when ofqdn is ip
>> 888 cacheurl_qstring.conf file is not generated when qstringIgnore=1 and
>> location profile parameter is missing for edge/mid cache
>> 885 Fix regression issues caused by TC-187
>> 882 Deleting a DS thru the TO API should also delete all SSL keys (if
>> applicable)
>> 
>> Traffic Router
>> --
>> 914 Failed to restart tomcat in Traffic Router when failed to get SSL keys
>> 907 Traffic Router requires at least one cache assigned to at least one DS
>> for TLD zone generation
>> 
>> 
>> There are 11 open issues in the 2.2 Milestone:
>> 
>> Code
>> --
>> 1777 TO golang -- api/1.3/parameters?orderby=id produces an error
>> 1620 API: Server API sets xmpp_id to null
>> 1397 ORT tries to get ats_uid before installing trafficserver
>> 1042 [TC-242] tm_user.username and role should be NOT NULL in the db
>> 986 [TC-544] TR should de-dupe Reponse Headers when sending a Steering
>> response.
>> 916 [TC-197] File descriptor leak caused by NIO protocol connector
>> 
>> Documentation
>> --
>> 1356 Step by Step installation guide for new users
>> 1173 'Multi Site Origin Algorithm' is removed in delivery service UI in
>> traffic_ops in TC-2.1
>> 1135 Traffic Server administration docs are out of date
>> 1130 cacheurl is deprecated in ATS 7.x
>> 
>> Licensing
>> --
>> 1750 /traffic_ops/app/public/images/ contains non-free images
>> 
>> 
>> Do we still feel like we can have all these fixed for a 2.2 Release in 2
>> weeks? Say, Monday, February 12?
>> 
>> Are there any cases moved to 2.3 that need to be moved back and done in
>> 2.2?
>> 
>> Thanks,
>> 

Re: Starting the 2.2 Branch for Next Release of TC

[Eric Friedrich (efriedri)]

Hey Rob-
  Thanks for taking on the RM duties!

We’ve got 22 major open bugs in Github Issues currently: 
https://github.com/apache/incubator-trafficcontrol/issues?q=is%3Aopen+is%3Aissue+label%3Abug+label%3Amajor

I see that a bunch of these bugs don’t have milestones assigned to them today. 
I think we should triage these into must-fix for 2.2 vs 2.3 and then make a 
judgment on when it makes the most sense to cut the release branch. Otherwise 
we might be stuck doing a bunch of back porting of fixes in the next month or 
so.

—Eric


On Jan 24, 2018, at 3:47 PM, Robert Butts 
> wrote:

Hi,

I'm your friendly neighbourhood Release Manager for Traffic Control 2.2.

We are getting ready to start the 2.2 branch of TC. We would like to do this in 
the next 2 weeks.

See https://github.com/apache/incubator-trafficcontrol/milestone/2

Are there any know issues that would prevent this from happening?

Are there any features that can be wrapped up and go into this version?

Any other comments or suggestions?

Thanks,

Robert Butts

Re: [VOTE] CHANGELOG.md file (second try)

[Eric Friedrich (efriedri)]

[X] +1 to adding a changelog.MD file
[] -1 to adding a changelog.MD file

[] +1 to adding a changelog label in github
[X] -1 to adding a changelog label in github


I don’t think an auto-generated changelog will provide enough value to our 
users. Asking for updates to changelog.md in each PR will make sure the author 
carefully considers what users need to know, rather than just tossing a label 
on the PR

—Eric

> On Jan 9, 2018, at 12:22 PM, Dave Neuman  wrote:
> 
> Looks like this thread died over the holidays.  Let me try to bring it back
> up before we cut a 2.2 branch.
> Can you please respond with *just* the following:
> 
> [] +1 to adding a changelog.MD file
> [] -1 to adding a changelog.MD file
> 
> [] +1 to adding a changelog label in github
> [] -1 to adding a changelog label in github
> 
> Thanks,
> Dave
> 
> On Fri, Dec 15, 2017 at 9:14 AM, Dewayne Richardson 
> wrote:
> 
>> +1 on the CHANGELOG.md as well, but I like having the changelog label
>> because it helps streamline it's creation.  I also think the labels are
>> valuable because they help summarize the parts of the repo that changed and
>> keeps the concept closer to the repository (where the real change is).
>> 
>> -Dew
>> 
>> On Thu, Dec 14, 2017 at 3:01 PM, Robert Butts 
>> wrote:
>> 
>>> +1. To clarify, I'm +1 on having the "changelog" label, to help whoever
>>> manually goes thru and builds the CHANGELOG.md.
>>> 
>>> But I agree with @neuman I don't think we can automate this. Titles are
>> too
>>> short, some changes need longer explanations and some don't, only a human
>>> can figure out how important something is to users; a high priority bug
>> fix
>>> could still be low-impact, or vica-versa. Much as I dislike manual
>> things,
>>> this really needs human judgement. And we absolutely need a good
>> Changelog
>>> with each Release.
>>> 
>>> On Thu, Dec 14, 2017 at 2:36 PM, Dave Neuman  wrote:
>>> 
 Thanks for putting that together Dewayne. I was just starting to do
>> that
 myself when I saw it was already done.
 As a developer this is fine, I can see a list of PRs and click on each
>>> one
 to see the whole conversation.  I can comb through them and get a
>> general
 sense for what changed.
 
 However, as an operator and user of Traffic Control (which I mostly am
 these days) I am -1 for the following reasons:
 1) only committers can assign labels to issues and pull requests.  This
 means that the committers need to be cognizant of the issues/PRs they
>> are
 reviewing and apply the change log label;
 2) the title of a PR only gives so much information about a change, if
>> I
 want more information I have to click on each individual one
 3) If I am not a developer, or someone who is very familiar with our
 project, it is hard for me to ascertain from this list which changes
>> are
 super-important or have some operational considerations attached to
>> them.
 These are the issues I really care about.
 4) When I go to do an upgrade, it's a lot easier to copy/paste a nice
 bulleted list of what changed then it is to try to take a list of PRs
>> and
 turn it into a high level list of what's important.
 
 We need a solution that gives someone what they need at a glance.
>>> Something
 that can be copy/pasted out or easily linked to.  IMO not all of our
>>> users
 are super technical or involved in the day to day so we shouldn't just
 point them at a list of PRs and say "figure it out"; we should make it
>>> easy
 for them to figure out.
 
 I have seen lots of alternatives suggested to what Steve has proposed,
>>> but
 I haven't seen anyone state why they don't like what Steve has
>>> proposed?  I
 think what he is proposing is pretty standard across major Github
>>> projects
 that I have seen.  I understand that we can introduce some additional
 inconvenience with having to manually write what your feature or bug
>> fix
 does, and there could be some conflicts, but isn't it important to
>>> clearly
 portray what has changed?  I don't think we need a changelog.md entry
>> to
 every PR, in fact I hope we don't...we need a changelog.md entry any
>>> time
 we add a new feature, any time we have some operational considerations
>>> that
 need to be communicated, or any time that we fix a bug from a previous
 release.
 
 
 On Thu, Dec 14, 2017 at 2:02 PM, Jeremy Mitchell <
>> mitchell...@gmail.com>
 wrote:
 
> This label idea would require everyone to go back and find their PRs
>>> that
> were closed after Aug 4th (the date 2.1 branch was created) and
>> attach
 the
> 'change log' label and the 2.2 milestone to the appropriate ones,
>>> right?
> And then that link dew provide would be an accurate picture of what
>> has
> changed between 21. and 2.2...
> 

Re: Changing max_dns_answers default

[Eric Friedrich (efriedri)]

Shoot sorry, I missed the huge word “DEFAULT” in the goose file!

Sounds like a reasonable change to me

—Eric

> On Dec 4, 2017, at 3:23 PM, Volz, Dylan <dylan_v...@comcast.com> wrote:
> 
> Eric, 
> It is modifying the table schema’s default it will not change existing values.
> 
> On 12/4/17, 1:22 PM, "Eric Friedrich (efriedri)" <efrie...@cisco.com> wrote:
> 
>It looks like this will modify existing values as well (so its not really 
> a default)?
> 
>> On Dec 4, 2017, at 3:03 PM, Durfey, Ryan <ryan_dur...@comcast.com> wrote:
>> 
>> With the capacity of our servers being in the 18 Gbps range I would say max 
>> dns answers should be between 2-4 unless volumes expected are > 250 Gbps.  I 
>> think 2 or 3 would be a good starting point for most services.
>> 
>> Ryan DurfeyM | 303-524-5099
>> CDN Support (24x7): 866-405-2993 or 
>> cdn_supp...@comcast.com<mailto:cdn_supp...@comcast.com>
>> 
>> 
>> From: Dylan Volz <dylan_v...@comcast.com>
>> Reply-To: "dev@trafficcontrol.incubator.apache.org" 
>> <dev@trafficcontrol.incubator.apache.org>
>> Date: Monday, December 4, 2017 at 12:58 PM
>> To: "dev@trafficcontrol.incubator.apache.org" 
>> <dev@trafficcontrol.incubator.apache.org>
>> Subject: Changing max_dns_answers default
>> 
>> Hi All,
>> 
>> The max_dns_answers has been defaulted to 0, which is an unlimited number of 
>> answers, which causes issues for deployments with large cache groups. I 
>> opened a PR 
>> (1611<https://github.com/apache/incubator-trafficcontrol/pull/1611><https://github.com/apache/incubator-trafficcontrol/pull/1611%3e>)
>>  to change the default from 0 to 5 which is hopefully a sensible value for 
>> most deployments. If this doesn’t seem like a sensible default please 
>> respond with alternatives.
>> 
>> Thanks,
>> Dylan
>> 
> 
> 
> 

Re: [VOTE] Release Apache Traffic Control (incubating) 2.1.0-RC2

[Eric Friedrich (efriedri)]

I’m +1 as well

Checked out:
- signatures/checksums (Hank is your key signed yet?)
- licenses
- build

—Eric


> On Nov 14, 2017, at 2:36 PM, Nir Sopher <n...@qwilt.com> wrote:
> 
> +1
> We were able to build traffic-control, install and connect OPs, Portal-V2,
> Monitor (Golang), Router and Stats.
> Also got a redirect.
> Note that I missed the last commit ("Change cdn.name to cdn.domain_name in
> DeliveryServiceInfoForDomainList"), but as far as I see it could not break
> the installation.
> Nir
> 
> On Tue, Nov 14, 2017 at 8:10 PM, Eric Friedrich (efriedri) <
> efrie...@cisco.com> wrote:
> 
>> Thanks Matt-
>>  I found another LEGAL ticket (https://issues.apache.org/
>> jira/browse/LEGAL-330) based on a Google version of the PATENTS file this
>> time.
>> 
>> Looks like things are OK to use then.
>> 
>> —Eric
>> 
>> On Nov 14, 2017, at 12:50 PM, Mills, Matthew <matthew_mi...@comcast.com<
>> mailto:matthew_mi...@comcast.com>> wrote:
>> 
>> FYI, Go itself has the same file
>> 
>> https://github.com/golang/go/blob/master/PATENTS
>> 
>> 
>> On 11/14/17, 10:36:43 AM, "Eric Friedrich (efriedri)" <efrie...@cisco.com>
>> wrote:
>> 
>>   I’ve been going through licensing for the 2.1 release and found this
>> file:
>>   ./traffic_stats/vendor/golang.org/x/net/PATENTS<http://
>> golang.org/x/net/PATENTS>
>> 
>>   This looks like it places some of the same restrictions that caused the
>> whole Facebook React.js and rocksDb controversy a few months ago.
>>   Fun reading here: https://issues.apache.org/jira/browse/LEGAL-303
>>   There’s some in depth discussion of the detailed Facebook license, I
>> can’t even begin to speculate how that compares to this Google conditional
>> patent grant.
>> 
>>   We can see what the IPMC/Legal thinks or maybe just remove the code?
>> 
>>   —Eric
>> 
>> 
>> 
>> 
>> 

Re: [VOTE] Release Apache Traffic Control (incubating) 2.1.0-RC2

[Eric Friedrich (efriedri)]

Thanks Matt-
  I found another LEGAL ticket 
(https://issues.apache.org/jira/browse/LEGAL-330) based on a Google version of 
the PATENTS file this time.

Looks like things are OK to use then.

—Eric

On Nov 14, 2017, at 12:50 PM, Mills, Matthew 
<matthew_mi...@comcast.com<mailto:matthew_mi...@comcast.com>> wrote:

FYI, Go itself has the same file

https://github.com/golang/go/blob/master/PATENTS


On 11/14/17, 10:36:43 AM, "Eric Friedrich (efriedri)" <efrie...@cisco.com> 
wrote:

   I’ve been going through licensing for the 2.1 release and found this file:
   
./traffic_stats/vendor/golang.org/x/net/PATENTS<http://golang.org/x/net/PATENTS>

   This looks like it places some of the same restrictions that caused the 
whole Facebook React.js and rocksDb controversy a few months ago.
   Fun reading here: https://issues.apache.org/jira/browse/LEGAL-303
   There’s some in depth discussion of the detailed Facebook license, I can’t 
even begin to speculate how that compares to this Google conditional patent 
grant.

   We can see what the IPMC/Legal thinks or maybe just remove the code?

   —Eric

Re: [VOTE] Release Apache Traffic Control (incubating) 2.1.0-RC2

[Eric Friedrich (efriedri)]

I’ve been going through licensing for the 2.1 release and found this file:
./traffic_stats/vendor/golang.org/x/net/PATENTS

This looks like it places some of the same restrictions that caused the whole 
Facebook React.js and rocksDb controversy a few months ago.
Fun reading here: https://issues.apache.org/jira/browse/LEGAL-303
There’s some in depth discussion of the detailed Facebook license, I can’t even 
begin to speculate how that compares to this Google conditional patent grant.

We can see what the IPMC/Legal thinks or maybe just remove the code?

—Eric



On Nov 13, 2017, at 3:14 PM, Dewayne Richardson 
> wrote:

+1

- Installed from scratch using the TO Install guide
- Used TO UI and logged in with success and saw data was seeded


-Dew



On Mon, Nov 13, 2017 at 6:14 AM, Hank Beatty 
> wrote:

Now that everyone has had the weekend to review RC2 we should be ready to
cut the release. I didn't see any -1. I'm going to take that as acceptance
and move forward.

Regards,
Hank

:) Just kidding.

On 11/10/2017 03:30 PM, Dewayne Richardson wrote:

Nice work 72 hours on a Friday!

On Fri, Nov 10, 2017 at 11:21 AM, Hank Beatty 
> wrote:

Hello All,

I've prepared a release for v2.1.0-RC1

The vote is open for at least 72 hours and passes if a majority of at
least 3 +1 PPMC votes are cast.

[ ] +1 Approve the release

[ ] -1 Do not release this package because ...

Changes since 2.0.0:
https://github.com/apache/incubator-trafficcontrol/compare/
2.0.x...RELEASE-2.1.0-RC2

This corresponds to git:
  Hash: cac666ef7f0626ea8180e976b07fa841d53f755f
  Tag: RELEASE-2.1.0-RC2

Which can be verified with the following: git tag -v RELEASE-2.1.0-RC1

My code signing key is available here:
https://pgp.mit.edu/pks/lookup?op=get=0x920152B94E0CC77C

The source .tar.gz file, pgp signature (.asc signed with my key from
above), md5 and sha512 checksums are provided here:
https://dist.apache.org/repos/dist/dev/incubator/trafficcont
rol/2.1.0/RC2


Thanks!

Re: [VOTE] Release Apache Traffic Control (incubating) 2.1.0-RC1

[Eric Friedrich (efriedri)]

Do PRs/issues that don’t have a milestone assigned still end up in this 
changelog? I know theres a bunch more that went into 2.1 that isn’t in this 
list. 

Should we make sure that every PR/Issue is assigned a milestone before its 
merged?

> On Oct 26, 2017, at 12:52 PM, Hank Beatty  wrote:
> 
> Thanks Phil this is very nice.
> 
> Both of these scripts use the "milestones" to generate the changelog. 
> Currently there are 21 issues associated with the 2.1.0 milestone.
> 
> Here is the output of the one in TC:
> 
> Changes with Traffic Control 2.1.0
>  #878 - [TC-488] Docs - Multi Site Origin not up to date
>  #879 - [TC-490] mso.qstring_handling parameter is checked but not documented
>  #880 - [TC-489] Multi Site Origin - Invalid default values for multiple 
> config params
>  #901 - [TC-377] Default profiles for EDGE and MID are missing after initial 
> install
>  #906 - [TC-327] ConfigFiles.pm detects blank as not null and tries to gen 
> files GH #1090
>  #909 - [TC-301] creating https delivery service and not setting to active 
> still looks for cert. Github Issue #1086
>  #912 - [TC-169] TR download the RGB file continuously when the same RGB file 
> on server
>  #915 - [TC-116] remap.config order is different on master (postgres) than it 
> is on 1.8.
>  #980 - [TC-552] Global parameters may be duplicated when seeds.sql is run
>  #988 - [TC-514] ORT: Change Traffic Ops hostname in middle of ORT run
>  #1001 - [TC-408] Documentation for creating ssl keys is missing a field.
>  #1090 - [TC-518] ToCDUCheck and ToCHRCheck: Value formatted as float instead 
> of int
>  #1115 - [TC-429] - TP - removes map due to license incompatibility
>  #1118 - POST /api/1.2/deliveryserviceserver doesn't update header rewrite, 
> regex remap and cacheurl
>  #1167 - [BACKPORT][TC-518] ToCDUCheck and ToCHRCheck: Value formatted as 
> float instead of int #1090
>  #1168 - [BACKPORT][TC-514] ORT: Change Traffic Ops hostname in middle of ORT 
> run
>  #1195 - [Issue-1189] - Backport to 2.1.x - delivery service tenancy is 
> forced on creation and update if use_tenancy is on
>  #1375 - BACKPORT - fix docs for Deliveryservice/sslkeys/generate and 
> deliveryservice/ssl…
>  #1386 - Traffic Portal V2 main menu has two rows labeled "Tenants"
> 
> 
> On 10/26/2017 11:55 AM, Phil Sorber wrote:
>> I believe this one has had a little more love recently and does things like
>> only include merged pull requests, etc.
>> https://github.com/apache/trafficserver/blob/master/tools/changelog.pl
>> On Thu, Oct 26, 2017 at 9:52 AM Phil Sorber  wrote:
>>> You guys mean like this?
>>> 
>>> 
>>> https://github.com/apache/incubator-trafficcontrol/blob/master/misc/changelog.pl
>>> 
>>> On Thu, Oct 26, 2017 at 8:39 AM Hank Beatty  wrote:
>>> 
 I was thinking of starting with something like this:
 https://github.com/skywinder/github-changelog-generator#output-example.
 I will also look at the github-changes that Steve mentions.
 
 Hank
 
 On 10/26/2017 08:48 AM, Steve Malenfant wrote:
> Do we have a sample from a script that would create a change log based
 on
> pull request only? I tried `github-changes` yesterday but it seemed to
 work
> only with the older releases (<1.8). Although I haven't spent much time
 on
> it.
> 
> We should probably only list the high level changes such as new
 features,
> improvements, important fixes and breaking changes (relates to
 operations
> action required).
> 
> Few things we should note as well :
> - How to upgrade from pre-2.0 releases
> - What are the upgrade procedures (run postinstall for example)
> - Profiles locations (not included in 2.x basic install anymore)
> 
> I'm sure some of those are documented already, might just requires
> references.
> 
> Steve
> 
> 
> 
> On Wed, Oct 25, 2017 at 8:06 AM, Hank Beatty 
 wrote:
> 
>> Hello Steve,
>> 
>> The release notes are still being worked on. I am also looking for
>> suggestions on how the release notes should be formatted, and how they
>> might be auto-generated. If you have any, please let me know.
>> 
>> Regards,
>> Hank
>> 
>> On 10/24/2017 07:26 AM, Steve Malenfant wrote:
>> 
>>> Is there any Release Notes associated with this release? 1,337
 changes and
>>> the link above will only display 250 of them.
>>> 
>>> Steve
>>> 
>>> On Mon, Oct 23, 2017 at 4:01 PM, Hank Beatty 
 wrote:
>>> 
>>> Hello All,
 
 I've prepared a release for v2.1.0-RC1
 
 The vote is open for at least 72 hours and passes if a majority of at
 least 3 +1 PPMC votes are cast.
 
 [ ] +1 Approve the release
 
 [ ] -1 Do not release this package because ...
 
 Changes since 

Re: Anonymous IP Blocking Flowchart

[Eric Friedrich (efriedri)]

Where do we think our “design docs” (diagrams and text) belong? 

 One option is the Wiki, but that can get stale so I’d toss it out there that 
PRs should include the obvious user facing docs, but also a design doc section 
as well

-Eric


> On Oct 20, 2017, at 10:23 AM, Jeff Elsloo <els...@apache.org> wrote:
> 
> Hey Eric,
> 
> Thanks for sending this out. Hopefully we can build more of these
> diagrams in the future to include in the docs. I'll try to put
> something similar together for the stuff I've been working on.
> --
> Thanks,
> Jeff
> 
> 
> On Thu, Oct 19, 2017 at 1:55 PM, Eric Friedrich (efriedri)
> <efrie...@cisco.com> wrote:
>> Just realized the first diagram I put up was outdated
>> 
>> The response to anonymous IP blocking is actually configurable between a 
>> slate (302 redirect to a new URL) and a 403
>> 
>> —Eric
>> 
>>> On Oct 19, 2017, at 10:53 AM, Eric Friedrich (efriedri) 
>>> <efrie...@cisco.com> wrote:
>>> 
>>> Here is flowchart requested at the summit. I’ll put this diagram along with 
>>> the rest of the slides up soon
>>> 
>>> Its a link to a PNG despite the horribly formatted URL
>>> https://cisco.box.com/s/4rwd6kk069vdmzxpp2ak0vt2elds0ufc
>>> 
>>> —Eric
>>> 
>>> 
>> 

Re: Traffic Ops API Semantic Versioning

[Eric Friedrich (efriedri)]

 should get rid of it.
>> Then at least the version itself won't be confusing and painful.
>> 
>> What's the consensus here? Does everyone agree with Semantic Versioning? Do
>> we want to commit to requiring it? Is there a consensus? Or should we take
>> a vote, whether to require Semantic Versioning, Absolute Versioning, or No
>> Version?
>> 
>> 
>> On Thu, Oct 12, 2017 at 7:39 AM, Dave Neuman <neu...@apache.org> wrote:
>> 
>>> Traffic ops currently does not handle versioning very well.  I think we
>> do
>>> support 1.1 and 1.2 versions of the API, but I think there are only a few
>>> (maybe asns and deliveryservices) that are actually different.
>>> Versioning is something we look to improve as we move to the golang
>> version
>>> of the API.
>>> 
>>> On Thu, Oct 12, 2017 at 6:50 AM, Eric Friedrich (efriedri) <
>>> efrie...@cisco.com> wrote:
>>> 
>>>> Does Traffic Ops expose a semantic version number as part of its API?
>>>> 
>>>> http://semver.org/
>>>> "Given a version number MAJOR.MINOR.PATCH, increment the:
>>>> 
>>>>  1.  MAJOR version when you make incompatible API changes,
>>>>  2.  MINOR version when you add functionality in a
>> backwards-compatible
>>>> manner, and
>>>>  3.  PATCH version when you make backwards-compatible bug fixes.
>>>> 
>>>> “
>>>> 
>>>> We have some TO clients and would like to improve their backwards
>>>> compatibility. Without this version number, it is not easy to determine
>>>> which fields in the API are supported by any given version.
>>>> 
>>>> Thanks,
>>>> Eric
>>>> 
>>>> 
>>> 
>> 

Anonymous IP Blocking Flowchart

[Eric Friedrich (efriedri)]

Here is flowchart requested at the summit. I’ll put this diagram along with the 
rest of the slides up soon

Its a link to a PNG despite the horribly formatted URL
https://cisco.box.com/s/4rwd6kk069vdmzxpp2ak0vt2elds0ufc

—Eric

Traffic Ops API Semantic Versioning

[Eric Friedrich (efriedri)]

Does Traffic Ops expose a semantic version number as part of its API?

http://semver.org/
"Given a version number MAJOR.MINOR.PATCH, increment the:

  1.  MAJOR version when you make incompatible API changes,
  2.  MINOR version when you add functionality in a backwards-compatible 
manner, and
  3.  PATCH version when you make backwards-compatible bug fixes.

“

We have some TO clients and would like to improve their backwards 
compatibility. Without this version number, it is not easy to determine which 
fields in the API are supported by any given version.

Thanks,
Eric

Re: Apache Cwiki vs. Github Wiki vs. Github Docs

[Eric Friedrich (efriedri)]

I’m pretty solidly against getting rid of the Wiki altogether for the reasons 
Jan laid out. 

I’d also like to know more about the Github Wiki workflow before we make a 
decision. For example, can someone actually open a PR against a Wiki page?

—Eric

> On Sep 26, 2017, at 8:46 PM, Durfey, Ryan  wrote:
> 
> And per Nick’s feedback I think most users would be able to submit PRs for 
> document modifications which would need to be accepted by committers.  I 
> think that solves the access issue.
> 
> The logic of the move from jira to github was to simplify/unify everything to 
> make it so there was less need for system access and less overhead of 
> maintaining multiple sites.  In moving to “full” github, everything collapses 
> down into one single system and we can focus on keeping it up to date.  It 
> also forces us to use the github publishing system more regularly which 
> should improve documentation output.  I think we could still keep informal 
> information in the site we would just create a new section for it that calls 
> out what it is.
> 
> If it makes a difference, Ashish and I could port the documentation over to 
> either location so the lift on the developers would be nil aside from 
> feedback.
> 
> 
> Ryan DurfeyM | 303-524-5099
> CDN Support (24x7): 866-405-2993 or 
> cdn_supp...@comcast.com
> 
> 
> From:  on behalf of NerdyNick 
> Reply-To: "dev@trafficcontrol.incubator.apache.org" 
> 
> Date: Tuesday, September 26, 2017 at 6:30 PM
> To: "dev@trafficcontrol.incubator.apache.org" 
> 
> Subject: Re: Apache Cwiki vs. Github Wiki vs. Github Docs
> 
> If having the Github wiki populated is a desire. You could publish the mb,
> rst, etc docs, already being worked on given Ryan's statement, via github's
> wiki repo methodology.
> https://help.github.com/articles/adding-and-editing-wiki-pages-locally/
> 
> On Tue, Sep 26, 2017 at 5:48 PM, Dave Neuman 
> > wrote:
> 
> I agree with Jan.  I don't see what moving to github wiki really does for
> us that cwiki can't, as a matter of fact it will probably cause other
> issues like only committers can edit it.   I also agree that we should have
> better and more formal docs; especially ones that help new users get ATC
> installed quickly.
> 
> 
> 
> On Tue, Sep 26, 2017 at 17:10 Jan van Doorn 
> > wrote:
> 
>> I think we still need a place to document things like meet ups and the
>> likes? I like the “less formal” feel of a wiki, and think the docs should
>> be more “official”, and part of the release.
>> 
>> Some people have started putting documentation in the README.md with the
>> code, and while I think that’s better than no documentation, I think a
>> project like ours should have official user and admin guides as part of
> the
>> release. We have been letting that part slip a little bit, but just
> giving
>> up on it seems too easy for me…
>> 
>> So I don’t like 1. Or 1. :-) I think we should stay on the apache Cwiki
>> for informal notes and beef up our rst docs for admins and users.
>> 
>> Rgds,
>> JvD
>> 
>> 
>>> On Sep 26, 2017, at 2:56 PM, Durfey, Ryan 
>>> >
>> wrote:
>>> 
>>> All,
>>> 
>>> Given the move to Github, I think we should consider moving out of
>> Apache Cwiki.  While I think this is a far superior wiki to the offering
>> from Github, I think unifying everything in one environment is a better
>> overall approach.  I also wanted to consider foregoing the wiki
> altogether
>> and suggest pushing this documentation into a new “beta” or “design”
>> section under our docs in
>> http://trafficcontrol.apache.org/docs/latest/index.html .  This would
>> unify us into a single documentation format, simplify the transition from
>> design to publication, and eliminate the need to support a wiki
> altogether.
>>> 
>>> Would love feedback on this:
>>> 
>>> 1.  Move to Github wiki
>>> 
>>> Or
>>> 
>>> 1.  Move to a new “beta” section under
>> http://trafficcontrol.apache.org/docs/latest/index.html
>>> 
>>> Ryan Durfey
>>> Sr. Product Manager - CDN | Comcast Technology Solutions
>>> 1899 Wynkoop Ste. 550 | Denver, CO 80202
>>> M | 303-524-5099
>>> ryan_dur...@comcast.com
>>> CDN Support (24x7): 866-405-2993 or 
>>> cdn_supp...@comcast.com> cdn_supp...@comcast.com>
>>> 
>> 
>> 
> 
> 

Re: Removing installation dependencies

[Eric Friedrich (efriedri)]

Jan-
  No need to apologize! 

One of my favorite things about open source is that we solve problems for each 
other and can improve on each solutions. 

I’m really glad you took the time to move those mid rewrite params out into a 
separate profile and even happier that there was automation for it. We have 
some different goals than you do, so we’re glad to iterate on this and 
hopefully contribute something useful for others in a similar position to us

—Eric

> On Sep 14, 2017, at 12:20 PM, Jan van Doorn <j...@knutsel.com> wrote:
> 
> The go migration is my fault…. I couldn’t figure out how to do some of that 
> stuff in SQL, and I think anyone would be hard pressed to do that. 
> 
> Wasn’t aware we need the go compiler for that when I went down that path 
> though. 
> 
> Maybe another alternative is to make a “light migration”? Here’s my thinking 
> - if I remember correctly most of that go code deals with the MSO config 
> moving from parameter overrides in header_rewrite to parent.config. If you 
> don’t have MSO, or are willing to migrate that stuff by hand, we can create a 
> simple SQL migration that just does the schema changes… 
> 
> I’ll be more careful pulling in something new like that next time, sorry… 
> 
> Cheers,
> JvD
> 
> 
>> On Sep 14, 2017, at 10:06 AM, Eric Friedrich (efriedri) <efrie...@cisco.com> 
>> wrote:
>> 
>> As we’re moving to TC2.1, we’ve found that the goose migration requires not 
>> just the goose binary to be installed, but also the go compiler and a fairly 
>> large set of dependencies. Most of these are a result of the migration of 
>> the MSO parent_retry parameters from the DS table into the 
>> profile_parameters table.
>> 
>> We have a very hard requirement that our product CANNOT require Internet 
>> access for installation.
>> 
>> We’d like to avoid vendoring (in one form or another) all of the goose 
>> dependencies, the same way we’ve had to do with web-deps and CPAN.
>> 
>> We are considering two solutions:
>> 1) Replace the .go migration with a PL/SQL file that does not require all of 
>> the go dependencies. In this case we would compile goose and ship it as a 
>> binary in our RPM to eliminate the ‘go get goose'
>> 
>> 2) Switch to a different fork of goose (https://github.com/pressly/goose) 
>> that can execute binary migrations. Here we would compile the .go migration 
>> into a binary and ship both the new goose binary and the migration binary in 
>> our RPM.
>> 
>> —Eric
>> 
>> 
> 

Removing installation dependencies

[Eric Friedrich (efriedri)]

As we’re moving to TC2.1, we’ve found that the goose migration requires not 
just the goose binary to be installed, but also the go compiler and a fairly 
large set of dependencies. Most of these are a result of the migration of the 
MSO parent_retry parameters from the DS table into the profile_parameters table.

We have a very hard requirement that our product CANNOT require Internet access 
for installation.

We’d like to avoid vendoring (in one form or another) all of the goose 
dependencies, the same way we’ve had to do with web-deps and CPAN.

We are considering two solutions:
1) Replace the .go migration with a PL/SQL file that does not require all of 
the go dependencies. In this case we would compile goose and ship it as a 
binary in our RPM to eliminate the ‘go get goose'

2) Switch to a different fork of goose (https://github.com/pressly/goose) that 
can execute binary migrations. Here we would compile the .go migration into a 
binary and ship both the new goose binary and the migration binary in our RPM.

—Eric

Re: Traffic Controller on RedHat Servers

[Eric Friedrich (efriedri)]

We’ve successfully run Traffic Control on Red Hat Servers. Some of the RPM 
versions are slightly different but there are no major changes required

—Eric

> On Sep 13, 2017, at 1:24 PM, Burak Sarp  wrote:
> 
> Hi all,
> I know that traffic controller requires Centos,Unfortunately we dont have 
> Centos servers, so we are going with RedHat servers.
> Do you have any experience and comment about that?
> Thanks,Sarp

Re: Configuration Management - Rules Engine

[Eric Friedrich (efriedri)]

Thanks Matt-

I do think we’ll have rules on both the TR and the caches. 

I also agree that there are some pieces that are simply configuration that 
don’t have associated rules

—Eric

> On Sep 6, 2017, at 2:59 PM, Mills, Matthew <matthew_mi...@comcast.com> wrote:
> 
> Hey Eric,
> 
> I think the intention of the “parameters” was to have a place to store the 
> pieces that don’t fit into rules; I don’t think this has been very well 
> planned out yet, and I also don’t think the examples listed make sense… I 
> think a much better example is initial dispersion, which, while affecting the 
> delivery services behavior, is applied in Traffic Router, where it wouldn’t 
> make much sense to represent things in the rule format that might be used on 
> the cache side. Another better example is the long_desc fields, they don’t 
> have any configuration effect on the cache, but are used for various things.
> 
> -MM
> 
> On 9/5/17, 7:01:29 AM, "Eric Friedrich (efriedri)" <efrie...@cisco.com> wrote:
> 
>Actual Wiki link is here: 
> https://cwiki.apache.org/confluence/display/TC/Configuration+Management#ConfigurationManagement-Rules_Engine
> 
> 
>What is the difference between a parameter and a service rule? From the 
> examples, it looks like parameters are all the legacy behaviors we have today 
> and rules are many of the new behaviors we will shortly be adding.
> 
>Today we have a delivery service configuration that has many parameters 
> (configuration) that are associated with implicit service rules (behaviors). 
> For example, setting the Geoblock field contains configuration (CZF-only, 
> which countries to allow) and there is always a hardcoded "service rule” to 
> make use of that configuration.
> 
>The two “parameter” examples below follow the same pattern
>  - For active/non-active the configuration is boolean and the service 
> rule either processes requests on that DS or rejects them
>  - For Query Strings, the configuration is “use”, “ignore”, “drop” and 
> there is a service rule that implements the logic for those 3 options.
> 
>Similarly all of the Service Rules example can be broken into 
> configuration + behavior.
> 
>It will be more work in the short term to get to this model, but IMO is 
> more cohesive than having some behavior be parameters and some behaviors be 
> service rules.
> 
> 
> 
> 
> 
> 
> 

Re: Configuration Management - Rules Engine

[Eric Friedrich (efriedri)]

Actual Wiki link is here: 
https://cwiki.apache.org/confluence/display/TC/Configuration+Management#ConfigurationManagement-Rules_Engine


What is the difference between a parameter and a service rule? From the 
examples, it looks like parameters are all the legacy behaviors we have today 
and rules are many of the new behaviors we will shortly be adding.

Today we have a delivery service configuration that has many parameters 
(configuration) that are associated with implicit service rules (behaviors). 
For example, setting the Geoblock field contains configuration (CZF-only, which 
countries to allow) and there is always a hardcoded "service rule” to make use 
of that configuration.

The two “parameter” examples below follow the same pattern
  - For active/non-active the configuration is boolean and the service rule 
either processes requests on that DS or rejects them
  - For Query Strings, the configuration is “use”, “ignore”, “drop” and there 
is a service rule that implements the logic for those 3 options.

Similarly all of the Service Rules example can be broken into configuration + 
behavior.

It will be more work in the short term to get to this model, but IMO is more 
cohesive than having some behavior be parameters and some behaviors be service 
rules.





On Sep 1, 2017, at 4:16 PM, Durfey, Ryan 
> wrote:

Opening a new thread on the rules engine. Please keep responses in the email 
thread (vs. wiki). I will summarize once we conclude debate.


Rules Engine

 *   The rules engine is a part of the generic configuration concept but 
separates out trigger/action type configurations from parameter type 
configurations
 *   It allows non-technical users access to powerful customizations that can 
be validated before use and implemented in non-blocking LUA co-routines
 *   Service Parameters are stored in the config database as per usual operation
*   Parameter Example:
   *   Serivce is Active or Non-Active
   *   Service considers Query Strings in URLs or Does Note
 *   Service Rules are stored in a specially formatted data table
*   Rules Example:
*
Trigger

Action

Request to Origin

Sign URL using AWS v4 Signature and Key

URL format regex

Strip content from URL and add to header

Failure to connect to origin

Add cache control header to 502 response with 10 second value


*
 *   Rules Engine Provides
*   List of Available Triggers
*   List of Available Actions
*   List of Required Parameters for Each Trigger / Action
 *   Rules for ATS and NGINX can be implemented with LUA Plugins to the C/C++ 
Code using LUA Scripting
 *   We can also potentially offer raw script fields for advanced users like 
CDN Owners
 *   Rules are generic and are interpreted in similar fashion to generic 
configuration parameters
 *   Rules engine builds in validation rules to ensure non-breaking changes



Ryan Durfey
Sr. Product Manager - CDN | Comcast Technology Solutions
1899 Wynkoop Ste. 550 | Denver, CO 80202
M | 303-524-5099
ryan_dur...@comcast.com
CDN Support (24x7): 866-405-2993 or 
cdn_supp...@comcast.com

Re: Github Issue Organization and Admin Document Updates

[Eric Friedrich (efriedri)]

The JIRA site is currently marked as read-only. I would expect that any changes 
there will fail. 

—Eric

> On Aug 31, 2017, at 10:49 AM, Durfey, Ryan  wrote:
> 
> With the transition of issues to Github, we will be making efforts to 
> organize the github issues and close out the jira site.  We will try to 
> complete by COB Friday 9/15/17.
> 
> To Do:
> 
>  1.  Figure out how to organize the github issues using new tools (labels)
> *   Figure out what reporting is possible
>  2.  Figure out how to close out or archive the jira site so there isn’t 
> confusion
>  3.  Update admin documents and wikis to point to github links
>  4.  Document how to open a github issue properly
> 
> There may be a lot of notifications generated from each site so you may need 
> to turn off notifications temporarily as we proceed.
> 
> Ryan Durfey
> Sr. Product Manager - CDN | Comcast Technology Solutions
> 1899 Wynkoop Ste. 550 | Denver, CO 80202
> M | 303-524-5099
> ryan_dur...@comcast.com
> CDN Support (24x7): 866-405-2993 or 
> cdn_supp...@comcast.com
> 

Re: [VOTE] Bugtracking in Github Issues

[Eric Friedrich (efriedri)]

Thanks Leif-
  I’ll check with the Incubator gurus. We already pulled the trigger, so at 
this point its more about asking for forgiveness.

—Eric
On Aug 29, 2017, at 6:47 PM, Leif Hedstrom 
<zw...@apache.org<mailto:zw...@apache.org>> wrote:


On Aug 28, 2017, at 10:38 AM, Eric Friedrich (efriedri) 
<efrie...@cisco.com<mailto:efrie...@cisco.com>> wrote:

We currently use JIRA Issues to track all of the Traffic Control bugs.

Now that we have write access to Github, we can move back to GH Issues for bug 
tracking.

This will be a better workflow because its one fewer tool and account to have 
to interact with. This will hopefully lower the bar for new contributors to 
file bugs and engage with the product. We can also help use it (along with the 
milestones) to create more useful changelogs and release notes.

A possible downside is that the Issues are maybe less flexible than JIRA in 
terms of permissions, workflow, fields, etc. However, we were using Issues 
before we entered the incubator and that was fine for us. Hopefully no one has 
developed an attachment for JIRA in the last year.


Please Vote +1 to move to Github Issues or -1 to stay on Jira. I’ll assume a 
lazy consensus if there aren’t enough votes.



I’m +1 on this, but did you verify with the Incubator powers that this is 
allowed? Another option is to defer this until you are down with incubation, 
and make the migration to a TLP include moving more of your workflow over to 
Github proper.

— Leif

[VOTE] Bugtracking in Github Issues

[Eric Friedrich (efriedri)]

We currently use JIRA Issues to track all of the Traffic Control bugs. 

Now that we have write access to Github, we can move back to GH Issues for bug 
tracking. 

This will be a better workflow because its one fewer tool and account to have 
to interact with. This will hopefully lower the bar for new contributors to 
file bugs and engage with the product. We can also help use it (along with the 
milestones) to create more useful changelogs and release notes. 

A possible downside is that the Issues are maybe less flexible than JIRA in 
terms of permissions, workflow, fields, etc. However, we were using Issues 
before we entered the incubator and that was fine for us. Hopefully no one has 
developed an attachment for JIRA in the last year. 


Please Vote +1 to move to Github Issues or -1 to stay on Jira. I’ll assume a 
lazy consensus if there aren’t enough votes. 

—Eric

Re: Preventing routing to individual caches

[Eric Friedrich (efriedri)]

This is more of a general question, not related to this specific feature:

What determines when something can be a new column in the profile table versus 
a parameter? (this also goes back to our DB normalization discussion)

--Eric


From: Mark Torluemke <mtorlue...@apache.org>
Sent: Thursday, August 24, 2017 12:11 PM
To: dev@trafficcontrol.incubator.apache.org
Subject: Re: Preventing routing to individual caches

I'm good with a new column on the profile table. Also, I don't share the
concern on this slowing down any queries significantly.

On Thu, Aug 24, 2017 at 8:52 AM, Gelinas, Derek <derek_geli...@comcast.com>
wrote:

> I think profile is right out - that means a profile lookup for each server
> that we process, and that’s going to make an already slow subroutine a lot
> slower.
>
> DG
>
> > On Aug 24, 2017, at 10:40 AM, Gelinas, Derek <derek_geli...@comcast.com>
> wrote:
> >
> > I’m not sure it would work, but I’ll look into it.
> >
> > Assuming it does not, does anyone have any strong feelings about any of
> the choices?  My personal preference is to use option 3 or option 1, or to
> use ccr_ignore.
> >
> > 1) Server table flag - when marked, nothing is routed to the host at
> > all.  Not as configurable as option 3, but more so than option 2.  Faster
> > than option 2 as it would be returned with existing search results and
> > could be easily filtered on.  Minor UI change only.
> > 2) Profile parameter - when marked, nothing is routed to any host
> > with this profile.  Heavy handed, and would require additional profile
> > parameter lookups when generating the crconfig, so it'd slow it down. No
> UI
> > change.
> > 3) deliveryservice_servers table flag - an additional column that is
> > true by default.  When desired, the user could pull up a sub-window
> within
> > the delivery service configuration that would present a list of the hosts
> > which have been assigned to the delivery service (and are not of org
> > type).  The user could deselect the desired hosts, setting the DSS
> routing
> > value to false.  This server would then be ignored when generating the
> > crconfig data for that specific delivery service.  This would be the most
> > configurable option, and should be as quick as option 1, but would
> require
> > the most extensive code changes.
> > 4) Column in the “type” table. Like option 1, this would apply at the
> server level.
> > 5) Column in the “profile” table.  Like option 2, this would apply at
> the profile level.
> >
> >
> >> On Aug 23, 2017, at 5:53 PM, rawlin.pet...@gmail.com <
> rawlin_pet...@comcast.com> wrote:
> >>
> >> What about the server status CCR_IGNORE ("Server is ignored by traffic
> router.") that already exists? It doesn't appear to be checked when
> generating CRConfig right now, but maybe it should be?
> >>
> >> --Rawlin
> >>
> >> On 2017-08-22 11:45, "Gelinas, Derek" <derek_geli...@comcast.com>
> wrote:
> >>> Iâ?Td agree with you if this was designed to drain, but this is
> intended as a permanent state for a pretty good long list of caches.
> >>>
> >>> DG
> >>>
> >>>> On Aug 22, 2017, at 1:28 PM, Eric Friedrich (efriedri) <
> efrie...@cisco.com> wrote:
> >>>>
> >>>> What about a modification of option 1- adding a new state per server.
> >>>>
> >>>> Instead of ADMIN_DOWN, it could be â?oREPORTED_DRAINâ?  to indicate
> the difference
> >>>>
> >>>> â?"Eric
> >>>>
> >>>>> On Aug 22, 2017, at 1:14 PM, Gelinas, Derek <
> derek_geli...@comcast.com> wrote:
> >>>>>
> >>>>> Thatâ?Ts actually the workaround weâ?Tre using at the moment -
> setting them to admin_down.  Thatâ?Ts a temporary measure, though - we want
> something more permanent.
> >>>>>
> >>>>> DG
> >>>>>> On Aug 22, 2017, at 1:09 PM, Eric Friedrich (efriedri) <
> efrie...@cisco.com> wrote:
> >>>>>>
> >>>>>> How does your use case differ from marking a server as offline in
> Traffic Ops and snapshotting?
> >>>>>>
> >>>>>> Thats the easiest way I can think of to get a server in this state
> >>>>>>
> >>>>>> â?"Eric
> >>>>>>
> >>>>>>> On Aug 22, 2017, at 1:00 PM, Gelinas, Derek <
> derek_geli...@comcast.com> wrote:
> >

RE: Preventing routing to individual caches

[Eric Friedrich (efriedri)]

Could this be a new DS type or does it apply to a whole server?

From: Gelinas, Derek [derek_geli...@comcast.com]
Sent: Tuesday, August 22, 2017 6:22 PM
To: dev@trafficcontrol.incubator.apache.org
Subject: Re: Preventing routing to individual caches

The use case is fairly specific.  Suffice it to say we have reverse proxies 
that need configuration without being treated as potential destinations by 
traffic router.

DG

On Aug 22, 2017, at 3:19 PM, Nir Sopher <n...@qwilt.com<mailto:n...@qwilt.com>> 
wrote:

Hi Derek,

Could you please shade more light on the problem you are trying to solve?

As I see it, option #3 is indeed more flexible - as it can work in a DS
granularity.
It is even more powerful when you combine it with other extensions for this
table suggested in the "Drop Server -> Delivery Service assignments".

However, as you describe options #1,#2 as valid options, it seems that the
problem you are dealing with completely resides in the "servers" domain -
as the server should have the same behavior for all delivery-services.
Therefore, option #1 might be more suitable.

Nir

On Tue, Aug 22, 2017 at 8:45 PM, Gelinas, Derek 
<derek_geli...@comcast.com<mailto:derek_geli...@comcast.com>>
wrote:

I'd agree with you if this was designed to drain, but this is intended as
a permanent state for a pretty good long list of caches.

DG

On Aug 22, 2017, at 1:28 PM, Eric Friedrich (efriedri) <
efrie...@cisco.com<mailto:efrie...@cisco.com>> wrote:

What about a modification of option 1- adding a new state per server.

Instead of ADMIN_DOWN, it could be "REPORTED_DRAIN" to indicate the
difference

-Eric

On Aug 22, 2017, at 1:14 PM, Gelinas, Derek 
<derek_geli...@comcast.com<mailto:derek_geli...@comcast.com>>
wrote:

That's actually the workaround we're using at the moment - setting them
to admin_down.  That's a temporary measure, though - we want something more
permanent.

DG
On Aug 22, 2017, at 1:09 PM, Eric Friedrich (efriedri) <
efrie...@cisco.com<mailto:efrie...@cisco.com>> wrote:

How does your use case differ from marking a server as offline in
Traffic Ops and snapshotting?

Thats the easiest way I can think of to get a server in this state

-Eric

On Aug 22, 2017, at 1:00 PM, Gelinas, Derek <
derek_geli...@comcast.com<mailto:derek_geli...@comcast.com>> wrote:

We've run across a situation in which we need certain caches to
simultaneously have map rules for a delivery service, but not actually have
those caches routed to when requests are made via traffic router.
Essentially, this means removing the delivery service from the cache's info
in the crconfig file.

There's been a bit of internal debate about the best ways to do this,
and I'd like to collect some opinions on the matter.

1) Server table flag - when marked, nothing is routed to the host at
all.  Not as configurable as option 3, but more so than option 2.  Faster
than option 2 as it would be returned with existing search results and
could be easily filtered on.  Minor UI change only.
2) Profile parameter - when marked, nothing is routed to any host
with this profile.  Heavy handed, and would require additional profile
parameter lookups when generating the crconfig, so it'd slow it down. No UI
change.
3) deliveryservice_servers table flag - an additional column that is
true by default.  When desired, the user could pull up a sub-window within
the delivery service configuration that would present a list of the hosts
which have been assigned to the delivery service (and are not of org
type).  The user could deselect the desired hosts, setting the DSS routing
value to false.  This server would then be ignored when generating the
crconfig data for that specific delivery service.  This would be the most
configurable option, and should be as quick as option 1, but would require
the most extensive code changes.

Personally, I like option 3, but would very much like to hear
opinions, arguments, and other options that I haven't thought of or listed
here.

Derek

Re: Preventing routing to individual caches

[Eric Friedrich (efriedri)]

What about a modification of option 1- adding a new state per server. 

Instead of ADMIN_DOWN, it could be “REPORTED_DRAIN” to indicate the difference

—Eric

> On Aug 22, 2017, at 1:14 PM, Gelinas, Derek <derek_geli...@comcast.com> wrote:
> 
> That’s actually the workaround we’re using at the moment - setting them to 
> admin_down.  That’s a temporary measure, though - we want something more 
> permanent.
> 
> DG
>> On Aug 22, 2017, at 1:09 PM, Eric Friedrich (efriedri) <efrie...@cisco.com> 
>> wrote:
>> 
>> How does your use case differ from marking a server as offline in Traffic 
>> Ops and snapshotting?
>> 
>> Thats the easiest way I can think of to get a server in this state
>> 
>> —Eric
>> 
>>> On Aug 22, 2017, at 1:00 PM, Gelinas, Derek <derek_geli...@comcast.com> 
>>> wrote:
>>> 
>>> We’ve run across a situation in which we need certain caches to 
>>> simultaneously have map rules for a delivery service, but not actually have 
>>> those caches routed to when requests are made via traffic router.  
>>> Essentially, this means removing the delivery service from the cache’s info 
>>> in the crconfig file.
>>> 
>>> There’s been a bit of internal debate about the best ways to do this, and 
>>> I’d like to collect some opinions on the matter.
>>> 
>>> 1) Server table flag - when marked, nothing is routed to the host at all.  
>>> Not as configurable as option 3, but more so than option 2.  Faster than 
>>> option 2 as it would be returned with existing search results and could be 
>>> easily filtered on.  Minor UI change only.
>>> 2) Profile parameter - when marked, nothing is routed to any host with this 
>>> profile.  Heavy handed, and would require additional profile parameter 
>>> lookups when generating the crconfig, so it’d slow it down. No UI change.
>>> 3) deliveryservice_servers table flag - an additional column that is true 
>>> by default.  When desired, the user could pull up a sub-window within the 
>>> delivery service configuration that would present a list of the hosts which 
>>> have been assigned to the delivery service (and are not of org type).  The 
>>> user could deselect the desired hosts, setting the DSS routing value to 
>>> false.  This server would then be ignored when generating the crconfig data 
>>> for that specific delivery service.  This would be the most configurable 
>>> option, and should be as quick as option 1, but would require the most 
>>> extensive code changes.
>>> 
>>> Personally, I like option 3, but would very much like to hear opinions, 
>>> arguments, and other options that I haven’t thought of or listed here.
>>> 
>>> Derek
>> 
> 

Re: Preventing routing to individual caches

[Eric Friedrich (efriedri)]

How does your use case differ from marking a server as offline in Traffic Ops 
and snapshotting?

Thats the easiest way I can think of to get a server in this state

—Eric

> On Aug 22, 2017, at 1:00 PM, Gelinas, Derek  wrote:
> 
> We’ve run across a situation in which we need certain caches to 
> simultaneously have map rules for a delivery service, but not actually have 
> those caches routed to when requests are made via traffic router.  
> Essentially, this means removing the delivery service from the cache’s info 
> in the crconfig file.
> 
> There’s been a bit of internal debate about the best ways to do this, and I’d 
> like to collect some opinions on the matter.
> 
> 1) Server table flag - when marked, nothing is routed to the host at all.  
> Not as configurable as option 3, but more so than option 2.  Faster than 
> option 2 as it would be returned with existing search results and could be 
> easily filtered on.  Minor UI change only.
> 2) Profile parameter - when marked, nothing is routed to any host with this 
> profile.  Heavy handed, and would require additional profile parameter 
> lookups when generating the crconfig, so it’d slow it down. No UI change.
> 3) deliveryservice_servers table flag - an additional column that is true by 
> default.  When desired, the user could pull up a sub-window within the 
> delivery service configuration that would present a list of the hosts which 
> have been assigned to the delivery service (and are not of org type).  The 
> user could deselect the desired hosts, setting the DSS routing value to 
> false.  This server would then be ignored when generating the crconfig data 
> for that specific delivery service.  This would be the most configurable 
> option, and should be as quick as option 1, but would require the most 
> extensive code changes.
> 
> Personally, I like option 3, but would very much like to hear opinions, 
> arguments, and other options that I haven’t thought of or listed here.
> 
> Derek

Re: Adding support for per-DeliveryService routing names

[Eric Friedrich (efriedri)]

> On Aug 4, 2017, at 3:09 PM, Peters, Rawlin <rawlin_pet...@comcast.com> wrote:
> 
> Ok, it makes sense that if it can support fully customized DS domain names, 
> there would be
> nothing stopping the user from entering a HOST_REGEXP 
> “foo.ds.cdn.company.com” to
> essentially pick “foo” as the custom routing name. However, I think that 
> misses the point
> of custom routing names. If the xml_id or CDN domain name changes, that 
> HOST_REGEXP
> would no longer work and would need updated to keep the DS running, right?
EF> Ok I understand now! The purpose is not to be able to use something instead 
of “edge” or “tr” but to keep the HOST_REGEXP static when xml_id or domain_name 
change. 

> 
> Custom routing names would be for users who continue to use the default 
> “.*\.ds\..*”
> HOST_REGEXP in their delivery service rather than a fully-customized domain. 
> That way
> they can change their DS more freely without the HOST_REGEXP requiring 
> constant updating.
> 
> --Rawlin
> 
> On 8/4/17, 10:50 AM, "Eric Friedrich (efriedri)" <efrie...@cisco.com> wrote:
> 
>As I understand Zhilin’s changes, they are a superset of changing the 
> routing name.
> 
>Whereas today we must have 
> “tr.ds.cdn.company.com<http://tr.ds.cdn.company.com>” or “edge.”, with 
> Zhilin’s changes you can set a completely custom DS FQDN.
> 
>As he puts it in his original email, "And 
> “my-subdomain.topdomain.com<http://my-subdomain.topdomain.com>” in pattern 2) 
> will be used as the RFQDN instead of 
> “tr.my-subdomain.cdndomain.com<http://tr.my-subdomain.cdndomain.com>” or 
> “edge.my-subdomain.cdndomain.com<http://edge.my-subdomain.cdndomain.com>”. 
> This way, user can fully customize the whole domain for a delivery service.”
> 
>This should (I hope!) extend to being able to set 
> “customroutingname.ds.cdn.company.com<http://customroutingname.ds.cdn.company.com>”
>  on a per-DS basis (or really any other custom Delivery Service FQDN someone 
> could want).
> 
>The motivation behind his proposal is to better support wholesale 
> customers who want to bring their own domain without needing to CNAME over to 
> the CDN (and play the corresponding games with HTTPS certs)
> 
>—Eric
> 
> 
>On Aug 4, 2017, at 12:37 PM, Peters, Rawlin 
> <rawlin_pet...@comcast.com<mailto:rawlin_pet...@comcast.com>> wrote:
> 
>@Dave @Eric
>I have my current code pushed to my fork, and it can be compared against 
> apache/master here [1].
> 
>I did see Zhilin’s original proposal on the mailing list, and I also 
> thought it seemed similar
>but not the same thing as what I’d been working on. In his example, there 
> is a reference
>to “tr.test.ipcdn.mycompany.com<http://tr.test.ipcdn.mycompany.com>.”, 
> which means the routing names are not being
>completely replaced, and custom DS domain support would be added alongside 
> the
>current functionality of using routing names like “tr” and “edge”.
> 
>- Rawlin
> 
>[1] 
> https://github.com/apache/incubator-trafficcontrol/compare/master...rawlinp:per-cdn-routing-names_2
> 
>On 8/4/17, 9:39 AM, "Dave Neuman" 
> <neu...@apache.org<mailto:neu...@apache.org>> wrote:
> 
>   @Eric
>   It looks like Zhilin's proposal is for custom delivery service domains 
> (eg
>   instead of "tr.foo.domain.com<http://tr.foo.domain.com>" you can have 
> "tr.foo.otherdomain.com<http://tr.foo.otherdomain.com>") while
>   Rawlins is for custom routing names (eg instead of 
> "tr.foo.domain.com<http://tr.foo.domain.com>" you
>   can have "bar.foo.domain.com<http://bar.foo.domain.com>").  I think the 
> two features are similar but
>   different.  Would you agree or am I misunderstanding?
>   @Rawlin and @Zhilin if you have WIP code maybe you could open a WIP PR 
> and
>   we can take a look to see if there will be conflicts?
> 
>   --Dave
> 
>   On Fri, Aug 4, 2017 at 8:59 AM, Durfey, Ryan 
> <ryan_dur...@comcast.com<mailto:ryan_dur...@comcast.com>>
>   wrote:
> 
>Yeah, I just rethought that.
> 
>I was envisioning   http://servicename.customername.cdn_domain/  where we
>could get a cert for “*.customername.cdn_domain/” for multiple customer
>services.
> 
>However, we currently have to use the format http://servicename-
>cusotmername.cdn_domain/ where a wildcard cert would not be applicable.
> 
>Apologies for the confusion.
> 
> 
>Ryan DurfeyM | 303-524-5099
>CDN Support (24x7):

Re: Adding support for per-DeliveryService routing names

[Eric Friedrich (efriedri)]

dn_domain”?

Ryan DurfeyM | 303-524-5099
CDN Support (24x7): 866-405-2993 or 
cdn_supp...@comcast.com<mailto:cdn_supp...@comcast.com>mailto:cdn_supp...@comcast.com>>mailto:cdn_supp...@comcast.com><mailto:cdn_supp...@comcast.com>>


From: Jan van Doorn 
<j...@knutsel.com<mailto:j...@knutsel.com><mailto:j...@knutsel.com>>
Reply-To: 
"dev@trafficcontrol.incubator.apache.org<mailto:dev@trafficcontrol.incubator.apache.org><mailto:dev@
trafficcontrol.incubator.apache.org<http://trafficcontrol.incubator.apache.org>>"
 <
dev@trafficcontrol.incubator.apache.org<mailto:dev@trafficcontrol.incubator.apache.org><mailto:dev@
trafficcontrol.incubator.apache.org<http://trafficcontrol.incubator.apache.org>>>
Date: Friday, August 4, 2017 at 8:04 AM
To: 
"dev@trafficcontrol.incubator.apache.org<mailto:dev@trafficcontrol.incubator.apache.org><mailto:dev@
trafficcontrol.incubator.apache.org<http://trafficcontrol.incubator.apache.org>>"
 <
dev@trafficcontrol.incubator.apache.org<mailto:dev@trafficcontrol.incubator.apache.org><mailto:dev@
trafficcontrol.incubator.apache.org<http://trafficcontrol.incubator.apache.org>>>
Subject: Re: Adding support for per-DeliveryService routing names

Agree with Dave on

[*DN] we should default the database column to "edge" for DNS and "tr" for*
*http.  Then we don't have to do the null check.*

If we do that, we can make the columns mandatory, and it makes sense
they're not in the DS_PROFILE. Also makes it so we don't have to have a CDN
wide setting. (and Rawlin, I think you mean to say DS_PROFILE rather than
TR_PROFILE type to add the param to if we chose to do that?? Or was it the
default that goes into TR_PROFILE and the override into DS_PROFILE?).
In any case - if we make the columns NOT NULL and default them to "tr" and
"edge", I'm +1 on columns in the deliveryservice table.

Cheers,
JvD

On Fri, Aug 4, 2017 at 7:12 AM Eric Friedrich (efriedri) <
efrie...@cisco.com<mailto:efrie...@cisco.com><mailto:efrie...@cisco.com><mailto:efrie...@cisco.com
<mailto:efrie...@cisco.com%3e>>
wrote:

Hey Rawlin-
   Zhilin has also been working on a very similar feature which was
proposed on this mailer last month:
https://lists.apache.org/thread.html/51d7ed1ae65a3697c39edd00236e6f
3897da37ef5b24ac452a17cabb@%3Cdev.trafficcontrol.apache.org%3E
<
https://lists.apache.org/thread.html/51d7ed1ae65a3697c39edd00236e6f
3897da37ef5b24ac452a17cabb@
>

Can you please work him to ensure we don’t duplicate work and that if both
solutions are needed they will work together?

On Aug 3, 2017, at 3:57 PM, Peters, Rawlin <rawlin_pet...@comcast.com<
mailto:rawlin_pet...@comcast.com><
mailto:rawlin_pet...@comcast.com>
<mailto:rawlin_pet...@comcast.com><mailto:rawlin_pet...@comcast.com
%3e><mailto:rawlin_pet...@comcast.com%3e%3cmailto:Rawlin_Peters@
comcast.com%3e%3e>>
wrote:

Sorry, Outlook converted my numbered list poorly. I’ve corrected the
numbering (items 1-3) below.

On 8/3/17, 1:52 PM, "Peters, Rawlin" <rawlin_pet...@comcast.commailto:rawlin_pet...@comcast.com>>mailto:rawlin_pet...@comcast.com>><mailto:rawlin_pet...@comcast.com%3e%3e>>
wrote:

Hello All,

I’ve been working on adding support for configurable per-CDN and
per-DeliveryService routing names [1] (what are currently
hardcoded/defaulted to ‘edge’ and ‘tr’ for DNS and HTTP Delivery Services,
respectively), and I have a few things to propose.


  1.  Add a column to the CDN table for the DNS and HTTP routing
names.



I’ve currently been working off the assumption that per-CDN routing
names would be configurable by adding ‘http.routing.name’ and ‘
dns.routing.name’ parameters to a profile of type TR_PROFILE using the
‘CRConfig.json’ config file. To me this seems like bad UX because the user
has to click through multiple steps and fill in multiple fields in the UI
just to change the CDN’s routing names. It also requires joining a few
different tables in the DB just to find the parameters per-CDN. For that
reason, I think it would be better if ‘dns_routing_name’ and
‘http_routing_name’ were added as columns of the ‘cdn’ table, and changing
them via the UI would follow the same process as choosing the CDN’s domain
name. Because the routing names would be the CDN-wide defaults, the ‘Edit
CDN’ window feels like the most natural place to put it.


  2.  Values for per-DeliveryService routing names could live in one
of
a couple different areas:
 *   New columns in the delivery_service table
 *   Parameters in a DS Profile

As the developer, my vote would be for Option A because it seems like
it would lead to cleaner code in Traffic Ops because the routing names
would be readily-available when handling a DeliveryService. You wouldn’t
have to

Re: Traffic Ops Golang Migration Proposal

[Eric Friedrich (efriedri)]

 likely to exist for most
users, or check for a package. Both approaches require platform
specific code and assumptions.

Because the traffic_ops_golang package will depend on traffic_ops, not
the other way around, it makes more sense to place the configuration
piece in the golang package. When the golang package is installed, we
can "take over" the port in the listen directive of cdn.conf, because
we know for a fact that it is on disk because of the RPM dependency on
traffic_ops. We also know that cdn.conf will be left alone if/when
traffic_ops is upgraded due to being marked as a config file. If the
user has installed either component outside of the normal RPM process,
they will have to figure out how to run the golang package separately,
as one would expect.

We can do the configuration during the postinstall step of the
traffic_ops_golang RPM. It's advantageous to manage that piece within
the RPM, because if, for example, one wanted to remove the golang
portion, we could have a postuninstall step that reverts changes made
to cdn.conf (put the port we took over back into cdn.conf). We could
seamlessly add and remove the traffic_ops_golang component without
disturbing anything in traffic_ops, and without having to run some
script manually. The platform specific things that would need to be
done in postinstall should be done in the RPM, because then we know
for sure which platform we're on, and assumptions about packages and
paths will be accurate.

Ideally we should be moving away from any manual run of any script
after an installation or upgrade, including postinstall, if that work
can be done within the RPM.
--
Thanks,
Jeff


On Wed, Jul 19, 2017 at 9:08 AM, Robert Butts <
robert.o.bu...@gmail.com<mailto:robert.o.bu...@gmail.com>>
wrote:
It sounds like the only -1 is having a unified Service and RPM.

How about the following compromise: a separate RPM and Service for
the
New
TO, and the New TO RPM is a dependency of Old TO, and likewise the
Service
is a dependency of `traffic_ops`. This way, upgrades will still
require
building the New TO RPM and adding it to Yum, but `yum upgrade` will
automatically install it without additional ops work, and `service
traffic_ops start` will also start the New TO. Bearing in mind this
double-service awkwardness will go away when all endpoints are
migrated.

Also, @alficles suggested configuring the New TO Config in
Postinstall
(which must be run after upgrading anyway). Because the New is a
dependency
of the Old, we're guaranteed `/opt/traffic_ops_golang` exists in
Postinstall, and can populate its config.

Also, I realized the New TO needs moved from
`/traffic_ops/traffic_ops_golang` to `/traffic_ops_golang`, because
`build_all.sh` requires projects be in the root directory.

Also, I will add tests, docs, and configurable logging before the PR
is
merged. (Just wanted to wait until we had consensus before putting
more
work
into it.)

How does that sound?


On Thu, Jul 13, 2017 at 1:00 PM, Robert Butts <
robert.o.bu...@gmail.com<mailto:robert.o.bu...@gmail.com>>
wrote:

@dewrich It's not that putting both services in the same Service and
RPM
is a good long-term solution; it's that it's easier to configure and
deploy.
A separate RPM and service is another step to install, and another
step
to
start the service. It's not that it's good, it's that it's the
lesser
of two
evils. My fear is, the more complex we make this, the less chance of
getting
it done at all.

@efriedri
1. Same answer: long-term, I absolutely agree proxies should be
separate.
But this is the simplest way to deploy an incremental migration.
2. Yes, the Perl GUI can transparently request what it needs.
That's a
goal of this: transparent new endpoints, that existing services
don't
know
or care that they come from a different backend.

Also bear in mind, this "proxy" only exists until Perl TO goes away.
Having it in the same Service and RPM as old TO, and having the
proxy
be the
same binary as the new endpoints, makes it easier to completely
remove
the
Proxy and Perl, once every endpoint is rewritten. If we make the
Service and
RPM separate, you have to change configs and uninstall the separate
Service
and RPM of the Perl TO. If the Proxy is a separate binary, you have
to
uninstall that, and then change the config of the Golang TO to
serve on
the
real port (443). As proposed, once all endpoints are rewritten, we
simply
remove the old TO from the RPM and Service, and users just upgrade,
and
it
keeps working, with no changes to config, Puppet, RPM, or anything
else.

I'd fully support different RPMs, different Services, and a separate
Proxy, once this is deployed. The fear is that the more complex we
make
this, the less chance it gets deployed at all.


On Thu, Jul 13, 2017 at 12:23 PM, Eric Friedrich (efriedri)
<efrie...@cisco.com<mailto:efrie...@cisco.com>> wrote:

Rob-
 Two minor questions:

1)  An alternative approach could have been using an LB/proxy to
cho

Re: 2.1 RM

[Eric Friedrich (efriedri)]

Hey Hank-
  Many Thanks. Your RM baseball cap will be in the mail!

I need to clean up the Release Management wiki page a bit for you. I’ll try to 
do that in the next few days.

When’s the release branch get pulled? ;-)

—Eric



> On Jul 17, 2017, at 3:03 PM, Dan Kirkwood  wrote:
> 
> Thanks,  Hank..feel free to ping either me or Eric if anything is
> unclear in the release manager notes.
> 
> -dan
> 
> On Mon, Jul 17, 2017 at 12:56 PM, Hank Beatty  wrote:
>> Hi Dave,
>> 
>> I would like to volunteer.
>> 
>> Thanks,
>> Hank
>> 
>> On 07/05/2017 03:38 PM, Dave Neuman wrote:
>>> 
>>> Hey All,
>>> Now that 2.0 has officially passed the project and IPMC vote (YAY!), it's
>>> time to start thinking about 2.1.  I don't think we have identified a
>>> release manager for the 2.1 release yet, would anyone like to volunteer?
>>> 
>>> Thanks,
>>> Dave
>>> 
>> 

Re: 2.1 RM

[Eric Friedrich (efriedri)]

Any takers for 2.1 release manager? Dan and I will both make ourselves 
available to help out and the process is pretty well known at this point. 

—Eric



> On Jul 6, 2017, at 2:12 PM, Dan Kirkwood  wrote:
> 
> fwiw,  1.8.0 was lengthy because of the transition to Apache Incubator
> -- we had a lot of work there.  And 2.0.0 was as well because of the
> transition to postgres from mysql.   2.1.0 is expected to be
> significantly less trouble,   and Eric and I have paved the way to
> make it so.   We'll be available to assist whomever takes on the role
> of release manager.
> 
> -dan
> 
> On Wed, Jul 5, 2017 at 1:38 PM, Dave Neuman  wrote:
>> Hey All,
>> Now that 2.0 has officially passed the project and IPMC vote (YAY!), it's
>> time to start thinking about 2.1.  I don't think we have identified a
>> release manager for the 2.1 release yet, would anyone like to volunteer?
>> 
>> Thanks,
>> Dave

Re: Promote Golang Traffic Monitor to Default

[Eric Friedrich (efriedri)]

I think I remember Rob making this point in Miami, but all of TMs APIs (REST, 
CRConfig, Health.json, etc…) are identical between the Java and Golang version, 
right?

What about compatibility with earlier versions of TC?

For example:
- Can a TC1.7 traffic ops configure a Golang TM?
- Does the Golang TM have any dependencies on a certain version of 
TrafficServer or astats?
- Whats the minimum required version of Traffic Router to use the Golang TM?
- I know Golang TMs can gossip with Java TMs, but can we mix versions here too? 
(i.e. TC1.7 Java TM with TC2.1 Golang TM)?

—Eric


> On Jul 14, 2017, at 1:00 PM, Jeff Elsloo  wrote:
> 
> Hi all,
> 
> We currently have two versions of Traffic Monitor: Java and golang.
> When we build all components, as far as I know, it results in a race
> condition between the two, as the resulting RPMs have the same
> filename. A PR[1] was opened to address the issue and the approach was
> to add `_go` to the version string used for the golang version's RPM.
> 
> Rob and I both think we (Comcast) have enough experience running the
> golang version that we have identified and corrected any major issues
> and that it is stable enough to be the preferred Traffic Monitor hence
> forth.
> 
> Therefore, I propose that within the project's directory structure, we:
>  1) rename traffic_monitor to traffic_monitor_legacy
>  2) rename traffic_monitor_golang to traffic_monitor
> 
> ..then work with the person that submitted the PR to take the same
> approach, except change the Java version's RPM name to contain
> `_legacy`.
> 
> I realize that this is a fairly significant change, the type that we
> typically reserve for major releases. The next major release, 3.0.0,
> is likely to be some time out in the future, and I don't know that we
> need to wait for it in order to make this change.
> 
> How does the group feel about the above proposal, and executing on it
> prior to the 3.0.0 release (i.e.: for 2.1.0)? Then, when we do
> actually prepare the 3.0.0 release, we can remove the Java version
> from the codebase entirely. Obviously this could impact anyone that
> has automated CI systems building components, in addition to the
> Apache CI we use ourselves.
> 
> Thoughts?
> 
> [1] https://github.com/apache/incubator-trafficcontrol/pull/731
> --
> Thanks,
> Jeff

Re: Support custom routing selection logics

[Eric Friedrich (efriedri)]

Thanks John-
  A few more questions:

   - Are there any extensions to the Track class needed here? 

   - Can we indicate in the access log how the plugins caused the request to be 
routed? If a plugin wanted to add additional detail to a Track object is that 
possible? 

   - Can a plugin interact with the stats API at all (if we want to see how 
many requests the plugin is serving or what the status of the connection 
between plugin and external system is)?

   - I assume the plugin may need to have some context associated with it. Do 
you have a prototype of what the callback made to the plugin will look like?

Also just for information because I think its an important point- such a plugin 
would be statically compiled into the TR “jar"

—Eric



> On Jun 23, 2017, at 10:49 AM, John Shen (weifensh)  wrote:
> 
> Traffic Router currently support Coverage Zone File based routing and 
> Geolocation based routing. We are planning to add another type of routing 
> selection logic. Since this part of code will not be put into open source 
> code, we are considering how to integrate the new routing selection code into 
> current TR.
> 
> One solution is to make Traffic Router support routing selection plugins. 
> Then we implement our new routing selection logic as a plugin, which is still 
> statically built into the TR package. Other types of new routing selection 
> plugins could also be integrated into TR easily. The plan is to add several 
> hooks in the method of:
> protected List TrafficRouter::selectCaches(final Request request, 
> final DeliveryService ds, final Track track);
> 
> There could be 4 possible hooks to be added: PRE_CZF_HOOK, POST_CZF_HOOK, 
> PRE_GEO_HOOK, and POST_GEO_HOOK. They will be processed during different 
> stages in TrafficRouter::selectCaches(). Plugins could be registered in each 
> hook. If there are multiple plugins in each hook, they will be processed in 
> the order of how they are registered. Any plugin returning non-empty caches 
> means a cache selection is successful for a certain request, thus no further 
> plugin or CZF/GEO based routing is needed for the request.
> 
> Any plugin will be created and registered during system initialization. A 
> possible point to do so is when ConfigHandler is initialized. And a plugin 
> would have following interfaces:
> loadConfig(); // called when CRConfig is updated
> selectCaches(); // called in TrafficRouter::selectCaches() hooks
> 
> Please help to comment this proposal. Thanks.
> 
> Best regards,
> John
> 

Re: Custom Delivery Service Domain Support

[Eric Friedrich (efriedri)]

Thanks Zhilin-
  Could I use a domain of just “topdomain-cdn.com” or does it require a minimum 
of three levels?

Is configuration just in delivery service or does the domain_name parameter 
need to be modified too?


> On Jun 21, 2017, at 4:56 AM, Zhilin Huang (zhilhuan)  
> wrote:
> 
> Hi all,
> 
> I am working on a feature to support custom delivery service domain.
> 
> Currently in traffic ops, you can configure a “HOST_REGEXP” for a delivery 
> service with pattern like:
> 1)  .*\.my-subdomain\..*
> 2) my-subdomain.topdomain.com
> 
> I think pattern 1) is well known by all of us. However pattern 2) is allowed 
> in traffic ops code, but not well supported. The Custom Delivery Service 
> Domain feature will fully support pattern 2) and allow “topdomain.com” be 
> different with the domain name of a CDN. This will allow to support different 
> domains within a single CDN, and share the router/monitor/cache servers with 
> a CDN for different (top level) domains.
> 
> And “my-subdomain.topdomain.com” in pattern 2) will be used as the RFQDN 
> instead of “tr.my-subdomain.cdndomain.com” or 
> “edge.my-subdomain.cdndomain.com”. This way, user can fully customize the 
> whole domain for a delivery service. The NS records for pattern 2) will be 
> the fqdn of the routers in this CDN.
> 
> Any comments/suggestions will be welcome. The draft code is ready and under 
> testing. And an example is attached in the end of this email.
> 
> Thanks,
> Zhilin
> 
> 
> 
> Here is the example, suppose we have a CDN with domain name 
> “ipcdn.mycompany.com”. And configured 2 delivery services with the following 
> “HOST_REGEXP”:
> DS1:   .*\.test\..*
> DS2:   test.wholesaler.com
> 
> Then in traffic router, the following zone files will be generated:
> 
> # ls /opt/traffic_router/var/auto-zones/
> test.ipcdn.mycompany.com.  test.wholesaler.com. ipcdn.mycompany.com.
> 
> # cat /opt/traffic_router/var/auto-zones/test.ipcdn.mycompany.com.
> test.ipcdn.mycompany.com.   3600IN  NS  
> crdc-router-1.ipcdn.mycompany.com.
> test.ipcdn.mycompany.com.   3600IN  NS  
> crdc-router-2.ipcdn.mycompany.com.
> test.ipcdn.mycompany.com.   86400   IN  SOA 
> crdc-router-1.ipcdn.mycompany.com. traffic_ops.test.ipcdn.mycompany.com. 
> 2017062108 28800 7200 604800 30
> crdc-edge-se-1.test.ipcdn.mycompany.com.3600IN  A   
> 192.168.122.112
> crdc-edge-se-2.test.ipcdn.mycompany.com.3600IN  A   
> 192.168.122.172
> crdc-edge-se-3.test.ipcdn.mycompany.com.3600IN  A   
> 192.168.122.110
> crdc-edge-se-4.test.ipcdn.mycompany.com.3600IN  A   
> 192.168.122.252
> crdc-router-1.test.ipcdn.mycompany.com. 3600IN  A   
> 192.168.122.175
> crdc-router-2.test.ipcdn.mycompany.com. 3600IN  A   192.168.122.96
> tr.test.ipcdn.mycompany.com.3600IN  A   192.168.122.96
> tr.test.ipcdn.mycompany.com.3600IN  A   192.168.122.175
> 
> # cat /opt/traffic_router/var/auto-zones/test.wholesaler.com.
> test.wholesaler.com.3600IN  A   192.168.122.96
> test.wholesaler.com.3600IN  A   192.168.122.175
> test.wholesaler.com.3600IN  NS  
> crdc-router-1.ipcdn.mycompany.com.
> test.wholesaler.com.3600IN  NS  
> crdc-router-2.ipcdn.mycompany.com.
> test.wholesaler.com.86400   IN  SOA 
> crdc-router-1.ipcdn.mycompany.com. traffic_ops.test.wholesaler.com. 
> 2017062108 28800 7200 604800 30
> crdc-edge-se-1.test.wholesaler.com. 3600IN  A   
> 192.168.122.112
> crdc-edge-se-2.test.wholesaler.com. 3600IN  A   
> 192.168.122.172
> crdc-edge-se-3.test.wholesaler.com. 3600IN  A   
> 192.168.122.110
> crdc-edge-se-4.test.wholesaler.com. 3600IN  A   
> 192.168.122.252
> crdc-router-1.test.wholesaler.com.  3600IN  A   
> 192.168.122.175
> crdc-router-2.test.wholesaler.com.  3600IN  A   192.168.122.96
> 
> # cat /opt/traffic_router/var/auto-zones/ipcdn.mycompany.com.
> ipcdn.mycompany.com.3600IN  NS  
> crdc-router-1.ipcdn.mycompany.com.
> ipcdn.mycompany.com.3600IN  NS  
> crdc-router-2.ipcdn.mycompany.com.
> ipcdn.mycompany.com.86400   IN  SOA 
> crdc-router-1.ipcdn.mycompany.com. traffic_ops.ipcdn.mycompany.com. 
> 2017062108 28800 7200 604800 30
> crdc-router-1.ipcdn.mycompany.com.  3600IN  A   192.168.122.175
> crdc-router-2.ipcdn.mycompany.com.  3600IN  A   192.168.122.96
> 
> 
> And in one of the cache server, the following content will be generated for 
> remap.config:
> 
> # cat /opt/trafficserver/etc/trafficserver/remap.config
> # DO NOT EDIT - Generated for crdc-edge-se-2 by Traffic Ops 
> (https://localhost) on Tue Jun 13 05:59:22 UTC 2017
> map http://crdc-edge-se-2.test.ipcdn.mycompany.com/ 
> http://192.168.122.156/ 

Re: [VOTE] Release Apache Traffic Control 2.0.0-incubating (RC6)

[Eric Friedrich (efriedri)]

Correction to the original vote deadline: 
Instead of Friday June 21, 2017, I meant that the vote will run until Wednesday 
June 21, 2017 (72 hrs)

Please get your votes in within the next day or so- we need at least one more 
+1 vote to pass the release on to the IPMC

Thanks,
Eric



> On Jun 19, 2017, at 5:07 PM, Dave Neuman <neu...@apache.org> wrote:
> 
> +1
> Things I tested (Centos 7.2 VM):
> -- verified md5
> -- verified gpg sig
> -- built all RPMs from source tar
> -- Traffic Stats installs, starts, and runs
> -- Traffic Router installs, starts, serves digs and curls
> -- Traffic Ops can be installed and started.
> 
> I found a couple issues that should be addressed before the next release:
> [1] Traffic Stats install can't create traffic_stats user.
> 
> Thanks,
> Dave
> 
> [1] https://issues.apache.org/jira/browse/TC-397
> 
> 
> On Mon, Jun 19, 2017 at 12:39 PM, Dan Kirkwood <dang...@gmail.com> wrote:
> 
>> +1
>> 
>> I checked:
>> - git tag verified
>> - gpg sig is good
>> - checksums match source tarball
>> - tarball has correct name and structure
>> - rpms build from tarball
>> - traffic_ops installation and postinstall on a clean Centos7 VM
>> - some basic traffic_ops UI functionality
>> - docs no longer mention build area to download rpms
>> 
>> One caveat -- gpg signature is not signed and so not in the web of
>> trust,  but according to
>> http://www.apache.org/dev/release-distribution.html#sigs-and-sums :
>> 
>> "Signing keys SHOULD be linked into a strong web of trust."
>> 
>> We should get Eric's key signed at the earliest opportunity,  but it's
>> not a requirement for the release.
>> 
>> On Fri, Jun 16, 2017 at 10:31 AM, Eric Friedrich (efriedri)
>> <efrie...@cisco.com> wrote:
>>> Hello All,
>>> 
>>> I've prepared the next candidate release for incubator-trafficcontrol
>> v2.0.0 (RC6)
>>> 
>>> Changes since 1.8.1:
>>> https://github.com/apache/incubator-trafficcontrol/
>> compare/RELEASE-1.8.1...RELEASE-2.0.0-RC6<https://
>> github.com/apache/incubator-trafficcontrol/compare/
>> RELEASE-1.8.1-RC0...RELEASE-2.0.0-RC5>
>>> 
>>> This corresponds to git:
>>> Hash: 85d14ebe2a4ac71236f86f70349481d2b3dc784b
>>> Tag: RELEASE-2.0.0-RC6
>>> 
>>> Which can be verified with the following:git tag -v RELEASE-2.0.0-RC6
>>> 
>>> My code signing key is available here:
>>> http://pgp.mit.edu/pks/lookup?op=get=0xF2200BAB9AB7BDD5
>>> 
>>> and here:
>>> https://dist.apache.org/repos/dist/dev/incubator/trafficcontrol/KEYS
>>> 
>>> Make sure you refresh from a key server to get all relevant signatures.
>>> 
>>> The source .tar.gz file, pgp signature (.asc signed with my key from
>>> above), and md5 and sha512 checksums are provided here:
>>> https://dist.apache.org/repos/dist/dev/incubator/
>> trafficcontrol/2.0.0/RC6
>>> 
>>> 
>>> Docs are available here: https://trafficcontrol.apache.org/docs/2.0.x/
>>> 
>>> 
>>> The vote will remain open until Friday, June 21, 2017.
>>> 
>>> 
>>> Thanks,
>>> Eric Friedrich
>> 

[VOTE] Release Apache Traffic Control 2.0.0-incubating (RC6)

[Eric Friedrich (efriedri)]

Hello All,

I've prepared the next candidate release for incubator-trafficcontrol v2.0.0 
(RC6)

Changes since 1.8.1:
https://github.com/apache/incubator-trafficcontrol/compare/RELEASE-1.8.1...RELEASE-2.0.0-RC6

This corresponds to git:
Hash: 85d14ebe2a4ac71236f86f70349481d2b3dc784b
Tag: RELEASE-2.0.0-RC6

Which can be verified with the following:git tag -v RELEASE-2.0.0-RC6

My code signing key is available here:
http://pgp.mit.edu/pks/lookup?op=get=0xF2200BAB9AB7BDD5

and here:
https://dist.apache.org/repos/dist/dev/incubator/trafficcontrol/KEYS

Make sure you refresh from a key server to get all relevant signatures.

The source .tar.gz file, pgp signature (.asc signed with my key from
above), and md5 and sha512 checksums are provided here:
https://dist.apache.org/repos/dist/dev/incubator/trafficcontrol/2.0.0/RC6


Docs are available here: https://trafficcontrol.apache.org/docs/2.0.x/


The vote will remain open until Friday, June 21, 2017.


Thanks,
Eric Friedrich

[VOTE] Release Apache Traffic Control 2.0.0-incubating (RC5)

[Eric Friedrich (efriedri)]

Hello All,

I've prepared the next candidate release for incubator-trafficcontrol v2.0.0 
(RC5)

Changes since 1.8.0:
https://github.com/apache/incubator-trafficcontrol/compare/RELEASE-1.8.1-RC0...RELEASE-2.0.0-RC5

This corresponds to git:
Hash: b64848e38a09ee372c9a21a3652ea210962ccffa
Tag: RELEASE-2.0.0-RC5

Which can be verified with the following:git tag -v RELEASE-2.0.0-RC5

My code signing key is available here:
http://pgp.mit.edu/pks/lookup?op=get=0xF2200BAB9AB7BDD5

and here:
https://dist.apache.org/repos/dist/dev/incubator/trafficcontrol/KEYS

Make sure you refresh from a key server to get all relevant signatures.

The source .tar.gz file, pgp signature (.asc signed with my key from
above), and md5 and sha512 checksums are provided here:
https://dist.apache.org/repos/dist/dev/incubator/trafficcontrol/2.0.0/RC5

Docs are available here: https://trafficcontrol.apache.org/docs/2.0.x/


The vote will remain open until Thursday, June 20, 2017.

This RC fixes some comments from the IPMC made about 1.8.1 and also some minor 
bugs in the post install.

Thanks,
Eric Friedrich

Re: Traffic Ops Default Profile Management

[Eric Friedrich (efriedri)]

Can we include an ORIGIN profile?

Does TRAFFIC_PORTAL need a profile too?  (I’ve never set it up)

—Eric

> On Jun 12, 2017, at 1:14 PM, Dewayne Richardson  wrote:
> 
> Based upon the discussion around how we manage the default profiles Traffic
> Ops profiles for 2.0 and 2.1, I propose we add instructions that under the
> TC Website documentation "Downloads"
> http://trafficcontrol.incubator.apache.org/downloads/index.html.  The
> profiles will be in the format that allows Traffic Ops to import and export
> them, either through the UI or the API and that is the only format we
> manage.
> 
> The naming convention for the profiles (as well as the corresponding file
> names) are up for discussion as well.  The following naming convention
> (unless a better one comes up) will be:
> 
> *Default Profiles*
> 
> EDGE_.json
> MID_.json
> TRAFFIC_MONITOR.json
> TRAFFIC_ROUTER.json
> TRAFFIC_STATS.json
> TRAFFIC_VAULT.json
> 
> I didn't include version numbers on the TC components because I didn't know
> if that was overkill.  I will move the ball forward with this ASAP if we
> have enough consensus.
> 
> Thank you,
> 
> -Dewayne

Re: [VOTE] Release Apache Traffic Control 2.0.0-incubating (RC4)

[Eric Friedrich (efriedri)]

The time period for the vote is up, but I’m looking for a few more votes. 

So far we have 2 +1s, but I don’t think anyone has actually installed the RC 
yet. 

Thanks,
Eric
> On Jun 8, 2017, at 12:40 PM, Dan Kirkwood <dang...@gmail.com> wrote:
> 
> you're right -- it shouldn't..   thanks!
> 
> On Thu, Jun 8, 2017 at 10:39 AM, Hank Beatty <hbea...@apache.org> wrote:
>> I just realized it is our cdn.conf that doesn't have the secrets in it not
>> the default. Still don't think it should error. I'll fix it and submit the
>> PR.
>> 
>> On 06/08/2017 11:30 AM, Dan Kirkwood wrote:
>>> 
>>> not that I'm aware of...  please do file an issue for it,  though and
>>> attach that to the PR.
>>> 
>>> thanks!  Dan
>>> 
>>> On Thu, Jun 8, 2017 at 9:18 AM, Hank Beatty <hbea...@apache.org> wrote:
>>>> 
>>>> postinstall might still be broken:
>>>> 
>>>> Password for database server admin:
>>>> Re-Enter Password for database server admin:
>>>> Download Maxmind Database? [yes]:
>>>> Download Maxmind Database?: yes
>>>> ===/opt/traffic_ops/app/conf/cdn.conf===
>>>> Generate a new secret? [yes]:
>>>> Generate a new secret?: yes
>>>> Number of secrets to keep? [10]:
>>>> Number of secrets to keep?: 10
>>>> Can't use an undefined value as an ARRAY reference at
>>>> /opt/traffic_ops/install/bin/_postinstall line 211,  line 13.
>>>> 
>>>> And I've already started looking into it...
>>>> 
>>>> It is trying to load the secrets from the cdn.conf file. The initial
>>>> cdn.conf file doesn't have any secrets. I can fix the _postinstall to
>>>> correct this and do a PR.
>>>> 
>>>> Unless someone else has already fixed it or it needs to be done a
>>>> different
>>>> way.
>>>> 
>>>> Thanks,
>>>> Hank
>>>> 
>>>> On 06/05/2017 02:16 PM, Eric Friedrich (efriedri) wrote:
>>>>> 
>>>>> 
>>>>> Hello All,
>>>>> 
>>>>> I've prepared the next candidate release for incubator-trafficcontrol
>>>>> v2.0.0 (RC4)
>>>>> 
>>>>> Changes since 1.8.0:
>>>>> 
>>>>> 
>>>>> https://github.com/apache/incubator-trafficcontrol/compare/RELEASE-1.8.1-RC0...RELEASE-2.0.0-RC4<https://github.com/apache/incubator-trafficcontrol/compare/RELEASE-1.8.1-RC0...RELEASE-2.0.0-RC3>
>>>>> 
>>>>> This corresponds to git:
>>>>> Hash: 795ea3adf2003dd27523b6b9ff4691f23d41ce30
>>>>> Tag: RELEASE-2.0.0-RC4
>>>>> 
>>>>> Which can be verified with the following:git tag -v
>>>>> RELEASE-2.0.0-RC4
>>>>> 
>>>>> My code signing key is available here:
>>>>> http://pgp.mit.edu/pks/lookup?op=get=0xF2200BAB9AB7BDD5
>>>>> 
>>>>> and here:
>>>>> https://dist.apache.org/repos/dist/dev/incubator/trafficcontrol/KEYS
>>>>> 
>>>>> Make sure you refresh from a key server to get all relevant signatures.
>>>>> 
>>>>> The source .tar.gz file, pgp signature (.asc signed with my key from
>>>>> above), and md5 and sha512 checksums are provided here:
>>>>> 
>>>>> https://dist.apache.org/repos/dist/dev/incubator/trafficcontrol/2.0.0/RC4
>>>>> 
>>>>> 
>>>>> The vote will remain open until Thursday, June 8, 2017.
>>>>> 
>>>>> This RC fixes some packaging issues in RC2 and RC3, there are no other
>>>>> changes. The git tag hash is the same, but due to changes in the tarball
>>>>> the
>>>>> release signatures HAVE changes.
>>>>> 
>>>>> Thanks,
>>>>> Eric Friedrich
>>>>> 
>>>> 
>> 

RE: Update on RFC7871 - Client Subnet in DNS Support

[Eric Friedrich (efriedri)]

I should also add this will be disabled by default. Even if enabled it will not 
change TR behavior unless the resolver includes the ECS field 

From: Steve Malenfant [smalenf...@gmail.com]
Sent: Tuesday, June 6, 2017 5:03 PM
To: dev@trafficcontrol.incubator.apache.org
Subject: Re: Update on RFC7871 - Client Subnet in DNS Support

I've been able to start some discussions internally and looks like it would
be considered for certain domains (Like the one served by Traffic Routers).
We definitevely have cache groups which are not associated with some local
caching DNS.

On Tue, Jun 6, 2017 at 2:50 PM, Eric Friedrich (efriedri) <
efrie...@cisco.com> wrote:

> Thanks Ryan-
>We will certainly have the option to disable use of this if you don’t
> want to use it.
>
> This is useful feedback though and I’ll be sure to push on the caching
> aspects of our specific use case more.
>
> —Eric
>
> > On Jun 6, 2017, at 2:43 PM, Durfey, Ryan <ryan_dur...@comcast.com>
> wrote:
> >
> > FYI, I followed up with one of our staff informally about implementation
> of RFC7871 and got similar feedback.  They ran a test and found a lot of
> impacts at scale.
> >
> > Feedback:
> > We found that EDNS0 Client Subnet, as currently standardized and
> implemented, doesn't work well with our distribution/scale.  Could actually
> increase the number of upstream queries and memory required to store
> responses by 1000x.
> >
> > Ryan DurfeyM | 303-524-5099
> >
> >
> > From: Steve Malenfant <smalenf...@gmail.com>
> > Reply-To: "dev@trafficcontrol.incubator.apache.org" <
> dev@trafficcontrol.incubator.apache.org>
> > Date: Monday, June 5, 2017 at 6:15 AM
> > To: "dev@trafficcontrol.incubator.apache.org" <
> dev@trafficcontrol.incubator.apache.org>
> > Subject: Re: Update on RFC7871 - Client Subnet in DNS Support
> >
> > +1.  We have a hard time convincing our DNS team to enable EDNS0 in our
> > caching DNS (because of the caching implications). I do see great
> benefits
> > to localize DNS DS.
> >
> > On Sun, Jun 4, 2017 at 5:29 AM, Ori Finkelman <o...@qwilt.com o...@qwilt.com>> wrote:
> >
> > +1
> > This will also work nicely with the effort to have a localized TR DNS
> > selection.
> > If Traffic Routers are selected the same way a traffic server is
> selected,
> > by the client proximity from CZF or Geo, then using EDNS0 client-subnet
> > will enable this choice to be more accurate based on real client subnet
> > rather than DSN resolver.
> >
> >
> >
> > On Fri, Jun 2, 2017 at 10:46 PM, David Neuman <david.neuma...@gmail.com<
> mailto:david.neuma...@gmail.com>>
> > wrote:
> >
> >> +1, excited to see this one come through.
> >>
> >> On Fri, Jun 2, 2017 at 12:15 PM, Eric Friedrich (efriedri) <
> >> efrie...@cisco.com<mailto:efrie...@cisco.com>> wrote:
> >>
> >>> We are planning to add support for RFC7871 to Traffic Router. Here is a
> >>> brief description of the feature. Comments appreciated!
> >>>
> >>> Background
> >>> Clients do not make DNS requests directly to TR. Typically TR requests
> >>> come from DNS resolvers within the infrastructure. Today, Cache Group
> >>> selection for DNS Delivery Services is based on the IP address of the
> > DNS
> >>> resolver making the request to TR. RFC7871 includes the client subnet
> > in
> >> an
> >>> EDNS0 option within the DNS query. When the ECS OPT is present (and the
> >>> feature is enabled via a TR parameter), Traffic Router will use this IP
> >> in
> >>> place of the source IP of the DNS packet (the IP of the resolver). This
> >> IP
> >>> will be used in the CZF and maxmind lookups.
> >>>
> >>> Requirements
> >>>
> >>>  1.  If DNS query includes ECS option in the Optional Record, Traffic
> >>> Router will use the IP address included in the ECS option as the client
> >>> address for Cache Group Selection
> >>>  2.  If there are multiple ECS options in the Optional Record, the one
> >>> with the longest IP prefix - i.e. the ECS option with largest Source
> > Net
> >>> Mask will be used
> >>>  3.  If DNS query includes ECS Option, then DNS response from Traffic
> >>> Router will also include the ECS Option. In the response the Scope Net
> >> Mask
> >>> is set to be same as the Source Net Mask. This is required by

[CANCEL] [VOTE] Release Apache Traffic Control 2.0.0-incubating (RC3)

[Eric Friedrich (efriedri)]

This vote was short lived due to more problems in the build. 

This will be fixed in RC4. 

—Eric

> On Jun 5, 2017, at 1:17 PM, Dan Kirkwood <dang...@gmail.com> wrote:
> 
> unfortunately,  have to -1.
> 
> Building traffic_ops from the tarball using docker-compose gives me this 
> error:
> traffic_ops_build_1  | The build area has been initialized.
> traffic_ops_build_1  | Building the rpm.
> traffic_ops_build_1  | error: line 26: Tag takes single token
> only: Release:  Not in git repository and no BUILD_NUMBER
> present -- aborting!.el7
> 
> The tarball is missing the BUILD_NUMBER file which gives it the
> - identifier for the rpm.   Using the
> docker-compose build method or `./build/build.sh source` will create
> the tarball with that file created and included.
> 
> -dan
> 
> 
> On Mon, Jun 5, 2017 at 9:53 AM, Jeff Elsloo <els...@apache.org> wrote:
>> +1 on this, signature and hashes validate.
>> --
>> Thanks,
>> Jeff
>> 
>> 
>> On Mon, Jun 5, 2017 at 7:03 AM, Eric Friedrich (efriedri)
>> <efrie...@cisco.com> wrote:
>>> Hello All,
>>> 
>>> I've prepared the next candidate release for incubator-trafficcontrol 
>>> v2.0.0 (RC3)
>>> 
>>> Changes since 1.8.0:
>>> https://github.com/apache/incubator-trafficcontrol/compare/RELEASE-1.8.1-RC0...RELEASE-2.0.0-RC3
>>> 
>>> This corresponds to git:
>>> Hash: 795ea3adf2003dd27523b6b9ff4691f23d41ce30
>>> Tag: RELEASE-2.0.0-RC3
>>> 
>>> Which can be verified with the following:git tag -v RELEASE-2.0.0-RC3
>>> 
>>> My code signing key is available here:
>>> http://pgp.mit.edu/pks/lookup?op=get=0xF2200BAB9AB7BDD5
>>> 
>>> and here:
>>> https://dist.apache.org/repos/dist/dev/incubator/trafficcontrol/KEYS
>>> 
>>> Make sure you refresh from a key server to get all relevant signatures.
>>> 
>>> The source .tar.gz file, pgp signature (.asc signed with my key from
>>> above), and md5 and sha512 checksums are provided here:
>>> https://dist.apache.org/repos/dist/dev/incubator/trafficcontrol/2.0.0/RC3<https://dist.apache.org/repos/dist/dev/incubator/trafficcontrol/2.0.0/RC2>
>>> 
>>> 
>>> The vote will remain open until Thursday, June 8, 2017.
>>> 
>>> This RC fixes some packaging issues in RC2, there are no other changes. The 
>>> git tag hash is the same, but due to changes in the tarball the release 
>>> signatures HAVE changes.
>>> 
>>> Thanks,
>>> Eric Friedrich

[VOTE] Release Apache Traffic Control 2.0.0-incubating (RC3)

[Eric Friedrich (efriedri)]

Hello All,

I've prepared the next candidate release for incubator-trafficcontrol v2.0.0 
(RC3)

Changes since 1.8.0:
https://github.com/apache/incubator-trafficcontrol/compare/RELEASE-1.8.1-RC0...RELEASE-2.0.0-RC3

This corresponds to git:
Hash: 795ea3adf2003dd27523b6b9ff4691f23d41ce30
Tag: RELEASE-2.0.0-RC3

Which can be verified with the following:git tag -v RELEASE-2.0.0-RC3

My code signing key is available here:
http://pgp.mit.edu/pks/lookup?op=get=0xF2200BAB9AB7BDD5

and here:
https://dist.apache.org/repos/dist/dev/incubator/trafficcontrol/KEYS

Make sure you refresh from a key server to get all relevant signatures.

The source .tar.gz file, pgp signature (.asc signed with my key from
above), and md5 and sha512 checksums are provided here:
https://dist.apache.org/repos/dist/dev/incubator/trafficcontrol/2.0.0/RC3


The vote will remain open until Thursday, June 8, 2017.

This RC fixes some packaging issues in RC2, there are no other changes. The git 
tag hash is the same, but due to changes in the tarball the release signatures 
HAVE changes.

Thanks,
Eric Friedrich

[CANCEL][VOTE] Release Apache Traffic Control 2.0.0-incubating (RC2)

[Eric Friedrich (efriedri)]

This vote is canceled due to issues in the release packaging. 

I will be opening a new vote for RC3 shortly

—Eric
> On Jun 2, 2017, at 1:32 PM, Dave Neuman <neu...@apache.org> wrote:
> 
> Unfortunately I have to be -1 due to
> https://issues.apache.org/jira/browse/TC-365.
> The source tar ball does not extract a proper directory structure.
> 
> Thanks,
> Dave
> 
> On Thu, Jun 1, 2017 at 9:31 AM, Jeff Elsloo <jeff.els...@gmail.com> wrote:
> 
>> I'm +1 on this. Thanks for creating the RC Eric!
>> --
>> Thanks,
>> Jeff
>> 
>> 
>> On Thu, Jun 1, 2017 at 9:30 AM, Eric Friedrich (efriedri)
>> <efrie...@cisco.com> wrote:
>>> Hello All,
>>> 
>>> I've prepared the next candidate release for incubator-trafficcontrol
>> v2.0.0 (RC2)
>>> 
>>> Changes since 1.8.0:
>>> https://github.com/apache/incubator-trafficcontrol/
>> compare/RELEASE-1.8.1-RC0...RELEASE-2.0.0-RC2
>>> 
>>> This corresponds to git:
>>> Hash: 795ea3adf2003dd27523b6b9ff4691f23d41ce30
>>> Tag: RELEASE-2.0.0-RC2
>>> 
>>> Which can be verified with the following:git tag -v RELEASE-2.0.0-RC2
>>> 
>>> My code signing key is available here:
>>> http://pgp.mit.edu/pks/lookup?op=get=0xF2200BAB9AB7BDD5
>>> 
>>> and here:
>>> https://dist.apache.org/repos/dist/dev/incubator/trafficcontrol/KEYS
>>> 
>>> Make sure you refresh from a key server to get all relevant signatures.
>>> 
>>> The source .tar.gz file, pgp signature (.asc signed with my key from
>>> above), and md5 and sha512 checksums are provided here:
>>> https://dist.apache.org/repos/dist/dev/incubator/
>> trafficcontrol/2.0.0/RC2
>>> 
>>> 
>>> The vote will remain open until Tuesday, June 6, 2017.
>>> 
>>> Thanks!
>>> Eric Friedrich
>>> 
>> 

Update on RFC7871 - Client Subnet in DNS Support

[Eric Friedrich (efriedri)]

We are planning to add support for RFC7871 to Traffic Router. Here is a brief 
description of the feature. Comments appreciated!

Background
 Clients do not make DNS requests directly to TR. Typically TR requests come 
from DNS resolvers within the infrastructure. Today, Cache Group selection for 
DNS Delivery Services is based on the IP address of the DNS resolver making the 
request to TR. RFC7871 includes the client subnet in an EDNS0 option within the 
DNS query. When the ECS OPT is present (and the feature is enabled via a TR 
parameter), Traffic Router will use this IP in place of the source IP of the 
DNS packet (the IP of the resolver). This IP will be used in the CZF and 
maxmind lookups.

Requirements

  1.  If DNS query includes ECS option in the Optional Record, Traffic Router 
will use the IP address included in the ECS option as the client address for 
Cache Group Selection
  2.  If there are multiple ECS options in the Optional Record, the one with 
the longest IP prefix - i.e. the ECS option with largest Source Net Mask will 
be used
  3.  If DNS query includes ECS Option, then DNS response from Traffic Router 
will also include the ECS Option. In the response the Scope Net Mask is set to 
be same as the Source Net Mask. This is required by RFC 7871 for DNS caching to 
work properly.
  4.  It is assumed that customers/operators will ensure that Source Net Mask 
for a subnet specified in the ECS is at greater than or equal to the net mask 
for the corresponding subnet entry in the CZF file. e.g. if ECS Option has 
192.168.10.0/8, then 192.168.0.0/16 in CZF will work, but 192.168.10./28 will 
not work.
  5.  Add a TR parameter to disable use of ECS field even when present

Design

To support this feature new logic will be added to NameServer.query() function. 
The new logic will be responsible for parsing ECS option in the OptionalRecord 
of the incoming DNS Request, and retrieving the Client IP address and the 
associated Source Net Mask (Scope Net Mask must be 0 in the incoming Request 
per RFC 7871). Please note that the underlying DNS library xbill/dnsjava 
already has support for parsing the ECS Options. These functions from the 
library will be leveraged.

  1.  Message.getOPT().getOptions(EDNSOption.Code.CLIENT_SUBNET) will return 
list of ClientSubnetOption for the incoming Request (Message)
  2.  ClientSubnetOption has public methods to retrieve netmask and Client IP 
address:  getSourceNetmask(), getAddress()

If ECS option is present, then the IP address retrieved from the ECS option, 
and will be passed as the Client IP address to the Traffic Router (through 
getZone call) for CZF/geo lookup

If ECS option is present, then ClientSubnetOption will be created and included 
in the DNS response. In the Response the Scope Net Mask of the 
ClientSubnetOption is set as the Source Net Mask

Testing
  We’ll test against all of the various CZF, National, Regional, VPN Blocking 
features and will do our best to check with DNSSEC

—Eric

[VOTE] Release Apache Traffic Control 2.0.0-incubating (RC2)

[Eric Friedrich (efriedri)]

Hello All,

I've prepared the next candidate release for incubator-trafficcontrol v2.0.0 
(RC2)

Changes since 1.8.0:
https://github.com/apache/incubator-trafficcontrol/compare/RELEASE-1.8.1-RC0...RELEASE-2.0.0-RC2

This corresponds to git:
 Hash: 795ea3adf2003dd27523b6b9ff4691f23d41ce30
 Tag: RELEASE-2.0.0-RC2

Which can be verified with the following:git tag -v RELEASE-2.0.0-RC2

My code signing key is available here:
http://pgp.mit.edu/pks/lookup?op=get=0xF2200BAB9AB7BDD5

and here:
https://dist.apache.org/repos/dist/dev/incubator/trafficcontrol/KEYS

Make sure you refresh from a key server to get all relevant signatures.

The source .tar.gz file, pgp signature (.asc signed with my key from
above), and md5 and sha512 checksums are provided here:
https://dist.apache.org/repos/dist/dev/incubator/trafficcontrol/2.0.0/RC2


The vote will remain open until Tuesday, June 6, 2017.

Thanks!
Eric Friedrich

Re: LDAP Access

[Eric Friedrich (efriedri)]

Is there an option to entirely block someone from even basic TO access despite 
authenticating with LDAP?


> On May 31, 2017, at 11:24 AM, Robert Butts  wrote:
> 
> We have a PR https://github.com/apache/incubator-trafficcontrol/pull/627 to
> change Traffic Ops to only allow LDAP users _not_ in the Traffic Ops
> database to view non-sensitive information, like graphs and total CDN
> bandwidth.
> 
> To be clear, users will still be able to authenticate with LDAP, as long as
> their user name is in the database. This only prevents access for LDAP
> users whose name is not in the database.
> 
> If you have LDAP-only users who need access, you can simply add their user
> name to the Traffic Ops database to allow continued access. They don't even
> need a password, simply inserting the username is sufficient.
> 
> LDAP is a security risk, especially for large organizations. Allowing all
> non-CDN personnel in the organization full information access, even
> read-only, means an attacker has only to compromise a single account in the
> organization, and they can see the full list of CDN server IPs and FDQNs,
> as well as the specific ATS and CentOS versions, in order to take advantage
> of known exploits against those versions.
> 
> Does anyone have any issues with that? Is anyone using LDAP without
> usernames in the database, who needs continued access? We just want to make
> sure we're not breaking anyone before we merge this, and figure out a
> solution if we are. Thanks,

Re: Getting CZF data from BGP?

[Eric Friedrich (efriedri)]

Hey Jan- 

Are you looking to build a static CZF based off of BGP inputs? 

Are you looking for something that will listen to BGP and create a “real-time 
CZF” that responds to routing/CG changes? 

Or something else?

> On May 30, 2017, at 1:00 PM, Jan van Doorn  wrote:
> 
> Hi,
> 
> We are considering building something to get the CZF data directly from the
> network, possibly using BGP and open source routing software. Before we get
> going, wanted to ask if anyone has any experience with this, or tools
> already built to do this?
> 
> Rgds,
> JvD

Re: Duplicate TO API routes

[Eric Friedrich (efriedri)]

Thumbs up on removal
> On May 19, 2017, at 1:58 PM, Jeremy Mitchell  wrote:
> 
> @Eric_Friedrich - any concerns from you regarding removal of these
> duplicate routes? Here they are to summarize:
> 
> remove GET /api/$version/deliveryservices/list in favor of GET
> /api/$version/deliveryservices
> remove GET /api/$version/deliveryservices/:id/get in favor of GET
> /api/$version/deliveryservices/:id
> remove POST /api/$version/deliveryservices/create in favor of POST
> /api/$version/deliveryservices
> remove PUT /api/$version/deliveryservices/:id/update in favor of PUT
> /api/$version/deliveryservices/:id
> remove GET /api/$version/cachegroups/list in favor of GET
> /api/$version/cachegroups
> remove POST /api/$version/cachegroups/create in favor of POST
> /api/$version/cachegroups
> remove PUT /api/$version/cachegroups/:id/update in favor of PUT
> /api/$version/cachegroups/:id
> remove DELETE /api/$version/cachegroups/:id/delete in favor of DELETE
> /api/$version/cachegroups/:id
> remove POST /api/$version/servers/create in favor of POST
> /api/$version/servers
> remove PUT /api/$version/servers/:id/update in favor of PUT
> /api/$version/servers/:id
> 
> On Fri, Apr 28, 2017 at 10:32 AM, Dewayne Richardson 
> wrote:
> 
>> And as long as the "duplicate" routes point to the same Perl module, I'm ok
>> as well
>> 
>> On Tue, Apr 18, 2017 at 4:27 PM, Dave Neuman  wrote:
>> 
>>> I assume you did a check of the client to make sure it wasn't using those
>>> routes?
>>> Other than that, no concerns here.
>>> 
>>> On Tue, Apr 18, 2017 at 12:37 PM, Jeremy Mitchell <
>> mitchell...@apache.org>
>>> wrote:
>>> 
 As we move towards a new authorization model on the TO API which
>> includes
 roles, capabilities and tenancy, it's important to remove any cruft
>> from
 the API to ease this new implementation. I've put some effort into
 organizing the TO routes file (
 https://github.com/apache/incubator-trafficcontrol/blob/
 master/traffic_ops/app/lib/TrafficOpsRoutes.pm#L373)
 but I'd also like to remove any duplicate routes.
 
 We have a few duplicates mainly in the area of deliveryservice,
>> servers,
 and cachegroups. This jira outlines the duplicates and suggests which
 routes to remove - https://issues.apache.org/jira/browse/TC-82
 
 If these routes are being used, they would not be removed until 2.1
>>> giving
 everyone time to use the standard ones.
 
 Concerns?
 
 Jeremy
 
>>> 
>> 

Re: [VOTE] Move Traffic Control to full GitHub

[Eric Friedrich (efriedri)]

I don’t think we can begin moving stuff over until we have write access to the 
Github repo. It looks like the Issues and Wiki tabs are disabled currently

—Eric

On May 19, 2017, at 11:34 AM, Durfey, Ryan 
> wrote:

Unless there are any objections:

  *   Ashish and I will begin migrating the conf wiki pages to github on Monday.
 *   We also need to add a release notation to the page titles so we know 
to which release the discussions apply.
 *   As we copy things over, we will move the existing conf wiki pages 
under an “Archive” page so we can recover anything missed.
 *   Not sure how easy it will be to pull the bottom of page and in-line 
comments in some of the conf wiki pages.  We may just copy over some of the 
more critical or unresolved items and abandon the rest in the conf wiki archive.
  *   Once that is finished we will also work to move the bugs and features 
over and setup labels.


Ryan DurfeyM | 303-524-5099


From: Jason Tucker >
Reply-To: 
"dev@trafficcontrol.incubator.apache.org"
 
>,
 "jasonwtuc...@gmail.com" 
>
Date: Friday, May 19, 2017 at 9:18 AM
To: 
"us...@trafficcontrol.incubator.apache.org"
 
>
Cc: 
"dev@trafficcontrol.incubator.apache.org"
 
>
Subject: Re: [VOTE] Move Traffic Control to full GitHub

+1

On Fri, May 19, 2017 at 1:41 PM, Dremin, Sergey 
>
wrote:

+1

On 5/18/17, 2:32 PM, "Jan van Doorn" 
> wrote:

 In
 https://lists.apache.org/thread.html/5bdb9b073343f49c1d5b85147eb9d2
60bf7ad15d61384929993c7e1d@%3Cdev.trafficcontrol.apache.org%3E
 Dave
 mentioned that we can now move to "full" GitHub. Some more information
in
 that thread if you are not familiar. I would like to call an official
vote
 on that.

 This vote will be open for at least 72 hours.

  [ ] +1 Move Traffic Control to use full GitHub
  [ ]  0 No opinion
  [ ] -1 Do not Move Traffic Control to use full GitHub because...

 Rgds,
 JvD

Traffic Control SLA

[Eric Friedrich (efriedri)]

Hi All-
  We had some discussion around what level of support we want to offer to users 
of our software. 


  I'd like to suggest a policy that for all releases we will fix all security 
issues and we will fix regressions at discretion of the release manager. 

  There was also discussion about how long we should support a release for. 

  Options included:
  - Doing only 1-2 LTS releases per year 
  - Having many smaller point releases with 1 or 2 LTS releases per year. 

  We should also define the support timeline of an LTS release to say 
supporting the two most recent LTS at all times. 

  Thoughts?

Re: [VOTE] Adding a CHANGELOG.md file

[Eric Friedrich (efriedri)]

The Influx Changelog contains sections for Features vs Bugfixes. 

I personally find that useful and would like to keep something similar in our 
Changelog. Hopefully with JIRA or Issues we can indicate with a label or tag 
which section this item should go into (defaulting to Bugfix to not be a pain)

—Eric


> On May 17, 2017, at 2:34 PM, Dave Neuman <neu...@apache.org> wrote:
> 
> yeah, something that creates an automated Changelog.MD file is better than
> putting it in a github release.  If I understood the talks I went to
> earlier, Apache does not like it when you create a "release" before you
> actually vote on release candidates, etc.
> 
> I am +1 with an automated release, once we move to "full" github.
> 
> On Wed, May 17, 2017 at 1:22 PM, Dan Kirkwood <dang...@gmail.com> wrote:
> 
>> yeah -- what Hank said...
>> 
>> On Wed, May 17, 2017 at 11:17 AM, Hank Beatty <hbea...@apache.org> wrote:
>>> -1 for a manual changelog - doing a compare between branches/commits in
>>> github is relatively simple.
>>> 
>>> +1 for a scripted changelog -
>>> https://github.com/skywinder/github-changelog-generator - There is even
>> a
>>> list of alternatives:
>>> https://github.com/skywinder/Github-Changelog-Generator/
>> wiki/Alternatives
>>> 
>>> On 05/17/2017 12:52 PM, Phil Sorber wrote:
>>>> 
>>>> Here is a link to an example script generated CHANGES file from Jira:
>>>> 
>>>> https://raw.githubusercontent.com/apache/trafficserver/6.0.x/CHANGES
>>>> 
>>>> On Wed, May 17, 2017 at 10:48 AM Phil Sorber <sor...@apache.org> wrote:
>>>> 
>>>>> The script can be updated to do Jira. ATS used a Jira version before
>> they
>>>>> went to github. You can also separate out easily. In fact, we did it
>> more
>>>>> easily with Jira than with github, since those categories are mutually
>>>>> exclusive in Jira and labels in github are not. You could also have a
>>>>> developer run the script regularly, or have CI do it.
>>>>> 
>>>>> To Eric's comment, if you can make that indication in Jira/GitHub then
>>>>> you
>>>>> can transition that to the script. For example, a "Changelog" label in
>>>>> github that would mean to have it included.
>>>>> 
>>>>> On Wed, May 17, 2017 at 10:37 AM Eric Friedrich (efriedri) <
>>>>> efrie...@cisco.com> wrote:
>>>>> 
>>>>>> What about a compromise where developer chooses whether or not a
>>>>>> feature/important fix is worth mentioning in the release notes. This
>>>>>> would
>>>>>> be at feature granularity not individual commit.
>>>>>> 
>>>>>> Then at release build time, a script gathers from JIRA/Github API all
>>>>>> fixes that were committed in that release and checks that into repo.
>>>>>> 
>>>>>> —Eric
>>>>>> 
>>>>>>> On May 17, 2017, at 12:18 PM, Phil Sorber <sor...@apache.org> wrote:
>>>>>>> 
>>>>>>> Don't we have a script that can generate this? ATS had this for a
>> long
>>>>>> 
>>>>>> time
>>>>>>> 
>>>>>>> and it became a huge hassle. It caused merge conflicts all the time,
>>>>>> 
>>>>>> that
>>>>>>> 
>>>>>>> while easy to address, were a huge nuisance. It also ended up out of
>>>>>> 
>>>>>> date
>>>>>>> 
>>>>>>> often.
>>>>>>> 
>>>>>>> On Wed, May 17, 2017 at 10:11 AM Gelinas, Derek <
>>>>>> 
>>>>>> derek_geli...@comcast.com>
>>>>>>> 
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> +1 for sure. It'll also give us a way to scan the notes and see what
>>>>>> 
>>>>>> needs
>>>>>>>> 
>>>>>>>> documenting and what doesn't yet have it.
>>>>>>>> 
>>>>>>>>> On May 17, 2017, at 11:44 AM, Dave Neuman <neu...@apache.org>
>> wrote:
>>>>>>>>> 
>>>>>>>>> Hey All,
>>>>>>>>> One thing we discussed at the meetup was the addition of a
>>>>>> 
>>>>>> CHANGELOG.md
>>>>>>>>> 
>>>>>>>>> file to the project.   This file will contain changes that are made
>>>>>>>>> to
>>>>>>>> 
>>>>>>>> the
>>>>>>>>> 
>>>>>>>>> project including bug fixes and new features. (e.g.
>>>>>>>>> https://github.com/influxdata/influxdb/blob/master/CHANGELOG.md).
>>>>>>>> 
>>>>>>>> Adding
>>>>>>>>> 
>>>>>>>>> this file means that we will now require each PR to contain an
>> update
>>>>>> 
>>>>>> to
>>>>>>>>> 
>>>>>>>>> the CHANGELOG.md file, and our documentation will need to be
>> updated
>>>>>>>>> accordingly.
>>>>>>>>> I thought it would be good to open a vote for adding this file, and
>>>>>> 
>>>>>> if it
>>>>>>>>> 
>>>>>>>>> passes, I will update the documentation and add a CHANGELOG.md
>> file.
>>>>>>>>> 
>>>>>>>>> Thanks,
>>>>>>>>> Dave
>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>>> 
>>>> 
>>> 
>> 

Re: [VOTE] Adding a CHANGELOG.md file

[Eric Friedrich (efriedri)]

What about a compromise where developer chooses whether or not a 
feature/important fix is worth mentioning in the release notes. This would be 
at feature granularity not individual commit. 

Then at release build time, a script gathers from JIRA/Github API all fixes 
that were committed in that release and checks that into repo. 

—Eric

> On May 17, 2017, at 12:18 PM, Phil Sorber  wrote:
> 
> Don't we have a script that can generate this? ATS had this for a long time
> and it became a huge hassle. It caused merge conflicts all the time, that
> while easy to address, were a huge nuisance. It also ended up out of date
> often.
> 
> On Wed, May 17, 2017 at 10:11 AM Gelinas, Derek 
> wrote:
> 
>> +1 for sure. It'll also give us a way to scan the notes and see what needs
>> documenting and what doesn't yet have it.
>> 
>>> On May 17, 2017, at 11:44 AM, Dave Neuman  wrote:
>>> 
>>> Hey All,
>>> One thing we discussed at the meetup was the addition of a CHANGELOG.md
>>> file to the project.   This file will contain changes that are made to
>> the
>>> project including bug fixes and new features. (e.g.
>>> https://github.com/influxdata/influxdb/blob/master/CHANGELOG.md).
>> Adding
>>> this file means that we will now require each PR to contain an update to
>>> the CHANGELOG.md file, and our documentation will need to be updated
>>> accordingly.
>>> I thought it would be good to open a vote for adding this file, and if it
>>> passes, I will update the documentation and add a CHANGELOG.md file.
>>> 
>>> Thanks,
>>> Dave
>> 

Re: 2.0 release?

[Eric Friedrich (efriedri)]

There is an issue that Jeff E will take care of later this week that is a 
showstopper. 

Also Dan was going to look into seeing if we needed more post install/postgres 
fixes back ported to 2.0.x so it could be useful. 

—Eric



> On May 17, 2017, at 11:02 AM, Dave Neuman <neu...@apache.org> wrote:
> 
> Hey All,
> We had some great discussion about the 2.0 release at the summit, I was
> wondering if anyone had a summary of that discussion and a list of what's
> left to do that could be added to this thread?  I think we discussed that
> we were going to take another look at 2.0 and see if it is a viable release
> that we should move forward with, is that everyone else's understanding as
> well?
> Does anyone know of any showstopper issues that still exist?
> 
> Thanks,
> Dave
> 
> On Mon, Apr 10, 2017 at 9:19 PM, Eric Friedrich (efriedri) <
> efrie...@cisco.com> wrote:
> 
>> Update:
>>  - License issue has been fixed- Thanks Rob!
>>  - Postinstall script is broken, Jeff and Dan are looking at it.
>> 
>> Once post install is fixed, I will cut an RC
>> 
>> —Eric
>> 
>> 
>> 
>>> On Apr 6, 2017, at 2:35 PM, Dewayne Richardson <dewr...@gmail.com>
>> wrote:
>>> 
>>> +1
>>> 
>>> On Thu, Apr 6, 2017 at 9:43 AM, Robert Butts <robert.o.bu...@gmail.com>
>>> wrote:
>>> 
>>>> +1
>>>> I didn't realize it was new.
>>>> 
>>>> On Thu, Apr 6, 2017 at 8:49 AM, Dan Kirkwood <dang...@gmail.com> wrote:
>>>> 
>>>>> +1
>>>>> 
>>>>> On Thu, Apr 6, 2017 at 7:43 AM, David Neuman <david.neuma...@gmail.com
>>> 
>>>>> wrote:
>>>>>> Since the Cookie Jar functionality is new to 2.0 and 2.0 is not yet
>>>>>> released, why don't we just remove the `ResumeSession` method all
>>>>> together
>>>>>> and eliminate the dependency?  Otherwise we are deprecating something
>>>>> that
>>>>>> we never formally released.
>>>>>> 
>>>>>> On Tue, Apr 4, 2017 at 2:30 PM, Robert Butts <
>> robert.o.bu...@gmail.com
>>>>> 
>>>>>> wrote:
>>>>>> 
>>>>>>> Regarding `TC-119: traffic_ops/client dependency license issue`:
>>>>>>> 
>>>>>>> It looks like the persistent cookie jar is only needed by Traffic Ops
>>>>>>> Client `ResumeSession(toURL string, insecure bool) (*Session,
>> error)`.
>>>>>>> Nothing in Traffic Control uses `ResumeSession`, and I doubt anyone
>>>>> else is
>>>>>>> using it. Because it returns an error, and persisted cookies have
>>>>>>> lifetimes, any current users already must handle errors from
>> persisted
>>>>>>> cookies being expired. Thus, we can change it to always return an
>>>> error
>>>>>>> with only degraded performance (and not much, login is cheap),
>> without
>>>>> loss
>>>>>>> of functionality.
>>>>>>> 
>>>>>>> To fix TC-119, I propose we document `ResumeSession` as deprecated,
>>>> and
>>>>>>> change it to always return an error, which lets us remove the
>>>>> dependency,
>>>>>>> without the development cost of writing our own persistent cookie
>>>> store
>>>>>>> that no one is using.
>>>>>>> 
>>>>>>> Any objections?
>>>>>>> 
>>>>>>> 
>>>>>>> On Tue, Apr 4, 2017 at 1:35 PM, Jeremy Mitchell <
>>>> mitchell...@gmail.com>
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> These all got fixed and backported to 2.0:
>>>>>>>> 
>>>>>>>> TC-203: Mojo doesn’t set cachable headers on public files”
>>>>>>>> TC-190: TTL type mismatch in CrConfig
>>>>>>>> TC-189: ssl_multicert.config too slow
>>>>>>>> 
>>>>>>>> So Jan and Dave just need to close the issues.
>>>>>>>> 
>>>>>>>> On Tue, Apr 4, 2017 at 12:22 PM, Jeffrey Martin <
>>>>> martin.jef...@gmail.com
>>>>>>>> 
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> Hi Eric,
>>>>>&g

Re: Moving Traffic Control the "full" github

[Eric Friedrich (efriedri)]

I am all for one less tool to use. Also I think it will lower bar to bringing 
more people into our project if they don’t have to sign up for the ASF JIRA 
separately. 

—Eric

> On May 17, 2017, at 10:57 AM, Mark Torluemke  wrote:
> 
> Also +1. Part of the move from github.com/Comcast to ASF JIRA included a
> 'scrub', so the move to github.com/apache can likely be scripted.
> 
> On Wed, May 17, 2017 at 8:55 AM, Robert Butts 
> wrote:
> 
>> +1
>> 
>> IMO Github issues, wiki, etc are much, much easier to use than Atlassian
>> tools.
>> 
>> On Wed, May 17, 2017 at 8:51 AM, Dave Neuman  wrote:
>> 
>>> While at ApaceCon, a few of us attended a talk about navigating the
>>> incubator where we were informed that "full" Github is now available for
>>> podlings.  This gives us the ability to use github issues, to use github
>>> wiki, to assign PRs, to add tags to PRs, and the "merge PR" button among
>>> other things.  It sounds like the process would take our repo down for a
>>> short period - minutes not hours - but the URL shouldn't change.  I know
>> we
>>> just got all of our issues moved to Jira, but we would need to move them
>>> over to github as well.
>>> 
>>> Since the apache way is to have a discussion before a vote, I thought I
>>> would start the discussion on this topic now and if we feel like this is
>>> worth pursuing, we start a vote.  Sothoughts?
>>> 
>>> 
>>> Thanks,
>>> Dave
>>> 
>> 

Re: API GW route configuration

[Eric Friedrich (efriedri)]

Will it actually be the API Gateway that creates the auth token? If not, it 
would be useful to show the presence of an auth service that actually services 
the auth requests to the gateway.

I don’t know if we got to consensus on validating the JWT token. I think API GW 
should certainly validate, but I don’t think it is sufficient in all cases. An 
endpoint in the underlying “API Namespace” might need to perform an additional 
token auth. We should make sure to allow that option (but not make it mandatory)

—Eric


> On May 12, 2017, at 3:23 PM, Chris Lemmons <alfic...@gmail.com> wrote:
> 
> Awesome diagram. Worth at least 1k words, I think. :)
> 
> I've got two observations:
> 1. In "checks capabilities found in JWT", that should include "validate JWT
> with auth server", which can be as simple as checking a timestamp.
> 2. I may have missed it, but how is the route from the Gateway to TO
> secured?
> 
> On Fri, May 12, 2017 at 8:41 AM David Neuman <david.neuma...@gmail.com>
> wrote:
> 
>> +1 on keeping in on the mailing list
>> 
>> On Fri, May 12, 2017 at 7:52 AM, Eric Friedrich (efriedri) <
>> efrie...@cisco.com> wrote:
>> 
>>> Can we please keep the discussion on the dev@ list? Apache rules and
>> all.
>>> 
>>> “If it didn’t happen on the email list, it didn’t happen”
>>> 
>>> 
>>> On 5/12/17, 9:50 AM, "Amir Yeshurun" <am...@qwilt.com> wrote:
>>> 
>>>Moving discussion to the wiki page. Commented on Jeremy's notes
>>> 
>>>On Fri, May 12, 2017 at 7:41 AM Shmulik Asafi <shmul...@qwilt.com>
>>> wrote:
>>> 
>>>> Regarding sharing signing key with other services raised by Eric -
>>> notice
>>>> jwt support asymmetric keys. I.e. only the auth server has the
>>> private key
>>>> and other services have a public key.
>>>> 
>>>> Also, there's another solution for "global invalidation" besides
>>> switching
>>>> keys, and that is setting a threshold for token issue date.
>> Whenever
>>>> there's an attack or whatever and we want to invalidate all tokens
>>> we just
>>>> need to set the threshold for 'now' thus forcing all users to issue
>>> a new
>>>> token.
>>>> I think this is more reasonable and scalable than switching keys.
>>>> 
>>>> On 12 May 2017 05:15, "Jeremy Mitchell" <mitchell...@gmail.com>
>>> wrote:
>>>> 
>>>>> Here was the image I was trying to attach:
>>>>> 
>>>>> https://cwiki.apache.org/confluence/display/TC/API+Gateway
>>>>> 
>>>>> Jeremy
>>>>> 
>>>>> On Thu, May 11, 2017 at 2:14 PM, Amir Yeshurun <am...@qwilt.com>
>>> wrote:
>>>>> 
>>>>>> Hi Jeremy,
>>>>>> Note that attachments seems to be stripped off on this list and
>>> the
>>>> image
>>>>>> is unavailable.
>>>>>> 
>>>>>> Your assumptions are correct. We need to figure out the easiest
>>>> topology
>>>>>> for UI routes to bypass the GW. Please reattach the picture so
>>> we can
>>>> get
>>>>>> more specific.
>>>>>> 
>>>>>> Thanks
>>>>>> /amiry
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Thu, May 11, 2017, 20:06 Jeremy Mitchell <
>>> mitchell...@gmail.com>
>>>>> wrote:
>>>>>> 
>>>>>>> What is of utmost importance to me is the ability to ease
>> into
>>> this.
>>>> We
>>>>>>> have a TO UI right now that needs to be unaffected by the API
>>> gateway
>>>>> in
>>>>>> my
>>>>>>> opinion. Granted the old UI might go away at some point but
>>> until
>>>> that
>>>>>> time
>>>>>>> it needs to function as-is.
>>>>>>> 
>>>>>>> To me, the simplest approach is to key off request URL.
>>> anything that
>>>>>>> starts with /api gets api gateway treatment, the rest passes
>> on
>>>>>>> thru...Here's a fancy picture to communicate what I
>> envision...
>>>>>>> 
>>>>>>> [image: Inline image 1]
>>>>>>> 
>>>>>>> I'm assuming all r

Re: API GW route configuration

[Eric Friedrich (efriedri)]

> > > >> malicious,
> > > >> > or
> > > >> > > just cause mayhem. The longer he can keep control, the better
> for
> > > him.
> > > >> > >
> > > >> > > So this attacker uses the local box to sniff the token off the
> > > >> network.
> > > >> > If
> > > >> > > the communication with the Gateway is encrypted, he might have
> to
> > do
> > > >> some
> > > >> > > ARP poisoning or something else to trick a host into talking to
> > the
> > > >> local
> > > >> > > box instead. (Properly implemented TLS also migates this 
angle.)
> > He
> > > >> knows
> > > >> > > that as soon as he starts his nefarious deed, alarms are going
> to
> > go
> > > >> off,
> > > >> > > so he also uses this local box to DOS the Auth Server. It's a
> lot
> > > >> easier
> > > >> > to
> > > >> > > take a box down from the outside than to actually gain control.
> > > >> > >
> > > >> > > If the Gateway "fails open" when it can't contact the Auth
> server,
> > > the
> > > >> > > attacker remains in control. If it "fails closed", the attacker
> > has
> > > to
> > > >> > > actually compromise the auth server (which is harder) to remain
> in
> > > >> > control.
> > > >> > >
> > > >> > > > Do we block all API calls if the auth service is temporarily
> > down
> > > >> > (being
> > > >> > > upgraded, container restarting, etc…)?
> > > >> > >
> > > >> > > Yes, I think we have to. Authentication is integral to reliable
> > > >> > operation.
> > > >> > >
> > > >> > > We've been talking in some fairly wild hypotheticals, though. 
Is
> > > >> there a
> > > >> > > specific auth service you're envisioning?
> > > >> > >
> > > >> > > On Wed, May 10, 2017 at 12:50 AM Shmulik Asafi <
> > shmul...@qwilt.com>
> > > >> > wrote:
> > > >> > >
> > > >> > > > Regarding the communication issue Chris raised - there is 
more
> > > than
> > > >> one
> > > >> > > > possible pattern to this, e.g.:
> > > >> > > >
> > > >> > > >- Blacklisted tokens can be communicated via a pub-sub
> > > mechanism
> > > >> > > >- The API gateway can poll for the latest list of tokens
> at a
> > > >> > regular
> > > >> > > >interval (which can be very short ~1sec, much shorter than
> > the
> > > >> time
> > > >> > it
> > > >> > > >takes devops to detect and react to malign tokens)
> > > >> > > >
> > > >> > > > Regarding hitting the blacklist datastore - this only sounds
> > > >> similar to
> > > >> > > > hitting to auth database; but the simplicity of a blacklist
> > > function
> > > >> > > allows
> > > >> > > > you to employ more efficient datastores, e.g. Redis or just a
> > > >> hashmap
> > > >> > in
> > > >> > > > the API gateway process memory.
> > > >> > > >
> > > >> > > > Regarding maliciously delayed message or such - I don't fully
> > > >> > understand
> > > >> > > > the point; if an attacker has such capabilities she can 
simply
> > > >> > > > prevent/delay devop users from updating the auth database
> itself
> > > >> thus
> > > >> > > > enabling the attack.
> > > >> > > >
> > > >> > > >
> > > >> > > > On Wed, May 10, 2017 at 4:25 AM, Eric Friedrich (efriedri) &l

Re: Backup Cache Group Selection

[Eric Friedrich (efriedri)]

backupList is not planned because the coordinates approach was sufficient

—Eric

On May 9, 2017, at 6:57 AM, Ori Finkelman 
<o...@qwilt.com<mailto:o...@qwilt.com>> wrote:

Hi again,
I understand now that the "backupList" feature does not exist yet.
What is the status of this feature ? is it planned ?

Thanks,
Ori

On Mon, May 8, 2017 at 4:18 PM, Ori Finkelman 
<o...@qwilt.com<mailto:o...@qwilt.com>> wrote:

Hi,
Following up on this one, it seems that both czf attributes described in
this thread, the "coordinates" and the "backupList" are not documented in
the official docs in
http://trafficcontrol.incubator.apache.org/docs/latest/admin/traffic_ops_
using.html#the-coverage-zone-file-and-asn-table

Is there a plan to update the documentation ? should I open a JIRA for it ?

Thanks,
Ori

On Thu, Mar 30, 2017 at 8:45 PM, Jeff Elsloo <jeff.els...@gmail.com>
wrote:

Yes, that's correct.
--
Thanks,
Jeff


On Thu, Mar 30, 2017 at 11:20 AM, Eric Friedrich (efriedri)
<efrie...@cisco.com> wrote:
Thanks Jeff-
 Could I think of it as the following? Echoing back to be sure I
understand...

If there is a lat/long for a cache group in the CZF file, any client
hit to that CG should use the CZF lat/long as it client’s lat/long instead
of using geolocation.

For the purposes of finding closest cache group, the client’s location
(from CZF as above or from Geolocation provider) will be compared against
the location of the cache’s as configuration in Traffic Op’s CG record?

—Eric


On Mar 30, 2017, at 1:07 PM, Jeff Elsloo <jeff.els...@gmail.com>
wrote:

It could now be considered the "average" of the location of the
clients within that section of the CZF, however, it should be noted
that the addition of the geo coordinates to the CZF is relatively new.
Previously we never had the ability to specify lat/long on those
cachegroups, and we solely relied on those specified in edgeLocations,
meaning that the matches had to be 1:1. Adding the coordinates allowed
us to cover edge cases and miss scenarios and stick to the CZF
whenever possible. Previously when we had no coordinates, and we had a
hit in the CZF but not corresponding hit within the edgeLocations
(health, assignments, etc), we would fall back to the Geolocation
provider.
--
Thanks,
Jeff


On Thu, Mar 30, 2017 at 5:29 AM, John Shen (weifensh)
<weife...@cisco.com> wrote:
Thanks Jeff and Oren for the discussion. I agree now that lat/long
from CZF is the “average” location of clients, and lat/long from Ops is the
location of a certain Cache Group. So it appears to be reasonable to use
them as source and dest to calculate the distance.

Thanks,
John


On 30/03/2017, 6:55 PM, "Oren Shemesh" <or...@qwilt.com> wrote:

  Jeff, having read this conversation more than once, I believe
there is a
  misunderstanding regarding the ability to provide coordinates for
cache
  groups both in the CZF and in the TO DB.

  Here is what I believe is a description which may help
understanding the
  current behaviour:

  The coordinates specified in the CZF for a cache group are not
supposed to
  be the exactly same as the coordinates in the TO DB for the same
cache
  group.
  This is because they do not represent the location of the caches
of the
  group.
  They represent the (average) location of clients found in the
subnets
  specified for this cache group.

  This, I believe, explains both the behaviour of the code (Why the
  coordinates from the CZF are used for the source, but the
coordinates from
  the TO DB are used for the various candidate cache groups), and
the fact
  that there is a 'duplication'.

  Is this description true ?



  On Wed, Mar 29, 2017 at 7:02 PM, Jeff Elsloo <els...@apache.org>
wrote:

The cachegroup settings in the Traffic Ops GUI end up in the
`edgeLocations` section of the CRConfig. This is the source of truth
for where caches are deployed, logically or physically. We do not
provide a means to generate a CZF in Traffic Ops, so it's up to the
end user to craft one to match what is in Traffic Ops.

There are several cases that need to be accounted for where a hit in
the CZF does match what's in `edgeLocations`, but cannot be served
there due to cache health, delivery service health, or delivery
service assignments. The other edge case is a hit where no
`edgeLocation` exists, which again, must be accounted for. Presumably
we have higher fidelity data in our CZF than we would in our
Geolocation provider and we should use it whenever possible.

Think about this: what if you use the same CZF for two configured
CDNs, but one of the two CDNs only has caches deployed to 50% of the
cache groups defined in the CZF. Would we want to use the Geolocation
provider in the event that our source address matches a cachegroup
that does not have any assigned caches? We would ideally have as much
granularity as possible in the CZF, then use that to inform the
dec

Re: API GW route configuration

[Eric Friedrich (efriedri)]

From a higher level- what is purpose of the API Gateway?  It seems like there 
may have been some previous discussions about API Gateway. Are there any notes 
or description that I can catch up on?

How will it be deployed? (Is it a standalone service or something that runs 
inside the experimental Traffic Ops)?

Is this new component required or optional? 

—Eric



> On May 7, 2017, at 8:28 PM, Jan van Doorn  wrote:
> 
> I looked into this a year or so ago, and I couldn't make nginx or http do
> what we need.
> 
> We can still have the services check the auth as well after the proxy auth,
> and make things better than today, where we have the same problem that if
> the TO mojo app is compromised, everything is compromised.
> 
> If we always route to TO, we don't untangle the mess of being dependent on
> the monolithic TO for everything. Many services today, and more in the
> future really just need a check to see if the user is authorized, and
> nothing more.
> 
> On Sun, May 7, 2017 at 11:55 AM Robert Butts 
> wrote:
> 
>> What are the advantages of these config files, over an existing reverse
>> proxy, like Nginx or httpd? It's just as much work as configuring and
>> deploying an existing product, but more code we have to write and maintain.
>> I'm having trouble seeing the advantage.
>> 
>> -1 on auth rules as a part of the proxy. Making a proxy care about auth
>> violates the Single Responsibility Principle, and further, is a security
>> risk. It creates unnecessary attack surface. If your proxy app or server is
>> compromised, the entire framework is now compromised. An attacker could
>> simply rewrite the proxy config to make all routes no-auth.
>> 
>> The simple alternative is for the proxy to always route to TO, and TO
>> checks the token against the auth service (which may also be proxied), and
>> redirects unauthorized requests to a login endpoint (which may also be
>> proxied).
>> 
>> The TO service (and any other service that requires auth) MUST hit the
>> database (or the auth service, which itself hits the database) to verify
>> valid tokens' users still have the permissions they did when the token was
>> created. Otherwise, it's impossible to revoke tokens, e.g. if an employee
>> quits, or an attacker gains a token, or a user changes their password.
>> 
>> 
>> On Sun, May 7, 2017 at 4:35 AM, Amir Yeshurun  wrote:
>> 
>>> Seems that attachments are stripped on this list. Examples pasted below
>>> 
>>> *rules.json*
>>> [
>>>{ "host": "localhost", "path": "/login",   "forward":
>>> "localhost:9004", "scheme": "https", "auth": false },
>>>{ "host": "localhost", "path": "/api/1.2/innovation/", "forward":
>>> "localhost:8004", "scheme": "http",  "auth": true, "routes-file":
>>> "innovation.json" },
>>>{ "host": "localhost", "path": "/api/1.2/","forward":
>>> "localhost:3000", "scheme": "http",  "auth": true, "routes-file":
>>> "traffic-ops-routes.json" },
>>>{ "host": "localhost", "path": "/internal/api/1.2/",   "forward":
>>> "localhost:3000", "scheme": "http",  "auth": true, "routes-file":
>>> "internal-routes.json" }
>>> ]
>>> 
>>> *traffic-ops-routes.json (partial)*
>>> .
>>> .
>>> .
>>>{ "match": "/cdns/health","auth": { "GET":
>>> ["cdn-health-read"] }},
>>>{ "match": "/cdns/capacity",  "auth": { "GET":
>>> ["cdn-health-read"] }},
>>>{ "match": "/cdns/usage/overview","auth": { "GET":
>>> ["cdn-stats-read"] }},
>>>{ "match": "/cdns/name/dnsseckeys/generate",  "auth": { "GET":
>>> ["cdn-security-keys-read"] }},
>>>{ "match": "/cdns/name/[^\/]+/?", "auth": { "GET":
>>> ["cdn-read"] }},
>>>{ "match": "/cdns/name/[^\/]+/sslkeys",   "auth": { "GET":
>>> ["cdn-security-keys-read"] }},
>>>{ "match": "/cdns/name/[^\/]+/dnsseckeys","auth": { "GET":
>>> ["cdn-security-keys-read"] }},
>>>{ "match": "/cdns/name/[^\/]+/dnsseckeys/delete", "auth": { "GET":
>>> ["cdn-security-keys-write"] }},
>>>{ "match": "/cdns/[^\/]+/queue_update",   "auth": { "POST":
>>> ["queue-updates-write"] }},
>>>{ "match": "/cdns/[^\/]+/snapshot",   "auth": { "PUT":
>>> ["cdn-config-snapshot-write"] }},
>>>{ "match": "/cdns/[^\/]+/health", "auth": { "GET":
>>> ["cdn-health-read"] }},
>>>{ "match": "/cdns/[^\/]+/?",  "auth": { "GET":
>>> ["cdn-read"], "PUT":  ["cdn-write"], "PATCH": ["cdn-write"], "DELETE":
>>> ["cdn-write"] }},
>>>{ "match": "/cdns",   "auth": { "GET":
>>> ["cdn-read"], "POST": ["cdn-write"] }},
>>> 
>>> .
>>> .
>>> .
>>> 
>>> 
>>> On Sun, May 7, 2017 at 12:39 PM Amir Yeshurun  wrote:
>>> 
 Attached please find examples for forwarding rules file (rules.json)
>> and
 the authorization rules file (traffic-ops-routes.json)
 
 
 On Sun, May 7, 2017 at 

Re: Delivery-Service Configuration Versioning

[Eric Friedrich (efriedri)]

Thanks Nir-
Comments inline
> On May 1, 2017, at 1:12 PM, Nir Sopher  wrote:
> 
> Dear all,
> 
> Planning the efforts toward "self-service", we are considering
> "delivery-service configuration versioning" (DSCV) as one of our next
> steps.
> In a very high level, by DSCV we refer to the ability to hold multiple
> configuration versions/revisions per delivery-service, and actively choose
> which version should be deployed.
> 
> A significant portion of the value we would like to bring when working
> toward "self-service" can be achieved using the initial step of
> configuration versioning:
> 
>   1. As the amount of delivery-services handled by TC is increasing,
>   denying the "non dev-ops" user from changing delivery-services
>   configuration by himself, and require a "dev-ops" user to actually make the
>   changes in the DB, put an increasing load on the operations team.
>   Via DSCV the operator may allow the users to really push configurations
>   into the DB, as it separates the provisioning phase from the deployment.
>   Once commited, the CDN's "dev-ops" user is able to examine the changes
>   and choose which version should be deployed, subject to the operator's
>   acceptance policy.
EF> How do we get from DSCV to the ultimate self-service goals where the CDN 
operator is no longer in the critical path for deploying DS changes?

>   2. DSCV brings improved auditing and troubleshooting capabilities, which
>   is important for supporting TC deployment growth, as well as allow users to
>   be more independent.
>   It allows to investigate issues using versions associated log records,
>   as well as the data in the DB itself: Examining the delivery-service
>   versions, their meta data (e.g. "deployed dates") as well as use tools for
>   versions comparisons.
>   3. DSCV allows a simple delivery service configuration rollback, which
>   provides a quick remedy for configuration errors issues.
> 
> Moreover, we suggest to allow the deployment of multiple versions of the
> same delivery service simultaneously, on the same caches. Doing so, and
> allowing the operator to orchestrate the usage of the different
> versions (for example, via "steering"), the below become available:
EF> This feature will extend to both caches and TR, right? Lots of DS-specific 
policy is evaluated by the TR.

> 
>   1. Manual testing of a new delivery-service configuration, via dedicated
>   URL or using request headers.
>   2. Staging / Canary testing of new versions, applying them only for a
>   specific content path, or filtering base on source IP.
>   3. Gradual transition between the different configuration versions.
>   4. Configuration versions A/B testing (assuming the reporting/stats also
>   becomes "version aware").
>   5. Immediate (no CRON wait, cr-config change only) delivery-service
>   version"switch", and specifically immediate rollback capabilities.
EF> Does #5 imply that it will be the TR choosing between the versions of DS’ 
deployed on the caches? How will this modify the format of requests to 
TrafficServer?
This will have impacts to log analysis, HTTPS, DNSSEC, and many other aspects 
of the system. 

> 
> Note that, engineering wise, one may consider DSCV as a building block for
> other "self-service" steps. It allows the system to identify what
> configuration is deployed on which server, as well as allows the servers to
> identify configuration changes with DS granularity. Therefore, it can help
> to decouple the individual delivery services deployment as well as reduce
> the load derived from the caches update process.
> We would greatly appreciate community input on the subject.
> 
> Many thanks,
> Nir

Re: Access Control - Limiting Roles / Capabilities Tenant Admins can Assign to Users

[Eric Friedrich (efriedri)]

Could we further differentiate the user creation capabilities to:
- Create CDN Admin user
- Create CDN Ops user
- Create CDN Viewer user
- Create Tenant Admin user
- Create Tenant Ops user
- Create Tenant Viewer user

Then only the CDN-Admin role would have the capability to create a cdn admin 
user. Would be good to see the capabilities assigned at a granularity below API 
endpoint in this case.

As for creation of new roles, I like #2 and #3. Users should not be able to 
level-up anyone’s capabilities beyond their own. Further, capabilities are 
enforced by code, so we should not allow creation of new capabilities by API

- - Eric



On May 3, 2017, at 9:44 AM, Durfey, Ryan 
> wrote:

Moving this active debate into the mailing list.
-Jeremy makes a good point.  We need a method for making restricting roles and 
capabilities for lower tier staff that can create new users.  Jeremy has 
suggested a point system or a hierarchy.  I think either of these would work if 
applied correctly.   I am open to any approach that works.

My thoughts:
1. We need to limit which users can build new roles from capabilities or new 
capabilities from APIs.  This could be limited to a master role like “CDN 
Admin”.  Otherwise other admins could circumvent the system by matching APIs to 
lower tier roles.
2. Another simple approach may be to only allow non-CDN Admins to assign roles 
to users which they have access.  Basically you can’t give anyone more rights 
than you have.
3. Perhaps with this approach we allow non-CDN Admins to build roles from 
existing capabilities to which they have access, but not create capabilities 
from APIs.  Then they can build new roles and assign any capabilities or roles 
to which they already have access.



From: Jeremy Mitchell

I like this model of a user has a role which has capabilities which map to API 
endpoints, however, there seems to be one flaw or at least one unaccounted for 
use case.
Let's look at the roles listed above:

  *   CDN-Admin
  *   CDN-Ops
  *   CDN-Viewer
  *   Tenant-Admin
  *   Tenant-Ops
  *   Tenant-Viewer

Jeremy is a CDN-Admin which has the user-create capability (among others) so he 
creates Bob, a Tenant-Admin. Being a Tenant-Admin, Bob also has user-create so 
he creates Sally and he can give her ANY role so he decides to give Sally the 
CDN-Admin rolewhoops, we don't want that...
Bob should be limited to creating users with role=Tenant-Admin (his role), 
Tenant-Ops or Tenant-Viewer...but how do we correlate one role with another? 
Currently, we have "privilege level" attached to a role. So I guess we could 
use that like so:

  *   CDN-Admin (100)
  *   CDN-Ops (50)
  *   CDN-Viewer (40)
  *   Tenant-Admin (30)
  *   Tenant-Ops (20)
  *   Tenant-Viewer (10)

Now, being a Tenant-Admin with the user-create capability, Bob can only create 
users where role.priv_level is 30 or below. I feel like this might be the 
easiest solution.
Thoughts?


...
Now, being a Tenant-Admin with the user-create capability, Bob can only create 
users where role.priv_level is 30 or below. I feel like this might be the 
easiest solution.
Or...you could make roles hierarchical the way that tenants are hierarchical
-CDN-Admin
--CDN-Ops
--CDN-Viewer
--Tenant-Admin
---Tenant-Ops
---Tenant-Viewer
And in this scenario, if you have the user-create capability you can create 
users with your role or a child of your role...
Thoughts?


Ryan Durfey
Sr. Product Manager - CDN | Comcast Technology Solutions
1899 Wynkoop Ste. 550 | Denver, CO 8020
M | 303-524-5099
ryan_dur...@comcast.com
24x7 CDN Support: 866-405-2993  or 
cdn_supp...@comcast.com

Re: Goose installer script

[Eric Friedrich (efriedri)]

I know licenses are the issue more than lines of code. 

I’m not a fan of submodules, but if licensing comes through, it might make 
sense to add goose and its deps as git submodules. 

One other thing to discuss :-)




> On May 1, 2017, at 2:04 PM, Dan Kirkwood <dang...@gmail.com> wrote:
> 
> It's not a trivial problem.  Yes -- we could include the source --
> goose itself is actually fairly small and MIT licensed.   Its
> dependencies have licenses that would need to be vetted,   so
> vendoring is also not trivial -- no matter how many lines of code are
> involved.
> 
> We could potentially compile goose and distribute within the rpm --
> I've also suggested that before.   Unfortunately,  we have a migration
> written in go,  which requires a go installation at run time.That
> means the requirement of a go installation is not avoidable without
> rewriting that as .sql.
> 
> There are alternatives we could use that may not suffer from the same issues.
> 
> I suggest we sit down together at the Summit and decide where to go with 
> this..
> 
> -dan
> 
> On Mon, May 1, 2017 at 10:37 AM, Robert Butts <robert.o.bu...@gmail.com> 
> wrote:
>> +1 on Vendoring. I don't see a difference if it's 375,000 lines or
>> 10,000,000 lines. What does it matter if it's 375k lines in someone else's
>> repo or our own? It does matter from a security standpoint. It means we're
>> now vulnerable if their repo is compromised. We shouldn't be pulling
>> _anything_ from the internet at install time.
>> 
>> Question for the Apache Gurus: If we include the Goose source, can we also
>> include a binary built from that source? I don't see a legal or
>> philosophical reason we shouldn't be able to, if we include a hash of the
>> binary and the LICENSE file. That lets us avoid requiring Go as a
>> dependency, which is difficult since few package managers have a modern Go
>> package. Goose is MIT,
>> https://www.apache.org/dev/licensing-howto.html#binary suggests we can, yes?
>> 
>> 
>> On Mon, May 1, 2017 at 8:31 AM, Dan Kirkwood <dang...@gmail.com> wrote:
>> 
>>> ughh.. I'd forgotten I'd done that in all this..
>>> 
>>> Again -- catch-22.
>>> 
>>> 
>>> 
>>> On Sun, Apr 30, 2017 at 10:20 PM, Mark Torluemke <mtorlue...@apache.org>
>>> wrote:
>>>> On Sun, Apr 30, 2017 at 7:05 PM, Gelinas, Derek <
>>> derek_geli...@comcast.com>
>>>> wrote:
>>>> 
>>>>> +1 on both of these.
>>>>> 
>>>>>> On Apr 30, 2017, at 8:50 PM, Eric Friedrich (efriedri) <
>>>>> efrie...@cisco.com> wrote:
>>>>>> 
>>>>>> Assuming we stick with goose, why not bundle goose source into the
>>>>> traffic ops RPM? This will pin the version for us and prevent users from
>>>>> needing to run go get
>>>>> 
>>>> 
>>>> Dan had put in a PR to add the Goose source:
>>>> https://github.com/apache/incubator-trafficcontrol/pull/157
>>>> 
>>>> We ended up closing it, as 375,000 lines felt a bit excessive...
>>>> 
>>>> 
>>>> 
>>>>>> 
>>>>>> We are allowed to bundle code with the MIT license into our releases.
>>>>>> 
>>>>>> As for the go installation, what about modifying the RPM spec file to
>>>>> list GoLang as a dependency of the traffic ops RPM?
>>>>>> 
>>>>>> —Eric
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>>> On Apr 28, 2017, at 4:46 PM, Dewayne Richardson <dewr...@gmail.com>
>>>>> wrote:
>>>>>>> 
>>>>>>> They are, but makes the tooling easier if we are all in Golang
>>>>>>> 
>>>>>>>> On Fri, Apr 28, 2017 at 1:44 PM, Dave Neuman <neu...@apache.org>
>>>>> wrote:
>>>>>>>> 
>>>>>>>> I don't see why re-writing the APIs in something like golang would
>>> mean
>>>>>>>> that we also need to re-write the database admin script.  I think
>>>>> those two
>>>>>>>> things are mutually exclusive, right?
>>>>>>>> 
>>>>>>>> On Fri, Apr 28, 2017 at 12:29 PM, Dewayne Richardson <
>>>>> dewr...@gmail.com>
>>>>>>>> wrote:
>>>>>>>> 
>&g

Re: Proposed changes to xml_id on a delivery service (from the API perspective)

[Eric Friedrich (efriedri)]

I’m also OK with making xml_id immutable.

I’d like us to look at having less restrictions in naming of DS Regexs rather 
than more. We have many use cases where the existing DS Regex is not sufficient 
and I think fixing it to xmlID would only worsen the problem.

A specific case: xml_id is unique per Traffic Ops, but ds regex is only unique 
per CDN.  Today I can have tr.sports.cdn1.com and 
tr.sports.cdn2.com.
Would I be able to do this if we fixed the DS names to xml_ids?

—Eric

On Apr 28, 2017, at 3:48 PM, Dave Neuman 
> wrote:

I believe we should allow users to create a host regex at create time if
they want to, otherwise we will create it for them.  This reduces issues
with DNSSEC keys, SSL keys, etc.
We also need to be cognizant that if the host regex is edited, the DNSSEC
keys and SSL keys will need to be updated as well.

On Fri, Apr 28, 2017 at 12:37 PM, Jeremy Mitchell 
>
wrote:

Great feedback. It sounds like people are OK with making xml_id immutable
on a delivery service... but not so much about making the host header regex
at position 0 immutable (the regex that requires this format -
.*\xml-id\..*).

Just to revisit:

You create a delivery service with xml_id=foo-bar and by default you get a
host header regex in this format at position 0 - .*\xml-id\..* so you end
up with URL=edge|tr.foo-bar.cdn.domain.com

Maybe you're happy with that and life is good.

Maybe you decide you want your URL to be edge|
tr.something-else.cdn.domain.com 
instead.so you need the ability to
edit the host header regex at position 0 to .*\something-else\..*

What about this? You can attach Static DNS entries to a delivery service.
Can Static DNS entries be used to accomplish this use case while leaving
.*\xml-id\..* alone? (Elsloo keyed me into that possibility)

Jeremy


On Fri, Apr 28, 2017 at 9:44 AM, Dave Neuman 
> wrote:

I don't think we should be forcing users into using their xml_id as the
host_regex used to create the delivery URL.  I think we can offer to
create
it for them, but we should also let users have the ability to define what
their regex needs to be when they create their delivery service.  We have
instances in our environment where the host regex does not match the
xml_id
a delivery service and I don't think we should take that functionality
away.
You are correct, changing an xml_id or a host_regex could have unintended
consequences in the system, but I agree with Ori that not letting users
change it can cause other problems.  I think we should allow users to
change it but maybe we need to figure out a way to only allow "super
special advanced" users to do it.

--Dave

On Thu, Apr 27, 2017 at 6:40 AM, Ori Finkelman 
> wrote:

Hi Jeremy,

I would like to refer to bullet 2 in your proposal.
Coupling between the xml_id and the host regexp, making the host regexp
immutable may create undesired limitations.
For example, if, for any reason, the content provider has to change the
host regexp, she will have to create a new DS. This limitation prevents
long term reports on DS volume, running long term analytics and other
operations that you may want to run over long periods of time.
In general, the host regexp is an attribute of the DS and therefore,
unlike
the id which should be immutable, I think it should be mutable, like
any
other attribute.

A side issue is that we would like to suggest promoting TC-55

in order to allow DS to
be
matched by a combination of host regexp and path regexp. TC-55
 requires it should be
allowed
to have the same host regexp for different DS, using the path regexp as
differentiator. This option has the benefit of using a single
certificate
for multiple DSs, it also reduces the complexity of delegating traffic
between CDNs, like the use cases of CDNi and Open Caching.
Unless I misunderstand the proposal, binding the first host regexp with
the
xml_id makes it impossible to have the same host for different DS, as
required by TC-55 .

Thanks,
Ori

On Thu, Apr 27, 2017 at 6:37 AM, Zhilin Huang (zhilhuan) <
zhilh...@cisco.com
wrote:

Hi Jeremy,

I like the idea that make xml_id and ds.regex at position 0
immutable.
Actually we have suffered some issues on HTTPs delivery services,
and I
had
ever created issue TC-187 with PR #360.

Besides ds.regex at position 0, the ds.regex at other position of
type
HOST_REGEXP is not working for HTTPs. I think the reason is SAN is no
supported in currently implementation in “generate_ssl_keys”. Do we
have
any plan to support multiple 

Re: Goose installer script

[Eric Friedrich (efriedri)]

Assuming we stick with goose, why not bundle goose source into the traffic ops 
RPM? This will pin the version for us and prevent users from needing to run go 
get

 We are allowed to bundle code with the MIT license into our releases. 

As for the go installation, what about modifying the RPM spec file to list 
GoLang as a dependency of the traffic ops RPM?

—Eric





> On Apr 28, 2017, at 4:46 PM, Dewayne Richardson <dewr...@gmail.com> wrote:
> 
> They are, but makes the tooling easier if we are all in Golang
> 
> On Fri, Apr 28, 2017 at 1:44 PM, Dave Neuman <neu...@apache.org> wrote:
> 
>> I don't see why re-writing the APIs in something like golang would mean
>> that we also need to re-write the database admin script.  I think those two
>> things are mutually exclusive, right?
>> 
>> On Fri, Apr 28, 2017 at 12:29 PM, Dewayne Richardson <dewr...@gmail.com>
>> wrote:
>> 
>>> I had that thought, as well as there are more recent versions like
>>> https://github.com/mattes/migrate.  The question becomes if we ever get
>>> around to rewriting TrafficOps APIs in golang, will the Perl version then
>>> become obsolete?
>>> 
>>> On Fri, Apr 28, 2017 at 11:58 AM, Dave Neuman <neu...@apache.org> wrote:
>>> 
>>>> Maybe it's time we take a look at what goose really buys us and
>> consider
>>>> writing our own database migration tool.  We already have admin.pl, it
>>>> could probably fit in with that?
>>>> 
>>>> On Fri, Apr 28, 2017 at 11:45 AM, Eric Friedrich (efriedri) <
>>>> efrie...@cisco.com> wrote:
>>>> 
>>>>> Hey Dew-
>>>>>  What calls this script?
>>>>> 
>>>>> If its called from the Traffic Ops Spec file, then this will cause
>> some
>>>>> pain for those of us that need to install without internet access.
>>>>> 
>>>>> —Eric
>>>>> 
>>>>>> On Apr 28, 2017, at 12:41 PM, Dewayne Richardson <
>> dewr...@gmail.com>
>>>>> wrote:
>>>>>> 
>>>>>> I'm working toward a more streamlined installation process for
>>> Traffic
>>>>> Ops
>>>>>> (internally) and publicly. Of course, the same hiccups that
>> everyone
>>>> else
>>>>>> runs into I am as well.  Installation of Golang (proper version)
>> and
>>>>>> installation of Goose.  Goose has been the most challenging for
>>> several
>>>>>> reasons.  The maintainer hasn't made any real changes since 2015,
>> and
>>>> has
>>>>>> not "branched" his code to allow for explicit version download.
>> Per
>>>> his
>>>>>> installation instructions "go get bitbucket.org/liamstask/goose/
>>>>> cmd/goose"
>>>>>> 
>>>>>> So I'm I'm proposing to write an installer script in bash to help
>>>>> automate
>>>>>> the Golang install as well as the Goose install.  My only concern
>> (as
>>>>> well
>>>>>> as most of yours) is "go get" will grab the latest, but since no
>> real
>>>>>> changes have happened I'm left with no other option.
>>>>>> 
>>>>>> Proposed:
>>>>>> 
>>>>>> /opt/traffic_ops/install/bin/install_goose.sh
>>>>>> 
>>>>>> - Install Golang (version 1.8.x)
>>>>>> - go get bitbucket.org/liamstask/goose/cmd/goose
>>>>>> 
>>>>>> Thoughts?
>>>>>> 
>>>>>> -Dew
>>>>> 
>>>>> 
>>>> 
>>> 
>> 

Re: Goose installer script

[Eric Friedrich (efriedri)]

Hey Dew-
  What calls this script? 

If its called from the Traffic Ops Spec file, then this will cause some pain 
for those of us that need to install without internet access.

—Eric

> On Apr 28, 2017, at 12:41 PM, Dewayne Richardson  wrote:
> 
> I'm working toward a more streamlined installation process for Traffic Ops
> (internally) and publicly. Of course, the same hiccups that everyone else
> runs into I am as well.  Installation of Golang (proper version) and
> installation of Goose.  Goose has been the most challenging for several
> reasons.  The maintainer hasn't made any real changes since 2015, and has
> not "branched" his code to allow for explicit version download.  Per his
> installation instructions "go get bitbucket.org/liamstask/goose/cmd/goose"
> 
> So I'm I'm proposing to write an installer script in bash to help automate
> the Golang install as well as the Goose install.  My only concern (as well
> as most of yours) is "go get" will grab the latest, but since no real
> changes have happened I'm left with no other option.
> 
> Proposed:
> 
> /opt/traffic_ops/install/bin/install_goose.sh
> 
> - Install Golang (version 1.8.x)
> - go get bitbucket.org/liamstask/goose/cmd/goose
> 
> Thoughts?
> 
> -Dew

Re: Proposal for CDN definition file based configuration management

[Eric Friedrich (efriedri)]

I think this sounds good Nir. 

Its not so much the size that is the main concern. Rather, people tend to have 
strong reactions to “its permanent, it will be there forever”. As long as we 
give some way to delete and preferably with a batch mode we should be all set.

—Eric

> On Apr 13, 2017, at 3:08 PM, Nir Sopher <n...@qwilt.com> wrote:
> 
> Hi Eric,
> 
> I thought to start with saving for each configuration the range of dates it
> was the "head" revision, and the range of dates it was deployed.
> This will allow the operator to remove old versions via designated script
> using criteria like "configuration age", "ds history length" or "was it
> deployed". For example "Leave all deployed revisions and up to 100 non
> deployed revisions".
> I haven't thought of the option to support the marking of configuration
> versions as "never delete", but it can surely be added.
> 
> I did not intended to create something more sophisticated, and believe that
> the mentioned script will be used only on rare cases that something is
> trashing the DB, as the math I did lead me to believe it is a none issue:
> Judging from the kable-town example, a delivery-service configuration size
> is less than 500B. Lets say the average is *1K *to support future growth.
> Lets also say we support *10K *DSs (which is much much more than any TC
> deployment I'm aware of has) and we have *1K *revisions per DS.
> In such a case versioning will use 10GB, which I believe is not an issue
> for postgres to hold (yet, I'm not a postgres expert).
> 
> Nir
> 
> 
> On Thu, Apr 13, 2017 at 3:53 PM, Eric Friedrich (efriedri) <
> efrie...@cisco.com> wrote:
> 
>> Hey Nir-
>>  If we keep all DS versions in the DB, are there any concerns about the
>> amount of data retained? I know Delivery Services don’t change very often,
>> but over time do we really need to keep the last 1000 revisions of a
>> delivery service?
>> 
>> Its more of an implementation detail, but I think it would be useful to
>> give some control over version retention policies (i.e. keep last n based
>> on quantity or dates, mark some as “never delete”)
>> 
>> More inline
>>> On Apr 12, 2017, at 12:53 AM, Nir Sopher <n...@qwilt.com> wrote:
>>> 
>>> Thanks Derek for the clarification.
>>> 
>>> So the definition file is a global file for the CDN.
>>> Does it contain the information of which server has which DS?
>>> Does it hold all CDN's DSs configuration together?
>>> On a single DS change, will all servers in the CDN download the entire
>> file
>>> for every DS change?
>>> 
>>> What I'm practically asking is, if it is not already your intention, "can
>>> the definition file hold only the information of which server holds which
>>> DS (and configuration version when we add it), and the DS configuration
>> be
>>> held and pulled separately on a DS level granularity?"
>>> 
>>> When discussing "self-service" we would like to decouple the operations
>> of
>>> the different users / content providers. Ultimately, when a DS is
>> changed,
>>> the change should be deployed immediately to the CDN - with no dependency
>>> with other DSs, and possibly with "no buffering" by the operator
>> deploying
>>> batch of DS changes together. This allows to improve the user experience
>>> and independence when working on the CDN.
>>> Following the change you are suggesting, will the DS configuration
>>> deployment coupling get tighter? We prefer not to have the need to
>> "finish
>>> your work and not start additional work before the queued run has
>>> completed".
>> EF> Agree. The less steps our users have to take, the happier they are. If
>> it was a common workflow to batch a bunch of DS changes and them roll them
>> out together, I would probably be a stronger advocate for keeping the queue
>> update/snapshot crconfig steps around. In our discussion, that doesn’t seem
>> to be often used. Should we consider deprecating those stages and
>> immediately (excepting ORT polling interval of course) applying config
>> changes when the DS is changed?
>> 
>>> 
>>> Another requirement is to be able to rollback changes on a DS level and
>> not
>>> only on a CDN level, as it is not desired to rollback changes of user "A"
>>> because of errors of user "B". If I understand correctly, the definition
>>> file does not support that.
>> EF> I thi

Re: Proposal for CDN definition file based configuration management

[Eric Friedrich (efriedri)]

Hey Nir-
  If we keep all DS versions in the DB, are there any concerns about the amount 
of data retained? I know Delivery Services don’t change very often, but over 
time do we really need to keep the last 1000 revisions of a delivery service? 

Its more of an implementation detail, but I think it would be useful to give 
some control over version retention policies (i.e. keep last n based on 
quantity or dates, mark some as “never delete”)

More inline
> On Apr 12, 2017, at 12:53 AM, Nir Sopher <n...@qwilt.com> wrote:
> 
> Thanks Derek for the clarification.
> 
> So the definition file is a global file for the CDN.
> Does it contain the information of which server has which DS?
> Does it hold all CDN's DSs configuration together?
> On a single DS change, will all servers in the CDN download the entire file
> for every DS change?
> 
> What I'm practically asking is, if it is not already your intention, "can
> the definition file hold only the information of which server holds which
> DS (and configuration version when we add it), and the DS configuration be
> held and pulled separately on a DS level granularity?"
> 
> When discussing "self-service" we would like to decouple the operations of
> the different users / content providers. Ultimately, when a DS is changed,
> the change should be deployed immediately to the CDN - with no dependency
> with other DSs, and possibly with "no buffering" by the operator deploying
> batch of DS changes together. This allows to improve the user experience
> and independence when working on the CDN.
> Following the change you are suggesting, will the DS configuration
> deployment coupling get tighter? We prefer not to have the need to "finish
> your work and not start additional work before the queued run has
> completed".
EF> Agree. The less steps our users have to take, the happier they are. If it 
was a common workflow to batch a bunch of DS changes and them roll them out 
together, I would probably be a stronger advocate for keeping the queue 
update/snapshot crconfig steps around. In our discussion, that doesn’t seem to 
be often used. Should we consider deprecating those stages and immediately 
(excepting ORT polling interval of course) applying config changes when the DS 
is changed? 

> 
> Another requirement is to be able to rollback changes on a DS level and not
> only on a CDN level, as it is not desired to rollback changes of user "A"
> because of errors of user "B". If I understand correctly, the definition
> file does not support that.
EF> I think the definition file can support this- only the rolled-back DS would 
change inside that file. No other users would be affected because their DS 
configs would not change. 

> 
> Last, using the definition file, must all servers in a CDN work with the
> same set of DSs? One of the reasons we consider "DS versioning" is to allow
> deployment of a change in a DS only for a subset of the server for canary
> testing.
EF> When I think of canary testing today, my mind first goes to Steering DS. 
With those steering delivery services, do we still need ability to set 
per-cache DS versions?

—Eric


> 
> Thanks,
> Nir
> 
> 
> 
> On Wed, Apr 12, 2017 at 3:00 AM, Dewayne Richardson <dewr...@gmail.com>
> wrote:
> 
>> +1 I was just about to formulate that response.  The "dev" list is our
>> discussion forum.
>> 
>> On Tue, Apr 11, 2017 at 9:35 AM, Dave Neuman <neu...@apache.org> wrote:
>> 
>>> @Ryan, I think its better to have conversations on the dev list than a
>> wiki
>>> page...
>>> 
>>> On Tue, Apr 11, 2017 at 9:01 AM, Durfey, Ryan <ryan_dur...@comcast.com>
>>> wrote:
>>> 
>>>> Started a new wiki page to discuss this here https://cwiki.apache.org/
>>>> confluence/display/TC/Configuration+Management
>>>> 
>>>> I will do my best to summarize the discussion below later today.
>>>> 
>>>> Ryan M | 303-524-5099
>>>> 
>>>> 
>>>> -Original Message-
>>>> From: Eric Friedrich (efriedri) [mailto:efrie...@cisco.com]
>>>> Sent: Tuesday, April 11, 2017 8:55 AM
>>>> To: dev@trafficcontrol.incubator.apache.org
>>>> Subject: Re: Proposal for CDN definition file based configuration
>>>> management
>>>> 
>>>> A few questions/thoughts, apologies for not in-lining:
>>>> 
>>>> 1) If we move away from individually queued updates, we give up the
>>>> ability to make changes and then selectively deploy them. How often do
>> TC
>>>> operations teams make config chan

Re: Recent changes to ATS config file generation

[Eric Friedrich (efriedri)]

Thanks Derek-

- ORT will not update regex_revalidate unless the parents have been cleared.
EF> Is this based on the parent’s upd_pending or the parent’s reval_pending?

- ORT will ignore the upd_pending state of the parents during syncds.
EF> Isn’t it important that parent’s upd_pending be cleared before doing a 
syncds on the child?

  
> On Apr 11, 2017, at 9:57 AM, Gelinas, Derek  wrote:
> 
> EF> What does this mean? Can edge servers now use other reverse proxies as 
> parents? How do we configure this on TO?
> 
> DG> Edge and mid servers can use other reverse proxies as their connection to 
> Traffic Ops.  This is configured in the GLOBAL parameter profile using the 
> parameter tm.rev_proxy_url.  When this is set, the metadata for the server 
> will show a toRevProxyUrl entry under the info section of the JSON.  Info 
> will include profileId, toRevProxyUrl, toUrl, serverIpv4, ServerTcpPort, 
> serverName, cdnId, cdnName, serverId, and profileName.  If ORT detects the 
> toRevProxyUrl, it will attempt to use that as the source for all 
> configuration file requests in that run.  Should the specified proxy be down 
> or unreachable, it will revert back to the URL used in the command line when 
> ORT was started.  While there’s no reason a delivery service for this purpose 
> couldn’t be created, we’ve chosen to run an ATS instance, configured 
> manually, on each Traffic Ops instance using a short (~3 minute) TTL.  This 
> allows us to keep the caching for Traffic Ops separate from the CDN we are 
> administering.
> 
> EF> Are parent server update flags are still required for syncds? We only 
> ignore them for the “new revalidate”/“instant invalidation” (I assume these 
> are same thing) options?
> 
> DG> The upd_pending flags still exist and are still used by syncds.  We’ve 
> added a new column, reval_pending, which is used for invalidation.  When 
> use_reval_pending is set in the GLOBAL profile with a value of 1, the 
> following happens:
>   - Invalidation jobs set by either the UI or the API will flag the 
> reval_pending column instead of upd_pending
>   - The update page checked by ORT will show a reval_pending entry with a 
> true/false value. (Value is null when use_reval_pending is disabled)
>   - ORT will perform revalidation checks at the start of each syncds 
> dispersion and every 60 seconds while the dispersion is running.
>   - ORT will not update regex_revalidate unless the parents have been 
> cleared.
>   - ORT will not perform updates of the regex_revalidate file during 
> syncds.
>   - ORT will ignore the upd_pending state of the parents during syncds.
> 
> If use_reval_pending is disabled, syncds will function as it does today.  If 
> an older version of Traffic Ops is in use and the API returns a 404, ORT will 
> use the currently released method for config files. 
> 
> EF> Where can we see the new API? Is scope an input/output/both parameter to 
> the API now?
> 
> DG> The API has been merged into master, under 
> app/lib/API/Configs/ApacheTrafficServer.pm  Scope is set in the get_scope() 
> sub.  The filename is passed to this sub and the correct scope is returned.  
> This is primarily used to validate requests - should the incorrect scope be 
> used, the api will return an error with the correct scope needed for that 
> file.  Each config file’s entry in the metadata will include the following:
>   -fNameOnDisk - the configuration file name
>   -location - the file’s location on the cache itself
>   -apiUri - the URI to use to download the file from Traffic Ops.  
> Example: “/api/1.2/cdn/title-vi/configfiles/ats/set_dscp_28.config"
>   -scope - the scope of the file
> 
> Proper names or the IDs of the server, profile, or cdn can be used 
> interchangeably.  The API will output names rather than IDs for readability.
> 
> EF> Will existing API remain as “deprecated” for a few more releases? Upgrade 
> path is very important. I assume we would upgrade TO to 2.1 and then upgrade 
> caches. Will 1.7,1.8,2.0 caches be able to request config files as they do 
> today from a TO2.1? 
> 
> DG> Both methods of requesting configuration files remain supported at this 
> time, and the code changes have been designed so either the caches or Traffic 
> Ops can be upgraded first.  Use of all new features will require upgrading 
> both, of course.
> 
> Hope that answers your questions.  Feel free to send any more you might have 
> my way.
> 
> Derek 
> 

Re: API GW, new AAA model and legacy AAA model in Traffic Ops

[Eric Friedrich (efriedri)]

Thanks Amir-
> On Apr 3, 2017, at 11:03 AM, Amir Yeshurun <am...@qwilt.com> wrote:
> 
> Hi Eric, please see response inline
> 
> On Mon, Apr 3, 2017 at 3:20 PM Eric Friedrich (efriedri) <efrie...@cisco.com>
> wrote:
> 
>> Hey Amir-
>>  Makes sense.
>> 
>> 1) Will API Gateway be a service external to TO (like a
>> trafficserver/nginx proxy responsible just for authorization)?
>> 
> [AY] Yes, the API GW and the authentication service are both external to
> TO. In a futuristic vision, TO functionality is split into a set of
> (micro)services.
> 
>> 
>> 2) The API GW will be performing the authorization, so will it need access
>> to the TO DB to read the policy stored there? This seems like it may be
>> breaking module boundaries?
>> 
> [AY] Yes, API GW use TO API to read AAA policy in order to enforce it. IMO
> that is valid. Can you explain your concern about module boundaries?
> (Note that performance wise, since policy changes are rare, the policy is
> cached in the API GW)
[EF] Glad to hear that API GW is using the TO API- no concerns about this. If 
API GW was making direct reads to the TO DB, I would have been concerned. 

> 
> 3) What authentication methods with the auth service support (user/pass,
>> API token, OAuth)?
>> 
> [AY] The current implementation is naive, and currently serve as a proof of
> concept. Login is done via user/pass, JWT is used as the API token.
> Note that you can also use any 3rd party to maintain your users DB. The 3rd
> party service should return a JWT that contains the user's capabilities.
If we are using a 3rd party authentication service, will there be documentation 
that will allow us to generate the same JWT?

> 
>> 
>> 
>> —Eric
>> 
>> 
>> 
>>> On Apr 2, 2017, at 8:44 AM, Amir Yeshurun <am...@qwilt.com> wrote:
>>> 
>>> Hi,
>>> 
>>> This email relates to an the API GW and to the new AAA model that are
>> under
>>> development for post 2.0 TC.
>>> 
>>> The purpose is to explain how new AAA model and existing AAA model live
>>> together then using the new API GW.
>>> 
>>> Currently TO handles Authentication and Authorization.
>>> Authentication is handled by Mojolicious.
>>> Authorization is spread in many places in the code (is_admin(),
>> is_oper(),
>>> is_federation()...)
>>> 
>>> When introducing API GW, authentication is done via an authentication
>>> service external to legacy TO.  Authorization is done in the API GW.
>>> 
>>> A new AAA model was introduced.
>>> 
>>>  - Each user in the users table has roles. Each role has capabilities.
>>>  - Capabilities define the API calls that the user is allowed to
>> perform.
>>>  - TO DB contains a table that maps each API route to the list of
>>>  capabilities required to perform this API call
>>> 
>>> Brief summary of the new AAA flow
>>> 
>>>  - Authentication service provides login endpoint, verify credentials
>>>  against TO DB, read the user's authorized capabilities from TO DB and
>> set
>>>  them as claims on a JWT.
>>>  - Per each API call, the API GW verifies the JWT and match the user's
>>>  capabilities against the required capabilities for the route.
>>>  - If authorized, the API GW forwards API calls to TO.
>>> 
>>> Note that the API GW only handles "/api" routes. It does not handle UI
>>> routes. There was no change in the current UI app. The Current UI is not
>>> affected by the new AAA model and keeps playing by the old rules.
>>> 
>>> As long as TO is enforcing the current AAA model, a session has to be
>>> authenticated against TO even when using API GW.
>>> 
>>> To achieve that, a simple design would be
>>> 
>>>  - Upon a "/login" to the new authentication service, the authentication
>>>  service will perform a login operation against TO to get a "legacy"
>>>  authentication token. The auth server sets the legacy token as a claim
>> in
>>>  the JWT.
>>>  - Legacy authentication is done *in addition *(and *not instead of*)
>> to
>>>  verifying the user tmUser and providing capabilities using the new AAA
>>>  model.
>>>  - The legacy token expiration should be longer than the JWT expiration.
>>>  - The API GW authorizes the call using the new model.
>>>  - If the request is authorized, API GW pulls the legacy token from the
>>>  JWT and set the request "Cookie:" header accordingly, before
>> forwarding the
>>>  request.
>>> 
>>> When all TO functionality is supported via API, and TO UI is replaces to
>>> the new UI built on top of the API, we can consider deprecating the old
>>> model as well as the Mojolitions app.
>>> 
>>> Thanks
>>> /amiry
>> 
>> 

Re: API GW, new AAA model and legacy AAA model in Traffic Ops

[Eric Friedrich (efriedri)]

Hey Amir-
  Makes sense. 

1) Will API Gateway be a service external to TO (like a trafficserver/nginx 
proxy responsible just for authorization)?

2) The API GW will be performing the authorization, so will it need access to 
the TO DB to read the policy stored there? This seems like it may be breaking 
module boundaries?

3) What authentication methods with the auth service support (user/pass, API 
token, OAuth)? 


—Eric



> On Apr 2, 2017, at 8:44 AM, Amir Yeshurun  wrote:
> 
> Hi,
> 
> This email relates to an the API GW and to the new AAA model that are under
> development for post 2.0 TC.
> 
> The purpose is to explain how new AAA model and existing AAA model live
> together then using the new API GW.
> 
> Currently TO handles Authentication and Authorization.
> Authentication is handled by Mojolicious.
> Authorization is spread in many places in the code (is_admin(), is_oper(),
> is_federation()...)
> 
> When introducing API GW, authentication is done via an authentication
> service external to legacy TO.  Authorization is done in the API GW.
> 
> A new AAA model was introduced.
> 
>   - Each user in the users table has roles. Each role has capabilities.
>   - Capabilities define the API calls that the user is allowed to perform.
>   - TO DB contains a table that maps each API route to the list of
>   capabilities required to perform this API call
> 
> Brief summary of the new AAA flow
> 
>   - Authentication service provides login endpoint, verify credentials
>   against TO DB, read the user's authorized capabilities from TO DB and set
>   them as claims on a JWT.
>   - Per each API call, the API GW verifies the JWT and match the user's
>   capabilities against the required capabilities for the route.
>   - If authorized, the API GW forwards API calls to TO.
> 
> Note that the API GW only handles "/api" routes. It does not handle UI
> routes. There was no change in the current UI app. The Current UI is not
> affected by the new AAA model and keeps playing by the old rules.
> 
> As long as TO is enforcing the current AAA model, a session has to be
> authenticated against TO even when using API GW.
> 
> To achieve that, a simple design would be
> 
>   - Upon a "/login" to the new authentication service, the authentication
>   service will perform a login operation against TO to get a "legacy"
>   authentication token. The auth server sets the legacy token as a claim in
>   the JWT.
>   - Legacy authentication is done *in addition *(and *not instead of*)  to
>   verifying the user tmUser and providing capabilities using the new AAA
>   model.
>   - The legacy token expiration should be longer than the JWT expiration.
>   - The API GW authorizes the call using the new model.
>   - If the request is authorized, API GW pulls the legacy token from the
>   JWT and set the request "Cookie:" header accordingly, before forwarding the
>   request.
> 
> When all TO functionality is supported via API, and TO UI is replaces to
> the new UI built on top of the API, we can consider deprecating the old
> model as well as the Mojolitions app.
> 
> Thanks
> /amiry

Re: Backup Cache Group Selection

[Eric Friedrich (efriedri)]

017 at 9:12 AM, John Shen (weifensh)
>>> <weife...@cisco.com> wrote:
>>>> Hi Jeff,
>>>> 
>>>> Thank you for the detail. I am wondering why there are two sets of
>>> lat/lang,
>>>> i.e. one in CZF, the other is in Ops GUI Cache Group setting. To
>>> calculate
>>>> the closest CG when matched CG in CZF is not available, the source
>>> lat/long
>>>> is from mathced CZF, and the dest lat/long is from Ops setting, which
>>> doesnt
>>>> seem to be consistent. Is there any reason why TR has this behavior?
>>>> 
>>>> Since there are two sets of lat/long in TR, can we just use the lat/long
>>> all
>>>> from Ops CG settings to get the closest, and do not care about the values
>>>> set in CZF? At least this will avoid inconsistent config for lat/long.
>>>> 
>>>> Thanks,
>>>> John
>>>> 
>>>> ---Original---
>>>> From: "Jeff Elsloo "<els...@apache.org>
>>>> Date: 2017/3/29 22:45:12
>>>> To:
>>>> "dev@trafficcontrol.incubator.apache.org"<dev@trafficcontrol.incubator.
>>> apache.org>;
>>>> Subject: Re: Backup Cache Group Selection
>>>> 
>>>> Yes, it's expected behavior. What you're describing sounds like a
>>>> cachegroup in the CZF without any corresponding configuration in
>>>> Traffic Ops, or a cachegroup with configuration in Traffic Ops, but
>>>> with no available caches (DS assignments, health, etc).
>>>> 
>>>> Presuming we have configured geolocation coordinates within the CZF,
>>>> we know the lat/long of the cachegroup within the CZF that contains
>>>> the source address. We can then order our list of cachegroups by
>>>> lat/long, then select the "next best" cache group by distance and
>>>> availability. That will be the actual cachegroup to serve the request;
>>>> this prevents a miss on the CZF that would normally be routed to the
>>>> Geolocation service selected for the DS.
>>>> 
>>>> We do have a slight gap around logging, and maybe that's part of the
>>>> question. What we see in the log is the selected lat/long, not the
>>>> source lat/long of the hit, so we can't easily tell when we're in this
>>>> case by simply looking at logs. This could be an area of improvement,
>>>> however, we'll need to be careful to not conflate the logs with
>>>> unnecessary information. In most cases the hit is the selected
>>>> cachegroup, so we need to be careful to not just add "source" and
>>>> "actual" coordinates to the log because it'll be identical in most CZF
>>>> hit cases.
>>>> 
>>>> Thanks,
>>>> Thanks,
>>>> Jeff
>>>> 
>>>> 
>>>> On Wed, Mar 29, 2017 at 7:02 AM, John Shen (weifensh)
>>>> <weife...@cisco.com> wrote:
>>>>> Hi Jeff,
>>>>> 
>>>>> I have just tried the getClosestCacheLocation() logic. It appears the
>>> CZF
>>>>> matched lat/long does come from CZF, but the lat/long of the “closest”
>>> Cache
>>>>> Groups is from the configuration by Ops. This means to calculate the
>>>>> distance from the matched CG and “closest” CG, the source lat/long is
>>> from
>>>>> CZF, but the dest lat/long is not from CZF but from CG settings on Ops.
>>> Is
>>>>> this expected behavior?
>>>>> 
>>>>> Thanks,
>>>>> John
>>>>> 
>>>>> 
>>>>> On 27/01/2017, 10:51 PM, "Jeff Elsloo" <jeff.els...@gmail.com> wrote:
>>>>> 
>>>>>Steve: I don't think the patch is required, however, as Eric found,
>>>>>without the patch there could be some gaps depending on the
>>> scenario.
>>>>>That specific scenario revolved around the "next best cache group"
>>> not
>>>>>having a DS assigned, or a healthy cache with the DS assigned. In
>>> that
>>>>>case, despite the hits, you would still end up falling through to
>>> the
>>>>>geolocation provider. The patch addresses that.
>>>>> 
>>>>>Eric: The rloc field is set via the Geolocation associated with the
>>>>>CacheLocation, which ultimately comes from the edgeLocations section
&

Re: adding new edge server

[Eric Friedrich (efriedri)]

Hey Burak-
  Here’s some documentation: 
https://trafficcontrol.apache.org/docs/latest/admin/traffic_ops_config.html?highlight=allow_ip#profile-parameters

In Traffic Ops go to Parameters->All Profiles. In the search box enter 
“allow_ip”. You might see several entries, you want to edit the entry for the 
cache group your edge servers belong to.

—Eric



On Monday, March 20, 2017 8:36 PM, Burak Sarp 
<sarp_bu...@yahoo.com<mailto:sarp_bu...@yahoo.com>> wrote:


Hi Eric,

Firstly, I solved the Live Tv problem thanks to you,

How can i find ip_allow config parameter?

thanks,
burak


On Monday, March 20, 2017 8:31 PM, Eric Friedrich (efriedri) 
<efrie...@cisco.com<mailto:efrie...@cisco.com>> wrote:


Did you add the IP of the Traffic Monitor to the ip_allow config parameter in 
Traffic Ops?

—Eric

> On Mar 20, 2017, at 1:20 PM, Burak Sarp 
> <sarp_bu...@yahoo.com.INVALID<mailto:sarp_bu...@yahoo.com.INVALID>> wrote:
>
> Hi all,
> I added new Edge server, I can reach contents from edge servee, so traffic 
> server is running and serving contents, but in monitor it is shown like that
> | REPORTED - java.util.concurrent.CancellationException
>
> |
>
>
> the monitor's event.log is here
> 1490022381.956 host="edge1", type=EDGE, available=false, msg="REPORTED - 
> java.net.ConnectException: Connection refused: /104.131.132.178:80 to 
> http://edge1.tvplus.com/_astats?application==eth0"1490027740.610 
> host="edge1", type=Delivery Service, available=true, msg="REPORTED - 
> available"1490027740.644 host="tvplus", type=Delivery Service, 
> available=true, msg="REPORTED - available"1490027741.994 host="edge", 
> type=EDGE, available=true, msg="ONLINE - available, monitoring 
> disabled"1490027742.632 host="edge1", type=EDGE, available=false, 
> msg="REPORTED - java.util.concurrent.CancellationException"
>
> when i trying to reach router the access.log is here1490029703.739 qtype=HTTP 
> chi=195.142.178.157 url="http://tr.origin1.cdn1.tvplus.com/; cqhm=GET 
> cqhv=HTTP/1.1 rtype=MISS rloc="-" rdtl=- rerr="-" rgb="-" pssc=503 ttms=1.863 
> rurl="-" rh="-"
> Is there anyway to fix this?
> thanks,burak

Re: adding new edge server

[Eric Friedrich (efriedri)]

Did you add the IP of the Traffic Monitor to the ip_allow config parameter in 
Traffic Ops?

—Eric

> On Mar 20, 2017, at 1:20 PM, Burak Sarp  wrote:
> 
> Hi all,
> I added new Edge server, I can reach contents from edge servee, so traffic 
> server is running and serving contents, but in monitor it is shown like that
> | REPORTED - java.util.concurrent.CancellationException
> 
> |
> 
> 
> the monitor's event.log is here
> 1490022381.956 host="edge1", type=EDGE, available=false, msg="REPORTED - 
> java.net.ConnectException: Connection refused: /104.131.132.178:80 to 
> http://edge1.tvplus.com/_astats?application==eth0"1490027740.610 
> host="edge1", type=Delivery Service, available=true, msg="REPORTED - 
> available"1490027740.644 host="tvplus", type=Delivery Service, 
> available=true, msg="REPORTED - available"1490027741.994 host="edge", 
> type=EDGE, available=true, msg="ONLINE - available, monitoring 
> disabled"1490027742.632 host="edge1", type=EDGE, available=false, 
> msg="REPORTED - java.util.concurrent.CancellationException"
> 
> when i trying to reach router the access.log is here1490029703.739 qtype=HTTP 
> chi=195.142.178.157 url="http://tr.origin1.cdn1.tvplus.com/; cqhm=GET 
> cqhv=HTTP/1.1 rtype=MISS rloc="-" rdtl=- rerr="-" rgb="-" pssc=503 ttms=1.863 
> rurl="-" rh="-"
> Is there anyway to fix this?
> thanks,burak

Re: Update Delivery Service URL

[Eric Friedrich (efriedri)]

You will also need to queue updates for the caches assigned to this delivery 
service and run the ORT script in “syncds” mode to update the configuration on 
the caches. This is how remap.config is changed. 

CRConfig only goes out to Traffic Monitor and Traffic Router. 

This is pretty safe to do on the fly, but once you run the ORT script the 
config will be changes on caches, so the caches will no longer respond to the 
original URL remap rule. If this is a system in production, you might want to 
consider setting up a second delivery service that shares the same origin 
server. When rolling out a new DS be sure to run ORT on the caches before 
SnapShot CRConfig.

—Eric
  


 
> On Mar 17, 2017, at 2:43 AM, Burak Sarp  wrote:
> 
> Hi,
> I needed to update Delivery Service URL,Is it enough to update delivery 
> service in ops adn CR Config? I mean after updating Cr Config the 
> remap.config in Traffic Server will be changed?lastly, is it safe to update 
> delivery service url on the fly?
> thanks,burak 

Re: Public CI Builds for Traffic Control

[Eric Friedrich (efriedri)]

robably
even
very occasional contributors. But stick builds on the most frequent
contributors' clones and we get 95% of the benefit without solving any of
the actually hard problems.

We'd need a host, though.

On Tue, Mar 14, 2017 at 5:06 PM Leif Hedstrom <zw...@apache.org> wrote:


On Mar 13, 2017, at 8:44 AM, Chris Lemmons <alfic...@gmail.com> wrote:

To me, the key features of CI are that a) it builds each branch
automatically, b) notifies affected parties when all is not well, and
c)
manages the artefacts in a reasonable way. Additionally, we're a lot
more
useful when we're writing neat software and not spending out time
managing
CI, so it should be as automatic as reasonable. We're using github for
PRs,
so if it's at all possible to get automatic PR tagging with build
information, that is greatly desirable. Knowing that the PR breaks the
build prior to merging it can save quite a bit of time. :)


My $0.25: My experience is that making as much CI build / tests on pull
requests, *before* they are landed, gives the most bang for the buck.
But
that might not work well for you, since you can’t use Github right?

— leif


I've not used BuildBot, but it feels... svney?

Jenkins can do all of the above, though basically all those features
are
plugins. There's a "build per branch" plugin that uses branches to
automatically make builds. There's a variety of notifier plugins. There
are
some artefact management plugins. There is a "build PRs" plugin that's
specifically designed for GitHub. Jenkins isn't much on its own, but
properly adorned with plugins, it can do most of what we'd want, I
think.

I've also had extensive experience with Bamboo, Atlassian's
closed-source
solution in the same suite as Jira. Bamboo is usable gratis for OSS in
the
same way we currently use Apache Jira.

Bamboo also has plugins, but the majority of it's features are good to
go
immediately. There's an automatic checkbox for "build per branch". The
artefacts are managed fairly automatically, especially if you fill in
the
"build expiration" boxes. Notification is automatic and easy to
configure.
It's got a (free) plugin for PR statuses.

In any case, we'll need to manually configure the valid lists of
contributors. For security reasons, we probably can't just run whatever
PR
is created without any prior contact. :/ In Jenkins, this looks like
maintaining a "white list" of legal contributors for a PR inside the PR
plugin. In Bamboo, it looks like listing the committer forks as copied
projects. Either way is fairly manageable.

Jenkins and Bamboo both run on Java. So... no winner there. :)

I think the major question is one of hosting. No matter what solution
we
use, we need a few cycles, a network interface, and a bit of disk space
to
run it. The apache jenkins appears to have almost all the stuff we
need.
It
does not appear to have docker-compose, which we're leveraging fairly
significantly at the moment. docker-compose, however, is Apache
licensed,
so we could theoretically ship it inside the repo. Not sure I like that
option, though. We could also switch off of compose and put the
relevant
options into our build script directly. Not sure I like that option,
either.

We'd have the most flexibility if we could get one or more companies to
donate a publicly accessible host (or even theoretically, a build
slave),
assuming that doesn't break Apache's rules. The CI doesn't need a ton
of
gas, but the more oomph it has, the more granularly it can build and
more
aggressively we can test.

On Sun, Mar 12, 2017 at 6:54 PM Eric Friedrich (efriedri) <
efrie...@cisco.com> wrote:

Hey All-
I’d played around before with Travis CI for Continuous Integration
builds, but never actually set it up for the public repo.

I know some others on the list have tried out comparable services. Does
anyone have experience or suggestions to share?

Also, we can now get access to Apache Buildbot (
https://ci.apache.org/buildbot.html) and an Apache hosted Jenkins (
https://cwiki.apache.org/confluence/display/INFRA/Jenkins)

I think Traffic Server has their own separate Jenkins server so they
can
hit more platforms, but with the latest build changes, pretty much all
we
require is access to a docker daemon

—Eric

Re: Public CI Builds for Traffic Control

[Eric Friedrich (efriedri)]

but the more oomph it has, the more granularly it can build and more
aggressively we can test.

On Sun, Mar 12, 2017 at 6:54 PM Eric Friedrich (efriedri) <
efrie...@cisco.com<mailto:efrie...@cisco.com>> wrote:

Hey All-
I’d played around before with Travis CI for Continuous Integration
builds, but never actually set it up for the public repo.

I know some others on the list have tried out comparable services. Does
anyone have experience or suggestions to share?

Also, we can now get access to Apache Buildbot (
https://ci.apache.org/buildbot.html) and an Apache hosted Jenkins (
https://cwiki.apache.org/confluence/display/INFRA/Jenkins)

I think Traffic Server has their own separate Jenkins server so they can
hit more platforms, but with the latest build changes, pretty much all we
require is access to a docker daemon

—Eric