Great write up Dewayne.  Given the length and forward looking nature of the 
topic I created a wiki page around this under traffic ops.  Please continue 
discussion in the email thread and I will summarize any changes there.

Ryan Durfey    M | 303-524-5099
CDN Support (24x7): 866-405-2993 or<>

From: Dewayne Richardson <>
Reply-To: "" 
Date: Wednesday, August 9, 2017 at 1:03 PM
To: "" 
Subject: Traffic Ops Golang Rewrite

Sorry for the TL;DR, but a lot of information needed to be conveyed.  So,
based upon the TO Rewrite discussions Rob Butts and I have been working on
a Golang proxy (Mark Torluemke, affectionately calls "the Vampire Proxy")
that initially implements the */monitoring* endpoint to lay down the
foundation for rewriting more Traffic Ops endpoints in Go.  The goal is to
do a straight rewrite in Go (that implements any old API's as well as any
new ones that follows the /api/1.2 format).  The intent is to make this
proxy 100% backward compatible (including any HTTP header requirements) to
keep the existing TO API clients from breaking.

This PR is significant because when postinstall runs (after this PR is
merged) it switches the ports according to the following:

*TO Port Change Overview*
Port *443* will be owned by Golang proxy

Port *60443* will now be owned by the Mojolicious/Perl Hypnotoad service

See */opt/traffic_ops/app/conf/cdn.conf* for a new property
=> '443'*.

*Important Operational Changes:*

*traffic_ops service*
The Golang Proxy Service is now combined with the *traffic_ops* service, so
when traffic_ops is restarted so is the Golang Proxy Service.  Since the
Golang service is a proxy any APIs that are not implemented in the Golang
Service will be forwarded to the existing TO Perl API.  Also, If the API is
implemented in the Golang Service the response is serviced by the Proxy
(where it will access the Postgres database as needed).

*traffic_ops logs*
*access.log - *old access.log is now renamed to *perl_access.log*, and the
Go proxy now takes over the *access.log* while logging in the *exact* format
as before (no monitoring or tooling changes are required)
*traffic_ops_golang.log* - this is a new file where any errors/debug will
be logged from the Go proxy.
*perl_access.log - the existing Mojolicious access.log gets a new name*
*traffic_ops.log* - existing Mojolicious debug file (no change)

There was a lot of debate and discussion about how to move forward and this
approach was less impactful to operations (which basically means less work
to move toward Go).  Overtime, the goal is to do a rewrite of all of the
relevant endpoints that are in Mojolicious into Go (with a heavy focus on
modularity and unit testing, for a future release with Micro Services).

*What about the Qwilt contributions of API Gateway, Capabilities, and
Tenancy wasn't that a thing?  *
Yes, and it still is.  As for the API Gateway we will start "absorbing" the
code that *Amir* Yeshurun, so graciously contributed for the APIGW and
JWT.  For the API Capabilities that *Naama Shoresh* also generously
contributed (all capabilities will be directly accessed from those
capability tables), please see the *Roadmap* below.  And last but not
least, once once we are confident that tenancy works as designed, we will
begin porting all the tenancy hooks as well as any apis related to tenancy
(tenant apis, user apis, delivery service apis). All of the features that
Qwilt built will be used in some form or fashion.

*How do I jump in?*
We are going to take a "Biggest Bang for our Buck" approach to the TPv2 UI
and TO Golang API.  The following roadmap is an initial plan of attack (of
course subject to debate).

- The TO API is READ heavy so all READ endpoints will be rewritten first
(avoiding Tenancy endpoints, like Users and Delivery Services)
- Circle back to the CREATE, UPDATE, DELETEs where more Golang foundation
will have to be developed (avoiding Tenancy endpoints)
- Implement Capabilities in the new TPv2 to help manage the
Roles/Capabilities with ease
- Evaluate Capabilities and design the api_capabilities data around that
- Hook in capabilities (once the capabilities design has settled and
database is correct in production)
- Evaluating the existing Tenancy Rules
- Rewrite Tenancy in Golang
- Tackle the 'Tenancy' endpoints in Golang, /deliveryservices, /users etc.

*Reasons to get excited!*

- *Improved PR's: *(Dan Kirkwood is helping us automate Jenkins/CI for
better PR quality so components like Documentation, Unit Testing are first
class citizens.  The more required items checked off the PR checklist makes
PR merging MUCH easier.

- *Testing:*  We have plans in the works to move testing to the forefront
of our development discussions.  The hope is a fresh approach to Unit
Testing (in a modern Golang world) where Go tools for topics like "Code
Coverage" will help us improve our automated testing, and allow us to
"Refactor Mercilessly"

- *Performance:* My initial testing for the new* /monitoring* Go endpoint
shows an average 20x faster than the Mojolicous Perl equivalent running on
the same host.  More efficiency endpoints mean more "scale" and less
production $$.

I will work to produce a list of initial endpoints and publish to the dev
list (along with the JIRA Tickets with the requirements to help manage the
priorities).  The following PR (which I plan on merging today, if not
tomorrow) has a section that highlights the foundational features that the
existing TO API  had that we had to build in Go,  ie:  logging, config
files, etc.

See the high level features implemented in the section *Things left before
this can be merged:*

TC JIRA for this Change:

Expect more information about our progress and we hope to be as transparent
as possible,

-Dewayne Richardson

Reply via email to