Great write up Dewayne. Given the length and forward looking nature of the topic I created a wiki page around this under traffic ops. Please continue discussion in the email thread and I will summarize any changes there.
https://cwiki.apache.org/confluence/display/TC/Golang+Traffic+Ops+Replacement+-+Vampire+Proxy Ryan Durfey M | 303-524-5099 CDN Support (24x7): 866-405-2993 or [email protected]<mailto:[email protected]> From: Dewayne Richardson <[email protected]> Reply-To: "[email protected]" <[email protected]> Date: Wednesday, August 9, 2017 at 1:03 PM To: "[email protected]" <[email protected]> Subject: Traffic Ops Golang Rewrite Sorry for the TL;DR, but a lot of information needed to be conveyed. So, based upon the TO Rewrite discussions Rob Butts and I have been working on a Golang proxy (Mark Torluemke, affectionately calls "the Vampire Proxy") that initially implements the */monitoring* endpoint to lay down the foundation for rewriting more Traffic Ops endpoints in Go. The goal is to do a straight rewrite in Go (that implements any old API's as well as any new ones that follows the /api/1.2 format). The intent is to make this proxy 100% backward compatible (including any HTTP header requirements) to keep the existing TO API clients from breaking. This PR is significant because when postinstall runs (after this PR is merged) it switches the ports according to the following: *TO Port Change Overview* Port *443* will be owned by Golang proxy Port *60443* will now be owned by the Mojolicious/Perl Hypnotoad service See */opt/traffic_ops/app/conf/cdn.conf* for a new property *traffic_ops_golang_port => '443'*. *Important Operational Changes:* *traffic_ops service* The Golang Proxy Service is now combined with the *traffic_ops* service, so when traffic_ops is restarted so is the Golang Proxy Service. Since the Golang service is a proxy any APIs that are not implemented in the Golang Service will be forwarded to the existing TO Perl API. Also, If the API is implemented in the Golang Service the response is serviced by the Proxy (where it will access the Postgres database as needed). *traffic_ops logs* *access.log - *old access.log is now renamed to *perl_access.log*, and the Go proxy now takes over the *access.log* while logging in the *exact* format as before (no monitoring or tooling changes are required) *traffic_ops_golang.log* - this is a new file where any errors/debug will be logged from the Go proxy. *perl_access.log - the existing Mojolicious access.log gets a new name* *traffic_ops.log* - existing Mojolicious debug file (no change) There was a lot of debate and discussion about how to move forward and this approach was less impactful to operations (which basically means less work to move toward Go). Overtime, the goal is to do a rewrite of all of the relevant endpoints that are in Mojolicious into Go (with a heavy focus on modularity and unit testing, for a future release with Micro Services). *What about the Qwilt contributions of API Gateway, Capabilities, and Tenancy wasn't that a thing? * Yes, and it still is. As for the API Gateway we will start "absorbing" the code that *Amir* Yeshurun, so graciously contributed for the APIGW and JWT. For the API Capabilities that *Naama Shoresh* also generously contributed (all capabilities will be directly accessed from those capability tables), please see the *Roadmap* below. And last but not least, once once we are confident that tenancy works as designed, we will begin porting all the tenancy hooks as well as any apis related to tenancy (tenant apis, user apis, delivery service apis). All of the features that Qwilt built will be used in some form or fashion. *How do I jump in?* We are going to take a "Biggest Bang for our Buck" approach to the TPv2 UI and TO Golang API. The following roadmap is an initial plan of attack (of course subject to debate). *Roadmap* - The TO API is READ heavy so all READ endpoints will be rewritten first (avoiding Tenancy endpoints, like Users and Delivery Services) - Circle back to the CREATE, UPDATE, DELETEs where more Golang foundation will have to be developed (avoiding Tenancy endpoints) - Implement Capabilities in the new TPv2 to help manage the Roles/Capabilities with ease - Evaluate Capabilities and design the api_capabilities data around that - Hook in capabilities (once the capabilities design has settled and database is correct in production) - Evaluating the existing Tenancy Rules - Rewrite Tenancy in Golang - Tackle the 'Tenancy' endpoints in Golang, /deliveryservices, /users etc. *Reasons to get excited!* - *Improved PR's: *(Dan Kirkwood is helping us automate Jenkins/CI for better PR quality so components like Documentation, Unit Testing are first class citizens. The more required items checked off the PR checklist makes PR merging MUCH easier. - *Testing:* We have plans in the works to move testing to the forefront of our development discussions. The hope is a fresh approach to Unit Testing (in a modern Golang world) where Go tools for topics like "Code Coverage" will help us improve our automated testing, and allow us to "Refactor Mercilessly" - *Performance:* My initial testing for the new* /monitoring* Go endpoint shows an average 20x faster than the Mojolicous Perl equivalent running on the same host. More efficiency endpoints mean more "scale" and less production $$. I will work to produce a list of initial endpoints and publish to the dev list (along with the JIRA Tickets with the requirements to help manage the priorities). The following PR (which I plan on merging today, if not tomorrow) has a section that highlights the foundational features that the existing TO API had that we had to build in Go, ie: logging, config files, etc. See the high level features implemented in the section *Things left before this can be merged:* TC JIRA for this Change: https://issues.apache.org/jira/browse/TC-447 https://github.com/apache/incubator-trafficcontrol/pull/729 Expect more information about our progress and we hope to be as transparent as possible, -Dewayne Richardson
