Re: Traffic Ops Golang Rewrite

[Nir Sopher]

Got you. 10x,
Nir

On Mon, Aug 14, 2017 at 5:37 PM, Dewayne Richardson 
wrote:

> The goal is to hit all the low hanging fruit first to discover all the
> refactor points and infrastructure before we attempt to bite off the bigger
> more impactful areas of Traffic Control like Tenancy and Capabilities.
>
> -Dew
>
> On Mon, Aug 14, 2017 at 5:22 AM, Nir Sopher  wrote:
>
> > +1!
> >
> > As far as tenancy is concerned, the main logic is held in the
> > "Utils::Tenant" class.
> > I would be happy to use this class as an entry point to the Golang TO
> > development.
> >
> > BTW, the UT coverage of the tenancy checks is quite extensive.
> > Are we going to migrate the UTs as well, write new UTs in Golang, or keep
> > working with the Perl UTs?
> >
> > Nir
> >
> > On Wed, Aug 9, 2017 at 11:14 PM, Durfey, Ryan 
> > wrote:
> >
> > > Great write up Dewayne.  Given the length and forward looking nature of
> > > the topic I created a wiki page around this under traffic ops.  Please
> > > continue discussion in the email thread and I will summarize any
> changes
> > > there.
> > >
> > > https://cwiki.apache.org/confluence/display/TC/Golang+
> > > Traffic+Ops+Replacement+-+Vampire+Proxy
> > >
> > > Ryan DurfeyM | 303-524-5099
> > > CDN Support (24x7): 866-405-2993 or cdn_supp...@comcast.com > > cdn_supp...@comcast.com>
> > >
> > >
> > > From: Dewayne Richardson 
> > > Reply-To: "dev@trafficcontrol.incubator.apache.org" <
> > > dev@trafficcontrol.incubator.apache.org>
> > > Date: Wednesday, August 9, 2017 at 1:03 PM
> > > To: "dev@trafficcontrol.incubator.apache.org" <
> > > dev@trafficcontrol.incubator.apache.org>
> > > Subject: Traffic Ops Golang Rewrite
> > >
> > > Sorry for the TL;DR, but a lot of information needed to be conveyed.
> So,
> > > based upon the TO Rewrite discussions Rob Butts and I have been working
> > on
> > > a Golang proxy (Mark Torluemke, affectionately calls "the Vampire
> Proxy")
> > > that initially implements the */monitoring* endpoint to lay down the
> > > foundation for rewriting more Traffic Ops endpoints in Go.  The goal is
> > to
> > > do a straight rewrite in Go (that implements any old API's as well as
> any
> > > new ones that follows the /api/1.2 format).  The intent is to make this
> > > proxy 100% backward compatible (including any HTTP header requirements)
> > to
> > > keep the existing TO API clients from breaking.
> > >
> > > This PR is significant because when postinstall runs (after this PR is
> > > merged) it switches the ports according to the following:
> > >
> > > *TO Port Change Overview*
> > > Port *443* will be owned by Golang proxy
> > >
> > > Port *60443* will now be owned by the Mojolicious/Perl Hypnotoad
> service
> > >
> > > See */opt/traffic_ops/app/conf/cdn.conf* for a new property
> > > *traffic_ops_golang_port
> > > => '443'*.
> > >
> > > *Important Operational Changes:*
> > >
> > > *traffic_ops service*
> > > The Golang Proxy Service is now combined with the *traffic_ops*
> service,
> > so
> > > when traffic_ops is restarted so is the Golang Proxy Service.  Since
> the
> > > Golang service is a proxy any APIs that are not implemented in the
> Golang
> > > Service will be forwarded to the existing TO Perl API.  Also, If the
> API
> > is
> > > implemented in the Golang Service the response is serviced by the Proxy
> > > (where it will access the Postgres database as needed).
> > >
> > > *traffic_ops logs*
> > > *access.log - *old access.log is now renamed to *perl_access.log*, and
> > the
> > > Go proxy now takes over the *access.log* while logging in the *exact*
> > > format
> > > as before (no monitoring or tooling changes are required)
> > > *traffic_ops_golang.log* - this is a new file where any errors/debug
> will
> > > be logged from the Go proxy.
> > > *perl_access.log - the existing Mojolicious access.log gets a new name*
> > > *traffic_ops.log* - existing Mojolicious debug file (no change)
> > >
> > > There was a lot of debate and discussion about how to move forward and
> > this
> > > approach was less impactful to operations (which basically means less
> > work
> > > to move toward Go).  Overtime, the goal is to do a rewrite of all of
> the
> > > relevant endpoints that are in Mojolicious into Go (with a heavy focus
> on
> > > modularity and unit testing, for a future release with Micro Services).
> > >
> > > *What about the Qwilt contributions of API Gateway, Capabilities, and
> > > Tenancy wasn't that a thing?  *
> > > Yes, and it still is.  As for the API Gateway we will start "absorbing"
> > the
> > > code that *Amir* Yeshurun, so graciously contributed for the APIGW and
> > > JWT.  For the API Capabilities that *Naama Shoresh* also generously
> > > contributed (all capabilities will be directly accessed from those
> > > capability tables), please see the *Roadmap* below.  And last but not
> > > least, once once we are confident that tenancy works as designed, we
> will
> > > begin porting all the tenancy hooks as well as any apis related to
> >

Re: Traffic Ops Golang Rewrite

[Dewayne Richardson]

The goal is to hit all the low hanging fruit first to discover all the
refactor points and infrastructure before we attempt to bite off the bigger
more impactful areas of Traffic Control like Tenancy and Capabilities.

-Dew

On Mon, Aug 14, 2017 at 5:22 AM, Nir Sopher  wrote:

> +1!
>
> As far as tenancy is concerned, the main logic is held in the
> "Utils::Tenant" class.
> I would be happy to use this class as an entry point to the Golang TO
> development.
>
> BTW, the UT coverage of the tenancy checks is quite extensive.
> Are we going to migrate the UTs as well, write new UTs in Golang, or keep
> working with the Perl UTs?
>
> Nir
>
> On Wed, Aug 9, 2017 at 11:14 PM, Durfey, Ryan 
> wrote:
>
> > Great write up Dewayne.  Given the length and forward looking nature of
> > the topic I created a wiki page around this under traffic ops.  Please
> > continue discussion in the email thread and I will summarize any changes
> > there.
> >
> > https://cwiki.apache.org/confluence/display/TC/Golang+
> > Traffic+Ops+Replacement+-+Vampire+Proxy
> >
> > Ryan DurfeyM | 303-524-5099
> > CDN Support (24x7): 866-405-2993 or cdn_supp...@comcast.com > cdn_supp...@comcast.com>
> >
> >
> > From: Dewayne Richardson 
> > Reply-To: "dev@trafficcontrol.incubator.apache.org" <
> > dev@trafficcontrol.incubator.apache.org>
> > Date: Wednesday, August 9, 2017 at 1:03 PM
> > To: "dev@trafficcontrol.incubator.apache.org" <
> > dev@trafficcontrol.incubator.apache.org>
> > Subject: Traffic Ops Golang Rewrite
> >
> > Sorry for the TL;DR, but a lot of information needed to be conveyed.  So,
> > based upon the TO Rewrite discussions Rob Butts and I have been working
> on
> > a Golang proxy (Mark Torluemke, affectionately calls "the Vampire Proxy")
> > that initially implements the */monitoring* endpoint to lay down the
> > foundation for rewriting more Traffic Ops endpoints in Go.  The goal is
> to
> > do a straight rewrite in Go (that implements any old API's as well as any
> > new ones that follows the /api/1.2 format).  The intent is to make this
> > proxy 100% backward compatible (including any HTTP header requirements)
> to
> > keep the existing TO API clients from breaking.
> >
> > This PR is significant because when postinstall runs (after this PR is
> > merged) it switches the ports according to the following:
> >
> > *TO Port Change Overview*
> > Port *443* will be owned by Golang proxy
> >
> > Port *60443* will now be owned by the Mojolicious/Perl Hypnotoad service
> >
> > See */opt/traffic_ops/app/conf/cdn.conf* for a new property
> > *traffic_ops_golang_port
> > => '443'*.
> >
> > *Important Operational Changes:*
> >
> > *traffic_ops service*
> > The Golang Proxy Service is now combined with the *traffic_ops* service,
> so
> > when traffic_ops is restarted so is the Golang Proxy Service.  Since the
> > Golang service is a proxy any APIs that are not implemented in the Golang
> > Service will be forwarded to the existing TO Perl API.  Also, If the API
> is
> > implemented in the Golang Service the response is serviced by the Proxy
> > (where it will access the Postgres database as needed).
> >
> > *traffic_ops logs*
> > *access.log - *old access.log is now renamed to *perl_access.log*, and
> the
> > Go proxy now takes over the *access.log* while logging in the *exact*
> > format
> > as before (no monitoring or tooling changes are required)
> > *traffic_ops_golang.log* - this is a new file where any errors/debug will
> > be logged from the Go proxy.
> > *perl_access.log - the existing Mojolicious access.log gets a new name*
> > *traffic_ops.log* - existing Mojolicious debug file (no change)
> >
> > There was a lot of debate and discussion about how to move forward and
> this
> > approach was less impactful to operations (which basically means less
> work
> > to move toward Go).  Overtime, the goal is to do a rewrite of all of the
> > relevant endpoints that are in Mojolicious into Go (with a heavy focus on
> > modularity and unit testing, for a future release with Micro Services).
> >
> > *What about the Qwilt contributions of API Gateway, Capabilities, and
> > Tenancy wasn't that a thing?  *
> > Yes, and it still is.  As for the API Gateway we will start "absorbing"
> the
> > code that *Amir* Yeshurun, so graciously contributed for the APIGW and
> > JWT.  For the API Capabilities that *Naama Shoresh* also generously
> > contributed (all capabilities will be directly accessed from those
> > capability tables), please see the *Roadmap* below.  And last but not
> > least, once once we are confident that tenancy works as designed, we will
> > begin porting all the tenancy hooks as well as any apis related to
> tenancy
> > (tenant apis, user apis, delivery service apis). All of the features that
> > Qwilt built will be used in some form or fashion.
> >
> > *How do I jump in?*
> > We are going to take a "Biggest Bang for our Buck" approach to the TPv2
> UI
> > and TO Golang API.  The following roadmap is an initial plan of attack
>

Re: Traffic Ops Golang Rewrite

[Nir Sopher]

+1!

As far as tenancy is concerned, the main logic is held in the
"Utils::Tenant" class.
I would be happy to use this class as an entry point to the Golang TO
development.

BTW, the UT coverage of the tenancy checks is quite extensive.
Are we going to migrate the UTs as well, write new UTs in Golang, or keep
working with the Perl UTs?

Nir

On Wed, Aug 9, 2017 at 11:14 PM, Durfey, Ryan 
wrote:

> Great write up Dewayne.  Given the length and forward looking nature of
> the topic I created a wiki page around this under traffic ops.  Please
> continue discussion in the email thread and I will summarize any changes
> there.
>
> https://cwiki.apache.org/confluence/display/TC/Golang+
> Traffic+Ops+Replacement+-+Vampire+Proxy
>
> Ryan DurfeyM | 303-524-5099
> CDN Support (24x7): 866-405-2993 or cdn_supp...@comcast.com cdn_supp...@comcast.com>
>
>
> From: Dewayne Richardson 
> Reply-To: "dev@trafficcontrol.incubator.apache.org" <
> dev@trafficcontrol.incubator.apache.org>
> Date: Wednesday, August 9, 2017 at 1:03 PM
> To: "dev@trafficcontrol.incubator.apache.org" <
> dev@trafficcontrol.incubator.apache.org>
> Subject: Traffic Ops Golang Rewrite
>
> Sorry for the TL;DR, but a lot of information needed to be conveyed.  So,
> based upon the TO Rewrite discussions Rob Butts and I have been working on
> a Golang proxy (Mark Torluemke, affectionately calls "the Vampire Proxy")
> that initially implements the */monitoring* endpoint to lay down the
> foundation for rewriting more Traffic Ops endpoints in Go.  The goal is to
> do a straight rewrite in Go (that implements any old API's as well as any
> new ones that follows the /api/1.2 format).  The intent is to make this
> proxy 100% backward compatible (including any HTTP header requirements) to
> keep the existing TO API clients from breaking.
>
> This PR is significant because when postinstall runs (after this PR is
> merged) it switches the ports according to the following:
>
> *TO Port Change Overview*
> Port *443* will be owned by Golang proxy
>
> Port *60443* will now be owned by the Mojolicious/Perl Hypnotoad service
>
> See */opt/traffic_ops/app/conf/cdn.conf* for a new property
> *traffic_ops_golang_port
> => '443'*.
>
> *Important Operational Changes:*
>
> *traffic_ops service*
> The Golang Proxy Service is now combined with the *traffic_ops* service, so
> when traffic_ops is restarted so is the Golang Proxy Service.  Since the
> Golang service is a proxy any APIs that are not implemented in the Golang
> Service will be forwarded to the existing TO Perl API.  Also, If the API is
> implemented in the Golang Service the response is serviced by the Proxy
> (where it will access the Postgres database as needed).
>
> *traffic_ops logs*
> *access.log - *old access.log is now renamed to *perl_access.log*, and the
> Go proxy now takes over the *access.log* while logging in the *exact*
> format
> as before (no monitoring or tooling changes are required)
> *traffic_ops_golang.log* - this is a new file where any errors/debug will
> be logged from the Go proxy.
> *perl_access.log - the existing Mojolicious access.log gets a new name*
> *traffic_ops.log* - existing Mojolicious debug file (no change)
>
> There was a lot of debate and discussion about how to move forward and this
> approach was less impactful to operations (which basically means less work
> to move toward Go).  Overtime, the goal is to do a rewrite of all of the
> relevant endpoints that are in Mojolicious into Go (with a heavy focus on
> modularity and unit testing, for a future release with Micro Services).
>
> *What about the Qwilt contributions of API Gateway, Capabilities, and
> Tenancy wasn't that a thing?  *
> Yes, and it still is.  As for the API Gateway we will start "absorbing" the
> code that *Amir* Yeshurun, so graciously contributed for the APIGW and
> JWT.  For the API Capabilities that *Naama Shoresh* also generously
> contributed (all capabilities will be directly accessed from those
> capability tables), please see the *Roadmap* below.  And last but not
> least, once once we are confident that tenancy works as designed, we will
> begin porting all the tenancy hooks as well as any apis related to tenancy
> (tenant apis, user apis, delivery service apis). All of the features that
> Qwilt built will be used in some form or fashion.
>
> *How do I jump in?*
> We are going to take a "Biggest Bang for our Buck" approach to the TPv2 UI
> and TO Golang API.  The following roadmap is an initial plan of attack (of
> course subject to debate).
>
> *Roadmap*
> - The TO API is READ heavy so all READ endpoints will be rewritten first
> (avoiding Tenancy endpoints, like Users and Delivery Services)
> - Circle back to the CREATE, UPDATE, DELETEs where more Golang foundation
> will have to be developed (avoiding Tenancy endpoints)
> - Implement Capabilities in the new TPv2 to help manage the
> Roles/Capabilities with ease
> - Evaluate Capabilities and design the api_capabilities data around that
> - Hook

Re: Traffic Ops Golang Rewrite

[Durfey, Ryan]

Great write up Dewayne.  Given the length and forward looking nature of the 
topic I created a wiki page around this under traffic ops.  Please continue 
discussion in the email thread and I will summarize any changes there.

https://cwiki.apache.org/confluence/display/TC/Golang+Traffic+Ops+Replacement+-+Vampire+Proxy

Ryan DurfeyM | 303-524-5099
CDN Support (24x7): 866-405-2993 or 
cdn_supp...@comcast.com


From: Dewayne Richardson 
Reply-To: "dev@trafficcontrol.incubator.apache.org" 

Date: Wednesday, August 9, 2017 at 1:03 PM
To: "dev@trafficcontrol.incubator.apache.org" 

Subject: Traffic Ops Golang Rewrite

Sorry for the TL;DR, but a lot of information needed to be conveyed.  So,
based upon the TO Rewrite discussions Rob Butts and I have been working on
a Golang proxy (Mark Torluemke, affectionately calls "the Vampire Proxy")
that initially implements the */monitoring* endpoint to lay down the
foundation for rewriting more Traffic Ops endpoints in Go.  The goal is to
do a straight rewrite in Go (that implements any old API's as well as any
new ones that follows the /api/1.2 format).  The intent is to make this
proxy 100% backward compatible (including any HTTP header requirements) to
keep the existing TO API clients from breaking.

This PR is significant because when postinstall runs (after this PR is
merged) it switches the ports according to the following:

*TO Port Change Overview*
Port *443* will be owned by Golang proxy

Port *60443* will now be owned by the Mojolicious/Perl Hypnotoad service

See */opt/traffic_ops/app/conf/cdn.conf* for a new property
*traffic_ops_golang_port
=> '443'*.

*Important Operational Changes:*

*traffic_ops service*
The Golang Proxy Service is now combined with the *traffic_ops* service, so
when traffic_ops is restarted so is the Golang Proxy Service.  Since the
Golang service is a proxy any APIs that are not implemented in the Golang
Service will be forwarded to the existing TO Perl API.  Also, If the API is
implemented in the Golang Service the response is serviced by the Proxy
(where it will access the Postgres database as needed).

*traffic_ops logs*
*access.log - *old access.log is now renamed to *perl_access.log*, and the
Go proxy now takes over the *access.log* while logging in the *exact* format
as before (no monitoring or tooling changes are required)
*traffic_ops_golang.log* - this is a new file where any errors/debug will
be logged from the Go proxy.
*perl_access.log - the existing Mojolicious access.log gets a new name*
*traffic_ops.log* - existing Mojolicious debug file (no change)

There was a lot of debate and discussion about how to move forward and this
approach was less impactful to operations (which basically means less work
to move toward Go).  Overtime, the goal is to do a rewrite of all of the
relevant endpoints that are in Mojolicious into Go (with a heavy focus on
modularity and unit testing, for a future release with Micro Services).

*What about the Qwilt contributions of API Gateway, Capabilities, and
Tenancy wasn't that a thing?  *
Yes, and it still is.  As for the API Gateway we will start "absorbing" the
code that *Amir* Yeshurun, so graciously contributed for the APIGW and
JWT.  For the API Capabilities that *Naama Shoresh* also generously
contributed (all capabilities will be directly accessed from those
capability tables), please see the *Roadmap* below.  And last but not
least, once once we are confident that tenancy works as designed, we will
begin porting all the tenancy hooks as well as any apis related to tenancy
(tenant apis, user apis, delivery service apis). All of the features that
Qwilt built will be used in some form or fashion.

*How do I jump in?*
We are going to take a "Biggest Bang for our Buck" approach to the TPv2 UI
and TO Golang API.  The following roadmap is an initial plan of attack (of
course subject to debate).

*Roadmap*
- The TO API is READ heavy so all READ endpoints will be rewritten first
(avoiding Tenancy endpoints, like Users and Delivery Services)
- Circle back to the CREATE, UPDATE, DELETEs where more Golang foundation
will have to be developed (avoiding Tenancy endpoints)
- Implement Capabilities in the new TPv2 to help manage the
Roles/Capabilities with ease
- Evaluate Capabilities and design the api_capabilities data around that
- Hook in capabilities (once the capabilities design has settled and
database is correct in production)
- Evaluating the existing Tenancy Rules
- Rewrite Tenancy in Golang
- Tackle the 'Tenancy' endpoints in Golang, /deliveryservices, /users etc.

*Reasons to get excited!*

- *Improved PR's: *(Dan Kirkwood is helping us automate Jenkins/CI for
better PR quality so components like Documentation, Unit Testing are first
class citizens.  The more required items checked off the PR checklist makes
PR merging MUCH easier.

- *Testing:*  We have plans in the works to move testing to the forefront
of our development discussions.  The hope is a fresh a