Re: Traffic Ops Golang Migration Proposal

2017-07-20 Thread Robert Butts 
@jan theoretically yes, we could do the proxy part of this in the gateway.
But that requires more operations work and configuration to set up, and
will likely take more development as well.

The idea behind this is as little ops and config work as possible.

No, this doesn't make moving to the Gateway or alternative Auth harder.
Anywhere they would point to the old TO, they point to the new TO (which is
in fact the same port), without change. The idea is a drop-in replacement,
all clients can continue to make the same requests, without knowing the
underlying service has changed.

If anything, this makes the gateway easier, as once endpoints are moved to
Go, they're independent functions which are trivial to pull out into
microservices if we want.

On Thu, Jul 20, 2017 at 7:56 AM, Jan van Doorn  wrote:

> I'm a little late to this party, sorry
>
>
> Rob, you say: 'Note this does NOT replace or affect the "API Gateway"
> plans, or "JWT Auth" plans. This is a rewrite of strictly the existing
> Traffic Ops application, and can be implemented alongside the "API
> Gateway", or "JWT Auth", or without them.'
>
> Why? Isn't this something we could do in that gateway as well, and doesn't
> this make moving to that harder?
>
> Rgds,
> JvD
>
>
> On Thu, Jul 20, 2017 at 7:52 AM Dewayne Richardson 
> wrote:
>
>> When:   Read · Thu, Jul 20.
>> 
>> [image: Timyo expectation line]
>> @malenfent We plan to re-evaluate postinstall as we move forward as
>> well.  We've noted the feedback about making SSL configuration separate,
>> and over time the extreme pain of Carton will disappear as well, as for
>> "downloadwebdeps" will also disappear because of Jeremy's new TO UI v2.
>> So, really it'll be a small postinstall wrapper that basically sets up the
>> database (with a default admin user) and minimal data setup.
>>
>> On Thu, Jul 20, 2017 at 7:30 AM, Robert Butts 
>> wrote:
>>
>>> @malenfant it sounds like Dewayne and Jeff reached a consensus, so I
>>> think the plan now, unless you object, is to use a single RPM and Service.
>>> Which, I agree with Jeff, it does make a lot of things easier, it also
>>> makes the final Perl app removal much easier, if we're willing to tolerate
>>> the temporary coupling.
>>>
>>> I think that answers most of your questions, right?
>>>
>>> So, if they're in the same RPM and Service, the New TO can use the same
>>> configs, adding the new port to the old config--which won't require
>>> Postinstall if you set it in Ansible.
>>>
>>> The new binary will go in the same `/opt/traffic_ops` dir.
>>>
>>> On Jul 20, 2017 6:38 AM, "Eric Friedrich (efriedri)" 
>>> wrote:
>>>
 Sorry bout the nitpicking...

 Whatever approach we choose for naming, we should use “go” instead of
 “golang”.

 "The language is called Go, not Golang. http://golang.org
  is just the the web site address, not the
 name of the language.” - Rob Pike
 https://twitter.com/rob_pike/status/886054143235719169

 —Eric

 On Jul 20, 2017, at 8:04 AM, Steve Malenfant 
 wrote:

 +1 to move to Golang. Few questions.

 Configuration seems like a none issue for us since we are using
 Ansible, we
 would prefer less postinstall interaction as possible as long as the
 configuration is documented.
 As for the name, do we plan to keep "traffic_ops_golang" in the future?
 Have we thought about using "traffic_ops" for golang and
 "traffic_ops_something" for Perl? It seems like the "Perl TO" should be
 the
 dependency here and should be handled by the RPM at installation. There
 could be confusion about folks not installing "traffic_ops_golang" on
 upgrades since they are doing a "yum update traffic_ops".

 If the dependency is this way, the golang version will need to be
 installed
 and the perl version as well. But you don't need to run any postinstall
 and
 simply need to change golang configuration to change the proxy.

 Talking about postinstall and migrations (SQL), which package would
 contain
 it in the future? The Perl or TO version?

 Is there something preventing re-using the same /opt/traffic_ops
 directory
 for both?

 Steve

 On Wed, Jul 19, 2017 at 5:25 PM, Dewayne Richardson 
 wrote:

 When:   Read · Wed, Jul 19. If Cc: read
 
 [image: Timyo expectation line]
 I had a knee jerk reaction to separate the RPMs because of the potential
 for a "new" approach, but I've now been convinced otherwise based upon
 the
 "Strangler Pattern" where the Perl gets rewritten into Golang overtime
 and
 the TO clients know none the difference.

 +1

Re: Traffic Ops Golang Migration Proposal

2017-07-20 Thread Eric Friedrich (efriedri) 
Sorry bout the nitpicking...

Whatever approach we choose for naming, we should use “go” instead of “golang”.

"The language is called Go, not Golang. http://golang.org 
  is just the the web site address, not the name of 
the language.” - Rob Pike
https://twitter.com/rob_pike/status/886054143235719169

—Eric

On Jul 20, 2017, at 8:04 AM, Steve Malenfant 
> wrote:

+1 to move to Golang. Few questions.

Configuration seems like a none issue for us since we are using Ansible, we
would prefer less postinstall interaction as possible as long as the
configuration is documented.
As for the name, do we plan to keep "traffic_ops_golang" in the future?
Have we thought about using "traffic_ops" for golang and
"traffic_ops_something" for Perl? It seems like the "Perl TO" should be the
dependency here and should be handled by the RPM at installation. There
could be confusion about folks not installing "traffic_ops_golang" on
upgrades since they are doing a "yum update traffic_ops".

If the dependency is this way, the golang version will need to be installed
and the perl version as well. But you don't need to run any postinstall and
simply need to change golang configuration to change the proxy.

Talking about postinstall and migrations (SQL), which package would contain
it in the future? The Perl or TO version?

Is there something preventing re-using the same /opt/traffic_ops directory
for both?

Steve

On Wed, Jul 19, 2017 at 5:25 PM, Dewayne Richardson 
>
wrote:

When:   Read · Wed, Jul 19. If Cc: read

[image: Timyo expectation line]
I had a knee jerk reaction to separate the RPMs because of the potential
for a "new" approach, but I've now been convinced otherwise based upon the
"Strangler Pattern" where the Perl gets rewritten into Golang overtime and
the TO clients know none the difference.

+1

-Dew

On Wed, Jul 19, 2017 at 12:52 PM, Jeff Elsloo 
> wrote:

I see. If that's the case, it's a hard requirement to run the golang
portion from whenever this is introduced onward. As long as we have
discipline around removing migrated routes, that should work okay and
would solve the "two watches" issue Mark mentioned when we discussed
this in person: A man with two watches (old API route, golang route)
does not know what time it is.

I don't think that it makes a lot of sense to have a separate RPM
since the dependency goes the other direction, and users are required
to run the golang component no matter what. We might as well just
build that into the existing RPM build process for traffic_ops.

Do we really need to ask the user for the port to move mojo to?
Obviously we can ask them to provide a port, but we could also just
pick a random, unused high port, and have mojo listen only on the
loopback interface. Maybe that's too "magical"?

Does the golang app run as trafops:trafops and drop privileges after
opening :443?
--
Thanks,
Jeff


On Wed, Jul 19, 2017 at 11:58 AM, Robert Butts 
>
wrote:
This means that the traffic_ops side would have to check to see whether
traffic_ops_golang

Because the traffic_ops_golang package will depend on traffic_ops, not
the
other way around

I was suggesting the other way around - traffic_ops will depend on
traffic_ops_golang. Which means upgrading traffic_ops automatically
installs
traffic_ops_golang, and we don't need to do the check. It'd mean you
couldn't remove `traffic_ops_golang`, but the plan is to remove old
endpoints from old TO anyway. Which is another reason making
traffic_ops_golang a dependency of traffic_ops makes sense: it really
is,
traffic_ops really does require it for the migrated endpoints.

I agree with moving away from manual post-installation scripts, but I
don't
think we can avoid it here, because we need the user to set a new port.


On Wed, Jul 19, 2017 at 11:40 AM, Jeff Elsloo 
>
wrote:

I'm +1 on most of what you suggest, except for doing the takeover in
postinstall in traffic_ops.

While we can do whatever we want with postinstall, I think it's
awkward to have a tool within the traffic_ops package configuring
something under the traffic_ops_golang package, when the latter
package might not be installed. This means that the traffic_ops side
would have to check to see whether traffic_ops_golang is installed
outside of the normal RPM dependencies, adding more platform specific
code to postinstall. I don't see a generic way to implement the
"check" for the golang package within postinstall that will work. We
would have to check for a path that is not likely to exist for most
users, or check for a package. Both approaches require platform
specific code and assumptions.

Because the traffic_ops_golang package will depend on traffic_ops, not

Re: Traffic Ops Golang Migration Proposal

2017-07-20 Thread Steve Malenfant 
+1 to move to Golang. Few questions.

Configuration seems like a none issue for us since we are using Ansible, we
would prefer less postinstall interaction as possible as long as the
configuration is documented.
As for the name, do we plan to keep "traffic_ops_golang" in the future?
Have we thought about using "traffic_ops" for golang and
"traffic_ops_something" for Perl? It seems like the "Perl TO" should be the
dependency here and should be handled by the RPM at installation. There
could be confusion about folks not installing "traffic_ops_golang" on
upgrades since they are doing a "yum update traffic_ops".

If the dependency is this way, the golang version will need to be installed
and the perl version as well. But you don't need to run any postinstall and
simply need to change golang configuration to change the proxy.

Talking about postinstall and migrations (SQL), which package would contain
it in the future? The Perl or TO version?

Is there something preventing re-using the same /opt/traffic_ops directory
for both?

Steve

On Wed, Jul 19, 2017 at 5:25 PM, Dewayne Richardson 
wrote:

> When:   Read · Wed, Jul 19. If Cc: read
> 
> [image: Timyo expectation line]
> I had a knee jerk reaction to separate the RPMs because of the potential
> for a "new" approach, but I've now been convinced otherwise based upon the
> "Strangler Pattern" where the Perl gets rewritten into Golang overtime and
> the TO clients know none the difference.
>
> +1
>
> -Dew
>
> On Wed, Jul 19, 2017 at 12:52 PM, Jeff Elsloo  wrote:
>
>> I see. If that's the case, it's a hard requirement to run the golang
>> portion from whenever this is introduced onward. As long as we have
>> discipline around removing migrated routes, that should work okay and
>> would solve the "two watches" issue Mark mentioned when we discussed
>> this in person: A man with two watches (old API route, golang route)
>> does not know what time it is.
>>
>> I don't think that it makes a lot of sense to have a separate RPM
>> since the dependency goes the other direction, and users are required
>> to run the golang component no matter what. We might as well just
>> build that into the existing RPM build process for traffic_ops.
>>
>> Do we really need to ask the user for the port to move mojo to?
>> Obviously we can ask them to provide a port, but we could also just
>> pick a random, unused high port, and have mojo listen only on the
>> loopback interface. Maybe that's too "magical"?
>>
>> Does the golang app run as trafops:trafops and drop privileges after
>> opening :443?
>> --
>> Thanks,
>> Jeff
>>
>>
>> On Wed, Jul 19, 2017 at 11:58 AM, Robert Butts 
>> wrote:
>> >> This means that the traffic_ops side would have to check to see whether
>> >> traffic_ops_golang
>> >
>> >> Because the traffic_ops_golang package will depend on traffic_ops, not
>> the
>> >> other way around
>> >
>> > I was suggesting the other way around - traffic_ops will depend on
>> > traffic_ops_golang. Which means upgrading traffic_ops automatically
>> installs
>> > traffic_ops_golang, and we don't need to do the check. It'd mean you
>> > couldn't remove `traffic_ops_golang`, but the plan is to remove old
>> > endpoints from old TO anyway. Which is another reason making
>> > traffic_ops_golang a dependency of traffic_ops makes sense: it really
>> is,
>> > traffic_ops really does require it for the migrated endpoints.
>> >
>> > I agree with moving away from manual post-installation scripts, but I
>> don't
>> > think we can avoid it here, because we need the user to set a new port.
>> >
>> >
>> > On Wed, Jul 19, 2017 at 11:40 AM, Jeff Elsloo 
>> wrote:
>> >>
>> >> I'm +1 on most of what you suggest, except for doing the takeover in
>> >> postinstall in traffic_ops.
>> >>
>> >> While we can do whatever we want with postinstall, I think it's
>> >> awkward to have a tool within the traffic_ops package configuring
>> >> something under the traffic_ops_golang package, when the latter
>> >> package might not be installed. This means that the traffic_ops side
>> >> would have to check to see whether traffic_ops_golang is installed
>> >> outside of the normal RPM dependencies, adding more platform specific
>> >> code to postinstall. I don't see a generic way to implement the
>> >> "check" for the golang package within postinstall that will work. We
>> >> would have to check for a path that is not likely to exist for most
>> >> users, or check for a package. Both approaches require platform
>> >> specific code and assumptions.
>> >>
>> >> Because the traffic_ops_golang package will depend on traffic_ops, not
>> >> the other way around, it makes more sense to place the configuration
>> >> piece in the golang package. When the golang package is installed, we
>> >> can "take over" the port in the listen directive of cdn.conf, because
>> >> we know for a fact

Re: Traffic Ops Golang Migration Proposal

2017-07-19 Thread Dewayne Richardson 
When:   Read · Wed, Jul 19. If Cc: read

[image: Timyo expectation line]
I had a knee jerk reaction to separate the RPMs because of the potential
for a "new" approach, but I've now been convinced otherwise based upon the
"Strangler Pattern" where the Perl gets rewritten into Golang overtime and
the TO clients know none the difference.

+1

-Dew

On Wed, Jul 19, 2017 at 12:52 PM, Jeff Elsloo  wrote:

> I see. If that's the case, it's a hard requirement to run the golang
> portion from whenever this is introduced onward. As long as we have
> discipline around removing migrated routes, that should work okay and
> would solve the "two watches" issue Mark mentioned when we discussed
> this in person: A man with two watches (old API route, golang route)
> does not know what time it is.
>
> I don't think that it makes a lot of sense to have a separate RPM
> since the dependency goes the other direction, and users are required
> to run the golang component no matter what. We might as well just
> build that into the existing RPM build process for traffic_ops.
>
> Do we really need to ask the user for the port to move mojo to?
> Obviously we can ask them to provide a port, but we could also just
> pick a random, unused high port, and have mojo listen only on the
> loopback interface. Maybe that's too "magical"?
>
> Does the golang app run as trafops:trafops and drop privileges after
> opening :443?
> --
> Thanks,
> Jeff
>
>
> On Wed, Jul 19, 2017 at 11:58 AM, Robert Butts 
> wrote:
> >> This means that the traffic_ops side would have to check to see whether
> >> traffic_ops_golang
> >
> >> Because the traffic_ops_golang package will depend on traffic_ops, not
> the
> >> other way around
> >
> > I was suggesting the other way around - traffic_ops will depend on
> > traffic_ops_golang. Which means upgrading traffic_ops automatically
> installs
> > traffic_ops_golang, and we don't need to do the check. It'd mean you
> > couldn't remove `traffic_ops_golang`, but the plan is to remove old
> > endpoints from old TO anyway. Which is another reason making
> > traffic_ops_golang a dependency of traffic_ops makes sense: it really is,
> > traffic_ops really does require it for the migrated endpoints.
> >
> > I agree with moving away from manual post-installation scripts, but I
> don't
> > think we can avoid it here, because we need the user to set a new port.
> >
> >
> > On Wed, Jul 19, 2017 at 11:40 AM, Jeff Elsloo  wrote:
> >>
> >> I'm +1 on most of what you suggest, except for doing the takeover in
> >> postinstall in traffic_ops.
> >>
> >> While we can do whatever we want with postinstall, I think it's
> >> awkward to have a tool within the traffic_ops package configuring
> >> something under the traffic_ops_golang package, when the latter
> >> package might not be installed. This means that the traffic_ops side
> >> would have to check to see whether traffic_ops_golang is installed
> >> outside of the normal RPM dependencies, adding more platform specific
> >> code to postinstall. I don't see a generic way to implement the
> >> "check" for the golang package within postinstall that will work. We
> >> would have to check for a path that is not likely to exist for most
> >> users, or check for a package. Both approaches require platform
> >> specific code and assumptions.
> >>
> >> Because the traffic_ops_golang package will depend on traffic_ops, not
> >> the other way around, it makes more sense to place the configuration
> >> piece in the golang package. When the golang package is installed, we
> >> can "take over" the port in the listen directive of cdn.conf, because
> >> we know for a fact that it is on disk because of the RPM dependency on
> >> traffic_ops. We also know that cdn.conf will be left alone if/when
> >> traffic_ops is upgraded due to being marked as a config file. If the
> >> user has installed either component outside of the normal RPM process,
> >> they will have to figure out how to run the golang package separately,
> >> as one would expect.
> >>
> >> We can do the configuration during the postinstall step of the
> >> traffic_ops_golang RPM. It's advantageous to manage that piece within
> >> the RPM, because if, for example, one wanted to remove the golang
> >> portion, we could have a postuninstall step that reverts changes made
> >> to cdn.conf (put the port we took over back into cdn.conf). We could
> >> seamlessly add and remove the traffic_ops_golang component without
> >> disturbing anything in traffic_ops, and without having to run some
> >> script manually. The platform specific things that would need to be
> >> done in postinstall should be done in the RPM, because then we know
> >> for sure which platform we're on, and assumptions about packages and
> >> paths will be accurate.
> >>
> >> Ideally we should be moving away from any manual run of any script
> >> after an

Re: Traffic Ops Golang Migration Proposal

2017-07-19 Thread Jeff Elsloo 
I see. If that's the case, it's a hard requirement to run the golang
portion from whenever this is introduced onward. As long as we have
discipline around removing migrated routes, that should work okay and
would solve the "two watches" issue Mark mentioned when we discussed
this in person: A man with two watches (old API route, golang route)
does not know what time it is.

I don't think that it makes a lot of sense to have a separate RPM
since the dependency goes the other direction, and users are required
to run the golang component no matter what. We might as well just
build that into the existing RPM build process for traffic_ops.

Do we really need to ask the user for the port to move mojo to?
Obviously we can ask them to provide a port, but we could also just
pick a random, unused high port, and have mojo listen only on the
loopback interface. Maybe that's too "magical"?

Does the golang app run as trafops:trafops and drop privileges after
opening :443?
--
Thanks,
Jeff


On Wed, Jul 19, 2017 at 11:58 AM, Robert Butts  wrote:
>> This means that the traffic_ops side would have to check to see whether
>> traffic_ops_golang
>
>> Because the traffic_ops_golang package will depend on traffic_ops, not the
>> other way around
>
> I was suggesting the other way around - traffic_ops will depend on
> traffic_ops_golang. Which means upgrading traffic_ops automatically installs
> traffic_ops_golang, and we don't need to do the check. It'd mean you
> couldn't remove `traffic_ops_golang`, but the plan is to remove old
> endpoints from old TO anyway. Which is another reason making
> traffic_ops_golang a dependency of traffic_ops makes sense: it really is,
> traffic_ops really does require it for the migrated endpoints.
>
> I agree with moving away from manual post-installation scripts, but I don't
> think we can avoid it here, because we need the user to set a new port.
>
>
> On Wed, Jul 19, 2017 at 11:40 AM, Jeff Elsloo  wrote:
>>
>> I'm +1 on most of what you suggest, except for doing the takeover in
>> postinstall in traffic_ops.
>>
>> While we can do whatever we want with postinstall, I think it's
>> awkward to have a tool within the traffic_ops package configuring
>> something under the traffic_ops_golang package, when the latter
>> package might not be installed. This means that the traffic_ops side
>> would have to check to see whether traffic_ops_golang is installed
>> outside of the normal RPM dependencies, adding more platform specific
>> code to postinstall. I don't see a generic way to implement the
>> "check" for the golang package within postinstall that will work. We
>> would have to check for a path that is not likely to exist for most
>> users, or check for a package. Both approaches require platform
>> specific code and assumptions.
>>
>> Because the traffic_ops_golang package will depend on traffic_ops, not
>> the other way around, it makes more sense to place the configuration
>> piece in the golang package. When the golang package is installed, we
>> can "take over" the port in the listen directive of cdn.conf, because
>> we know for a fact that it is on disk because of the RPM dependency on
>> traffic_ops. We also know that cdn.conf will be left alone if/when
>> traffic_ops is upgraded due to being marked as a config file. If the
>> user has installed either component outside of the normal RPM process,
>> they will have to figure out how to run the golang package separately,
>> as one would expect.
>>
>> We can do the configuration during the postinstall step of the
>> traffic_ops_golang RPM. It's advantageous to manage that piece within
>> the RPM, because if, for example, one wanted to remove the golang
>> portion, we could have a postuninstall step that reverts changes made
>> to cdn.conf (put the port we took over back into cdn.conf). We could
>> seamlessly add and remove the traffic_ops_golang component without
>> disturbing anything in traffic_ops, and without having to run some
>> script manually. The platform specific things that would need to be
>> done in postinstall should be done in the RPM, because then we know
>> for sure which platform we're on, and assumptions about packages and
>> paths will be accurate.
>>
>> Ideally we should be moving away from any manual run of any script
>> after an installation or upgrade, including postinstall, if that work
>> can be done within the RPM.
>> --
>> Thanks,
>> Jeff
>>
>>
>> On Wed, Jul 19, 2017 at 9:08 AM, Robert Butts 
>> wrote:
>> > It sounds like the only -1 is having a unified Service and RPM.
>> >
>> > How about the following compromise: a separate RPM and Service for the
>> > New
>> > TO, and the New TO RPM is a dependency of Old TO, and likewise the
>> > Service
>> > is a dependency of `traffic_ops`. This way, upgrades will still require
>> > building the New TO RPM and adding it to Yum, but `yum upgrade` will
>> > automatically install it without additional ops

Re: Traffic Ops Golang Migration Proposal

2017-07-19 Thread Jeff Elsloo 
I'm +1 on most of what you suggest, except for doing the takeover in
postinstall in traffic_ops.

While we can do whatever we want with postinstall, I think it's
awkward to have a tool within the traffic_ops package configuring
something under the traffic_ops_golang package, when the latter
package might not be installed. This means that the traffic_ops side
would have to check to see whether traffic_ops_golang is installed
outside of the normal RPM dependencies, adding more platform specific
code to postinstall. I don't see a generic way to implement the
"check" for the golang package within postinstall that will work. We
would have to check for a path that is not likely to exist for most
users, or check for a package. Both approaches require platform
specific code and assumptions.

Because the traffic_ops_golang package will depend on traffic_ops, not
the other way around, it makes more sense to place the configuration
piece in the golang package. When the golang package is installed, we
can "take over" the port in the listen directive of cdn.conf, because
we know for a fact that it is on disk because of the RPM dependency on
traffic_ops. We also know that cdn.conf will be left alone if/when
traffic_ops is upgraded due to being marked as a config file. If the
user has installed either component outside of the normal RPM process,
they will have to figure out how to run the golang package separately,
as one would expect.

We can do the configuration during the postinstall step of the
traffic_ops_golang RPM. It's advantageous to manage that piece within
the RPM, because if, for example, one wanted to remove the golang
portion, we could have a postuninstall step that reverts changes made
to cdn.conf (put the port we took over back into cdn.conf). We could
seamlessly add and remove the traffic_ops_golang component without
disturbing anything in traffic_ops, and without having to run some
script manually. The platform specific things that would need to be
done in postinstall should be done in the RPM, because then we know
for sure which platform we're on, and assumptions about packages and
paths will be accurate.

Ideally we should be moving away from any manual run of any script
after an installation or upgrade, including postinstall, if that work
can be done within the RPM.
--
Thanks,
Jeff


On Wed, Jul 19, 2017 at 9:08 AM, Robert Butts  wrote:
> It sounds like the only -1 is having a unified Service and RPM.
>
> How about the following compromise: a separate RPM and Service for the New
> TO, and the New TO RPM is a dependency of Old TO, and likewise the Service
> is a dependency of `traffic_ops`. This way, upgrades will still require
> building the New TO RPM and adding it to Yum, but `yum upgrade` will
> automatically install it without additional ops work, and `service
> traffic_ops start` will also start the New TO. Bearing in mind this
> double-service awkwardness will go away when all endpoints are migrated.
>
> Also, @alficles suggested configuring the New TO Config in Postinstall
> (which must be run after upgrading anyway). Because the New is a dependency
> of the Old, we're guaranteed `/opt/traffic_ops_golang` exists in
> Postinstall, and can populate its config.
>
> Also, I realized the New TO needs moved from
> `/traffic_ops/traffic_ops_golang` to `/traffic_ops_golang`, because
> `build_all.sh` requires projects be in the root directory.
>
> Also, I will add tests, docs, and configurable logging before the PR is
> merged. (Just wanted to wait until we had consensus before putting more work
> into it.)
>
> How does that sound?
>
>
> On Thu, Jul 13, 2017 at 1:00 PM, Robert Butts 
> wrote:
>>
>> @dewrich It's not that putting both services in the same Service and RPM
>> is a good long-term solution; it's that it's easier to configure and deploy.
>> A separate RPM and service is another step to install, and another step to
>> start the service. It's not that it's good, it's that it's the lesser of two
>> evils. My fear is, the more complex we make this, the less chance of getting
>> it done at all.
>>
>> @efriedri
>> 1. Same answer: long-term, I absolutely agree proxies should be separate.
>> But this is the simplest way to deploy an incremental migration.
>> 2. Yes, the Perl GUI can transparently request what it needs. That's a
>> goal of this: transparent new endpoints, that existing services don't know
>> or care that they come from a different backend.
>>
>> Also bear in mind, this "proxy" only exists until Perl TO goes away.
>> Having it in the same Service and RPM as old TO, and having the proxy be the
>> same binary as the new endpoints, makes it easier to completely remove the
>> Proxy and Perl, once every endpoint is rewritten. If we make the Service and
>> RPM separate, you have to change configs and uninstall the separate Service
>> and RPM of the Perl TO. If the Proxy is a separate binary, you have to
>> uninstall that, and then change

Fwd: Traffic Ops Golang Migration Proposal

2017-07-13 Thread Robert Butts 
-- Forwarded message --
From: Durfey, Ryan <ryan_dur...@comcast.com>
Date: Thu, Jul 13, 2017 at 9:26 AM
Subject: Re: Traffic Ops Golang Migration Proposal
To: "us...@trafficcontrol.incubator.apache.org" <
us...@trafficcontrol.incubator.apache.org>


I am +1 on this.  We are actively discussing how config management should
work as we move forward and think a rewrite gives us a better opportunity
to overhaul how it works.



Also, should this go to dev@trafficcontrol.incubator.apache.org vs. users?



*Ryan Durfey*M | 303-524-5099 <(303)%20524-5099>

CDN Support (24x7): 866-405-2993 <(866)%20405-2993> or
cdn_supp...@comcast.com





*From: *Jason Tucker <jasonwtuc...@gmail.com>
*Reply-To: *"us...@trafficcontrol.incubator.apache.org" <
us...@trafficcontrol.incubator.apache.org>, "jasonwtuc...@gmail.com" <
jasonwtuc...@gmail.com>
*Date: *Thursday, July 13, 2017 at 9:18 AM
*To: *"us...@trafficcontrol.incubator.apache.org" <users@trafficcontrol.
incubator.apache.org>
*Subject: *Re: Traffic Ops Golang Migration Proposal



While I have sentimental attachments to perl, I'm +1 on moving to a
non-perl alternative. From an administrator's perspective, anything that
helps me avoid perl module dependency tangles is a win.

__Jason



On Thu, Jul 13, 2017 at 11:11 AM, Robert Butts <robert.o.bu...@gmail.com>
wrote:

I'd like to propose a plan for rewriting Traffic Ops in Golang.

It's been a goal of Traffic Control for years now to rewrite the aging Perl
Traffic Ops app. We've made numerous prototypes and attempts, none of which
have made it past the prototype stage.

See
https://github.com/apache/incubator-trafficcontrol/tree/master/traffic_ops/goto

https://github.com/apache/incubator-trafficcontrol/tree/master/traffic_ops/experimental/server

https://github.com/apache/incubator-trafficcontrol/tree/master/traffic_ops/experimental/postgrest

https://github.com/apache/incubator-trafficcontrol/tree/master/traffic_ops/experimental/traffic_ops_auth

https://github.com/apache/incubator-trafficcontrol/tree/master/traffic_ops/
experimental/url-rewriter-nginx
https://github.com/apache/incubator-trafficcontrol/tree/master/traffic_ops/
experimental/go-monitoring

I believe this is primarily because (1) Traffic Ops is too large to rewrite
all at once, and (2) Traffic Ops requires considerable operations tooling
(RPMs, Config files [Puppet/Ansible/etc], Postinstall, etc), and none of
the prototypes were operationally simple or complete.

To this end, I asked the question, "What is the operationally simplest
application we can create, which enables incrementally rewriting Traffic
Ops?" This is the answer I came up with:

A Golang application which serves endpoints which have been rewritten, and
reverse-proxies the old Perl Traffic Ops for all other requests. This app
can be included in the RPM and Service files for Traffic Ops. Then, the old
Traffic Ops config can be changed to a different port (say, 60443), and the
new Go Traffic Ops can be configured to serve on 443, and reverse-proxy to
the old application. Both applications will run on the same machine.

The new app will accept, understand, and authenticate Mojolicious cookies
the same way the old Perl TO does, including refreshing cookies with a new
expiration in the reply.



In this way, the new TO can be deployed trivially, only requiring a
one-line config change in the old TO, and a single new config file for the
new TO (e.g. in Puppet).



Then, API endpoints can be rewritten one-by-one, until the new TO has all
endpoints and the old Perl TO can simply be removed.

Note this does NOT replace or affect the "API Gateway" plans, or "JWT Auth"
plans. This is a rewrite of strictly the existing Traffic Ops application,
and can be implemented alongside the "API Gateway", or "JWT Auth", or
without them.



Everything I just proposed is already written, including the Auth, RPM, and
Service. Initially, it serves `/api/1.2/cdns/{cdn}/configs/monitoring.json`
and proxies everything else. The PR is at
https://github.com/apache/incubator-trafficcontrol/pull/729



I believe this to be the simplest, easiest, fastest, and cheapest way to
rewrite Traffic Ops, and likely the only way we will accomplish rewriting
the massive Perl application.



Thoughts?