Re: Goose installer script

[Dave Neuman]

I don't see why re-writing the APIs in something like golang would mean
that we also need to re-write the database admin script.  I think those two
things are mutually exclusive, right?

On Fri, Apr 28, 2017 at 12:29 PM, Dewayne Richardson 
wrote:

> I had that thought, as well as there are more recent versions like
> https://github.com/mattes/migrate.  The question becomes if we ever get
> around to rewriting TrafficOps APIs in golang, will the Perl version then
> become obsolete?
>
> On Fri, Apr 28, 2017 at 11:58 AM, Dave Neuman  wrote:
>
> > Maybe it's time we take a look at what goose really buys us and consider
> > writing our own database migration tool.  We already have admin.pl, it
> > could probably fit in with that?
> >
> > On Fri, Apr 28, 2017 at 11:45 AM, Eric Friedrich (efriedri) <
> > efrie...@cisco.com> wrote:
> >
> > > Hey Dew-
> > >   What calls this script?
> > >
> > > If its called from the Traffic Ops Spec file, then this will cause some
> > > pain for those of us that need to install without internet access.
> > >
> > > —Eric
> > >
> > > > On Apr 28, 2017, at 12:41 PM, Dewayne Richardson 
> > > wrote:
> > > >
> > > > I'm working toward a more streamlined installation process for
> Traffic
> > > Ops
> > > > (internally) and publicly. Of course, the same hiccups that everyone
> > else
> > > > runs into I am as well.  Installation of Golang (proper version) and
> > > > installation of Goose.  Goose has been the most challenging for
> several
> > > > reasons.  The maintainer hasn't made any real changes since 2015, and
> > has
> > > > not "branched" his code to allow for explicit version download.  Per
> > his
> > > > installation instructions "go get bitbucket.org/liamstask/goose/
> > > cmd/goose"
> > > >
> > > > So I'm I'm proposing to write an installer script in bash to help
> > > automate
> > > > the Golang install as well as the Goose install.  My only concern (as
> > > well
> > > > as most of yours) is "go get" will grab the latest, but since no real
> > > > changes have happened I'm left with no other option.
> > > >
> > > > Proposed:
> > > >
> > > > /opt/traffic_ops/install/bin/install_goose.sh
> > > >
> > > > - Install Golang (version 1.8.x)
> > > > - go get bitbucket.org/liamstask/goose/cmd/goose
> > > >
> > > > Thoughts?
> > > >
> > > > -Dew
> > >
> > >
> >
>

Compiling TC 2.0.0 Release Notes

[Eric Friedrich]

Hey All-
  I'm putting together the release notes for the 2.0 release. There's about
1,041 commits since 1.8, which are way too many for me to filter through
myself.

If you committed a feature you would like mentioned in the release notes
(or you think our users need to know about), please respond to the email
with a brief 1-2 sentence description of the feature and the commit hash if
you want.


Thanks,
Eric

Re: Proposed changes to xml_id on a delivery service (from the API perspective)

[Jeremy Mitchell]

Great feedback. It sounds like people are OK with making xml_id immutable
on a delivery service... but not so much about making the host header regex
at position 0 immutable (the regex that requires this format -
.*\xml-id\..*).

Just to revisit:

You create a delivery service with xml_id=foo-bar and by default you get a
host header regex in this format at position 0 - .*\xml-id\..* so you end
up with URL=edge|tr.foo-bar.cdn.domain.com

Maybe you're happy with that and life is good.

Maybe you decide you want your URL to be edge|
tr.something-else.cdn.domain.com instead.so you need the ability to
edit the host header regex at position 0 to .*\something-else\..*

What about this? You can attach Static DNS entries to a delivery service.
Can Static DNS entries be used to accomplish this use case while leaving
.*\xml-id\..* alone? (Elsloo keyed me into that possibility)

Jeremy


On Fri, Apr 28, 2017 at 9:44 AM, Dave Neuman  wrote:

> I don't think we should be forcing users into using their xml_id as the
> host_regex used to create the delivery URL.  I think we can offer to create
> it for them, but we should also let users have the ability to define what
> their regex needs to be when they create their delivery service.  We have
> instances in our environment where the host regex does not match the xml_id
> a delivery service and I don't think we should take that functionality
> away.
> You are correct, changing an xml_id or a host_regex could have unintended
> consequences in the system, but I agree with Ori that not letting users
> change it can cause other problems.  I think we should allow users to
> change it but maybe we need to figure out a way to only allow "super
> special advanced" users to do it.
>
> --Dave
>
> On Thu, Apr 27, 2017 at 6:40 AM, Ori Finkelman  wrote:
>
> > Hi Jeremy,
> >
> > I would like to refer to bullet 2 in your proposal.
> > Coupling between the xml_id and the host regexp, making the host regexp
> > immutable may create undesired limitations.
> > For example, if, for any reason, the content provider has to change the
> > host regexp, she will have to create a new DS. This limitation prevents
> > long term reports on DS volume, running long term analytics and other
> > operations that you may want to run over long periods of time.
> > In general, the host regexp is an attribute of the DS and therefore,
> unlike
> > the id which should be immutable, I think it should be mutable, like any
> > other attribute.
> >
> > A side issue is that we would like to suggest promoting TC-55
> > 
> > in order to allow DS to be
> > matched by a combination of host regexp and path regexp. TC-55
> >  requires it should be
> > allowed
> > to have the same host regexp for different DS, using the path regexp as
> > differentiator. This option has the benefit of using a single certificate
> > for multiple DSs, it also reduces the complexity of delegating traffic
> > between CDNs, like the use cases of CDNi and Open Caching.
> > Unless I misunderstand the proposal, binding the first host regexp with
> the
> > xml_id makes it impossible to have the same host for different DS, as
> > required by TC-55 .
> >
> > Thanks,
> > Ori
> >
> > On Thu, Apr 27, 2017 at 6:37 AM, Zhilin Huang (zhilhuan) <
> > zhilh...@cisco.com
> > > wrote:
> >
> > > Hi Jeremy,
> > >
> > > I like the idea that make xml_id and ds.regex at position 0 immutable.
> > > Actually we have suffered some issues on HTTPs delivery services, and I
> > had
> > > ever created issue TC-187 with PR #360.
> > >
> > > Besides ds.regex at position 0, the ds.regex at other position of type
> > > HOST_REGEXP is not working for HTTPs. I think the reason is SAN is no
> > > supported in currently implementation in “generate_ssl_keys”. Do we
> have
> > > any plan to support multiple subdomain for HTTPs?
> > >
> > > If we do plan to support SAN, I think we need also consider the
> > > modification for ds.regex at position other than 0.
> > >
> > >
> > > Thanks,
> > > Zhilin
> > >
> > >
> > >
> > > On 4/27/17, 1:41 AM, "Jeremy Mitchell"  wrote:
> > >
> > > As we port functionality from the existing TO UI to the TO API, I
> am
> > > always
> > > looking for ways to simplify things. I can't lie. I like simple.
> > Plus,
> > > simple == less user error. I would like to discuss the possibility
> of
> > > making changes to how xml_id works (at least thru the API). Here's
> > > what I
> > > propose. Feel free to poke holes in it.
> > >
> > > 1. make delivery service xml_id immutable. If you create a delivery
> > > service
> > > with xml_id = foo-bar, that is it. no changing itever...
> > > 2. the ds.regex in position 0 must always be of type HOST_REGEXP
> and
> > in
> > > this format - 

Re: Goose installer script

[Dewayne Richardson]

I had that thought, as well as there are more recent versions like
https://github.com/mattes/migrate.  The question becomes if we ever get
around to rewriting TrafficOps APIs in golang, will the Perl version then
become obsolete?

On Fri, Apr 28, 2017 at 11:58 AM, Dave Neuman  wrote:

> Maybe it's time we take a look at what goose really buys us and consider
> writing our own database migration tool.  We already have admin.pl, it
> could probably fit in with that?
>
> On Fri, Apr 28, 2017 at 11:45 AM, Eric Friedrich (efriedri) <
> efrie...@cisco.com> wrote:
>
> > Hey Dew-
> >   What calls this script?
> >
> > If its called from the Traffic Ops Spec file, then this will cause some
> > pain for those of us that need to install without internet access.
> >
> > —Eric
> >
> > > On Apr 28, 2017, at 12:41 PM, Dewayne Richardson 
> > wrote:
> > >
> > > I'm working toward a more streamlined installation process for Traffic
> > Ops
> > > (internally) and publicly. Of course, the same hiccups that everyone
> else
> > > runs into I am as well.  Installation of Golang (proper version) and
> > > installation of Goose.  Goose has been the most challenging for several
> > > reasons.  The maintainer hasn't made any real changes since 2015, and
> has
> > > not "branched" his code to allow for explicit version download.  Per
> his
> > > installation instructions "go get bitbucket.org/liamstask/goose/
> > cmd/goose"
> > >
> > > So I'm I'm proposing to write an installer script in bash to help
> > automate
> > > the Golang install as well as the Goose install.  My only concern (as
> > well
> > > as most of yours) is "go get" will grab the latest, but since no real
> > > changes have happened I'm left with no other option.
> > >
> > > Proposed:
> > >
> > > /opt/traffic_ops/install/bin/install_goose.sh
> > >
> > > - Install Golang (version 1.8.x)
> > > - go get bitbucket.org/liamstask/goose/cmd/goose
> > >
> > > Thoughts?
> > >
> > > -Dew
> >
> >
>

Re: Goose installer script

[Dewayne Richardson]

The user, we'd document the installation steps

On Fri, Apr 28, 2017 at 11:45 AM, Eric Friedrich (efriedri) <
efrie...@cisco.com> wrote:

> Hey Dew-
>   What calls this script?
>
> If its called from the Traffic Ops Spec file, then this will cause some
> pain for those of us that need to install without internet access.
>
> —Eric
>
> > On Apr 28, 2017, at 12:41 PM, Dewayne Richardson 
> wrote:
> >
> > I'm working toward a more streamlined installation process for Traffic
> Ops
> > (internally) and publicly. Of course, the same hiccups that everyone else
> > runs into I am as well.  Installation of Golang (proper version) and
> > installation of Goose.  Goose has been the most challenging for several
> > reasons.  The maintainer hasn't made any real changes since 2015, and has
> > not "branched" his code to allow for explicit version download.  Per his
> > installation instructions "go get bitbucket.org/liamstask/goose/
> cmd/goose"
> >
> > So I'm I'm proposing to write an installer script in bash to help
> automate
> > the Golang install as well as the Goose install.  My only concern (as
> well
> > as most of yours) is "go get" will grab the latest, but since no real
> > changes have happened I'm left with no other option.
> >
> > Proposed:
> >
> > /opt/traffic_ops/install/bin/install_goose.sh
> >
> > - Install Golang (version 1.8.x)
> > - go get bitbucket.org/liamstask/goose/cmd/goose
> >
> > Thoughts?
> >
> > -Dew
>
>

Re: Goose installer script

[Eric Friedrich (efriedri)]

Hey Dew-
  What calls this script? 

If its called from the Traffic Ops Spec file, then this will cause some pain 
for those of us that need to install without internet access.

—Eric

> On Apr 28, 2017, at 12:41 PM, Dewayne Richardson  wrote:
> 
> I'm working toward a more streamlined installation process for Traffic Ops
> (internally) and publicly. Of course, the same hiccups that everyone else
> runs into I am as well.  Installation of Golang (proper version) and
> installation of Goose.  Goose has been the most challenging for several
> reasons.  The maintainer hasn't made any real changes since 2015, and has
> not "branched" his code to allow for explicit version download.  Per his
> installation instructions "go get bitbucket.org/liamstask/goose/cmd/goose"
> 
> So I'm I'm proposing to write an installer script in bash to help automate
> the Golang install as well as the Goose install.  My only concern (as well
> as most of yours) is "go get" will grab the latest, but since no real
> changes have happened I'm left with no other option.
> 
> Proposed:
> 
> /opt/traffic_ops/install/bin/install_goose.sh
> 
> - Install Golang (version 1.8.x)
> - go get bitbucket.org/liamstask/goose/cmd/goose
> 
> Thoughts?
> 
> -Dew

Goose installer script

[Dewayne Richardson]

I'm working toward a more streamlined installation process for Traffic Ops
(internally) and publicly. Of course, the same hiccups that everyone else
runs into I am as well.  Installation of Golang (proper version) and
installation of Goose.  Goose has been the most challenging for several
reasons.  The maintainer hasn't made any real changes since 2015, and has
not "branched" his code to allow for explicit version download.  Per his
installation instructions "go get bitbucket.org/liamstask/goose/cmd/goose"

So I'm I'm proposing to write an installer script in bash to help automate
the Golang install as well as the Goose install.  My only concern (as well
as most of yours) is "go get" will grab the latest, but since no real
changes have happened I'm left with no other option.

Proposed:

/opt/traffic_ops/install/bin/install_goose.sh

- Install Golang (version 1.8.x)
- go get bitbucket.org/liamstask/goose/cmd/goose

Thoughts?

-Dew

Re: Configuration Management - Future State Design for Self Service

[Durfey, Ryan]

Great feedback Eric. My response is below.

Ryan DurfeyM | 303-524-5099
24x7 CDN Support: 866-405-2993 or cdn_supp...@comcast.com 
 

On 4/28/17, 8:18 AM, "Eric Friedrich (efriedri)"  wrote:


> On Apr 27, 2017, at 12:19 PM, Durfey, Ryan  
wrote:
> 
> As we move into Self-Service discussions, Configuration Management needs 
to be discussed in greater depth.  I wanted to get the conversation started and 
get feedback from everyone on what the future state should look like and how we 
get there from our current state.
> 
> TO 2.1 Changes In Progress
> Several changes are underway with TO 2.1 that will make a significant 
impact to current challenges. Derek G., keep me honest.
> 
>  1.  Invalidation is being decoupled from configuration updates
> *   Allows for invalidations on 1 min CRON jobs which should take 
approximately 3 min total to apply to mid and edge tier in succession.
> *   Allows configs to be applied to mid tier and edge tier 
simultaneously reducing time by half.
>  2.  Reduction in the overall number of config files generated per 
configuration update
> *   Allows configs to be applied on tighter than the current 15 min 
CRON intervals, though not sure how rapidly yet.  Will require testing.
> This still leaves us with a monolithic configuration management process 
which contains all Server and Service configs.  This presents some challenges 
to individual Self-Service.
> 
> Future State (v3?) Ideas
> 
>  1.  Separate out Server configs from Service configs so they can be:
> *   Tested separately
> *   Applied separately
> *   Logged & reported separately
> *   Reverted separately
> *   Non-blocking when issues are encountered.
>  2.  Separate out individual Service configs  for same reasons as above.
>  3.  Allow for instant push of new configs, no CRON wait time.
EF> We have to be careful with Pushes. Depending on how its implemented, 
this may open up a new API (and a new attack surface) on publicly accessible 
caches.
 
Any reason to believe a 1 minute cronjob would not be fast enough?

RD: I think instant push is the ideal especially if you need to roll something 
back that is broken.  But the reality is that a 1 min cronjob would be pretty 
close and I wouldn’t have issues with that.  I think we attempt to articulate 
the ideal situation and any time we can get an 80/20 solution that gets us 
close with significantly less headaches we are very happy with that.


>  4.  Allow for Service builds and changes to be staged for initial 
testing prior to production roll out.
>  5.  Allow for rollback of config changes in both staging and production 
environments.
>  6.  Log all Service changes so that a Tenant User can pull back a 
history of all changes related to their services through the API.
> 
> 
> Ryan Durfey
> Sr. Product Manager - CDN | Comcast Technology Solutions
> 1899 Wynkoop Ste. 550 | Denver, CO 8020
> M | 303-524-5099
> ryan_dur...@comcast.com
> 24x7 CDN Support: 866-405-2993  or 
cdn_supp...@comcast.com
>