Re: TO jobs_* Table Cleanup

[Dewayne Richardson]

Thanks all for the replies, we've been considering collapsing all those
tables into a single one for "invalidate_content".  Just getting a feel for
it's use.

-Dew

On Thu, May 3, 2018 at 9:59 AM, Steve Malenfant <smalenf...@gmail.com>
wrote:

> We haven't touched those tables @Cox. We use the built-in content
> invalidation.
>
> On Tue, May 1, 2018 at 6:05 PM, Hongfei Zhang (hongfezh) <
> hongf...@cisco.com
> > wrote:
>
> > @Dew - we did not touch job_result table. Thanks, -Hongfei
> >
> > On 5/1/18, 5:03 PM, "Dewayne Richardson" <dewr...@apache.org> wrote:
> >
> > @Hongfei
> >
> > What about the "job_result" table?  It doesn't seem to be used at
> all.
> >
> > - Dew
> >
> > On Tue, May 1, 2018 at 2:47 PM, Hongfei Zhang (hongfezh) <
> > hongf...@cisco.com
> > > wrote:
> >
> > > Hi Dew/Jan,
> > >
> > > We made some minor changes in content invalidation which were using
> > > Job/Status - specifically we:
> > > 1. added back CANCELLED job_status and added a Cancel subroutine
> in
> > > API/Job.pm
> > > 2. Exposed the cancel function via API endpoint DELETE
> > > /api/$version/jobs/:id  - which calls above cancel()
> > >
> > >
> > > Thanks,
> >     > -Hongfei
> > >
> > > On 5/1/18, 4:27 PM, "Jan van Doorn" <j...@knutsel.com> wrote:
> > >
> > > I think they should be nuked.
> > >
> > > Rgds,
> > > JvD
> > >
> > > > On May 1, 2018, at 2:04 PM, Dewayne Richardson <
> > dewr...@apache.org>
> > > wrote:
> > > >
> > > > I'm beginning to evaluate the Perl cleanup effort and noticed
> > that
> > > the
> > > > *job_agent*, *job_status*, and *job_result* are minimally
> used
> > > (except for
> > > > the "job" table which has the list of Purge Jobs").  I wanted
> > to get
> > > a feel
> > > > for the usage outside of Comcast and what the sentiment is to
> > those
> > > tables
> > > > (or not).
> > > >
> > > >
> > > >
> > > > -Dew
> > >
> > >
> > >
> > >
> >
> >
> >
>

Re: TO jobs_* Table Cleanup

[Dewayne Richardson]

@Hongfei

What about the "job_result" table?  It doesn't seem to be used at all.

- Dew

On Tue, May 1, 2018 at 2:47 PM, Hongfei Zhang (hongfezh) <hongf...@cisco.com
> wrote:

> Hi Dew/Jan,
>
> We made some minor changes in content invalidation which were using
> Job/Status - specifically we:
> 1. added back CANCELLED job_status and added a Cancel subroutine  in
> API/Job.pm
> 2. Exposed the cancel function via API endpoint DELETE
> /api/$version/jobs/:id  - which calls above cancel()
>
>
> Thanks,
> -Hongfei
>
> On 5/1/18, 4:27 PM, "Jan van Doorn" <j...@knutsel.com> wrote:
>
> I think they should be nuked.
>
> Rgds,
> JvD
>
> > On May 1, 2018, at 2:04 PM, Dewayne Richardson <dewr...@apache.org>
> wrote:
> >
> > I'm beginning to evaluate the Perl cleanup effort and noticed that
> the
> > *job_agent*, *job_status*, and *job_result* are minimally used
> (except for
> > the "job" table which has the list of Purge Jobs").  I wanted to get
> a feel
> > for the usage outside of Comcast and what the sentiment is to those
> tables
> > (or not).
> >
> >
> >
> > -Dew
>
>
>
>

TO jobs_* Table Cleanup

[Dewayne Richardson]

I'm beginning to evaluate the Perl cleanup effort and noticed that the
*job_agent*, *job_status*, and *job_result* are minimally used (except for
the "job" table which has the list of Purge Jobs").  I wanted to get a feel
for the usage outside of Comcast and what the sentiment is to those tables
(or not).



-Dew

Re: Traffic Ops go client responses

[Dewayne Richardson]

I think at one point there was some effort to expose some of that with the
"rawRequest" func, not sure who created a lot of these, I have just been
using what was there for the API Tests.  I think we do need to do some
refactoring to expose more for proper API Testing.

https://github.com/apache/incubator-trafficcontrol/blob/master/traffic_ops/client/v13/session.go#L285

-Dew

On Wed, Apr 25, 2018 at 3:53 PM, Rawlin Peters 
wrote:

> Hey folks,
>
> I noticed in our TO go client that we aren't decoding the JSON
> response returned from PUT/POST endpoints. If we actually decoded
> those responses, it would be quicker and more useful from a user's
> perspective, save bandwidth from all the unnecessary GETs after
> POST/PUTs, and also reduce load on the API server.
>
> Is there a reason why we don't decode those responses?
>
> - Rawlin
>

Re: Traffic Server Secondary Streaming IPs Design

[Dewayne Richardson]

I agree with Rob, please explore the Go implementation, we're reaching
critical mass (most of the critical foundational routes are complete) with
the API endpoints written in Go with the big one /deliveryservices being
the biggest and most important to complete.

-Dew

On Fri, Apr 20, 2018 at 8:37 AM, Robert Butts 
wrote:

> @weifensh We're hoping to have most of the API endpoints, not including ATS
> config files, in the next month or so. I'm currently working on
> `deliveryservices`, and @dylanvoltz is on `servers`, the two biggest, and
> they're both mostly done. They should be code-complete by the end of next
> week, and then maybe a week for testing. Most of the rest are
> straightforward DB->JSON->HTTP mappings without much logic, they should go
> pretty fast.
>
> Do you have a list of endpoints you'll have to modify for this feature? We
> can certainly prioritize them, if you give us a list.
>
> Will you be modifying any ATS config files generated by Traffic Ops? Those
> are lower on our list, but again, if you have some, we can definitely
> prioritize them.
>
> I'd highly recommend not doing anything new in Perl. At Comcast, we just
> recently allocated people for Self Service, and that's going to require all
> endpoints in Go, so the Perl->Go migration just got bumped to the top of
> our priority list. Since we're not using this "Secondary IP" feature, it's
> going to be harder for us to be sure it's working right when we
> transliterate Perl to Go, and we're likely to break it for you, and we'll
> have to figure out why. It'll just be more painful for everyone to do new
> things in Perl.
>
> You can also convert Perl endpoints to Go yourself, as you need to modify
> them, if you like. Just be sure to coordinate with us, so we don't
> duplicate work.
>
>
> On Fri, Apr 20, 2018 at 3:57 AM, John Shen (weifensh) 
> wrote:
>
> > Hi Nir and all,
> >
> > Do you know when will the full GO version of Traffic Ops be ready in the
> > master branch? The reason I raise this question is that we are starting
> to
> > implement this feature, and if some of the APIs involving this feature
> are
> > still in Perl and will not be ported to GO very soon, we are planning
> > implement them in the Perl. For other APIs, we will implement them in GO.
> > Is there any objection to this plan?
> >
> > Thanks,
> > John
> >
> >
> > On 2018/4/10, 4:22 PM, "Zhilin Huang (zhilhuan)" 
> > wrote:
> >
> > To be clear, "immutable" here means the IP could not be removed, but
> > allow modification.
> >
> > And streaming "priority" is better be Delivery Service based than
> > server based.
> >
> > Thanks,
> > Zhilin
> >
> >
> > On 10/04/2018, 4:12 PM, "Zhilin Huang (zhilhuan)" <
> zhilh...@cisco.com>
> > wrote:
> >
> > Hey Nir,
> >
> > Thanks a lot for your comments. Please see my replies inline.
> >
> > On 10/04/2018, 3:14 AM, "Nir Sopher"  wrote:
> >
> >  Hey Zhilin,
> >
> > Regarding the ports configuration. Even though I believe
> > modeling will be
> > cleaner if the port and IP are set together, you are probably
> > correct - it
> > is reasonable to consider the Port per IP flexibility as a
> > future extension
> > and avoid it for now.
> > Still, I would suggest to at-least module the cr-config with
> > the Port
> > specified per IP (delivery-unit). It is more flexible as well
> > as simplify
> > the router and monitor code.
> > ZH> I understand your consideration about the flexibility. But I
> > still think port is a server lever configuration, do not see the needs of
> > multiple ports in the near future. Anyway, if we want to add port
> > configured together with IP, it is easy to add a new field into the json
> of
> > RESTful API or cr-config, since adding new field is easy to be backward
> > compatible. So I would like to leave this change to future when there is
> > use case required.
> >
> >
> > Regarding the crud of the server configuration, I believe the
> > API should
> > change, but with backward compatibility.
> > Maybe we should have
> > (GET/POST/PUT/DELETE)  /api/1.2/servers/{:svrId}/interfaces
> > And
> > (GET/POST/PUT/DELETE)  /api/1.2/servers/{:svrId}/
> > interfaces/{:ifId}/delivery-units
> > ZH> I went thru those APIs again, and agree with your points. The
> > design doc has been updated to reflect this change in section 3.1.1.3 and
> > 3.1.1.4. A little different than your suggestion is I used "ips" instead
> of
> > "delivery-units" to allow manipulation of management IP and ILO IP as
> well.
> >
> > These APIs will allow us to manipulate all interfaces and all
> > IPs
> > (delivery-units). Note that as I see it, there is no special
> > "primary" IP
> > 

Re: Updated TO API guidelines

[Dewayne Richardson]

Thank you John for giving us "API Use Cases" to think about with these new
TO API Guidelines.  The main goal for these changes are to build TO API's
that are intuitive.  I'm of the opinion that if the API's are intuitive
(with easy to understand patterns) that prevents me from wasting time
looking up API docs.  A nice side effect of having simple standards and
patterns is that simplicity ripples into our API Go code which allows us to
easily build frameworks that do all of the work instead of the API
snowflakes that we have today.

I encourage everyone to shoot as many holes into our thoughts around this
new direction so that we can see the bigger picture.

-Dew

On Wed, Apr 4, 2018 at 10:29 PM, John Rushford  wrote:

> Why the change?  It’s my understanding that path parameters should be used
> to specify a particular resource
> and query parameters should be used to sort/filter the query.  Why use a
> query parameter to specify a particular
> resource?  Is this REST API best practice?
>
> What about sub resource queries such as using the following:
>
> GET api/1.3/deliveryservices/{xmlID}/urisigning
>
> where you are requesting a particular urisigning keys sub resource for the
> particular deliveryservice resource. You can make it work
> with an xmlid query parameter but what do you return if the query
> parameter is left off, all uri signing keys?  Is that useful?
>
> John
>
> > On Apr 4, 2018, at 3:23 PM, Jeremy Mitchell 
> wrote:
> >
> > tbh i'm not sure about versioning. I was just trying to suggest that new
> > routes be formulated this way per the new API guidelines:
> >
> > GET /foos[?id, name, etc=]
> > POST /foos
> > PUT /foos [?id, name, etc=]
> > DELETE /foos [?id, name, etc=]
> >
> > instead of the old way:
> >
> > GET /foos
> > GET /foos/:id
> > POST /foos
> > PUT /foos/:id
> > DELETE /foos/:id
> >
> > The difference being the use of query params over route/path params.
> >
> > Technically, adding new routes does not break old stuff right so i don't
> > think that warrants a major version roll.
> >
> > While we're on the subject, what does everyone think if we took this one
> > step further and made routes handle a request payload with one or more
> > items. For example:
> >
> > GET /foos[?id, name, etc=]
> > POST /foos <-- takes in an array of foos to create
> > PUT /foos <-- takes in an array of foos to update
> > DELETE /foos <-- takes in an array of foos to delete
> >
> > in this scenario, query params only pertain to the GET. The POST, PUT and
> > DELETE rely on the contents of the request json...
> >
> > Jeremy
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Wed, Apr 4, 2018 at 1:55 PM, Robert Butts 
> > wrote:
> >
> >> That document doesn't mention versions, 1.2 vs 1.3 vs 2.0.
> >>
> >> Just to clarify, changing to query parameters breaks compatibility with
> 1.2
> >> and older, so new APIs in that format have to be a new major version,
> i.e.
> >> 2.0, per Semantic Versioning, right?
> >>
> >> On Tue, Apr 3, 2018 at 3:26 PM, Jeremy Mitchell  >
> >> wrote:
> >>
> >>> FYI - I've updated the TO API guidelines to reflect our desire to move
> >> away
> >>> from route/path params and embrace query params in the Golang API.
> >>>
> >>> https://cwiki.apache.org/confluence/display/TC/API+Guidelines
> >>>
> >>> Jeremy
> >>>
> >>
>
>

Traffic Control Doc's Proposal

[Dewayne Richardson]

I've been going through the Traffic Ops API doc's and revamping them with
Swagger and noticed this trend with our documentation.   "Guides" (Howto's,
Installs, Developer's)  growing because they are easier to maintain
(because they are less fancy) as markdown files in the Github project.  My
concern is the RST equivalents here:
http://traffic-control-cdn.readthedocs.io/en/latest/development/index.html
are going stale and they are also being duplicated in the markdown files.
I also worry that the markdown files aren't surfaced properly to our
community because there is valuable information in them as well.

Therefore I propose the following documentation "rules of thumb":

1. Howto's, Installs, Developer/Admin Guides are done in Markdown but have
RST links (or some reference to an index page to surface all of them)
2. High level informative documentation like API docs, and Traffic Control
features are done in RST as we have been to surface that information for
"Users of Traffic Control"

Please provide your feedback,

-Dew


Here is a quick list of the Markdown files I've found in the project so far:


./misc/git/README.md
./CODE_OF_CONDUCT.md
./experimental/traffic_router_golang/README.md
./test/traffic_ops_cfg/Readme.md
./test/router/dnssec/Readme.md
./test/router/Readme.md
./CHANGELOG.md
./traffic_portal/v1/README.md
./traffic_portal/test/end_to_end/README.md
./traffic_portal/README.md
./traffic_portal/build/README.md
./traffic_monitor_java/README.md
./traffic_monitor/tools/testcaches/README.md
./traffic_monitor/README.md
./traffic_server/plugins/astats_over_http/README.md
./traffic_server/README.md
./README.md
./traffic_ops_db/pg-migration/README.md
./traffic_ops/experimental/goto/README.md
./traffic_ops/experimental/auth/README.md
./traffic_ops/experimental/server/README.md
./traffic_ops/experimental/webfront/README.md
./traffic_ops/INSTALL.md
./traffic_ops/README.md
./traffic_ops/testing/compare/README.md
./traffic_ops/testing/api/docker/README.md
./traffic_ops/testing/api/README.md
./traffic_ops/build/README.md
./traffic_ops/client_tests/README.md
./traffic_ops/traffic_ops_golang/README.md
./traffic_ops/traffic_ops_golang/swaggerdocs/v13/README.md
./CONTRIBUTING.md
./BUILD.md
./build/README.md
./infrastructure/docker/README.md
./infrastructure/docker/build/README.md
./infrastructure/test/integration/README.md
./infrastructure/test/README.md
./traffic_router/core/src/test/resources/Readme.md
./traffic_router/neustar/README.md

General TC Cleanup

[Dewayne Richardson]

As we get more visibility as we become an Apache project, I've been taking
a fresh eye at Github project so that we can tidy up a bit to present
ourselves well.  I highly recommend that each of you do as well so that we
can get more eyeballs on the matter.

One nit that I've noticed is the

Github README.md News
https://github.com/apache/incubator-trafficcontrol#news

This section always seems to get overlooked with each TC release.  The
question I have is do we think the "News" section is still valuable (it's
was only used to point out releases), or is there a better location we
should be directing from our Github project that is more Apache facing so
that we're making changes in one place?

I've created this PR as my suggestion to the change.  Please provide
feedback and suggestions on the appropriate location for our "Traffic
Control News".

https://github.com/apache/incubator-trafficcontrol/pull/2011

-Dew

Re: Traffic Ops Change log

[Dewayne Richardson]

IMO, I don't think it's necessary to pollute the Change Log with Delivery
Service Requests (DSRs) comments.  I think DSRs are turning out to be the
Change Log for Delivery Services and comments on those are Change Log noise.

-Dew

On Wed, Mar 21, 2018 at 10:24 AM, Jeremy Mitchell 
wrote:

> Currently, every POST (create), PUT (update) or DELETE (delete) TO API
> endpoint (or most of them) create a change log entry.
>
> For example when i PUT /api/cachegroups/4 it creates this CL entry:
>
> myusername - Updated Cachegroup named 'Foo' with id: 57
>
> Question: In the golang rewrite of the TO API should EVERY POST, PUT
> and DELETE trigger a change log entry? Or should it be optional and
> determine by the developer?
>
> I ask because I'm adding endpoints for create, update and delete of
> delivery service request comments and it feels like too much to create a
> change log entry for that. I'd hate to pollute the change log for minor
> stuff.
>
> To me, it seems like it should be optional and determined by the API
> developer.
>
> Jeremy
>

Re: Backup Cache Group Selection

[Dewayne Richardson]

As another thought, maybe we should take advantage of https://postgis.net/
and figure out how to flip CZF into it.

-Dew

On Tue, Mar 13, 2018 at 10:05 AM, David Neuman 
wrote:

> What happens when Geo Limit is set to "CZF Only"  with all backup Caches
> are unavailable and fallbackToClosest is set to True. Current
> implementation will fail this. Should we do Geo lookup now in this change?
>
> In this case we should fail. If the Geo Limit is set to “CZF Only” then
> that is all we use.
> ​
>
> On Tue, Mar 13, 2018 at 8:17 AM, Vijay Anand <
> vijayanand.jayaman...@gmail.com> wrote:
>
> > @Rawlin,
> >
> > Let me try and get the changes done for TR according to your suggestions.
> > This change would be like:
> > 1. CZF only contains cache groups which should map back to TO's Cache
> Group
> > configurations (cr-config)
> > 2. Backup configurations should reach TR via cr-config in the format you
> > detailed.
> > 3. fallbackToClosest will be True by default. If backupLocation config is
> > present, it will be assumed as false unless otherwise it is stated as
> TRUE
> > explicitly.
> > 4. This will work irrespective of the coverage Zones in CZF as long as
> the
> > backup Cache Group specified is in cr-config.
> >
> > I have a doubt in this as well.
> >
> > What happens when Geo Limit is set to "CZF Only"  with all backup Caches
> > are unavailable and fallbackToClosest is set to True. Current
> > implementation will fail this. Should we do Geo lookup now in this
> change?
> >
> > Shall i delete my existing PR and create a new one with these changes?
> >
> > I will try to get the necessary changes for TO (Perl Mojo) along with
> this.
> > Would require your help in TO (Golang) and TP. Will keep you posted.
> >
> > Thanks,
> > Vijayanand S
> >
> >
> >
> >
> > On Tue, Mar 13, 2018 at 3:04 AM, Rawlin Peters 
> > wrote:
> >
> > > If we start by putting this in the Cache Group API first, then later
> > > we really only have to worry about adding the CIDRs to the API. The
> > > backup config is really just relationships between cache groups, which
> > > makes perfect sense to model in a relational DB rather than the CZF.
> > > Why put something in the CZF to just tear it out later?
> > >
> > > - Rawlin
> > >
> > > On Mon, Mar 12, 2018 at 3:12 PM, Dave Neuman 
> wrote:
> > > > Good point Rawlin, but I think it does answer your questions.  CZF
> for
> > > now,
> > > > whatever the new CZF thing is after that.
> > > >
> > > > On Mon, Mar 12, 2018 at 1:44 PM, Rawlin Peters <
> > rawlin.pet...@gmail.com>
> > > > wrote:
> > > >
> > > >> The original scope of this thread was determining where the Backup
> > > >> Cache Group config should live (API vs CZF), not necessarily about
> > > >> building the entire CZF in the database, although I'm +1 on that
> idea
> > > >> as well. I think any decisions made about doing that should probably
> > > >> be started in a separate thread.
> > > >>
> > > >> - Rawlin
> > > >>
> > > >> On Mon, Mar 12, 2018 at 1:11 PM, Dave Neuman 
> > wrote:
> > > >> > +1 on building the CZF in the database.  Jan tried to go down that
> > > rabbit
> > > >> > hole but realized it was a pretty hard problem to solve.  I think
> > > this is
> > > >> > something we might want to re-visit.  Maybe this is something we
> > > should
> > > >> > discuss at our meetup and then update this thread with our
> > decisions?
> > > >> >
> > > >> > On Mon, Mar 12, 2018 at 11:25 AM, Rawlin Peters <
> > > rawlin.pet...@gmail.com
> > > >> >
> > > >> > wrote:
> > > >> >
> > > >> >> @VijayAnand:
> > > >> >>
> > > >> >> Right, a Coverage Zone that doesn't map to a Cache Group in TO
> > won't
> > > >> >> be chosen as a backup in case of failure, but you could have a
> > > >> >> Coverage-Zone-not-in-TO that configures Coverage-Zones-in-TO as
> > > >> >> backups. But, I think the general sentiment right now is that all
> > > >> >> Coverage Zones in the CZF should map back to Cache Groups in TO,
> so
> > > >> >> the backup config should also be done via the Cache Group API.
> > > >> >>
> > > >> >> So from the Traffic Router perspective, the process should
> become:
> > > >> >> 1. Rather than parsing from the CZF into the NetworkNode class,
> > parse
> > > >> >> Cache Group backup config from the CRConfig into the existing
> > > >> >> CacheLocation class
> > > >> >> 2. in the DS request flow, the NetworkNode will map back to a
> > > >> >> registered CacheLocation which would contain the backup CG config
> > > >> >>
> > > >> >> The rest of the PR's behavior should stay the same, it's just a
> > > matter
> > > >> >> of the config being located in a different class. To give you an
> > idea
> > > >> >> of how I would format it in the CRConfig (so you know how to
> parse
> > it
> > > >> >> out), here is a snippet of "edgeLocations" from CRConfig.json:
> > > >> >>
> > > >> >> "edgeLocations": {
> > > >> >> "edge-cg-1": {
> > > >> >>   

Re: Backup Cache Group Selection

[Dewayne Richardson]

Eric, I agree with this as well, maybe we build separate API's for
"loading" CZF formats (or any other types of external data) into the same
area of the data model (however that ultimately looks).  If we keep the CZF
data centralized it'll be easier to build relationships if needed.


-Dew

On Mon, Mar 12, 2018 at 7:14 AM, Eric Friedrich (efriedri) <
efrie...@cisco.com> wrote:

> Good point. I think it makes sense to move both the backupList and
> coordinates into the CG API. By move coordinates into the API, I’m implying
> that we consolidate into 1 set of coordinates per CG. The existing CG
> coordinates would now be used for both backup edge CG selection and initial
> edge cg selection when doing client geolocation. This can certainly happen
> in a different PR than the backupList getting committed.
>
> It looks as though the CZ file will give us more flexibility, but it is
> unneeded. Today that flexibility is only causing more complexity and more
> operational overhead.
>
> —Eric
>
>
> > On Mar 9, 2018, at 5:21 PM, Rawlin Peters 
> wrote:
> >
> > So in your CZF example, we can't actually have two CZs using the same
> > name ("a"), because when that JSON gets parsed one of the CZs will be
> > overwritten so that there is only one "a" key in the JSON. The names
> > would have to be "a1" and "a2" for instance and backupList [a, b, c]
> > and [a, c, b], but I see your point in how that might be useful. But
> > at that point maybe it's just better to create two empty cachegroups
> > "a1" and "a2" in the API that first fall back to cachegroup "a".
> >
> > If I'd been around at the time coordinates were added to the CZF, I
> > think I would've -1'd the idea. The coordinates for Client Source and
> > Cache Group Destination should be the same typically because you
> > should know based upon your network topology which cachegroup to route
> > clients to. By being able to specify backups, the coordinates in the
> > CZF would become even less relevant. And fwiw at Comcast we generate
> > the CZF using the coordinates from the Cache Group API anyways, so
> > they always match.
> >
> > Another major concern with this is the scalability of having the
> > config in the CZF alone. If we can change these backupZone relations
> > using the API, that means we can add it to Traffic Portal and have a
> > much better UX rather than hard-coding these relations in whatever
> > script we're using to generate the CZF. We'd get benefits like
> > validation and typo safety, and who knows maybe in the future we could
> > have a map in TP to visualize the relationships between cache groups
> > for troubleshooting.
> >
> > - Rawlin
> >
> > On Fri, Mar 9, 2018 at 2:22 PM, Eric Friedrich (efriedri)
> >  wrote:
> >> "I can't imagine why we'd ever want the two sets of coordinates to
> differ for the same Cache Group. “
> >> Maybe someone else can chime in about why coordinates were added to the
> CZF in the first place, but I’ve also thought of them like this:
> >> CG API Coordinates - Where the cache servers are. To be used as a
> destination location routing traffic towards
> >> CZF Coordinates - Where the clients are. To be used as the source
> location routing traffic from these coordinates to the CG API coordinates.
> >>
> >> I could see cases where:
> >> a) you might set the CG coordinates of DenCG to Denver’s lat long
> >> b) you might set the CZF Coordinates of that coverageZone to somewhere
> other than Denver (because the coverageZone is not a perfect circle
> centered on Denver, like the caches might be)
> >>
> >> So I think there are valid reason’s for two sets of coordinates to
> exist, but I’m not sure if people set them differently in practice? If they
> are always the same, for every CG it seems like they should get
> consolidated. (I don’t think we’re using CZF coordinates currently)
> >>
> >> Second Point:
> >>  By having the backupList in the CZF, we have more granularity on how
> clients are routed during failures. For example, could we create a CZF that
> looks like
> >> “coverageZones”: {
> >>  “a”: {
> >>“backupList”: [b,c]
> >>“networks": [1,2,3]
> >>  },
> >>
> >> “a”: {
> >>“backupList”: [c,b]
> >>“networks": [4,5,6]
> >>  }
> >> }
> >>
> >> Here all clients are part of the “a” CacheGroup/Coverage Zone, but
> depending on their specific subnet they have different backup policies.
> >>
> >> Our particular requirement for this feature is a backup at the
> CacheGroup level, not the CZ level as I’ve shown here- so perhaps we’re
> overbuilding it.
> >>
> >> —Eric
> >>
> >>
> >>
> >>
> >>
> >>> On Mar 9, 2018, at 4:05 PM, Rawlin Peters 
> wrote:
> >>>
> >>> Ok, so the coordinates in the CZF are only used when no available
> >>> cache is found in the matched cachegroup. Rather than using the
> >>> coordinates of the matched cachegroup that it already knows about from
> >>> the API, Traffic Router uses the coordinates from the CZF 

Re: [VOTE] CHANGELOG.md file (second try)

[Dewayne Richardson]

> others
> >>> can
> >>> >> simply put a milestone on their PR (if there is no related issue).
> >>> Either
> >>> >> way, it tags the items we want included in the changelog. And then
> the
> >>> RM
> >>> >> can build the changelog from the milestones. Easy peasy.
> >>> >>
> >>> >> Jeremy
> >>> >>
> >>> >>
> >>> >>
> >>> >>
> >>> >>
> >>> >>
> >>> >> On Tue, Jan 9, 2018 at 4:56 PM, Leif Hedstrom <zw...@apache.org>
> wrote:
> >>> >>
> >>> >>>
> >>> >>>
> >>> >>> > On Jan 9, 2018, at 10:22 AM, Dave Neuman <neu...@apache.org>
> wrote:
> >>> >>> >
> >>> >>> > Looks like this thread died over the holidays.  Let me try to
> bring
> >>> it
> >>> >>> back
> >>> >>> > up before we cut a 2.2 branch.
> >>> >>> > Can you please respond with *just* the following:
> >>> >>> >
> >>> >>> > [X] +1 to adding a changelog.MD file
> >>> >>> > [] -1 to adding a changelog.MD file
> >>> >>> >
> >>> >>> > [] +1 to adding a changelog label in github
> >>> >>> > [X] -1 to adding a changelog label in github
> >>> >>> >
> >>> >>>
> >>> >>>
> >>> >>> To add to the previous thread / thoughts. I feel reasonably
> strongly
> >>> that
> >>> >>> having a changelog label in Github is not useful. In the ATS
> >>> community, we
> >>> >>> can *barely* get people to set the Milestone properly, and I feel
> that
> >>> the
> >>> >>> Milestone is a much more important thing to keep accurate than
> anything
> >>> >>> else. And, to be normalized, you don’t want to duplicate this info
> :-).
> >>> >>>
> >>> >>> So, what we do is have the RM be like a hawk over the Milestones,
> and
> >>> then
> >>> >>> run Phil’s fancy pants script to generate the ChangeLog from the
> >>> Milestones
> >>> >>> on all PRs closed.
> >>> >>>
> >>> >>> Cheers,
> >>> >>>
> >>> >>> — leif
> >>> >>>
> >>> >>> > Thanks,
> >>> >>> > Dave
> >>> >>> >
> >>> >>> > On Fri, Dec 15, 2017 at 9:14 AM, Dewayne Richardson <
> >>> dewr...@gmail.com>
> >>> >>> > wrote:
> >>> >>> >
> >>> >>> >> +1 on the CHANGELOG.md as well, but I like having the changelog
> >>> label
> >>> >>> >> because it helps streamline it's creation.  I also think the
> labels
> >>> are
> >>> >>> >> valuable because they help summarize the parts of the repo that
> >>> changed
> >>> >>> and
> >>> >>> >> keeps the concept closer to the repository (where the real
> change
> >>> is).
> >>> >>> >>
> >>> >>> >> -Dew
> >>> >>> >>
> >>> >>> >> On Thu, Dec 14, 2017 at 3:01 PM, Robert Butts <
> >>> robert.o.bu...@gmail.com
> >>> >>> >
> >>> >>> >> wrote:
> >>> >>> >>
> >>> >>> >>> +1. To clarify, I'm +1 on having the "changelog" label, to help
> >>> whoever
> >>> >>> >>> manually goes thru and builds the CHANGELOG.md.
> >>> >>> >>>
> >>> >>> >>> But I agree with @neuman I don't think we can automate this.
> >>> Titles are
> >>> >>> >> too
> >>> >>> >>> short, some changes need longer explanations and some don't,
> only a
> >>> >>> human
> >>> >>> >>> can figure out how important something is to users; a high
> >>> priority bug
> >>> >>> >> fix
> >>> >>> >>> could still be low-impact, or vica-versa. Much as I dislike
&g

Re: [VOTE] CHANGELOG.md file (second try)

[Dewayne Richardson]

we're fastidious about
> > > >>> > requiring changelog updates on every single PR. A PR should
> either
> > > >>> > provide a reasonable changelog entry, or, in the body of the PR,
> > > >>> > justify it's absence. A simple "This bugfix does not require a
> > > >>> > changelog entry." is sufficient for trivial bugfixes. That's
> plenty
> > > >>> > for a reviewer to quickly either agree or decide to request (or
> > > >>> > provide) an entry.
> > > >>> >
> > > >>> > If we add the changelog file, we need to update the contributing
> > file
> > > >>> > to include the new guidelines.
> > > >>> >
> > > >>> > On Wed, Jan 10, 2018 at 9:14 AM, Jeremy Mitchell <
> > > mitchell...@gmail.com>
> > > >>> wrote:
> > > >>> >> Yes, I was about to mention milestones. Why not leverage Github
> > > >>> milestones
> > > >>> >> on issues/PRs? We talked about using milestones at the last TC
> > > summit to
> > > >>> >> determine the scope of a release. Now is our chance to do that.
> > > >>> >>
> > > >>> >> Milestones can be applied to issues or PRs. I tend to create
> > issues
> > > for
> > > >>> >> everything I do, so I would put the milestone on the issue but
> > > others
> > > >>> can
> > > >>> >> simply put a milestone on their PR (if there is no related
> issue).
> > > >>> Either
> > > >>> >> way, it tags the items we want included in the changelog. And
> then
> > > the
> > > >>> RM
> > > >>> >> can build the changelog from the milestones. Easy peasy.
> > > >>> >>
> > > >>> >> Jeremy
> > > >>> >>
> > > >>> >>
> > > >>> >>
> > > >>> >>
> > > >>> >>
> > > >>> >>
> > > >>> >> On Tue, Jan 9, 2018 at 4:56 PM, Leif Hedstrom <zw...@apache.org
> >
> > > wrote:
> > > >>> >>
> > > >>> >>>
> > > >>> >>>
> > > >>> >>> > On Jan 9, 2018, at 10:22 AM, Dave Neuman <neu...@apache.org>
> > > wrote:
> > > >>> >>> >
> > > >>> >>> > Looks like this thread died over the holidays.  Let me try to
> > > bring
> > > >>> it
> > > >>> >>> back
> > > >>> >>> > up before we cut a 2.2 branch.
> > > >>> >>> > Can you please respond with *just* the following:
> > > >>> >>> >
> > > >>> >>> > [X] +1 to adding a changelog.MD file
> > > >>> >>> > [] -1 to adding a changelog.MD file
> > > >>> >>> >
> > > >>> >>> > [] +1 to adding a changelog label in github
> > > >>> >>> > [X] -1 to adding a changelog label in github
> > > >>> >>> >
> > > >>> >>>
> > > >>> >>>
> > > >>> >>> To add to the previous thread / thoughts. I feel reasonably
> > > strongly
> > > >>> that
> > > >>> >>> having a changelog label in Github is not useful. In the ATS
> > > >>> community, we
> > > >>> >>> can *barely* get people to set the Milestone properly, and I
> feel
> > > that
> > > >>> the
> > > >>> >>> Milestone is a much more important thing to keep accurate than
> > > anything
> > > >>> >>> else. And, to be normalized, you don’t want to duplicate this
> > info
> > > :-).
> > > >>> >>>
> > > >>> >>> So, what we do is have the RM be like a hawk over the
> Milestones,
> > > and
> > > >>> then
> > > >>> >>> run Phil’s fancy pants script to generate the ChangeLog from
> the
> > > >>> Milestones
> > > >>> >>> on all PRs closed.
> > > >>> >>>
> > > >>> >>> Cheers,
&

Re: Traffic Ops API Swagger Doc

[Dewayne Richardson]

I mentioned a week ago if we should use Swagger for documenting the API
(with my example findings).  Since I have seen no -1's the plan is to move
forward with Swagger and only document the Golang Proxy routes (1.3), the
Perl routes will still use the existing 1.2 API docs.

In terms of how we integrate with our existing doc the hope is to generate
a new traffic-ops-api-1.3.rst file for the 1.3 api docs and hook into the
mainline docs accordingly.

Thanks,

-Dew

On Tue, Feb 20, 2018 at 10:50 AM, Jeremy Mitchell <mitchell...@gmail.com>
wrote:

> Eric, are you saying you never want to write RST anymore? me neither. yuck!
>
> On Tue, Feb 20, 2018 at 9:57 AM, Eric Friedrich (efriedri) <
> efrie...@cisco.com> wrote:
>
> > Is it possible to take the swagger generated documentation and have that
> > automatically included in the read-the-docs site?
> >
> > Asked another way: Can swagger generate docs in ReStructed Text (.rst)
> > format?
> >
> > —Eric
> >
> >
> > > On Feb 20, 2018, at 11:38 AM, Dave Neuman <neu...@apache.org> wrote:
> > >
> > > Sounds good.  I look forward to seeing it merged into our repo.
> > > I guess this means there will need to be a PR to remove our current API
> > > docs as they get moved to swagger.
> > >
> > > On Tue, Feb 20, 2018 at 8:40 AM, Jeremy Mitchell <
> mitchell...@gmail.com>
> > > wrote:
> > >
> > >> I think this all sounds very promising. Some advantages that I see
> are:
> > >>
> > >> - docs never drift from API implementation (currently our docs get out
> > of
> > >> sync real fast)
> > >> - this provides yet another interface -
> > >> https://app.swaggerhub.com/apis/dewrich/traffic-ops_api/1.3 -  (in
> > >> addition
> > >> to TP) to interact with the API
> > >> - a great way to demonstrate / educate developers on the capabilities
> of
> > >> the api
> > >>
> > >> plus, i heard some bonus features that could be really interesting:
> > >>
> > >> - auto generation of a TO golang client. can we deprecate the old TO
> > client
> > >> in favor of an auto-generated one? The current TO client never seems
> to
> > >> stay in sync with the api
> > >> - generating server stubs
> > >>
> > >> imo our api can never be fully auto-generated because of the business
> > logic
> > >> that needs to be accounted forbut it would be pretty neat if all
> we
> > had
> > >> to do was "define" the api in a yaml file and that yaml file would
> spit
> > out
> > >> documentation, tests, clients (be it golang or java or whatever), and
> > >> server side code (stubs) and then all we could focus on writing code
> > >> specific to business logic.
> > >>
> > >> my guess is to really do this right, we'd have to rev the api version
> to
> > >> 2.0 to make the api more swagger friendly...tools (swagger) like
> > >> consistency and our api is far from consistent...
> > >>
> > >> jeremy
> > >>
> > >> On Wed, Feb 14, 2018 at 10:08 AM, Durfey, Ryan <
> ryan_dur...@comcast.com
> > >
> > >> wrote:
> > >>
> > >>> I am +1 on the swagger concept.  This makes working with APIs much
> > easier
> > >>> for non-developer staff and makes it easier to educate customers as
> > well.
> > >>>
> > >>> Ryan DurfeyM | 303-524-5099
> > >>> CDN Support (24x7): 866-405-2993 or cdn_supp...@comcast.com > >>> cdn_supp...@comcast.com>
> > >>>
> > >>> From: Dewayne Richardson <dewr...@gmail.com>
> > >>> Reply-To: "dev@trafficcontrol.incubator.apache.org" <
> > >>> dev@trafficcontrol.incubator.apache.org>
> > >>> Date: Wednesday, February 14, 2018 at 9:34 AM
> > >>> To: "dev@trafficcontrol.incubator.apache.org" <
> > >>> dev@trafficcontrol.incubator.apache.org>
> > >>> Subject: Traffic Ops API Swagger Doc
> > >>>
> > >>> We've been working diligently on the TO Golang Rewrite
> > >>> <https://github.com/apache/incubator-trafficcontrol/projects/2>
> > project
> > >> to
> > >>> obviously rewrite the Perl into Go, but also to improve our Testing
> and
> > >>> Documentation efforts.  I presented the idea of using Swagger se

Re: Traffic Ops API Swagger Doc

[Dewayne Richardson]

Eric: Out of curiosity I was able to generate it with the tooling I
mentioned here:

https://github.com/dewrich/incubator-trafficcontrol/blob/swagger-demo/traffic_ops/traffic_ops_golang/docs/swagger.rst

-Dew

On Tue, Feb 20, 2018 at 10:38 AM, Dewayne Richardson <dewr...@gmail.com>
wrote:

> Yes the plan was to get the initial thumbs up, then figure out how we can
> deploy it (by default swagger generates interactive, meaning it needs a
> server, UI doc).  A quick internet search says there does look like tooling
> for converting swagger docs to .rst (https://github.com/Arello-
> Mobile/swagger2rst).
>
> Once we see that it's going to all work out then we'll start removing the
> old API docs and hook in the new ones.
>
> -Dew
>
> On Tue, Feb 20, 2018 at 9:57 AM, Eric Friedrich (efriedri) <
> efrie...@cisco.com> wrote:
>
>> Is it possible to take the swagger generated documentation and have that
>> automatically included in the read-the-docs site?
>>
>> Asked another way: Can swagger generate docs in ReStructed Text (.rst)
>> format?
>>
>> —Eric
>>
>>
>> > On Feb 20, 2018, at 11:38 AM, Dave Neuman <neu...@apache.org> wrote:
>> >
>> > Sounds good.  I look forward to seeing it merged into our repo.
>> > I guess this means there will need to be a PR to remove our current API
>> > docs as they get moved to swagger.
>> >
>> > On Tue, Feb 20, 2018 at 8:40 AM, Jeremy Mitchell <mitchell...@gmail.com
>> >
>> > wrote:
>> >
>> >> I think this all sounds very promising. Some advantages that I see are:
>> >>
>> >> - docs never drift from API implementation (currently our docs get out
>> of
>> >> sync real fast)
>> >> - this provides yet another interface -
>> >> https://app.swaggerhub.com/apis/dewrich/traffic-ops_api/1.3 -  (in
>> >> addition
>> >> to TP) to interact with the API
>> >> - a great way to demonstrate / educate developers on the capabilities
>> of
>> >> the api
>> >>
>> >> plus, i heard some bonus features that could be really interesting:
>> >>
>> >> - auto generation of a TO golang client. can we deprecate the old TO
>> client
>> >> in favor of an auto-generated one? The current TO client never seems to
>> >> stay in sync with the api
>> >> - generating server stubs
>> >>
>> >> imo our api can never be fully auto-generated because of the business
>> logic
>> >> that needs to be accounted forbut it would be pretty neat if all
>> we had
>> >> to do was "define" the api in a yaml file and that yaml file would
>> spit out
>> >> documentation, tests, clients (be it golang or java or whatever), and
>> >> server side code (stubs) and then all we could focus on writing code
>> >> specific to business logic.
>> >>
>> >> my guess is to really do this right, we'd have to rev the api version
>> to
>> >> 2.0 to make the api more swagger friendly...tools (swagger) like
>> >> consistency and our api is far from consistent...
>> >>
>> >> jeremy
>> >>
>> >> On Wed, Feb 14, 2018 at 10:08 AM, Durfey, Ryan <
>> ryan_dur...@comcast.com>
>> >> wrote:
>> >>
>> >>> I am +1 on the swagger concept.  This makes working with APIs much
>> easier
>> >>> for non-developer staff and makes it easier to educate customers as
>> well.
>> >>>
>> >>> Ryan DurfeyM | 303-524-5099
>> >>> CDN Support (24x7): 866-405-2993 or cdn_supp...@comcast.com> >>> cdn_supp...@comcast.com>
>> >>>
>> >>> From: Dewayne Richardson <dewr...@gmail.com>
>> >>> Reply-To: "dev@trafficcontrol.incubator.apache.org" <
>> >>> dev@trafficcontrol.incubator.apache.org>
>> >>> Date: Wednesday, February 14, 2018 at 9:34 AM
>> >>> To: "dev@trafficcontrol.incubator.apache.org" <
>> >>> dev@trafficcontrol.incubator.apache.org>
>> >>> Subject: Traffic Ops API Swagger Doc
>> >>>
>> >>> We've been working diligently on the TO Golang Rewrite
>> >>> <https://github.com/apache/incubator-trafficcontrol/projects/2>
>> project
>> >> to
>> >>> obviously rewrite the Perl into Go, but also to improve our Testing
>> and
>> >>> Documentation efforts.  I pres

Re: Traffic Ops API Swagger Doc

[Dewayne Richardson]

Yes the plan was to get the initial thumbs up, then figure out how we can
deploy it (by default swagger generates interactive, meaning it needs a
server, UI doc).  A quick internet search says there does look like tooling
for converting swagger docs to .rst (
https://github.com/Arello-Mobile/swagger2rst).

Once we see that it's going to all work out then we'll start removing the
old API docs and hook in the new ones.

-Dew

On Tue, Feb 20, 2018 at 9:57 AM, Eric Friedrich (efriedri) <
efrie...@cisco.com> wrote:

> Is it possible to take the swagger generated documentation and have that
> automatically included in the read-the-docs site?
>
> Asked another way: Can swagger generate docs in ReStructed Text (.rst)
> format?
>
> —Eric
>
>
> > On Feb 20, 2018, at 11:38 AM, Dave Neuman <neu...@apache.org> wrote:
> >
> > Sounds good.  I look forward to seeing it merged into our repo.
> > I guess this means there will need to be a PR to remove our current API
> > docs as they get moved to swagger.
> >
> > On Tue, Feb 20, 2018 at 8:40 AM, Jeremy Mitchell <mitchell...@gmail.com>
> > wrote:
> >
> >> I think this all sounds very promising. Some advantages that I see are:
> >>
> >> - docs never drift from API implementation (currently our docs get out
> of
> >> sync real fast)
> >> - this provides yet another interface -
> >> https://app.swaggerhub.com/apis/dewrich/traffic-ops_api/1.3 -  (in
> >> addition
> >> to TP) to interact with the API
> >> - a great way to demonstrate / educate developers on the capabilities of
> >> the api
> >>
> >> plus, i heard some bonus features that could be really interesting:
> >>
> >> - auto generation of a TO golang client. can we deprecate the old TO
> client
> >> in favor of an auto-generated one? The current TO client never seems to
> >> stay in sync with the api
> >> - generating server stubs
> >>
> >> imo our api can never be fully auto-generated because of the business
> logic
> >> that needs to be accounted forbut it would be pretty neat if all we
> had
> >> to do was "define" the api in a yaml file and that yaml file would spit
> out
> >> documentation, tests, clients (be it golang or java or whatever), and
> >> server side code (stubs) and then all we could focus on writing code
> >> specific to business logic.
> >>
> >> my guess is to really do this right, we'd have to rev the api version to
> >> 2.0 to make the api more swagger friendly...tools (swagger) like
> >> consistency and our api is far from consistent...
> >>
> >> jeremy
> >>
> >> On Wed, Feb 14, 2018 at 10:08 AM, Durfey, Ryan <ryan_dur...@comcast.com
> >
> >> wrote:
> >>
> >>> I am +1 on the swagger concept.  This makes working with APIs much
> easier
> >>> for non-developer staff and makes it easier to educate customers as
> well.
> >>>
> >>> Ryan DurfeyM | 303-524-5099
> >>> CDN Support (24x7): 866-405-2993 or cdn_supp...@comcast.com >>> cdn_supp...@comcast.com>
> >>>
> >>> From: Dewayne Richardson <dewr...@gmail.com>
> >>> Reply-To: "dev@trafficcontrol.incubator.apache.org" <
> >>> dev@trafficcontrol.incubator.apache.org>
> >>> Date: Wednesday, February 14, 2018 at 9:34 AM
> >>> To: "dev@trafficcontrol.incubator.apache.org" <
> >>> dev@trafficcontrol.incubator.apache.org>
> >>> Subject: Traffic Ops API Swagger Doc
> >>>
> >>> We've been working diligently on the TO Golang Rewrite
> >>> <https://github.com/apache/incubator-trafficcontrol/projects/2>
> project
> >> to
> >>> obviously rewrite the Perl into Go, but also to improve our Testing and
> >>> Documentation efforts.  I presented the idea of using Swagger several
> >>> summits (years) ago about using Swagger to help drive our Traffic Ops
> API
> >>> documentation.  Since then Swagger has evolved and is becoming the de
> >> facto
> >>> standard for building (the potential for generating TO Golang Client
> and
> >>> Server stubs is now available) and documenting REST APIs.
> >>>
> >>> I would like to propose going forward that we use Swagger for future
> API
> >>> level documentation (because it can be generated out of our Golang
> >>> code/structs).  The below resources point out a TO API version 1.3 (the
> >>> 

Traffic Ops API Swagger Doc

[Dewayne Richardson]

We've been working diligently on the TO Golang Rewrite
 project to
obviously rewrite the Perl into Go, but also to improve our Testing and
Documentation efforts.  I presented the idea of using Swagger several
summits (years) ago about using Swagger to help drive our Traffic Ops API
documentation.  Since then Swagger has evolved and is becoming the de facto
standard for building (the potential for generating TO Golang Client and
Server stubs is now available) and documenting REST APIs.

I would like to propose going forward that we use Swagger for future API
level documentation (because it can be generated out of our Golang
code/structs).  The below resources point out a TO API version 1.3 (the
version we decided to rev for the rewritten Golang endpoints).  The intent
behind 1.3 is obviously an improved version of the API (entirely backward
compatible to 1.2), but also to give us a starting point for building the
API doc in Swagger.

The following resources are my examples:

Swagger has several implementations and I chose go-swagger
 because it has more Golang
features.

*Sample TO API doc *

https://app.swaggerhub.com/apis/dewrich/traffic-ops_api/1.3


*Sample TO Golang code with embedded doc*

Generated from the combination of these Golang documentation "hooks"
(there's current a go-swagger bug that prevents the doc from being tied
directly into the handlers)
https://github.com/dewrich/incubator-trafficcontrol/tree/swagger-demo/traffic_ops/traffic_ops_golang/docs

And the *asns.go*, *cdns.go*, *divisions.go*, *regions.go* and *statuses.go*
structs in my branch here:
https://github.com/dewrich/incubator-trafficcontrol/tree/swagger-demo/lib/go-tc


*TO Client Generated from Swagger*

This new Golang package is only a sample of a TO client generated (based
upon the the code generated swagger.json

)

https://github.com/dewrich/incubator-trafficcontrol/tree/swagger-demo/traffic_ops/traffic_ops_golang/toclient
https://github.com/dewrich/incubator-trafficcontrol/tree/swagger-demo/traffic_ops/traffic_ops_golang/toclient/main.go

The hope with tying the documentation closer to the code will help with
keeping the API docs up-to-date, as well as providing more documentation
for developers.

Please give your thoughts on this idea as well as a vote by Feb 21 (a week
from today) so that we can move forward with building our TO API doc.

-Dew

Github PR/Issues Format Templates

[Dewayne Richardson]

I was working through the go-swagger repo and found a bug.  I submitted a
new issue and found this interesting approach I think the TC github should
adopt, "Issue and PR Templates".  I think the main value here is
consistency in our PRs/Issues and user friendly prompts to say "these are
the data points we need to help you solve your issue".

Working example:
https://github.com/go-swagger/go-swagger/issues/new

Github Doc on how to implement templates:
https://github.com/blog/2111-issue-and-pull-request-templates

If we think it's a good idea, then I'll respond with some examples for
Issues and PR's that we can discuss.

-Dew

Re: Golang Proxy Validation Dependencies

[Dewayne Richardson]

So, for the vote of using the validation dependencies we had

5 +1's for using the Validation libs

So the vote passes, that these dependencies will be used for the Golang
Proxy validation rules.

This PR will be can now be merged:
https://github.com/apache/incubator-trafficcontrol/pull/1766

Thank you for taking the time to respond,

-Dew

On Wed, Jan 17, 2018 at 9:25 AM, Volz, Dylan <dylan_v...@comcast.com> wrote:

> +1
>
> On 1/17/18, 5:43 AM, "John Rushford" <jjrushf...@gmail.com> wrote:
>
> +1
>
> Sent from my iPad
>
> > On Jan 16, 2018, at 2:07 PM, Dave Neuman <neu...@apache.org> wrote:
> >
> > +1
> >
> >> On Tue, Jan 16, 2018 at 12:58 Jan van Doorn <j...@knutsel.com>
> wrote:
> >>
> >> +1 on using libs.
> >>
> >>> On Jan 16, 2018, at 10:52 AM, Dan Kirkwood <dang...@gmail.com>
> wrote:
> >>>
> >>> +1 -- agree with Jeff -- the validation of the fields of
> >>> deliveryservice is something that is incomplete in the Perl
> >>> traffic_ops.
> >>>
> >>> These libraries make for concise code to do the validation so it
> will
> >>> be easier to extend without much extra code.   It will not be
> called
> >>> on every API function,  but only once on each POST/PUT which are a
> >>> small minority of calls.   It also need not be used in every case.
> >>> That, to me,  makes the reflection argument much less of a concern.
> >>>
> >>> I would like to see it go in sooner than Friday,  but won't argue
> that
> >> point..
> >>>
> >>> -dan
> >>>
> >>>
> >>> On Tue, Jan 16, 2018 at 10:10 AM, Dewayne Richardson <
> dewr...@gmail.com>
> >> wrote:
> >>>> So, it's been a few days on this topic and I'd like to call a
> vote for
> >> the
> >>>> dependencies listed in this thread.  Please vote +1 or -1 by Noon
> >> Friday so
> >>>> that we can move forward the Golang Proxy development.
> >>>>
> >>>> Thanks,
> >>>>
> >>>> -Dew
> >>>>
> >>>> On Thu, Jan 11, 2018 at 10:53 AM, Jeff Elsloo <els...@apache.org>
> >> wrote:
> >>>>
> >>>>> I don't think we should assume anything about the performance
> just
> >>>>> because it uses reflection. Yes, traditionally reflection is
> >>>>> computationally expensive, however, when used properly the
> penalty can
> >>>>> be negligible. I don't think we have enough understanding of
> these
> >>>>> libraries to know whether there is a concerning performance
> penalty.
> >>>>>
> >>>>> As Dewayne said, create, update and delete actions represent a
> tiny
> >>>>> fraction of the overall requests into TO. Given that the
> majority of
> >>>>> these actions are performed by humans, I would be shocked if
> there was
> >>>>> a perceptible performance difference with the reflection based
> >>>>> validation in place. It's not like we're trying to validate
> enormous
> >>>>> and complex objects here; we're talking 20 fields or so for any
> given
> >>>>> post.
> >>>>>
> >>>>> I'm +1 on using validation libraries such as these even if they
> use
> >>>>> reflection, provided that we do not see dramatic changes in
> >>>>> performance. I think that's highly unlikely in this case.
> >>>>> --
> >>>>> Thanks,
> >>>>> Jeff
> >>>>>
> >>>>>
> >>>>> On Thu, Jan 11, 2018 at 10:07 AM, Chris Lemmons <
> alfic...@gmail.com>
> >>>>> wrote:
> >>>>>> True, but how many of those out-of-the-box checks are both
> useful and
> >>>>>> relevantly complex?
> >>>>>>
> >>>>>> To me, the cool part of ozzo is the way it collects the output
> and
> >>>>>> formats it. That's unfortunately also the computationally
> expensive
> >>>>>> part.
> >>>>>>
> >>>

Re: [VOTE] Release Apache Traffic Control (incubating) 2.1.0-RC3

[Dewayne Richardson]

Awesome, great work!

On Fri, Jan 19, 2018 at 6:47 AM, Hank Beatty  wrote:

> Hello,
>
> It would appear RC3 has passed the IPMC vote. I'll do the release today.
>
> Regards,
> Hank
>
> On 2018-01-17 17:01, Phil Sorber  wrote:
> > +1 (binding). After having fought with docker for a bit I was able to
> build
> > packages. Sig and hashes check out as well. We should work on getting all
> > our GPG keys signed by each other next summit, but that is not a show
> > stopper in my opinion.
> >
> > Thanks!
> >
> > On Tue, Jan 16, 2018 at 11:45 AM Leif Hedstrom  wrote:
> >
> > > +1 (binding). Tested the build, RPMs successfully built, and double
> > > checked the RAT report.
> > >
> > > Cheers
> > >
> > > %u2014 Leif
> > >
> > > > On Jan 10, 2018, at 18:04, Justin Mclean 
> > > wrote:
> > > >
> > > > Hi,
> > > >
> > > > +1 (binding)
> > > >
> > > > I checked:
> > > > - incubating in name
> > > > - hashes and signatures good
> > > > - DISCLAIMER exists
> > > > - LICENSE good and NOTICE OK (see below)
> > > > - no unexpected binary files
> > > > - All ALv2 source files have ASF headers
> > > >
> > > > For the NOTICE don%u2019t forget to update the year to be the
> current one and
> > > there%u2019s no need to list information about the fonts inside it.
> > > >
> > > > Thanks,
> > > > Justin
> > > > 
> -
> > > > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> > > > For additional commands, e-mail: general-h...@incubator.apache.org
> > > >
> > >
> > >
> > > -
> > > To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org
> > > For additional commands, e-mail: general-h...@incubator.apache.org
> > >
> > >
> >
>

Re: Move Docs to readthedocs.io ?

[Dewayne Richardson]

+1for anything that will help us streamline doc,

-Dew

On Tue, Jan 2, 2018 at 12:23 PM, Jan van Doorn  wrote:

> I propose we move our docs to readthedocs - like :
> http://incubator-trafficcontrol.readthedocs.io/en/latest/ (this is setup
> w my clone).
>
> This will allow us to have multiple versions of docs online, do automatic
> updates of the released and latest docs and will get us the "Edit on
> Github" button that Ryan requested ( https://github.com/apache/incu
> bator-trafficcontrol/issues/1625 ).
>
>
> Thoughts?
>
>
> JvD
>
>

Re: [VOTE] CHANGELOG.md file (second try)

[Dewayne Richardson]

As a POC for the Change Log label follow this link:

https://github.com/apache/incubator-trafficcontrol/pulls?q=is%3Apr+is%3Aclosed+label%3A%22change+log%22+milestone%3A2.2.0

-Dew

On Thu, Dec 14, 2017 at 10:48 AM, Gelinas, Derek <derek_geli...@comcast.com>
wrote:

> I'm +1 for the label as well.
>
> > On Dec 14, 2017, at 12:29 PM, Robert Butts <robert.o.bu...@gmail.com>
> wrote:
> >
> > +1 on a "changelog" label. Seems like that would make it a lot easier for
> > the person writing it up. Easier to skip things like code maintenance
> that
> > have no interface effect.
> >
> > On Thu, Dec 14, 2017 at 10:14 AM, Dewayne Richardson <dewr...@gmail.com>
> > wrote:
> >
> >> Another idea would be to include a new CHANGELOG label that you could
> tack
> >> onto specific PR's that you wanted to be included (as well as grouped
> >> within Milestones) as the high level features that went into the
> release.
> >> As for the gotchas, I think those should be Github issues because to me
> >> those were bugs.
> >>
> >> -Dew
> >>
> >> On Thu, Dec 14, 2017 at 10:01 AM, Jeremy Mitchell <
> mitchell...@gmail.com>
> >> wrote:
> >>
> >>> I agree. Very readable. I also like the idea of a either creating a
> >>> CHANGELOG.md for each component...or one CHANGELOG.md with sections for
> >>> each component.
> >>>
> >>> I still do like the idea of creating issues with milestones but I'll
> let
> >>> that go. That seems to be a personal preference of mine.
> >>>
> >>> Jeremy
> >>>
> >>>> On Thu, Dec 14, 2017 at 9:45 AM, Dave Neuman <neu...@apache.org>
> wrote:
> >>>>
> >>>> Have you taken a look at some Changelogs of other github projects?
> >> Here
> >>>> are a few from other projects we use in Traffic Control:
> >>>> - Docker Compose: https://github.com/docker/
> >>> compose/blob/master/CHANGELOG.
> >>>> md
> >>>> - InfluxDB: https://github.com/influxdata/influxdb/blob/master/
> >>>> CHANGELOG.md
> >>>> - Grafana: https://github.com/grafana/grafana/blob/master/CHANGELOG.
> md
> >>>> - Ansible: https://github.com/ansible/ansible/blob/devel/CHANGELOG.md
> >>>>
> >>>> See how easy to read those are and how they provide a lot of
> >> information
> >>>> without having to wade through hundreds of issues and pull requests?
> >>> Some
> >>>> of them also have links to issues for new features, as well as bug
> >> fixes
> >>>> that are in the current release.  I think all of them are easy to read
> >>> and
> >>>> can give a user a quick overview of what is in the new release. I
> think
> >>>> it's fine to add a link to all the issues if we want, but I think
> >>>> ultimately we want to create something like what I have linked to
> >> above.
> >>>> We might also want to break out our CHANGELOG.md by component to
> >> provide
> >>>> even more readability.
> >>>>
> >>>> I know our first inclination is to try to automate everything, but
> >>>> sometimes it makes sense to do things manually so that you can create
> a
> >>>> better output.
> >>>>
> >>>>
> >>>>
> >>>> On Thu, Dec 14, 2017 at 8:55 AM, Jeremy Mitchell <
> >> mitchell...@gmail.com>
> >>>> wrote:
> >>>>
> >>>>> What if CHANGLOG.md includes 2 things:
> >>>>>
> >>>>> 1. a link to 2.2 issues (i.e.
> >>>>> https://github.com/apache/incubator-trafficcontrol/
> >>>>> issues?q=is%3Aopen+is%3Aissue+milestone%3A2.2.0
> >>>>> 2. a bulleted list of 2.2 gotchas (i.e. Traffic Ops Golang doesn't
> >>>> connect
> >>>>> to a Riak with self-signed certificates; Riak security grant needs
> >>>> updated;
> >>>>> upgrade param needed for ds routing names, etc, etc...)
> >>>>>
> >>>>> But again this requires everyone to create issues (with a milestone)
> >> in
> >>>>> which one or more PR's can be attached to.
> >>>>>
> >>>>> Dave, you said "If you are developing a new feature you could easily
> >>> end
> >>>> up
> >>>

Re: [VOTE] CHANGELOG.md file (second try)

[Dewayne Richardson]

Another idea would be to include a new CHANGELOG label that you could tack
onto specific PR's that you wanted to be included (as well as grouped
within Milestones) as the high level features that went into the release.
As for the gotchas, I think those should be Github issues because to me
those were bugs.

-Dew

On Thu, Dec 14, 2017 at 10:01 AM, Jeremy Mitchell 
wrote:

> I agree. Very readable. I also like the idea of a either creating a
> CHANGELOG.md for each component...or one CHANGELOG.md with sections for
> each component.
>
> I still do like the idea of creating issues with milestones but I'll let
> that go. That seems to be a personal preference of mine.
>
> Jeremy
>
> On Thu, Dec 14, 2017 at 9:45 AM, Dave Neuman  wrote:
>
> > Have you taken a look at some Changelogs of other github projects?  Here
> > are a few from other projects we use in Traffic Control:
> > - Docker Compose: https://github.com/docker/
> compose/blob/master/CHANGELOG.
> > md
> > - InfluxDB: https://github.com/influxdata/influxdb/blob/master/
> > CHANGELOG.md
> > - Grafana: https://github.com/grafana/grafana/blob/master/CHANGELOG.md
> > - Ansible: https://github.com/ansible/ansible/blob/devel/CHANGELOG.md
> >
> > See how easy to read those are and how they provide a lot of information
> > without having to wade through hundreds of issues and pull requests?
> Some
> > of them also have links to issues for new features, as well as bug fixes
> > that are in the current release.  I think all of them are easy to read
> and
> > can give a user a quick overview of what is in the new release. I think
> > it's fine to add a link to all the issues if we want, but I think
> > ultimately we want to create something like what I have linked to above.
> > We might also want to break out our CHANGELOG.md by component to provide
> > even more readability.
> >
> > I know our first inclination is to try to automate everything, but
> > sometimes it makes sense to do things manually so that you can create a
> > better output.
> >
> >
> >
> > On Thu, Dec 14, 2017 at 8:55 AM, Jeremy Mitchell 
> > wrote:
> >
> > > What if CHANGLOG.md includes 2 things:
> > >
> > > 1. a link to 2.2 issues (i.e.
> > > https://github.com/apache/incubator-trafficcontrol/
> > > issues?q=is%3Aopen+is%3Aissue+milestone%3A2.2.0
> > > 2. a bulleted list of 2.2 gotchas (i.e. Traffic Ops Golang doesn't
> > connect
> > > to a Riak with self-signed certificates; Riak security grant needs
> > updated;
> > > upgrade param needed for ds routing names, etc, etc...)
> > >
> > > But again this requires everyone to create issues (with a milestone) in
> > > which one or more PR's can be attached to.
> > >
> > > Dave, you said "If you are developing a new feature you could easily
> end
> > up
> > > with 5 or more PRs"
> > >
> > > So in this case, I'd expect 1 issue and 5 PR's linked to that 1
> issue...
> > >
> > > Jeremy
> > >
> > > On Thu, Dec 14, 2017 at 8:40 AM, Dave Neuman 
> wrote:
> > >
> > > > I think it's fine to include all of our PRs and issues in a github
> > > > milestone, and we should do that, but I don't think that we want to
> > > include
> > > > every single PR in our changelog.  When we have tried that in the
> past
> > we
> > > > have realized that it gets to be so much information that it's not
> > > useful.
> > > > A good example of why is a new feature. If you are developing a new
> > > feature
> > > > you could easily end up with 5 or more PRs: one to introduce the
> > feature,
> > > > one to add some more functionality, several to fix bugs with it, etc.
> > > > Instead of having a line in the changelog for each one of those PRs,
> we
> > > > should just have one line that says "introduced feature X" with
> maybe a
> > > > blurb about what it is and any release notes that an operator would
> > need
> > > to
> > > > know.  This way the file in concise and easy to understand by anyone
> > > > wanting to use a new version of our product.  I think it's also
> > important
> > > > to include what bug fixes (since the previous release) that we have
> > > fixed.
> > > > Those are the reasons why I tend to lean towards a manual changelog.
> > It
> > > > gives us the ability to control how much information goes into the
> > > > changelog and the flexibility to make sure it is useful.
> > > >
> > > > On Thu, Dec 14, 2017 at 8:13 AM, Jeremy Mitchell <
> > mitchell...@gmail.com>
> > > > wrote:
> > > >
> > > > > Why not leverage Github issues/milestones to determine the scope of
> > > each
> > > > > release? Per our CONTRIBUTING.md
> > > > >  > > > > master/CONTRIBUTING.md>
> > > > > :
> > > > >
> > > > > "If you have improvements (enhancements or bug fixes) for Traffic
> > > > Control,
> > > > > start by creating an issue
> > > > > ..."
> > > > >
> > > > > That implies that all 

Traffic OPS API Test Tool

[Dewayne Richardson]

I've been working on a PR to introduce a new Traffic Ops API tool based on
the existing traffic_ops/client_tests

area (if approved this area will be deprecated).  The hope is to use this
tool to build up more functionality within the Go TO Client to reach
feature parity with the Traffic Ops Go Rewrite evolves as well.  Also, the
reason for the rewrite was to brand this tool for more robust API tests and
to move away from the term "integration test" because that implies more
than two systems working together as a test.

Currently only the /cdns endpoint was implemented as a POC, and the intent
is the Go /cdns route will become an end-to-end example of how to implement
a Go CRUD endpoint, along with the test cases and documentation to follow.
Once approved this tool will hopefully be used for CI regression testing
against the TO API.

See the following for more info:

https://github.com/apache/incubator-trafficcontrol/pull/1604


Thanks,

-Dew

Re: Traffic Ops Golang Unit Tests Pattern

[Dewayne Richardson]

Hi Kevin,

Thanks for your question.  I'm not sure we want to change the method
signature that won't be used by the running Golang, but haven't said that
I'd like to see an example of what you are describing.  (Maybe a github
gist?)

-Dew

On Tue, Nov 7, 2017 at 2:34 PM, Kevin Mackenzie  wrote:

> Hello,
>
> I’m a student at RPI working with Eric Friedrich on some Golang routes for
> Traffic Ops and I am working on writing unit tests for my code.  I am using
> the same method as the other tests for mocking SQL queries for the
> functions that make them, but for the functions that call the functions
> that make the SQL queries, mocking the SQL queries seems kind of
> redundant.  I noticed this is how “TestGetMonitoringJson” in
> monitoring_test.go works.
>
> One method I found is to mock the functions that make the SQL queries, but
> this would require adding function arguments to the functions like
> “getMonitoringJson” for the sake of testing.
>
> For example, “getMonitoringJson” would require additional parameters of
> types:
>
> type MonitoringServercesGetter(db *sqlx.DB, cdn string) []Monitor,
> []Cache, []Router, error
> type CachegroupsGetter(db *sqlx.DB, cdn string) []Cachegroup, error
> type ProfilesGetter(db *sqlx.DB, caches []Cache, routers []Router)
> []Profile, error
> type DeliverServicesGetter(db *sql.DB, routers []Router)
> []DeliverServices, error
> type ConfigGetter(db *sqlx.DB) map[string]interface{}, error
>
> Since “getMonitoringJson” calls functions with above signatures, the
> caller of “getMonitoringJson” will need to pass the normal versions of
> these functions to it, but in the testing methods, mock versions of these
> methods that don’t make SQL queries can be passed.
>
> Would it be better to integrate this kind of pattern into my code to
> simplify the testing methods or to follow the existing pattern and mock all
> of the SQL queries required by functions like “getMonitoringServers” and
> “getCachegroups” when testing functions like “getMonitoringJson”?
>
> Thanks,
> Kevin Mackenzie
>

Re: [VOTE] Release Apache Traffic Control (incubating) 2.1.0-RC2

[Dewayne Richardson]

Nice work 72 hours on a Friday!

On Fri, Nov 10, 2017 at 11:21 AM, Hank Beatty  wrote:

> Hello All,
>
> I've prepared a release for v2.1.0-RC1
>
> The vote is open for at least 72 hours and passes if a majority of at
> least 3 +1 PPMC votes are cast.
>
> [ ] +1 Approve the release
>
> [ ] -1 Do not release this package because ...
>
> Changes since 2.0.0:
> https://github.com/apache/incubator-trafficcontrol/compare/
> 2.0.x...RELEASE-2.1.0-RC2
>
> This corresponds to git:
>   Hash: cac666ef7f0626ea8180e976b07fa841d53f755f
>   Tag: RELEASE-2.1.0-RC2
>
> Which can be verified with the following: git tag -v RELEASE-2.1.0-RC1
>
> My code signing key is available here:
> https://pgp.mit.edu/pks/lookup?op=get=0x920152B94E0CC77C
>
> The source .tar.gz file, pgp signature (.asc signed with my key from
> above), md5 and sha512 checksums are provided here:
> https://dist.apache.org/repos/dist/dev/incubator/trafficcontrol/2.1.0/RC2
>
>
> Thanks!
>

TC Golang structs refactor

[Dewayne Richardson]

FYI: There's been an ongoing need to refactor the Golang structs to an area
of the code base that affect Traffic Monitor, Traffic Stats, Traffic Ops
Client, and the Golang Proxy.  We've moved all common structs to this area
https://github.com/apache/incubator-trafficcontrol/tree/master/lib/go-tc
and changed the client areas accordingly (so the Golang code all
compiles).

We plan on doing extensive regression testing to ensure no breakage, but
this change was only a refactor of the structs no logic changes.

https://github.com/apache/incubator-trafficcontrol/pull/1367


-Dew

Re: Traffic Ops Rewrite Golang Dependency - sqlx

[Dewayne Richardson]

The deadline of Friday 09-15 at noon has passed, sqlx as a TO Golang
Dependency has been accepted. The vote tally is 6 Aye to 2 Nay's

Have a good weekend,

-Dew

Results:
Dan Kirkwood: +1
Derek Gelina: +1
Dewayne Richardson: +1
Dylan Volz: +1
Jan Van Doorn: +1
Jeremy Mitchell: +1

Chris Lemmons: -1
Rob Butts: -1

On Wed, Sep 13, 2017 at 10:05 AM, Volz, Dylan (Contractor) <
dylan_v...@comcast.com> wrote:

> I see, just looked at SliceScan, I mistakenly assumed it was how they
> implement selecting into a slice of structs which is one of the features
> that is very useful versus the straight sql implementation.
>
> On 9/13/17, 9:59 AM, "Chris Lemmons" <alfic...@gmail.com> wrote:
>
> That's not the problem that SliceScan helps with. SliceScan helps when
> you
> don't have control of the sql statement and you don't know the columns
> that
> will be returned, or how many will be returned. MapScan does the same,
> but
> puts it into a map, so you can iterate over them. That's a major
> feature
> that isn't available in the standard `sql` library, and would
> definitely
> justify the inclusion of `sqlx`. I don't think that's something we
> really
> want, though, because we control the sql statements.
>
> On Wed, Sep 13, 2017 at 8:54 AM, Volz, Dylan (Contractor) <
> dylan_v...@comcast.com> wrote:
>
> > Yes Select, or the SliceScan is one of the reasons I would use sqlx,
> I am
> > pulling out sets using IN and left outer joins to avoid the tight
> looping
> > issues in some endpoints in the perl.
> >
> > On 9/13/17, 8:42 AM, "Chris Lemmons" <alfic...@gmail.com> wrote:
> >
> > I see that there's a significant preference to not list the
> members of
> > the
> > struct in the Scan function. That works.
> >
> > I'm still not certain I understand why we'd prefer to bring in
> all of
> > `sqlx` instead of writing a single, fairly simple function to
> return
> > the
> > columns for the scan. Are there plans to use MapScan or
> SliceScan?
> > Cause
> > that's really where `sqlx` starts to be worth it's cost.
> >
> > Still, to be sure we're all on the same page wrt performance, the
> > reason
> > the performance is fairly good is because the cost is per-query,
> not
> > per-row. Since we're pretty good about our queries, I'm fairly
> happy
> > with
> > it's performance. If we're doing queries in a tight loop, it will
> > start to
> > bite us, but network delay would bite us harder.
> >
> > On Wed, Sep 13, 2017 at 8:18 AM, Dewayne Richardson <
> dewr...@gmail.com
> > >
> > wrote:
> >
> > > I also agree but I think your argument about future cost is
> more
> > > "speculative" than reality.  If we're smart about how we
> integrate
> > sqlx,
> > > then we can hedge the bet.
> > >
> > > -Dew
> > >
> > > On Tue, Sep 12, 2017 at 7:52 PM, Robert Butts <
> > robert.o.bu...@gmail.com>
> > > wrote:
> > >
> > > > I am a pretty big -1 on sqlx. Those PRs are extremely
> deceptive.
> > > >
> > > > Those lines are entirely unnecessary.
> > > >
> > > > I have created an example PR at
> https://github.com/apache/incu
> > > > bator-trafficcontrol/pull/1165
> > > >
> > > > The relevant commits are
> > > > https://github.com/apache/incubator-trafficcontrol/pull/1165
> > > > /commits/6fc735d7f97eaaffbf08e8457b7ccb6bf14baca0
> > > > https://github.com/apache/incubator-trafficcontrol/pull/1165
> > > > /commits/6939ee1d401c571af139db53b018a5e53f80c02a#diff-219ca
> > > > ea1a282285fe1cc21e53bf9dafbL26
> > > >
> > > > As you can see, the only difference is that
> `rows.StructScan()`
> > > becomes `
> > > > rows.Scan(, , ,
> , 
> > > > DomainName, , , , ,
> > ,
> > > 
> > > > IloIpGateway, , ,
> , 
> > > > InterfaceMtu, , , ,
> > > ,
> > > > , , ,
> , 
> > > > MgmtIpGateway, , ,
> > , 
> > > > PhysLocation

Re: Traffic Ops Rewrite Golang Dependency - sqlx

[Dewayne Richardson]

I also agree but I think your argument about future cost is more
"speculative" than reality.  If we're smart about how we integrate sqlx,
then we can hedge the bet.

-Dew

On Tue, Sep 12, 2017 at 7:52 PM, Robert Butts <robert.o.bu...@gmail.com>
wrote:

> I am a pretty big -1 on sqlx. Those PRs are extremely deceptive.
>
> Those lines are entirely unnecessary.
>
> I have created an example PR at https://github.com/apache/incu
> bator-trafficcontrol/pull/1165
>
> The relevant commits are
> https://github.com/apache/incubator-trafficcontrol/pull/1165
> /commits/6fc735d7f97eaaffbf08e8457b7ccb6bf14baca0
> https://github.com/apache/incubator-trafficcontrol/pull/1165
> /commits/6939ee1d401c571af139db53b018a5e53f80c02a#diff-219ca
> ea1a282285fe1cc21e53bf9dafbL26
>
> As you can see, the only difference is that `rows.StructScan()` becomes `
> rows.Scan(, , , , 
> DomainName, , , , , , 
> IloIpGateway, , , , 
> InterfaceMtu, , , , ,
> , , , , 
> MgmtIpGateway, , , , 
> PhysLocationId, , , , , 
> RevalPending, , , , ,
> , , , , , 
> XmppPasswd)`
>
> It is a one-line difference per endpoint, not 100 lines. (Plus column
> annotations on every struct field for sqlx)
>
> That said, I agree the former is better for readability. The issue is the
> maintenance cost, when-not-if sqlx stops being maintained. It will be
> embedded in Traffic Ops, in every single endpoint and query. We'll be in
> exactly the same position we are with Goose, stuck with an unmaintained and
> probably vulnerable library, which is very expensive in developer-hours to
> remove. Surely most of us here have been in this situation, with legacy
> unmaintained apps, libraries, compilers, etc?
>
> By `cloc` Sqlx is 3400 lines, which doesn't sound like a lot, but a big
> percentage of that is Go Reflection, which is exceedingly painful to write,
> debug, and maintain.
>
> Is standard Go really that much more difficult to write? The above is one
> of the worst cases (along with Deliveryservices), most of our tables aren't
> nearly that big. It doesn't seem likely to cause bugs, any mismatches
> should be immediately caught when running the first time, and certainly by
> the tests we've been mandating.
>
> I'm not wholesale against third-party libraries. Often the benefit
> outweighs the cost; for example, `sqlmock`, and in the future, `jwt`. But
> in this particular case, the maintenance cost far outweighs the benefit.
>
> This isn't a black-and-white issue, it's a cost-benefit analysis. Sqlx is
> marginally easier to write, for an unknowable and potentially enormous
> future cost.
>
>
> On Tue, Sep 12, 2017 at 6:54 PM, Volz, Dylan (Contractor) <
> dylan_v...@comcast.com> wrote:
>
> > It would be maintaining about a 1500 line codebase (excluding tests with
> > ~70% coverage), it uses reflection and tag introspection so it isn’t the
> > simplest go code but it does seem to be well commented.
> >
> > On 9/12/17, 6:36 PM, "Gelinas, Derek" <derek_geli...@comcast.com> wrote:
> >
> > After looking at the code, and given the work I've been doing with
> > rewriting the config file endpoints, I have to say sqlx all the way.
> > What's involved in the maintenance?
> >
> > Derek
> >
> > On Sep 12, 2017, at 8:28 PM, Dewayne Richardson <dewr...@gmail.com
> > <mailto:dewr...@gmail.com>> wrote:
> >
> > There has been quite a bit of discussion about how to move forward
> > with the
> > Traffic Ops Rewrite in terms of Golang dependencies.  Currently there
> > is
> > only one dependency for Mocking out the database for unit testing
> > called
> > https://github.com/DATA-DOG/go-sqlmock. Another that we want to
> > evaluate is
> > https://github.com/jmoiron/sqlx for accessing Postgres to help with
> > minimizing Golang boilerplate code.  The following are the Pros and
> > Cons
> > are listed to he
> > lp decide (please add any that I failed to include)
> >
> >
> > Pros
> > - Developer productivity increases (less boilerplate code)
> > - Less Developer errors through tedious field mapping
> > - Active Development
> >
> > Cons
> > - Another dependency to maintain if it is no longer supported
> >
> > Performance
> >  The performance penalty is neglible (I tested the /api/1.2/servers
> > endpoint, very loosely, in the Comcast Open Stack lab using the
> >  same VM and Apache Bench with 1000, 1, and 1 separate
> requests
> > and the performance was +/-5% depending on the cloud resources that
> > were
>

Re: Traffic Ops Rewrite Golang Dependency - sqlx

[Dewayne Richardson]

I'm also not 100% on sqlx 100% of the time, but what I'm 100% on is ease of
use, development productivity, and minimizing bug introduction.

-Dew

On Tue, Sep 12, 2017 at 9:08 PM, Chris Lemmons <alfic...@gmail.com> wrote:

> @dan, how do you feel about the middle-ground I proposed: a
> reflection-based function that would look like this:
> rows.Scan(FieldsOf()...) ?
>
> On Tue, Sep 12, 2017 at 9:05 PM, Dan Kirkwood <dang...@gmail.com> wrote:
>
> > As one ready to jump in and add more endpoints,  I'm a strong +1 on
> > using sqlx.  I agree that adding a new dependency should not be done
> > without consideration,  but I find the sqlx version much more readable
> > and easier to approach than either your or dew's version of non-sqlx
> > and would be much easier to approach for one unfamiliar with details
> > of this project.   For me,   it's worth it.
> >
> > strong +1
> >
> > -dan
> >
> >
> > On Tue, Sep 12, 2017 at 7:52 PM, Robert Butts <robert.o.bu...@gmail.com>
> > wrote:
> > > I am a pretty big -1 on sqlx. Those PRs are extremely deceptive.
> > >
> > > Those lines are entirely unnecessary.
> > >
> > > I have created an example PR at https://github.com/apache/incu
> > > bator-trafficcontrol/pull/1165
> > >
> > > The relevant commits are
> > > https://github.com/apache/incubator-trafficcontrol/pull/1165
> > > /commits/6fc735d7f97eaaffbf08e8457b7ccb6bf14baca0
> > > https://github.com/apache/incubator-trafficcontrol/pull/1165
> > > /commits/6939ee1d401c571af139db53b018a5e53f80c02a#diff-219ca
> > > ea1a282285fe1cc21e53bf9dafbL26
> > >
> > > As you can see, the only difference is that `rows.StructScan()`
> > becomes `
> > > rows.Scan(, , , , 
> > > DomainName, , , , , ,
> > 
> > > IloIpGateway, , , , 
> > > InterfaceMtu, , , ,
> > ,
> > > , , , , 
> > > MgmtIpGateway, , , , 
> > > PhysLocationId, , , , , 
> > > RevalPending, , , ,
> > ,
> > > , , , , ,
> 
> > > XmppPasswd)`
> > >
> > > It is a one-line difference per endpoint, not 100 lines. (Plus column
> > > annotations on every struct field for sqlx)
> > >
> > > That said, I agree the former is better for readability. The issue is
> the
> > > maintenance cost, when-not-if sqlx stops being maintained. It will be
> > > embedded in Traffic Ops, in every single endpoint and query. We'll be
> in
> > > exactly the same position we are with Goose, stuck with an unmaintained
> > and
> > > probably vulnerable library, which is very expensive in developer-hours
> > to
> > > remove. Surely most of us here have been in this situation, with legacy
> > > unmaintained apps, libraries, compilers, etc?
> > >
> > > By `cloc` Sqlx is 3400 lines, which doesn't sound like a lot, but a big
> > > percentage of that is Go Reflection, which is exceedingly painful to
> > write,
> > > debug, and maintain.
> > >
> > > Is standard Go really that much more difficult to write? The above is
> one
> > > of the worst cases (along with Deliveryservices), most of our tables
> > aren't
> > > nearly that big. It doesn't seem likely to cause bugs, any mismatches
> > > should be immediately caught when running the first time, and certainly
> > by
> > > the tests we've been mandating.
> > >
> > > I'm not wholesale against third-party libraries. Often the benefit
> > > outweighs the cost; for example, `sqlmock`, and in the future, `jwt`.
> But
> > > in this particular case, the maintenance cost far outweighs the
> benefit.
> > >
> > > This isn't a black-and-white issue, it's a cost-benefit analysis. Sqlx
> is
> > > marginally easier to write, for an unknowable and potentially enormous
> > > future cost.
> > >
> > >
> > > On Tue, Sep 12, 2017 at 6:54 PM, Volz, Dylan (Contractor) <
> > > dylan_v...@comcast.com> wrote:
> > >
> > >> It would be maintaining about a 1500 line codebase (excluding tests
> with
> > >> ~70% coverage), it uses reflection and tag introspection so it isn’t
> the
> > >> simplest go code but it does seem to be well commented.
> > >>
> > >> On 9/12/17, 6:36 PM, "Gelinas, Derek" <derek_geli...@comcast.com>
> > wrote:
> > >>
> > >> After looking at the code, and given the work I've been doing with
> > >> rewriting the config file end

Re: [VOTE] Bugtracking in Github Issues

[Dewayne Richardson]

+1

On Mon, Aug 28, 2017 at 10:44 AM, Dave Neuman  wrote:

> I thought we already had a vote on this?  Maybe I am thinking of the "move
> to github" vote which I assumed to be all-encompassing.
> Anyway, +1
>
> On Mon, Aug 28, 2017 at 10:43 AM, Dan Kirkwood  wrote:
>
> > +1
> >
> > On Mon, Aug 28, 2017 at 10:38 AM, Eric Friedrich (efriedri)
> >  wrote:
> > > We currently use JIRA Issues to track all of the Traffic Control bugs.
> > >
> > > Now that we have write access to Github, we can move back to GH Issues
> > for bug tracking.
> > >
> > > This will be a better workflow because its one fewer tool and account
> to
> > have to interact with. This will hopefully lower the bar for new
> > contributors to file bugs and engage with the product. We can also help
> use
> > it (along with the milestones) to create more useful changelogs and
> release
> > notes.
> > >
> > > A possible downside is that the Issues are maybe less flexible than
> JIRA
> > in terms of permissions, workflow, fields, etc. However, we were using
> > Issues before we entered the incubator and that was fine for us.
> Hopefully
> > no one has developed an attachment for JIRA in the last year.
> > >
> > >
> > > Please Vote +1 to move to Github Issues or -1 to stay on Jira. I’ll
> > assume a lazy consensus if there aren’t enough votes.
> > >
> > > —Eric
> > >
> > >
> >
>

Re: Adding support for per-DeliveryService routing names

[Dewayne Richardson]

This is a tough one because the obvious ways of breaking an API are "URL
format changes", "Request or Response format changes", etc.  But I think
the more obvious way to think about the API is, do the clients have to
change their code?  If the answer is yes, it's a breaking API change.

So, really the only way around this is to "default" the new field's value
and make it optional, otherwise the API needs to rev.

-Dew

On Tue, Aug 15, 2017 at 8:09 AM, Jeremy Mitchell 
wrote:

> I don't think you can add a new not null column to the ds table because
> that would break the DS api. For example, on DS create/update you are
> saying routing_name is now required, right? This is an API breaking change
> that would require a new version of the API, no?
>
> Hence my suggestion for a default. But, yes, i forgot about the per CDN
> thing.
>
> Jeremy
>
> On Mon, Aug 14, 2017 at 5:25 PM, Peters, Rawlin  >
> wrote:
>
> > Jeremy’s suggestion could work, but the param would probably be created
> in
> > a TR_PROFILE per-CDN. However, that still wouldn’t fix the visibility
> > problem. If a CDN isn’t using the default “tr” HTTP routing name,
> operators
> > would still need to know that there is a new profile parameter that needs
> > updating post-upgrade but before a snap/queue. So either way there needs
> to
> > be sufficient upgrade notes, but personally I still prefer keeping the
> > routing_name column non-null.
> >
> > That said, this is my current proposal for the DB migration which also
> > gets us past the upgrade issue:
> > 1. Add a routing_name column to the DeliveryService table.
> > 2. Update the routing_name for DNS Delivery Services to “edge”.
> > 3. Update the routing_name of non-DNS Delivery Services to the value of a
> > temporary upgrade parameter associated with the Delivery Service’s CDN
> (if
> > the upgrade parameter doesn’t exist, the routing_names will remain null).
> > 4. Update the remaining null routing_names to “tr”.
> > 5. Make the routing_name column non-null and add a non-empty constraint.
> >
> > So these would be an operator’s pre-upgrade steps:
> > 1. Verify if a custom http.routing.name has been configured for Traffic
> > Routers in their CDNs.
> > 2. If custom http.routing.name, do the following. Otherwise, no
> > pre-upgrade steps needed (for per-DS routing names at least):
> > a. create a parameter named “upgrade_http_routing_name” with the
> value
> > of the target CDN’s custom http.routing.name
> > b. associate this parameter to the TR_PROFILE belonging to the target
> > CDN
> > c. repeat steps 2a and 2b for each CDN using a custom
> > http.routing.name
> >
> > This would keep everything working the same post-upgrade as it did
> > pre-upgrade, and from that point on you’d be able to change a Delivery
> > Service’s routing name to any arbitrary hostname (without periods).
> >
> > --Rawlin
> >
> > On 8/14/17, 4:22 PM, "Dave Neuman"  wrote:
> >
> > I don't think that solves the issue Rawlin was describing.  The issue
> > that
> > Rawlin was describing is that someone has already defined a different
> > routing name in traffic_router/http.properties which is no longer
> > going to
> > be used after the upgrade.  The upgrade needs to somehow know about
> > this
> > other routing name and use that when it creates the database column
> and
> > populates it with the pre-defined defaults (edge, tr).
> >
> > Also, creating a global param doesn't help those with more than 1
> CDN.
> >
> > On Mon, Aug 14, 2017 at 4:09 PM, Jeremy Mitchell <
> > mitchell...@gmail.com>
> > wrote:
> >
> > > Adding a temp parameter would work but I worry that someone won't
> > read the
> > > upgrade documentation and forget to create this temporary parameter
> > before
> > > running the upgrade.
> > >
> > > Here's another option.
> > >
> > > Create 2 global TO parameters (http.routing.name and
> > dns.routing.name
> > > ) that default to tr and edge
> > respectively and
> > > make the ds.routing_name an optional field.
> > >
> > > in seeds.sql
> > >
> > > insert into parameter (name, config_file, value) values ('
> > > http.routing.name',
> > > 'global', 'tr') ON CONFLICT (name, config_file, value) DO NOTHING;
> > > insert into parameter (name, config_file, value) values ('
> > dns.routing.name
> > > ',
> > > 'global', 'edge') ON CONFLICT (name, config_file, value) DO
> NOTHING;
> > >
> > > in code (warning. ugly pseudo code to follow):
> > >
> > > function getRoutingName(ds) {
> > > return ds.routing_name if not null
> > > if (ds.type = HTTP) {
> > > return parameter.http.routing.name
> > > } else
> > > return parameter.dns.routing.name
> > > }
> > > }
> > >
> > > Just my 2 cents.
> > >
> > > On Mon, Aug 14, 2017 at 10:55 AM, Dave Neuman 

Re: Traffic Ops Golang Rewrite

[Dewayne Richardson]

The goal is to hit all the low hanging fruit first to discover all the
refactor points and infrastructure before we attempt to bite off the bigger
more impactful areas of Traffic Control like Tenancy and Capabilities.

-Dew

On Mon, Aug 14, 2017 at 5:22 AM, Nir Sopher <n...@qwilt.com> wrote:

> +1!
>
> As far as tenancy is concerned, the main logic is held in the
> "Utils::Tenant" class.
> I would be happy to use this class as an entry point to the Golang TO
> development.
>
> BTW, the UT coverage of the tenancy checks is quite extensive.
> Are we going to migrate the UTs as well, write new UTs in Golang, or keep
> working with the Perl UTs?
>
> Nir
>
> On Wed, Aug 9, 2017 at 11:14 PM, Durfey, Ryan <ryan_dur...@comcast.com>
> wrote:
>
> > Great write up Dewayne.  Given the length and forward looking nature of
> > the topic I created a wiki page around this under traffic ops.  Please
> > continue discussion in the email thread and I will summarize any changes
> > there.
> >
> > https://cwiki.apache.org/confluence/display/TC/Golang+
> > Traffic+Ops+Replacement+-+Vampire+Proxy
> >
> > Ryan DurfeyM | 303-524-5099
> > CDN Support (24x7): 866-405-2993 or cdn_supp...@comcast.com > cdn_supp...@comcast.com>
> >
> >
> > From: Dewayne Richardson <dewr...@gmail.com>
> > Reply-To: "dev@trafficcontrol.incubator.apache.org" <
> > dev@trafficcontrol.incubator.apache.org>
> > Date: Wednesday, August 9, 2017 at 1:03 PM
> > To: "dev@trafficcontrol.incubator.apache.org" <
> > dev@trafficcontrol.incubator.apache.org>
> > Subject: Traffic Ops Golang Rewrite
> >
> > Sorry for the TL;DR, but a lot of information needed to be conveyed.  So,
> > based upon the TO Rewrite discussions Rob Butts and I have been working
> on
> > a Golang proxy (Mark Torluemke, affectionately calls "the Vampire Proxy")
> > that initially implements the */monitoring* endpoint to lay down the
> > foundation for rewriting more Traffic Ops endpoints in Go.  The goal is
> to
> > do a straight rewrite in Go (that implements any old API's as well as any
> > new ones that follows the /api/1.2 format).  The intent is to make this
> > proxy 100% backward compatible (including any HTTP header requirements)
> to
> > keep the existing TO API clients from breaking.
> >
> > This PR is significant because when postinstall runs (after this PR is
> > merged) it switches the ports according to the following:
> >
> > *TO Port Change Overview*
> > Port *443* will be owned by Golang proxy
> >
> > Port *60443* will now be owned by the Mojolicious/Perl Hypnotoad service
> >
> > See */opt/traffic_ops/app/conf/cdn.conf* for a new property
> > *traffic_ops_golang_port
> > => '443'*.
> >
> > *Important Operational Changes:*
> >
> > *traffic_ops service*
> > The Golang Proxy Service is now combined with the *traffic_ops* service,
> so
> > when traffic_ops is restarted so is the Golang Proxy Service.  Since the
> > Golang service is a proxy any APIs that are not implemented in the Golang
> > Service will be forwarded to the existing TO Perl API.  Also, If the API
> is
> > implemented in the Golang Service the response is serviced by the Proxy
> > (where it will access the Postgres database as needed).
> >
> > *traffic_ops logs*
> > *access.log - *old access.log is now renamed to *perl_access.log*, and
> the
> > Go proxy now takes over the *access.log* while logging in the *exact*
> > format
> > as before (no monitoring or tooling changes are required)
> > *traffic_ops_golang.log* - this is a new file where any errors/debug will
> > be logged from the Go proxy.
> > *perl_access.log - the existing Mojolicious access.log gets a new name*
> > *traffic_ops.log* - existing Mojolicious debug file (no change)
> >
> > There was a lot of debate and discussion about how to move forward and
> this
> > approach was less impactful to operations (which basically means less
> work
> > to move toward Go).  Overtime, the goal is to do a rewrite of all of the
> > relevant endpoints that are in Mojolicious into Go (with a heavy focus on
> > modularity and unit testing, for a future release with Micro Services).
> >
> > *What about the Qwilt contributions of API Gateway, Capabilities, and
> > Tenancy wasn't that a thing?  *
> > Yes, and it still is.  As for the API Gateway we will start "absorbing"
> the
> > code that *Amir* Yeshurun, so graciously contributed for the APIGW and
> > JWT.  For 

Re: Open Issues against 2.1

[Dewayne Richardson]

I just mentioned that to Dan Kirkwood, I'm also looking for 2.2.0 as well.

-Dew

On Wed, Aug 9, 2017 at 10:43 AM, Eric Friedrich (efriedri) <
efrie...@cisco.com> wrote:

> Many of the bugs that got moved over from Github are now without a
> “Versions Affected” field.
>
> We should also go through the bugs without a version and either close them
> or mark them into the 2.1 release
>
> —Eric
>
> > On Aug 9, 2017, at 11:59 AM, Dewayne Richardson <dewr...@gmail.com>
> wrote:
> >
> > Technically only 20 (at the moment) Jeremy didn't filter out for Bugs
> only:
> >
> > https://issues.apache.org/jira/browse/TC-504?jql=
> project%20%3D%20TC%20AND%20issuetype%20%3D%20Bug%20AND%
> 20status%20in%20(Open%2C%20%22In%20Progress%22%2C%
> 20Reopened)%20AND%20priority%20in%20(Blocker%2C%20Critical%
> 2C%20Major)%20AND%20affectedVersion%20%3D%202.1.0
> >
> > On Wed, Aug 9, 2017 at 9:14 AM, Jeremy Mitchell <mitchell...@gmail.com>
> > wrote:
> >
> >> Here's a link to the major, critical or blocker issues filed against 2.1
> >> (35 of them as of this writing)
> >>
> >> https://issues.apache.org/jira/browse/TC-504?jql=
> project%20%3D%20TC%20AND%
> >> 20status%20in%20(Open%2C%20%22In%20Progress%22%2C%
> >> 20Reopened)%20AND%20priority%20in%20(Blocker%2C%20Critical%
> >> 2C%20Major)%20AND%20affectedVersion%20%3D%202.1.0
> >>
> >> Doesn't seem like a good idea to ship something with 35 major+ issues
> filed
> >> against it. Maybe our Release Manager (Hank) can reach out to the
> reporter
> >> of each issue to see if it can be downgraded or fixed prior to RC0? In
> the
> >> meantime, like Dave said, let's be proactive and address any issues
> you've
> >> reported. I'm looking at mine now.
> >>
> >> Jeremy
> >>
> >> On Wed, Aug 9, 2017 at 7:53 AM, Dave Neuman <neu...@apache.org> wrote:
> >>
> >>> Hey All,
> >>>
> >>> I was looking at Jira and noticed that we have 53 unresolved issues
> >> against
> >>> TC 2.1 [1].  Of these 53, 6 are Critical or Blocker.  I know that we
> Hank
> >>> just cut the 2.1 branch and is thinking about putting out an RC0.  In
> my
> >>> opinion we need to address these issues before RC0 is released for
> >>> testing.  If you have some time can you please take a look at the open
> >>> issues and help out by either submitting a PR, moving to a different
> >>> release, or closing the issue if it truly is resolved?
> >>>
> >>> Thanks,
> >>> Dave
> >>>
> >>>
> >>>
> >>> [1] https://issues.apache.org/jira/browse/TC-489?filter=12341560
> >>>
> >>
>
>

Re: Open Issues against 2.1

[Dewayne Richardson]

Technically only 20 (at the moment) Jeremy didn't filter out for Bugs only:

https://issues.apache.org/jira/browse/TC-504?jql=project%20%3D%20TC%20AND%20issuetype%20%3D%20Bug%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20priority%20in%20(Blocker%2C%20Critical%2C%20Major)%20AND%20affectedVersion%20%3D%202.1.0

On Wed, Aug 9, 2017 at 9:14 AM, Jeremy Mitchell 
wrote:

> Here's a link to the major, critical or blocker issues filed against 2.1
> (35 of them as of this writing)
>
> https://issues.apache.org/jira/browse/TC-504?jql=project%20%3D%20TC%20AND%
> 20status%20in%20(Open%2C%20%22In%20Progress%22%2C%
> 20Reopened)%20AND%20priority%20in%20(Blocker%2C%20Critical%
> 2C%20Major)%20AND%20affectedVersion%20%3D%202.1.0
>
> Doesn't seem like a good idea to ship something with 35 major+ issues filed
> against it. Maybe our Release Manager (Hank) can reach out to the reporter
> of each issue to see if it can be downgraded or fixed prior to RC0? In the
> meantime, like Dave said, let's be proactive and address any issues you've
> reported. I'm looking at mine now.
>
> Jeremy
>
> On Wed, Aug 9, 2017 at 7:53 AM, Dave Neuman  wrote:
>
> > Hey All,
> >
> > I was looking at Jira and noticed that we have 53 unresolved issues
> against
> > TC 2.1 [1].  Of these 53, 6 are Critical or Blocker.  I know that we Hank
> > just cut the 2.1 branch and is thinking about putting out an RC0.  In my
> > opinion we need to address these issues before RC0 is released for
> > testing.  If you have some time can you please take a look at the open
> > issues and help out by either submitting a PR, moving to a different
> > release, or closing the issue if it truly is resolved?
> >
> > Thanks,
> > Dave
> >
> >
> >
> > [1] https://issues.apache.org/jira/browse/TC-489?filter=12341560
> >
>

Re: Delivery Service based config generation and Cache Manager

[Dewayne Richardson]

When:   Read · Fri, Jul 28.

[image: Timyo expectation line]
If Postgres performance becomes a concern, we can make ORT send it's
"Update" to a different endpoint than the Read, we can also take advantage
of Postgres HA to have dedicated Postgres Replicas that will allow us to
scale even more.

-Dew

On Thu, Jul 27, 2017 at 8:28 AM, Robert Butts 
wrote:

> As long as we have the proper indexes, database performance shouldn't be an
> issue. Postgres is capable of handling, and I've personally worked with,
> databases several orders of magnitude larger than ours. Now, if you don't
> have the indexes, querying will absolutely be slow. But as long as we add
> indexes to the columns we query on, it won't be an issue.
>
> I'm likewise not concerned about size. Our current production database is
> 162mb. But I'm not opposed to writing a truncate utility, if we think it's
> necessary. The queries are simple, it should be easy enough to write a
> function and a GUI button. I would oppose it being used unless absolutely
> necessary though; history is invaluable.
>
> On Thu, Jul 27, 2017 at 8:15 AM, Gelinas, Derek  >
> wrote:
>
> > I'm down with this! But I'm worried about database performance, for one,
> > and table size. I think we need to have a utility for removing older
> > entries if we are to go this route.
> >
> > On Jul 27, 2017, at 10:12 AM, Robert Butts  > mailto:robert.o.bu...@gmail.com>> wrote:
> >
> > Can I propose an adjustment?
> >
> > If we add a timestamp to every table, we can generate the JSON
> on-the-fly.
> > Then, the snapshot becomes a timestamp field, `snapshot_time`, and all
> the
> > data `select` queries add `where timestamp <= snapshot_time limit 1`.
> > Instead of updating rows, we only ever insert new rows with new
> timestamps.
> > This gives us snapshots back to eternity, and if a snapshot ever breaks,
> > rolling back is as simple as updating the `snapshot_time`. Our data is so
> > tiny, space is almost certainly not a problem, but if it is, truncating
> is
> > as easy as `delete where count > X and timestamp < Y`.
> >
> > That gives us all the benefits of your plan, plus the benefits of
> > relational data, type safety, more powerful querying, etc. And it
> shouldn't
> > be much more work to implement: add timestamp columns, snapshotting
> updates
> > the snapshot field, and getting the config simply runs what the snapshot
> > otherwise would to create the JSON. If generation performance is an issue
> > (it may be in Perl, probably not in Go), we can always cache the latest
> > snapshot in memory, and only regenerate it when the `snapshot_time`
> > changes.
> >
> > On Wed, Jul 26, 2017 at 9:08 AM, Gelinas, Derek <
> derek_geli...@comcast.com
> > >
> > wrote:
> >
> > That’s not a terrible idea.  Fewer changes to the code that way for sure,
> > really just the DS interface page.
> >
> > On Jul 26, 2017, at 10:30 AM, Nir Sopher  > qwilt.com>> wrote:
> >
> > Hi Derek,
> >
> > As discussed in the summit, we also see significant value in
> >
> >  1. DS Deployment Granularity - using DS individual config files.
> >  2. Delivery Service Configuration Versioning (DSCV) -  separating the
> >  "provisioning" from the "deployment".
> >  3. Improving the roll-out procedure, joining the capabilities #1 & #2
> >
> > We are on the same page with these needs:)
> >
> > However, as I see it, these are #1 & #2 are 2 separate features, each has
> > different requirements.
> > For example, for DSCV,  I would suggest to manage the versions as
> > standard
> > rows in the Delivery-Service table, side by side with the "hot" DS
> > configuration.
> > This will allow the existing code (with minor adjustments) to properly
> > work
> > on these rows.
> > Furthermore, it also allows you to simply "restore" the DS "hot"
> > configuration to a specified revision.
> > It is also more resilient to DS table schema updates.
> >
> > I'll soon share, on another thread, a link to a "DSCV functional spec" I
> > was working on. It extends the presentation
> >  > 20Summit%20-%20Spring%202017%20-%20Self-Service.pptx?
> > version=1=1495451091000=v2>
> > we
> > had in the summit.
> > I would appreciate any inputs to this spec.
> >
> > Nir
> >
> > On Tue, Jul 25, 2017 at 10:13 PM, Gelinas, Derek <
> > derek_geli...@comcast.com>
> > wrote:
> >
> > At the summit, there was some talk about changing the manner in which we
> > generate configuration files.  The early stages of this idea had me
> > creating large CDN definition files, but in the course of our
> > discussion it
> > became clear that we would be better served by creating delivery service
> > configuration files instead.  This would shift us from a
> > 

Re: Traffic Ops Golang Migration Proposal

[Dewayne Richardson]

When:   Read · Wed, Jul 19. If Cc: read

[image: Timyo expectation line]
I had a knee jerk reaction to separate the RPMs because of the potential
for a "new" approach, but I've now been convinced otherwise based upon the
"Strangler Pattern" where the Perl gets rewritten into Golang overtime and
the TO clients know none the difference.

+1

-Dew

On Wed, Jul 19, 2017 at 12:52 PM, Jeff Elsloo  wrote:

> I see. If that's the case, it's a hard requirement to run the golang
> portion from whenever this is introduced onward. As long as we have
> discipline around removing migrated routes, that should work okay and
> would solve the "two watches" issue Mark mentioned when we discussed
> this in person: A man with two watches (old API route, golang route)
> does not know what time it is.
>
> I don't think that it makes a lot of sense to have a separate RPM
> since the dependency goes the other direction, and users are required
> to run the golang component no matter what. We might as well just
> build that into the existing RPM build process for traffic_ops.
>
> Do we really need to ask the user for the port to move mojo to?
> Obviously we can ask them to provide a port, but we could also just
> pick a random, unused high port, and have mojo listen only on the
> loopback interface. Maybe that's too "magical"?
>
> Does the golang app run as trafops:trafops and drop privileges after
> opening :443?
> --
> Thanks,
> Jeff
>
>
> On Wed, Jul 19, 2017 at 11:58 AM, Robert Butts 
> wrote:
> >> This means that the traffic_ops side would have to check to see whether
> >> traffic_ops_golang
> >
> >> Because the traffic_ops_golang package will depend on traffic_ops, not
> the
> >> other way around
> >
> > I was suggesting the other way around - traffic_ops will depend on
> > traffic_ops_golang. Which means upgrading traffic_ops automatically
> installs
> > traffic_ops_golang, and we don't need to do the check. It'd mean you
> > couldn't remove `traffic_ops_golang`, but the plan is to remove old
> > endpoints from old TO anyway. Which is another reason making
> > traffic_ops_golang a dependency of traffic_ops makes sense: it really is,
> > traffic_ops really does require it for the migrated endpoints.
> >
> > I agree with moving away from manual post-installation scripts, but I
> don't
> > think we can avoid it here, because we need the user to set a new port.
> >
> >
> > On Wed, Jul 19, 2017 at 11:40 AM, Jeff Elsloo  wrote:
> >>
> >> I'm +1 on most of what you suggest, except for doing the takeover in
> >> postinstall in traffic_ops.
> >>
> >> While we can do whatever we want with postinstall, I think it's
> >> awkward to have a tool within the traffic_ops package configuring
> >> something under the traffic_ops_golang package, when the latter
> >> package might not be installed. This means that the traffic_ops side
> >> would have to check to see whether traffic_ops_golang is installed
> >> outside of the normal RPM dependencies, adding more platform specific
> >> code to postinstall. I don't see a generic way to implement the
> >> "check" for the golang package within postinstall that will work. We
> >> would have to check for a path that is not likely to exist for most
> >> users, or check for a package. Both approaches require platform
> >> specific code and assumptions.
> >>
> >> Because the traffic_ops_golang package will depend on traffic_ops, not
> >> the other way around, it makes more sense to place the configuration
> >> piece in the golang package. When the golang package is installed, we
> >> can "take over" the port in the listen directive of cdn.conf, because
> >> we know for a fact that it is on disk because of the RPM dependency on
> >> traffic_ops. We also know that cdn.conf will be left alone if/when
> >> traffic_ops is upgraded due to being marked as a config file. If the
> >> user has installed either component outside of the normal RPM process,
> >> they will have to figure out how to run the golang package separately,
> >> as one would expect.
> >>
> >> We can do the configuration during the postinstall step of the
> >> traffic_ops_golang RPM. It's advantageous to manage that piece within
> >> the RPM, because if, for example, one wanted to remove the golang
> >> portion, we could have a postuninstall step that reverts changes made
> >> to cdn.conf (put the port we took over back into cdn.conf). We could
> >> seamlessly add and remove the traffic_ops_golang component without
> >> disturbing anything in traffic_ops, and without having to run some
> >> script manually. The platform specific things that would need to be
> >> done in postinstall should be done in the RPM, because then we know
> >> for sure which platform we're on, and assumptions about packages and
> >> paths will be accurate.
> >>
> >> Ideally we should be moving away from any manual run of any script
> >> after an 

Re: 2.1 RM

[Dewayne Richardson]

When:   Read · Mon, Jul 17.

[image: Timyo expectation line]
Yes, Hank thank you!

On Mon, Jul 17, 2017 at 1:28 PM, Eric Friedrich (efriedri) <
efrie...@cisco.com> wrote:

> Hey Hank-
>   Many Thanks. Your RM baseball cap will be in the mail!
>
> I need to clean up the Release Management wiki page a bit for you. I’ll
> try to do that in the next few days.
>
> When’s the release branch get pulled? ;-)
>
> —Eric
>
>
>
> > On Jul 17, 2017, at 3:03 PM, Dan Kirkwood  wrote:
> >
> > Thanks,  Hank..feel free to ping either me or Eric if anything is
> > unclear in the release manager notes.
> >
> > -dan
> >
> > On Mon, Jul 17, 2017 at 12:56 PM, Hank Beatty 
> wrote:
> >> Hi Dave,
> >>
> >> I would like to volunteer.
> >>
> >> Thanks,
> >> Hank
> >>
> >> On 07/05/2017 03:38 PM, Dave Neuman wrote:
> >>>
> >>> Hey All,
> >>> Now that 2.0 has officially passed the project and IPMC vote (YAY!),
> it's
> >>> time to start thinking about 2.1.  I don't think we have identified a
> >>> release manager for the 2.1 release yet, would anyone like to
> volunteer?
> >>>
> >>> Thanks,
> >>> Dave
> >>>
> >>
>
>

Re: [VOTE] Release Apache Traffic Control 2.0.0-incubating (RC6)

[Dewayne Richardson]

+1

- gpg sig is good
- checksums match source tarball
- tarball has correct name and structure
- rpms build from tarball using 'pkg' command
- installed using those rpms and ran postinstall on a clean Centos7 VM
- imported default profiles from here:
http://trafficcontrol.incubator.apache.org/downloads/profiles/2.0.x/default/
- some basic traffic_ops UI functionality


-Dewayne

On Tue, Jun 20, 2017 at 2:04 PM, Eric Friedrich (efriedri) <
efrie...@cisco.com> wrote:

> Correction to the original vote deadline:
> Instead of Friday June 21, 2017, I meant that the vote will run until
> Wednesday June 21, 2017 (72 hrs)
>
> Please get your votes in within the next day or so- we need at least one
> more +1 vote to pass the release on to the IPMC
>
> Thanks,
> Eric
>
>
>
> > On Jun 19, 2017, at 5:07 PM, Dave Neuman  wrote:
> >
> > +1
> > Things I tested (Centos 7.2 VM):
> > -- verified md5
> > -- verified gpg sig
> > -- built all RPMs from source tar
> > -- Traffic Stats installs, starts, and runs
> > -- Traffic Router installs, starts, serves digs and curls
> > -- Traffic Ops can be installed and started.
> >
> > I found a couple issues that should be addressed before the next release:
> > [1] Traffic Stats install can't create traffic_stats user.
> >
> > Thanks,
> > Dave
> >
> > [1] https://issues.apache.org/jira/browse/TC-397
> >
> >
> > On Mon, Jun 19, 2017 at 12:39 PM, Dan Kirkwood 
> wrote:
> >
> >> +1
> >>
> >> I checked:
> >> - git tag verified
> >> - gpg sig is good
> >> - checksums match source tarball
> >> - tarball has correct name and structure
> >> - rpms build from tarball
> >> - traffic_ops installation and postinstall on a clean Centos7 VM
> >> - some basic traffic_ops UI functionality
> >> - docs no longer mention build area to download rpms
> >>
> >> One caveat -- gpg signature is not signed and so not in the web of
> >> trust,  but according to
> >> http://www.apache.org/dev/release-distribution.html#sigs-and-sums :
> >>
> >> "Signing keys SHOULD be linked into a strong web of trust."
> >>
> >> We should get Eric's key signed at the earliest opportunity,  but it's
> >> not a requirement for the release.
> >>
> >> On Fri, Jun 16, 2017 at 10:31 AM, Eric Friedrich (efriedri)
> >>  wrote:
> >>> Hello All,
> >>>
> >>> I've prepared the next candidate release for incubator-trafficcontrol
> >> v2.0.0 (RC6)
> >>>
> >>> Changes since 1.8.1:
> >>> https://github.com/apache/incubator-trafficcontrol/
> >> compare/RELEASE-1.8.1...RELEASE-2.0.0-RC6 >> github.com/apache/incubator-trafficcontrol/compare/
> >> RELEASE-1.8.1-RC0...RELEASE-2.0.0-RC5>
> >>>
> >>> This corresponds to git:
> >>> Hash: 85d14ebe2a4ac71236f86f70349481d2b3dc784b
> >>> Tag: RELEASE-2.0.0-RC6
> >>>
> >>> Which can be verified with the following:git tag -v
> RELEASE-2.0.0-RC6
> >>>
> >>> My code signing key is available here:
> >>> http://pgp.mit.edu/pks/lookup?op=get=0xF2200BAB9AB7BDD5
> >>>
> >>> and here:
> >>> https://dist.apache.org/repos/dist/dev/incubator/trafficcontrol/KEYS
> >>>
> >>> Make sure you refresh from a key server to get all relevant signatures.
> >>>
> >>> The source .tar.gz file, pgp signature (.asc signed with my key from
> >>> above), and md5 and sha512 checksums are provided here:
> >>> https://dist.apache.org/repos/dist/dev/incubator/
> >> trafficcontrol/2.0.0/RC6
> >>>
> >>>
> >>> Docs are available here: https://trafficcontrol.apache.org/docs/2.0.x/
> >>>
> >>>
> >>> The vote will remain open until Friday, June 21, 2017.
> >>>
> >>>
> >>> Thanks,
> >>> Eric Friedrich
> >>
>
>

Re: Traffic Ops Default Profile Management

[Dewayne Richardson]

Here's a snag I just hit while testing.  It seems that there is a filename
filter on the "Import Profile" that assumes whichever file you import is
named "*.traffic_ops".  So for files named "*.json" you have to manually
change that filter.  I can open a JIRA for this to make the default look
for ".json" (not a show stopper, but not a good experience).

On Mon, Jun 12, 2017 at 12:33 PM, Dewayne Richardson <dewr...@gmail.com>
wrote:

> I actually had the thought of a new tab at the same level as Downloads
> called "Profiles" that we point to in the documentation, that way (for
> newbies) the "Profiles" concept seems like an important concept.  It also
> allows us to grow the "Profiles" area separate of the artifact downloads
> (.rpms, .tarballs)
>
> -Dewayne
>
> On Mon, Jun 12, 2017 at 12:20 PM, Dave Neuman <neu...@apache.org> wrote:
>
>> I would make sure we clearly mark a section on the Downloads page for
>> Default Profile Downloads.
>> I would also make sure we update the Traffic Ops install doc to reflect
>> that default profiles can be downloaded from the website; assuming Jan's
>> PR
>> doesn't do this already.
>> I am +1 on having ATS in the EDGE and MID profile names to reflect that
>> they are for Traffic Server.
>>
>>
>> On Mon, Jun 12, 2017 at 12:00 PM, Dewayne Richardson <dewr...@gmail.com>
>> wrote:
>>
>> > I'm ok with that unless someone says otherwise.
>> >
>> > On Mon, Jun 12, 2017 at 11:31 AM, Shmulik Asafi <shmul...@qwilt.com>
>> > wrote:
>> >
>> > > Just a small refinement on the names, perhaps be explicit about the
>> > > profiles being for ATS as well, as in:
>> > >
>> > > EDGE_ATS_.json
>> > > MID_ATS_.json
>> > >
>> > > So in the future if we have other caches supported, it already takes
>> this
>> > > under consideration (even if not, I don't think we're trading off
>> > anything
>> > > here)
>> > >
>> > > On Mon, Jun 12, 2017 at 8:21 PM, Eric Friedrich (efriedri) <
>> > > efrie...@cisco.com> wrote:
>> > >
>> > > > Can we include an ORIGIN profile?
>> > > >
>> > > > Does TRAFFIC_PORTAL need a profile too?  (I’ve never set it up)
>> > > >
>> > > > —Eric
>> > > >
>> > > > > On Jun 12, 2017, at 1:14 PM, Dewayne Richardson <
>> dewr...@gmail.com>
>> > > > wrote:
>> > > > >
>> > > > > Based upon the discussion around how we manage the default
>> profiles
>> > > > Traffic
>> > > > > Ops profiles for 2.0 and 2.1, I propose we add instructions that
>> > under
>> > > > the
>> > > > > TC Website documentation "Downloads"
>> > > > > http://trafficcontrol.incubator.apache.org/downloads/index.html.
>> > The
>> > > > > profiles will be in the format that allows Traffic Ops to import
>> and
>> > > > export
>> > > > > them, either through the UI or the API and that is the only
>> format we
>> > > > > manage.
>> > > > >
>> > > > > The naming convention for the profiles (as well as the
>> corresponding
>> > > file
>> > > > > names) are up for discussion as well.  The following naming
>> > convention
>> > > > > (unless a better one comes up) will be:
>> > > > >
>> > > > > *Default Profiles*
>> > > > >
>> > > > > EDGE_.json
>> > > > > MID_.json
>> > > > > TRAFFIC_MONITOR.json
>> > > > > TRAFFIC_ROUTER.json
>> > > > > TRAFFIC_STATS.json
>> > > > > TRAFFIC_VAULT.json
>> > > > >
>> > > > > I didn't include version numbers on the TC components because I
>> > didn't
>> > > > know
>> > > > > if that was overkill.  I will move the ball forward with this
>> ASAP if
>> > > we
>> > > > > have enough consensus.
>> > > > >
>> > > > > Thank you,
>> > > > >
>> > > > > -Dewayne
>> > > >
>> > > >
>> > >
>> > >
>> > > --
>> > > *Shmulik Asafi*
>> > > Qwilt | Work: +972-72-2221692| Mobile: +972-54-6581595|
>> > shmul...@qwilt.com
>> > > <y...@qwilt.com>
>> > >
>> >
>>
>
>

Re: Traffic Ops Default Profile Management

[Dewayne Richardson]

I actually had the thought of a new tab at the same level as Downloads
called "Profiles" that we point to in the documentation, that way (for
newbies) the "Profiles" concept seems like an important concept.  It also
allows us to grow the "Profiles" area separate of the artifact downloads
(.rpms, .tarballs)

-Dewayne

On Mon, Jun 12, 2017 at 12:20 PM, Dave Neuman <neu...@apache.org> wrote:

> I would make sure we clearly mark a section on the Downloads page for
> Default Profile Downloads.
> I would also make sure we update the Traffic Ops install doc to reflect
> that default profiles can be downloaded from the website; assuming Jan's PR
> doesn't do this already.
> I am +1 on having ATS in the EDGE and MID profile names to reflect that
> they are for Traffic Server.
>
>
> On Mon, Jun 12, 2017 at 12:00 PM, Dewayne Richardson <dewr...@gmail.com>
> wrote:
>
> > I'm ok with that unless someone says otherwise.
> >
> > On Mon, Jun 12, 2017 at 11:31 AM, Shmulik Asafi <shmul...@qwilt.com>
> > wrote:
> >
> > > Just a small refinement on the names, perhaps be explicit about the
> > > profiles being for ATS as well, as in:
> > >
> > > EDGE_ATS_.json
> > > MID_ATS_.json
> > >
> > > So in the future if we have other caches supported, it already takes
> this
> > > under consideration (even if not, I don't think we're trading off
> > anything
> > > here)
> > >
> > > On Mon, Jun 12, 2017 at 8:21 PM, Eric Friedrich (efriedri) <
> > > efrie...@cisco.com> wrote:
> > >
> > > > Can we include an ORIGIN profile?
> > > >
> > > > Does TRAFFIC_PORTAL need a profile too?  (I’ve never set it up)
> > > >
> > > > —Eric
> > > >
> > > > > On Jun 12, 2017, at 1:14 PM, Dewayne Richardson <dewr...@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > Based upon the discussion around how we manage the default profiles
> > > > Traffic
> > > > > Ops profiles for 2.0 and 2.1, I propose we add instructions that
> > under
> > > > the
> > > > > TC Website documentation "Downloads"
> > > > > http://trafficcontrol.incubator.apache.org/downloads/index.html.
> > The
> > > > > profiles will be in the format that allows Traffic Ops to import
> and
> > > > export
> > > > > them, either through the UI or the API and that is the only format
> we
> > > > > manage.
> > > > >
> > > > > The naming convention for the profiles (as well as the
> corresponding
> > > file
> > > > > names) are up for discussion as well.  The following naming
> > convention
> > > > > (unless a better one comes up) will be:
> > > > >
> > > > > *Default Profiles*
> > > > >
> > > > > EDGE_.json
> > > > > MID_.json
> > > > > TRAFFIC_MONITOR.json
> > > > > TRAFFIC_ROUTER.json
> > > > > TRAFFIC_STATS.json
> > > > > TRAFFIC_VAULT.json
> > > > >
> > > > > I didn't include version numbers on the TC components because I
> > didn't
> > > > know
> > > > > if that was overkill.  I will move the ball forward with this ASAP
> if
> > > we
> > > > > have enough consensus.
> > > > >
> > > > > Thank you,
> > > > >
> > > > > -Dewayne
> > > >
> > > >
> > >
> > >
> > > --
> > > *Shmulik Asafi*
> > > Qwilt | Work: +972-72-2221692| Mobile: +972-54-6581595|
> > shmul...@qwilt.com
> > > <y...@qwilt.com>
> > >
> >
>

Re: Traffic Ops Default Profile Management

[Dewayne Richardson]

I'm ok with that unless someone says otherwise.

On Mon, Jun 12, 2017 at 11:31 AM, Shmulik Asafi <shmul...@qwilt.com> wrote:

> Just a small refinement on the names, perhaps be explicit about the
> profiles being for ATS as well, as in:
>
> EDGE_ATS_.json
> MID_ATS_.json
>
> So in the future if we have other caches supported, it already takes this
> under consideration (even if not, I don't think we're trading off anything
> here)
>
> On Mon, Jun 12, 2017 at 8:21 PM, Eric Friedrich (efriedri) <
> efrie...@cisco.com> wrote:
>
> > Can we include an ORIGIN profile?
> >
> > Does TRAFFIC_PORTAL need a profile too?  (I’ve never set it up)
> >
> > —Eric
> >
> > > On Jun 12, 2017, at 1:14 PM, Dewayne Richardson <dewr...@gmail.com>
> > wrote:
> > >
> > > Based upon the discussion around how we manage the default profiles
> > Traffic
> > > Ops profiles for 2.0 and 2.1, I propose we add instructions that under
> > the
> > > TC Website documentation "Downloads"
> > > http://trafficcontrol.incubator.apache.org/downloads/index.html.  The
> > > profiles will be in the format that allows Traffic Ops to import and
> > export
> > > them, either through the UI or the API and that is the only format we
> > > manage.
> > >
> > > The naming convention for the profiles (as well as the corresponding
> file
> > > names) are up for discussion as well.  The following naming convention
> > > (unless a better one comes up) will be:
> > >
> > > *Default Profiles*
> > >
> > > EDGE_.json
> > > MID_.json
> > > TRAFFIC_MONITOR.json
> > > TRAFFIC_ROUTER.json
> > > TRAFFIC_STATS.json
> > > TRAFFIC_VAULT.json
> > >
> > > I didn't include version numbers on the TC components because I didn't
> > know
> > > if that was overkill.  I will move the ball forward with this ASAP if
> we
> > > have enough consensus.
> > >
> > > Thank you,
> > >
> > > -Dewayne
> >
> >
>
>
> --
> *Shmulik Asafi*
> Qwilt | Work: +972-72-2221692| Mobile: +972-54-6581595| shmul...@qwilt.com
> <y...@qwilt.com>
>

Re: Traffic Ops Default Profile Management

[Dewayne Richardson]

We can include anything we think is valuable for a "default" setup (do you
have an ORIGIN profile for me to include, just tell me where it is and I'll
take a lookg).  Re: TRAFFIC_PORTAL, we haven't come across the need as of
yet.

On Mon, Jun 12, 2017 at 11:21 AM, Eric Friedrich (efriedri) <
efrie...@cisco.com> wrote:

> Can we include an ORIGIN profile?
>
> Does TRAFFIC_PORTAL need a profile too?  (I’ve never set it up)
>
> —Eric
>
> > On Jun 12, 2017, at 1:14 PM, Dewayne Richardson <dewr...@gmail.com>
> wrote:
> >
> > Based upon the discussion around how we manage the default profiles
> Traffic
> > Ops profiles for 2.0 and 2.1, I propose we add instructions that under
> the
> > TC Website documentation "Downloads"
> > http://trafficcontrol.incubator.apache.org/downloads/index.html.  The
> > profiles will be in the format that allows Traffic Ops to import and
> export
> > them, either through the UI or the API and that is the only format we
> > manage.
> >
> > The naming convention for the profiles (as well as the corresponding file
> > names) are up for discussion as well.  The following naming convention
> > (unless a better one comes up) will be:
> >
> > *Default Profiles*
> >
> > EDGE_.json
> > MID_.json
> > TRAFFIC_MONITOR.json
> > TRAFFIC_ROUTER.json
> > TRAFFIC_STATS.json
> > TRAFFIC_VAULT.json
> >
> > I didn't include version numbers on the TC components because I didn't
> know
> > if that was overkill.  I will move the ball forward with this ASAP if we
> > have enough consensus.
> >
> > Thank you,
> >
> > -Dewayne
>
>

Traffic Ops Default Profile Management

[Dewayne Richardson]

Based upon the discussion around how we manage the default profiles Traffic
Ops profiles for 2.0 and 2.1, I propose we add instructions that under the
TC Website documentation "Downloads"
http://trafficcontrol.incubator.apache.org/downloads/index.html.  The
profiles will be in the format that allows Traffic Ops to import and export
them, either through the UI or the API and that is the only format we
manage.

The naming convention for the profiles (as well as the corresponding file
names) are up for discussion as well.  The following naming convention
(unless a better one comes up) will be:

*Default Profiles*

EDGE_.json
MID_.json
TRAFFIC_MONITOR.json
TRAFFIC_ROUTER.json
TRAFFIC_STATS.json
TRAFFIC_VAULT.json

I didn't include version numbers on the TC components because I didn't know
if that was overkill.  I will move the ball forward with this ASAP if we
have enough consensus.

Thank you,

-Dewayne

Re: Data patches during upgrade

[Dewayne Richardson]

Yea, it's just a new feature to admin.pl to support data conversions, to
keep the migrations clean.  Derek and I have been working through it.

-Dew

On Thu, Jun 8, 2017 at 7:40 AM, Jeremy Mitchell 
wrote:

> This seems to make sense to me but honestly, I'd probably defer to Dewayne.
>
> In theory, it would be nice if migrations only included "structural"
> changes (new tables, columns, changing column types or not  null, etc) and
> seeds focused on the "base" (or the minimum required) static data required
> of TO (types, statuses, roles, etc) and then yea, putting data fixing or
> data massaging as the last step makes sense to me. But you know what they
> say about theory...
>
> +1
>
> Jeremy
>
> On Wed, Jun 7, 2017 at 8:41 AM, Gelinas, Derek 
> wrote:
>
> > I'm adding a feature to traffic ops that creates a new column in
> > steering_target called type, that is populated with type ids from the
> type
> > table.  Using admin.pl upgrade, the column is created in migrations, and
> > the two types for this table are populated by seeds.sql.  None of this is
> > out of the ordinary.  Unfortunately I also need to populate the type
> column
> > based on data that isn't in there until after seeds.sql is run, so I
> can't
> > place this into the migration.  Seeds.sql needs to run after the
> migration
> > due to any structural changes that happen there.
> >
> > Dewayne and I have discussed this a bit this morning, and we're thinking
> > the best solution might be a third step, run after seeds.sql, called
> > patches.sql.  This would be specifically for data fixes like in this use
> > case.  The order would be as follows:
> >
> > migration - structure
> > seeds - static data
> > patches - data fixes
> >
> > Thoughts?
> >
> > Derek
>

Re: LDAP Access

[Dewayne Richardson]

I have a question in a similar vein, how often do we really use LDAP?  My
understanding is we created LDAP access to allow external users in to see
our TO Graphs.  Now that graphs are in Graphana is the need for LDAP still
needed?  If we require anyone using TO or the TO API to be in the database
it would alleviate this LDAP security issue entirely.

 I also wonder if we shouldn't try to leverage transitioning our user
management to Postgres.  Postgres has many options for authentication (as I
mentioned at the Summit), which would allow for more flexibility at TO
installations.

-Dewayne

On Wed, May 31, 2017 at 9:24 AM, Robert Butts 
wrote:

> We have a PR https://github.com/apache/incubator-trafficcontrol/pull/627
> to
> change Traffic Ops to only allow LDAP users _not_ in the Traffic Ops
> database to view non-sensitive information, like graphs and total CDN
> bandwidth.
>
> To be clear, users will still be able to authenticate with LDAP, as long as
> their user name is in the database. This only prevents access for LDAP
> users whose name is not in the database.
>
> If you have LDAP-only users who need access, you can simply add their user
> name to the Traffic Ops database to allow continued access. They don't even
> need a password, simply inserting the username is sufficient.
>
> LDAP is a security risk, especially for large organizations. Allowing all
> non-CDN personnel in the organization full information access, even
> read-only, means an attacker has only to compromise a single account in the
> organization, and they can see the full list of CDN server IPs and FDQNs,
> as well as the specific ATS and CentOS versions, in order to take advantage
> of known exploits against those versions.
>
> Does anyone have any issues with that? Is anyone using LDAP without
> usernames in the database, who needs continued access? We just want to make
> sure we're not breaking anyone before we merge this, and figure out a
> solution if we are. Thanks,
>

Re: TC Contributers to Publish list of Planned/WIP Items for Better Community Sync

[Dewayne Richardson]

+1 on tracking the Roadmap on Github, the "Labels" in github will make it
easier to track than a wiki (and even JIRA)

On Thu, May 25, 2017 at 12:39 PM, Shmulik Asafi  wrote:

> I think JIRA/Github are suitable candidates.
> It will require to decide on some way to tag these items so a convenient
> filter can produce the desired list (otherwise it will be hidden between
> lots of JIRAs and no use for anyone)
>
> What I personally like about the Wiki option, is that maintaining my list
> of items will involve only a single page (no mouse clicks); in contrast to
> JIRA where I'll need to create items and maintain each item in a separate
> page (many mouse clicks). I'm also more of a list-of-open-bullets guy than
> a table-with-fixed-columns guy when it comes to viewing information.
>
> On the other hand JIRA gives more power when it comes to stuff like
> order-by filter-by and etc.
>
> Ultimately, it's a question of which tool most people imagine will be
> convenient for them to update and view.
>
> On Thu, May 25, 2017 at 7:02 PM, Dave Neuman  wrote:
>
> > I am probably opening a can of worms that I don't know if I want to open,
> > but can't we just use Jira for this (and Github once we move there)?  If
> > someone is working on something it should be in Jira as an issue that
> > assigned to that person.  From there you can see who is working on what.
> > It's not going to be perfect, and everything is not always going to be in
> > there, but it should be good enough IMO.  I think that would be a better
> > solution than relying on people to keep a wiki up to date.
> >
> > On Thu, May 25, 2017 at 9:54 AM, Durfey, Ryan 
> > wrote:
> >
> > > I am +1 on this.  This dovetails well with the idea of a high-level
> > > roadmap illustrating active areas of work.
> > >
> > > I am open to ideas on what tool to use.  I agree with Shmulik we could
> > > make this work in the wiki if needed.
> > >
> > > Ryan DurfeyM | 303-524-5099
> > > CDN Support (24x7): 866-405-2993 or cdn_supp...@comcast.com > > cdn_supp...@comcast.com>
> > >
> > >
> > > From: Shmulik Asafi 
> > > Reply-To: "dev@trafficcontrol.incubator.apache.org" <
> > > dev@trafficcontrol.incubator.apache.org>
> > > Date: Wednesday, May 24, 2017 at 11:54 PM
> > > To: "dev@trafficcontrol.incubator.apache.org" <
> > > dev@trafficcontrol.incubator.apache.org>
> > > Subject: TC Contributers to Publish list of Planned/WIP Items for
> Better
> > > Community Sync
> > >
> > > Hi All,
> > >
> > > During the summit I've suggested to a few people that maybe each TC
> > > contributer should publish a list of the items she's working on
> presently
> > > and planning to work on in the near future.
> > >
> > > The goal is to share what "major stuff" is going to happen in the
> project
> > > (e.g. Jeff's work on Localized/Edge Traffic Routing; Derek's work on
> > > Delivery Service configuration; Dave/Matt's work on Lua scripts; etc.).
> > >
> > > I think this is very little effort by each contributer and will allow
> us
> > to
> > > be much more synced, make it possible to engage with existing efforts
> and
> > > provide a platform to detect conflicting/duplicated efforts. I also
> think
> > > this can be a first baby step towards building a roadmap in the long
> > term.
> > >
> > > The few people I talked with seemed open to this idea. I hope others
> will
> > > feel the same.
> > >
> > > I think Wiki pages is the best place for this, but it can also be in
> > > JIRA/Git, or whatever appropriate tools Apache provides (it would also
> be
> > > cool if we could make RSS feed no changes).
> > >
> > > What do you think?
> > >
> > > If all support, I'd be happy to create the "framework" for this in
> > whatever
> > > tool we choose.
> > >
> > > Thanks!
> > >
> > > --
> > > *Shmulik Asafi*
> > > Qwilt | Work: +972-72-2221692 <+972%2072-222-1692>| Mobile:
> > > +972-54-6581595
> > > <+972%2054-658-1595>| shmul...@qwilt.com <
> > > y...@qwilt.com>
> > >
> > >
> >
>
>
>
> --
> *Shmulik Asafi*
> Qwilt | Work: +972-72-2221692| Mobile: +972-54-6581595| shmul...@qwilt.com
> 
>

Re: 2.0 release?

[Dewayne Richardson]

i want to do some testing down the fresh install path to see what needs to
be backported, hopefully this week.

On Wed, May 17, 2017 at 9:02 AM Dave Neuman <neu...@apache.org> wrote:

> Hey All,
> We had some great discussion about the 2.0 release at the summit, I was
> wondering if anyone had a summary of that discussion and a list of what's
> left to do that could be added to this thread?  I think we discussed that
> we were going to take another look at 2.0 and see if it is a viable release
> that we should move forward with, is that everyone else's understanding as
> well?
> Does anyone know of any showstopper issues that still exist?
>
> Thanks,
> Dave
>
> On Mon, Apr 10, 2017 at 9:19 PM, Eric Friedrich (efriedri) <
> efrie...@cisco.com> wrote:
>
> > Update:
> >   - License issue has been fixed- Thanks Rob!
> >   - Postinstall script is broken, Jeff and Dan are looking at it.
> >
> > Once post install is fixed, I will cut an RC
> >
> > —Eric
> >
> >
> >
> > > On Apr 6, 2017, at 2:35 PM, Dewayne Richardson <dewr...@gmail.com>
> > wrote:
> > >
> > > +1
> > >
> > > On Thu, Apr 6, 2017 at 9:43 AM, Robert Butts <robert.o.bu...@gmail.com
> >
> > > wrote:
> > >
> > >> +1
> > >> I didn't realize it was new.
> > >>
> > >> On Thu, Apr 6, 2017 at 8:49 AM, Dan Kirkwood <dang...@gmail.com>
> wrote:
> > >>
> > >>> +1
> > >>>
> > >>> On Thu, Apr 6, 2017 at 7:43 AM, David Neuman <
> david.neuma...@gmail.com
> > >
> > >>> wrote:
> > >>>> Since the Cookie Jar functionality is new to 2.0 and 2.0 is not yet
> > >>>> released, why don't we just remove the `ResumeSession` method all
> > >>> together
> > >>>> and eliminate the dependency?  Otherwise we are deprecating
> something
> > >>> that
> > >>>> we never formally released.
> > >>>>
> > >>>> On Tue, Apr 4, 2017 at 2:30 PM, Robert Butts <
> > robert.o.bu...@gmail.com
> > >>>
> > >>>> wrote:
> > >>>>
> > >>>>> Regarding `TC-119: traffic_ops/client dependency license issue`:
> > >>>>>
> > >>>>> It looks like the persistent cookie jar is only needed by Traffic
> Ops
> > >>>>> Client `ResumeSession(toURL string, insecure bool) (*Session,
> > error)`.
> > >>>>> Nothing in Traffic Control uses `ResumeSession`, and I doubt anyone
> > >>> else is
> > >>>>> using it. Because it returns an error, and persisted cookies have
> > >>>>> lifetimes, any current users already must handle errors from
> > persisted
> > >>>>> cookies being expired. Thus, we can change it to always return an
> > >> error
> > >>>>> with only degraded performance (and not much, login is cheap),
> > without
> > >>> loss
> > >>>>> of functionality.
> > >>>>>
> > >>>>> To fix TC-119, I propose we document `ResumeSession` as deprecated,
> > >> and
> > >>>>> change it to always return an error, which lets us remove the
> > >>> dependency,
> > >>>>> without the development cost of writing our own persistent cookie
> > >> store
> > >>>>> that no one is using.
> > >>>>>
> > >>>>> Any objections?
> > >>>>>
> > >>>>>
> > >>>>> On Tue, Apr 4, 2017 at 1:35 PM, Jeremy Mitchell <
> > >> mitchell...@gmail.com>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> These all got fixed and backported to 2.0:
> > >>>>>>
> > >>>>>> TC-203: Mojo doesn’t set cachable headers on public files”
> > >>>>>> TC-190: TTL type mismatch in CrConfig
> > >>>>>> TC-189: ssl_multicert.config too slow
> > >>>>>>
> > >>>>>> So Jan and Dave just need to close the issues.
> > >>>>>>
> > >>>>>> On Tue, Apr 4, 2017 at 12:22 PM, Jeffrey Martin <
> > >>> martin.jef...@gmail.com
> > >>>>>>
> > >>>>>> wrote:
> > >>>>>>
> > >>>&g

Re: [VOTE] Move Traffic Control to full GitHub

[Dewayne Richardson]

Move to full github

On Thu, May 18, 2017 at 2:32 PM Jan van Doorn  wrote:

> In
>
> https://lists.apache.org/thread.html/5bdb9b073343f49c1d5b85147eb9d260bf7ad15d61384929993c7e1d@%3Cdev.trafficcontrol.apache.org%3E
> Dave
> mentioned that we can now move to "full" GitHub. Some more information in
> that thread if you are not familiar. I would like to call an official vote
> on that.
>
> This vote will be open for at least 72 hours.
>
>  [ ] +1 Move Traffic Control to use full GitHub
>  [ ]  0 No opinion
>  [ ] -1 Do not Move Traffic Control to use full GitHub because...
>
> Rgds,
> JvD
>

Re: API GW route configuration

[Dewayne Richardson]

If only the API GW authenticates/authorizes we also have a single point of
entry to test for security instead of having it sprinkled across services
in different ways.  It also simplifies the code on the service side and
makes them easier to test with automation.

-Dew

On Mon, May 8, 2017 at 8:42 AM, Robert Butts 
wrote:

> > couldn't make nginx or http do what we need.
>
> I was suggesting a different architecture. Not making the proxy do auth,
> only standard proxying.
>
> > We can still have the services check the auth as well after the proxy
> auth
>
> +1
>
>
> On Mon, May 8, 2017 at 3:36 AM, Amir Yeshurun  wrote:
>
> > Hi,
> >
> > Let me elaborate some more on the purpose of the API GW. I will put up a
> > wiki page following our discussions here.
> >
> > Main purpose is to allow innovation by creating new services that handle
> TO
> > functionality, not as a part of the monolithic Mojo app.
> > The long term vision is to de-compose TO into multiple microservices,
> > allowing new functionality easily added.
> > Indeed, the goal it to eventually deprecate the current AAA model, and
> > replace it with the new AAA model currently under work (user-roles,
> > role-capabilities)
> >
> > I think that handling authorization in the API layer is a valid approach.
> > Security wise, I don't see much difference between that, and having each
> > module access the auth service, as long as the auth service is deployed
> in
> > the backend.
> > Having another proxy (nginx?) fronting the world and forwarding all
> > requests to the backend GW mitigates the risk for compromising the
> > authorization service.
> > However, as mentioned above, we can still have the services check the
> auth
> > as well after the proxy auth.
> >
> > It is a standalone process, completely optional at this point. One can
> > choose to deploy it in order to allow integration with additional
> > services. Deployment
> > and management are still T.B.D, and feedback on this is most welcome.
> >
> > Regarding token validation and revocation:
> > Tokens have expiration time. Expired tokens do not pass token validation.
> > In production, expiration should be set to relatively short time, say 5
> > minute.
> > This way revocation is automatic. Re-authentication is handled via
> refresh
> > tokens (not implemented yet). Hitting the DB upon every API call cause
> > congestion on users DB.
> > To avoid that, we chose to have all user information self-contained
> inside
> > the JWT.
> >
> > Thanks
> > /amiry
> >
> > On Mon, May 8, 2017 at 5:42 AM Jan van Doorn  wrote:
> >
> > > It's the reverse proxy we've discussed for the "micro services" version
> > for
> > > a while now (as in
> > > https://cwiki.apache.org/confluence/display/TC/Design+Overview+v3.0).
> > >
> > > On Sun, May 7, 2017 at 7:22 PM Eric Friedrich (efriedri) <
> > > efrie...@cisco.com>
> > > wrote:
> > >
> > > > From a higher level- what is purpose of the API Gateway?  It seems
> like
> > > > there may have been some previous discussions about API Gateway. Are
> > > there
> > > > any notes or description that I can catch up on?
> > > >
> > > > How will it be deployed? (Is it a standalone service or something
> that
> > > > runs inside the experimental Traffic Ops)?
> > > >
> > > > Is this new component required or optional?
> > > >
> > > > —Eric
> > > >
> > > >
> > > >
> > > > > On May 7, 2017, at 8:28 PM, Jan van Doorn  wrote:
> > > > >
> > > > > I looked into this a year or so ago, and I couldn't make nginx or
> > http
> > > do
> > > > > what we need.
> > > > >
> > > > > We can still have the services check the auth as well after the
> proxy
> > > > auth,
> > > > > and make things better than today, where we have the same problem
> > that
> > > if
> > > > > the TO mojo app is compromised, everything is compromised.
> > > > >
> > > > > If we always route to TO, we don't untangle the mess of being
> > dependent
> > > > on
> > > > > the monolithic TO for everything. Many services today, and more in
> > the
> > > > > future really just need a check to see if the user is authorized,
> and
> > > > > nothing more.
> > > > >
> > > > > On Sun, May 7, 2017 at 11:55 AM Robert Butts <
> > robert.o.bu...@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > >> What are the advantages of these config files, over an existing
> > > reverse
> > > > >> proxy, like Nginx or httpd? It's just as much work as configuring
> > and
> > > > >> deploying an existing product, but more code we have to write and
> > > > maintain.
> > > > >> I'm having trouble seeing the advantage.
> > > > >>
> > > > >> -1 on auth rules as a part of the proxy. Making a proxy care about
> > > auth
> > > > >> violates the Single Responsibility Principle, and further, is a
> > > security
> > > > >> risk. It creates unnecessary attack surface. If your proxy app or
> > > > server is
> > > > >> compromised, the entire framework is now compromised. An 

Re: Goose installer script

[Dewayne Richardson]

I had that thought, as well as there are more recent versions like
https://github.com/mattes/migrate.  The question becomes if we ever get
around to rewriting TrafficOps APIs in golang, will the Perl version then
become obsolete?

On Fri, Apr 28, 2017 at 11:58 AM, Dave Neuman <neu...@apache.org> wrote:

> Maybe it's time we take a look at what goose really buys us and consider
> writing our own database migration tool.  We already have admin.pl, it
> could probably fit in with that?
>
> On Fri, Apr 28, 2017 at 11:45 AM, Eric Friedrich (efriedri) <
> efrie...@cisco.com> wrote:
>
> > Hey Dew-
> >   What calls this script?
> >
> > If its called from the Traffic Ops Spec file, then this will cause some
> > pain for those of us that need to install without internet access.
> >
> > —Eric
> >
> > > On Apr 28, 2017, at 12:41 PM, Dewayne Richardson <dewr...@gmail.com>
> > wrote:
> > >
> > > I'm working toward a more streamlined installation process for Traffic
> > Ops
> > > (internally) and publicly. Of course, the same hiccups that everyone
> else
> > > runs into I am as well.  Installation of Golang (proper version) and
> > > installation of Goose.  Goose has been the most challenging for several
> > > reasons.  The maintainer hasn't made any real changes since 2015, and
> has
> > > not "branched" his code to allow for explicit version download.  Per
> his
> > > installation instructions "go get bitbucket.org/liamstask/goose/
> > cmd/goose"
> > >
> > > So I'm I'm proposing to write an installer script in bash to help
> > automate
> > > the Golang install as well as the Goose install.  My only concern (as
> > well
> > > as most of yours) is "go get" will grab the latest, but since no real
> > > changes have happened I'm left with no other option.
> > >
> > > Proposed:
> > >
> > > /opt/traffic_ops/install/bin/install_goose.sh
> > >
> > > - Install Golang (version 1.8.x)
> > > - go get bitbucket.org/liamstask/goose/cmd/goose
> > >
> > > Thoughts?
> > >
> > > -Dew
> >
> >
>

Re: Goose installer script

[Dewayne Richardson]

The user, we'd document the installation steps

On Fri, Apr 28, 2017 at 11:45 AM, Eric Friedrich (efriedri) <
efrie...@cisco.com> wrote:

> Hey Dew-
>   What calls this script?
>
> If its called from the Traffic Ops Spec file, then this will cause some
> pain for those of us that need to install without internet access.
>
> —Eric
>
> > On Apr 28, 2017, at 12:41 PM, Dewayne Richardson <dewr...@gmail.com>
> wrote:
> >
> > I'm working toward a more streamlined installation process for Traffic
> Ops
> > (internally) and publicly. Of course, the same hiccups that everyone else
> > runs into I am as well.  Installation of Golang (proper version) and
> > installation of Goose.  Goose has been the most challenging for several
> > reasons.  The maintainer hasn't made any real changes since 2015, and has
> > not "branched" his code to allow for explicit version download.  Per his
> > installation instructions "go get bitbucket.org/liamstask/goose/
> cmd/goose"
> >
> > So I'm I'm proposing to write an installer script in bash to help
> automate
> > the Golang install as well as the Goose install.  My only concern (as
> well
> > as most of yours) is "go get" will grab the latest, but since no real
> > changes have happened I'm left with no other option.
> >
> > Proposed:
> >
> > /opt/traffic_ops/install/bin/install_goose.sh
> >
> > - Install Golang (version 1.8.x)
> > - go get bitbucket.org/liamstask/goose/cmd/goose
> >
> > Thoughts?
> >
> > -Dew
>
>

Goose installer script

[Dewayne Richardson]

I'm working toward a more streamlined installation process for Traffic Ops
(internally) and publicly. Of course, the same hiccups that everyone else
runs into I am as well.  Installation of Golang (proper version) and
installation of Goose.  Goose has been the most challenging for several
reasons.  The maintainer hasn't made any real changes since 2015, and has
not "branched" his code to allow for explicit version download.  Per his
installation instructions "go get bitbucket.org/liamstask/goose/cmd/goose"

So I'm I'm proposing to write an installer script in bash to help automate
the Golang install as well as the Goose install.  My only concern (as well
as most of yours) is "go get" will grab the latest, but since no real
changes have happened I'm left with no other option.

Proposed:

/opt/traffic_ops/install/bin/install_goose.sh

- Install Golang (version 1.8.x)
- go get bitbucket.org/liamstask/goose/cmd/goose

Thoughts?

-Dew

Re: Proposal for CDN definition file based configuration management

[Dewayne Richardson]

+1 I was just about to formulate that response.  The "dev" list is our
discussion forum.

On Tue, Apr 11, 2017 at 9:35 AM, Dave Neuman  wrote:

> @Ryan, I think its better to have conversations on the dev list than a wiki
> page...
>
> On Tue, Apr 11, 2017 at 9:01 AM, Durfey, Ryan 
> wrote:
>
> > Started a new wiki page to discuss this here https://cwiki.apache.org/
> > confluence/display/TC/Configuration+Management
> >
> > I will do my best to summarize the discussion below later today.
> >
> > Ryan M | 303-524-5099
> >
> >
> > -Original Message-
> > From: Eric Friedrich (efriedri) [mailto:efrie...@cisco.com]
> > Sent: Tuesday, April 11, 2017 8:55 AM
> > To: dev@trafficcontrol.incubator.apache.org
> > Subject: Re: Proposal for CDN definition file based configuration
> > management
> >
> > A few questions/thoughts, apologies for not in-lining:
> >
> > 1) If we move away from individually queued updates, we give up the
> > ability to make changes and then selectively deploy them. How often do TC
> > operations teams make config changes but do not immediately queue
> updates.
> > (I personally think that we currently have a bit of a tricky situation
> > where queuing updates much later can push down an unknowingly large
> config
> > change to a cache- i.e. many new DS added/removed since last time updates
> > were queued maybe months earlier). I wouldn't be sad to see queue updates
> > go away, but don't want to cause hardship on operators using that
> feature.
> >
> > 2) If we move away from individually queued updates, how does that affect
> > the implicit "config state machine"? Specifically, how will edges know
> when
> > their parents have been configured and are ready for service? Today we
> > don't config an edge cache with a new DS unless the mid is ready to
> handle
> > traffic as well.
> >
> > 3) If we move away from individually queued updates, how do we do things
> > like unassign a delivery service from a cache? Today we have to snapshot
> > CRConfig first to stop redirects to the cache before we queue the update.
> > If updates are immediately applied and snapshot is still separate, how do
> > we get TR to stop sending traffic to a cache that no longer has the remap
> > rule?
> >
> > 4) Also along the lines of the config state machine, we never really
> > closed on if we would make any changes to the queue update/snapshot
> > CRConfig flow. If we are looking at redoing how we generate config files,
> > it would be great to have consensus on an approach (if not an
> > implementation) to remove the need to sequence queue updates and snapshot
> > CRConfig. I think the requirement here would be to have Traffic Control
> > figure out on its own when to activate/deactivate routing to a cache from
> > TR.
> >
> > 5) I like the suggestion of cache-based config file generation.
> >   - Caches only retrieve relevant information, so scale proportional to
> > number of caches/DSs in the CDN is much better
> >   - We could modify TR/TM to use the same approach, rather than
> > snapshotting a CRConfig.
> >   - Cache/TR/TM-based config could play a greater role in config state
> > machine, rather than having Traffic Ops build static configuration ahead
> of
> > time.
> >
> > Downsides
> >   - Versioning is still possible, but more work than maintaining
> snapshots
> > of a config file
> >   - Have to be very careful with API changes, any breakage now impacts
> > cache updates.
> >
> > -Eric
> >
> > > On Apr 10, 2017, at 9:45 PM, Gelinas, Derek  >
> > wrote:
> > >
> > > Thanks Rob. To your point about scalability: I think that this is more
> > scaleable than the current crconfig implementation due to the caching.
> > However that is a very valid point and one that has been considered. I've
> > started looking into the problem from that angle and hope to have some
> more
> > solid data soon.  I still believe that this is ultimately more scaleable
> > than current config implementation, even with the scope caching, but the
> > proof will be in the data.
> > >
> > > Derek
> > >
> > >> On Apr 10, 2017, at 9:23 PM, Robert Butts 
> > wrote:
> > >>
> > >> I'd propose:
> > >> * Instead of storing the JSON as blob, use
> > >> https://www.postgresql.org/doc s/9.2/static/datatype-json.html
> > >> * Instead of version-then-file request, use a "latest" endpoint with
> > >> `If-Modified-Since`
> > >> (https://tools.ietf.org/html/rfc7232#section-3.3). We can also serve
> > >> each version at endpoints, but `If-Modified-Since` lets us determine
> > >> whether there's a new snapshot and get it in a single request, both
> > >> efficiently and using a standard. (We should do the same for the
> > CRConfig).
> > >>
> > >> Also for cache-side config generation, consider
> > >> https://github.com/apache/incubator-trafficcontrol/pull/151 . It's a
> > >> prototype and needs work to bring it to production, but the basic
> > >> 

Re: Traffic-Control Official PostgreSQL Version

[Dewayne Richardson]

We have started with 9.6.x, so I'd say we should assume > 9.6.x at the
moment.

On Tue, Apr 4, 2017 at 2:09 PM, Nir Sopher  wrote:

> Hi,
>
> Is there an official required PostgreSQL version?
>
> One reason it may be important is that supported syntax is added in newer
> versions.
> For example, recently one of the DB migrations scripts was added with "ADD
> COLUMN IF NOT EXISTS" statement, which is considered a syntax error (and
> breaks "admin.pl setup") in PostgreSQL versions < 9.6.
>
> Thanks,
> Nir
>

Re: TO API Versioning

[Dewayne Richardson]

Yes, the versioning wasn't consistent because we switched out the database
to Postgres which had several impacts to the API. The issue with API routes
being too granular in the versioning is tricky because the consumers of the
API will need to change as the versions of Traffic Ops moves forward.  Now
that we are on Postgres we plan on a new versioning scheme with /api/v2/*
which we will shortly introduce.  The goal with /api/v2 will be to keep the
response format the same, with the intention of deprecating the /api/1.x
routes.

Another goal we have is to build an API test tool (now that more CRUD
routes are available), which will help with consistency in the API as well
as help us with regression testing (more to come on this front).

-Dewayne

On Mon, Mar 20, 2017 at 11:20 AM, Hank Beatty  wrote:

> I recently ran into an issue where one of my scripts that uses the TO API
> stopped working correctly. It was failing in such a way that if I had not
> been in there updating something else I never would have known it was not
> working correctly. This failure was due to a change in the TO API version
> 1.2. I wrote the script using version 1.2 and the version I'm testing
> against now is also 1.2.
>
> I would like to suggest that we chose a versioning philosophy for the TO
> API. The site http://semver.org/ offers a good philosophy to follow. This
> also happens to be the one ATS uses.
>
> Let's please choose a versioning philosophy so that issues like the one I
> describe above can be avoided.
>
> Thanks,
> Hank
>

Re: Removing 'internal' from TO API

[Dewayne Richardson]

I think we should do as Dave mentioned, assess and rename.

> On Mar 15, 2017, at 2:18 PM, Jeremy Mitchell  wrote:
>
> I don't like duplicating routes either but I thought it would ease the
> transition rather than just changing the route. So no code duplication,
> just 2 routes that go to the same place:
>
> $r->get("/internal/api/$version/steering")->over( authenticated => 1 )->to(
> 'Steering#index', namespace => 'API::DeliveryService' );
> $r->get("/api/$version/steering")->over( authenticated => 1 )->to(
> 'Steering#index', namespace => 'API::DeliveryService' );
>
> And then we circle back and delete
>
> $r->get("/internal/api/$version/steering")->over( authenticated => 1 )->to(
> 'Steering#index', namespace => 'API::DeliveryService' );
>
> at some point.
>
> And yes, this internal namespace was introduced for comcast-specific
> reasons that I believe no longer exist.
>
> Jeremy
>
>
>
> On Wed, Mar 15, 2017 at 2:13 PM, David Neuman
> wrote:
>
> > At least a few of those (Steering, federations) were put in the "internal"
> > namespace to work around Comcast specific issues. I don't know that I like
> > the idea of duplicating routes, if anything we should see what is impacted
> > by moving them out of the internal namespace.
> >
> > On Wed, Mar 15, 2017 at 1:30 PM, Jeremy Mitchell
> > wrote:
> >
> > > Currently, we have a number of API routes scoped as "internal". Here are
> > a
> > > few examples:
> > >
> > > https://github.com/apache/incubator-trafficcontrol/blob/
> > > master/traffic_ops/app/lib/TrafficOpsRoutes.pm#L516
> > >
> > > I believe this is going to make it more difficult as we try to implement
> > > more granular roles / capabilities coupled with tenancy.
> > >
> > > So I'm proposing that we create a duplicate non-internal route like this,
> > > for example:
> > >
> > > $r->get("/api/$version/steering")->over( authenticated => 1 )->to(
> > > 'Steering#index', namespace => 'API::DeliveryService' );
> > >
> > > that way we can slowly move away from the "internal" routes and
> > eventually
> > > deprecate them.
> > >
> > > I think with our upcoming more robust role / tenancy model, there is no
> > > longer a need for "internal".
> > >
> > > Thoughts?
> > >
> > > Jeremy
> > >
> >

Re: Proposed change to Deliverservice API

[Dewayne Richardson]

Right sorry, 3.0

On Wed, Nov 30, 2016 at 8:40 AM, Jan van Doorn <j...@knutsel.com> wrote:

> > ... requirements on the list for 2.1.
>
> For 3.0, you mean? No API breaking changes on a minor rev, I thought?
>
>
> > On Nov 30, 2016, at 08:35, Dewayne Richardson <dewr...@gmail.com> wrote:
> >
> > I think once we get 2.0 in place we can start discussing the direction
> > forward.  I agree with the need for "natural keys" because I also was
> > working on an integration test tool and found the need to "lookup" the
> "id"
> > too cumbersome.  Once 2.0 is in place, we should put these on the
> > requirements on the list for 2.1.
> >
> > On Wed, Nov 30, 2016 at 8:28 AM, David Neuman <david.neuma...@gmail.com>
> > wrote:
> >
> >> Fair enough.  I agree we should use natural keys.  To me the ID thing is
> >> something internal to the DB that a client should not have to worry
> about.
> >> I can update the Traffic Ops client to support using IDs and keep the
> API
> >> as-is.
> >>
> >> On Wed, Nov 30, 2016 at 8:26 AM, Jan van Doorn <j...@knutsel.com> wrote:
> >>
> >>> Agree with Jeremey. And we can't just slip in a change to 2.0 that does
> >>> part of this.
> >>>
> >>> I'm -1 on neuman's change, at least for 2.0.
> >>>
> >>> Cheers,
> >>> JvD
> >>>
> >>>
> >>>
> >>>> On Nov 30, 2016, at 08:23, Jeremy Mitchell <mitchell...@gmail.com>
> >>> wrote:
> >>>>
> >>>> Let's look at an example of using ID's versus names for POSTS
> (creates)
> >>>>
> >>>> Here is the region table. A region belongs to a division.
> >>>>
> >>>> mysql> desc region;
> >>>> +--+-+--+-+-
> >>> --+-+
> >>>> | Field| Type| Null | Key | Default   | Extra
> >>>>   |
> >>>> +--+-+--+-+-
> >>> --+-+
> >>>> | id   | int(11) | NO   | PRI | NULL  |
> >>>> auto_increment  |
> >>>> | name | varchar(45) | NO   | UNI | NULL  |
> >>>>   |
> >>>> | division | int(11) | NO   | MUL | NULL  |
> >>>>   |
> >>>> | last_updated | timestamp   | YES  | | CURRENT_TIMESTAMP | on
> >> update
> >>>> CURRENT_TIMESTAMP |
> >>>> +--+-+--+-+-
> >>> --+-+
> >>>> 4 rows in set (0.01 sec)
> >>>>
> >>>> Currently, if you want to create a region, you have to provide a
> >> division
> >>>> "id" (as opposed to a division name)
> >>>>
> >>>> POST /api/version/regions
> >>>>
> >>>> {
> >>>> name: "myregion",
> >>>> division: 2
> >>>> }
> >>>>
> >>>> This allows the json to go directly into the table with one insert
> >> query.
> >>>>
> >>>> if we use a division name instead, like this
> >>>>
> >>>> {
> >>>> name: "myregion",
> >>>> division: "central"
> >>>> }
> >>>>
> >>>> then 2 queries have to happen on the server side. 1 query to fetch the
> >>>> division id for "central" and then the insert query to create the
> >> region.
> >>>>
> >>>> To do this right, imo, the ID for the central division WOULD be
> >> "central"
> >>>> instead of the number 2 - and this is why natural keys make a lot of
> >>> sense.
> >>>> So rather than changing the api to accept cdn name, profile name, and
> >>> type
> >>>> name, continue to send thru the ids and we need to make the effort to
> >> get
> >>>> to natural keys.
> >>>>
> >>>> my 2 cents
> >>>>
> >>>> On Wed, Nov 30, 2016 at 7:53 AM, Dave Neuman <neu...@apache.org>
> >> wrote:
> >>>>
> >>>>> Thanks Derek, it's actually al