+1 Makes sense to me. Is 2.0 the target for dropping the PERL implementation?
On 12/2/19, 12:19 PM, "Gray, Jonathan" <[email protected]> wrote: +1 I think the only point of contention was if the API v2.0 would be in ATC v4.0 or not. Either way is fine with me, and more time to know this is coming makes sense. Jonathan G On 12/2/19, 11:32 AM, "Rawlin Peters" <[email protected]> wrote: Hey all, A bunch of us got together and discussed this issue offline, and we got consensus to go with Option 1 - Graduate current API from 1.x to 2.x - allowing us to leave behind routes in 1.x that are dangerous/deprecated. Deprecated endpoints will not be rewritten from Perl to Go. So, we will graduate the latest minor version of existing 1.x Go endpoints to 2.0. For example, today we have /api/1.2/deliveryservices, /api/1.3/deliveryservices, and /api/1.4/deliveryservices. We will take the existing handler for /api/1.4/deliveryservices and promote it to /api/2.0/deliveryservices. We will follow this example for other 1.x Go endpoints with multiple minor versions as well. API endpoints that are deemed broken, dangerous, or deprecated will not be promoted to API 2.0. This will likely include the ~7 endpoints that I've mentioned in this thread already, but we haven't yet compiled the full list of endpoints that won't be promoted to 2.0. We will come up with that list (along with its criteria) soon and bring that list to the mailing list for everyone to look over and agree upon. Deprecated endpoints will be updated to return deprecation alerts where applicable. This API graduation will likely miss the boat for 4.0 but will hopefully land in 4.1 or 4.2 depending on how long it takes. So this would be the tentative timetable: TC 4.0: - API 1.x supported, some deprecation notices TC 4.1: - API 1.x still supported, deprecation notices added to endpoints not graduated to 2.0 - API 2.0 supported, consisting of 1.x endpoints that were graduated - starting with this release, you need to start migrating external clients off of 1.x over to 2.0 TC 4.2: - internal clients (e.g. TM, TR, etc) will be migrated off API 1.x over to 2.0. Doing this step after 4.1 adds confidence that 1.x is still supported alongside 2.0 in order to provide a smooth migration period for API clients. TC 5.0: - API 1.x no longer supported, only API 2.x is supported Those who took part in that offline discussion, please keep me honest here with a +1 if we did in fact come to a rough consensus on this option. If you weren't part of that offline discussion, feel free to voice your vote on this issue as well (I believe a bunch of people were already +1 on this option a few weeks ago). - Rawlin On Tue, Nov 19, 2019 at 12:23 PM Robert O Butts <[email protected]> wrote: > > > - /riak/stats > > Looking at the Perl, this looks easy to rewrite. It's just proxying a > request to Riak. > > +1 on rewriting this, for API stability, even if we don't "value" the > endpoint as a project. > > >- /cachegroup_fallbacks > > Likewise +1 on rewriting. Not as trivial as /riak/stats, but it also looks > easy to me, looks like a standard CRUD endpoint. +1 on leaving a > "deprecated, please prefer X" message in, and +1 in removing in the big API > rewrite/refactor/2.0. > > > - /capabilities (all methods except for GET) -- capabilities are > built-in, should not be changed at runtime > > - /api_capabilities (all methods except for GET) -- same reasons as > /capabilities > > I actually don't agree with this - because they're useful to plugins. IMO > it should be possible to add and modify Capabilities and API Capabilities, > because they can be used by plugins which add new endpoints, to control > access to them. If there's disagreement there, I'd vote we take that to a > separate thread. > > But in terms of the endpoints themselves, the Perl also looks easy to > rewrite, looks like standard CRUD. -1 on deprecating or removing, +1 on > rewriting. > > > - /cdns/{name}/configs/routing -- this was a half-baked attempt at > redoing CrConfig.json -- cannot possibly be of use to anyone in its current > state > > Ugh. I agree with the sentiment; and this one looks slightly less trivial. > But I personally don't see it as an unreasonable amount of work. For the > sake of API stability, I'm personally willing to volunteer to rewrite this > one, including API tests to make fixing easy if the schema changes (which > it appears to have in master). And hopefully it can just exist until the > fabled API Rewrite. Is that acceptable? > > > - the ATS config generation endpoints -- Rob actually proposed to remove > these from 1.x due to danger of being out of sync w/ cache-side config > generator > > I'm personally on the fence about this one. Like I said in that thread, I'm > the last one to want to break users, and I don't take this lightly. If, > after I made the case for it, I'm the only one, I might be inclined to > retract my proposal, and just continue to maintain them. Maybe some more > API tests would help with that. I think I'm the primary dev on the ATS > config gen at this point anyway. > > > - ATS Config Files Stuff Aren't these all already in Go anyway? Whether > or not they should be removed from the API aside, I don't think anything > actually needs to hit Perl anymore. > > Correct. > > > - /hwinfo/dtdata > > I think the 'render' call here is just rendering the JSON. But, the > endpoint itself is just hanging and not returning for me, I'm not sure why. > I'd like to volunteer to look into this endpoint as well. I think you might > be right, this one might be unreasonable to rewrite. But if it is > reasonable, I'd like to just do it. I'd like to at least look into it. > > > On Tue, Nov 19, 2019 at 12:05 PM ocket 8888 <[email protected]> wrote: > > > I'm generally +1 for option 2 > > > > regarding those specific endpoints: > > > > - /riak/stats if this is easy and doesn't expose anything dangerous, we > > should do it IMO. > > - /cachegroup_fallbacks should be pretty trivial, so I still say do it > > - /capabilities since these aren't actually used for anything, I think it's > > pretty safe (and easy) to rewrite. Plus there's already a PR open for > > exactly that > > - /api_capabilities dropping this one could be problematic, just because > > people may depend on it to tell them what endpoints exist. To be clear, I > > think it ought to be deprecated, but unfortunately I just don't think it's > > terribly unlikely that this is built into scripts and tools. The actual > > code for the rewrite should be easy too, although maintaining it is a pain. > > > > - ATS Config Files Stuff Aren't these all already in Go anyway? Whether or > > not they should be removed from the API aside, I don't think anything > > actually needs to hit Perl anymore. > > > > - /cdn/{{name}}/configs/routing CRConfigs are large and complex, so if this > > never worked anyway maybe we should just add a Go handler for it that > > returns a '410 Gone' response (or '501 Not Implemented' is the other one > > I've heard suggested) and leave the actual logic behind in the Perl > > - /hwinfo/dtdata if this calls into Mojolicious rendering, returns a > > non-json response, and/or contains information that in either content or > > encoding is meaningless to any sane client then it should also return a > > 410/501 and get left behind in Perl. > > > > On Tue, Nov 19, 2019 at 11:08 AM Rawlin Peters <[email protected]> wrote: > > > > > While I appreciate and mostly agree with your essay on the importance > > > of not breaking users (or rather, breaking them in predictable ways), > > > I for one would really like to meet at some middle ground with option > > > 2. I'm not asking to remove 50 high-usage endpoints from 1.x. IIRC > > > I've only specifically proposed "leaving behind" a few specific Perl > > > endpoints that I believe serve no purpose, have no valid reason to be > > > used, and/or are dangerous to the system: > > > > > > - /riak/stats -- informational, was used for debugging the initial > > > Riak integration. This one might be pretty easy to rewrite since it > > > just calls a corresponding riak API and returns the result via TO, so > > > I could be on board with rewriting it if necessary. > > > - /cachegroup_fallbacks -- these endpoints were released in 3.0 before > > > we managed to augment the existing /cachegroups API to handle > > > fallbacks, so I could get on board with rewriting these if we really > > > think it's important. > > > - /capabilities (all methods except for GET) -- capabilities are > > > built-in, should not be changed at runtime > > > - /api_capabilities (all methods except for GET) -- same reasons as > > > /capabilities > > > - /cdns/{name}/configs/routing -- this was a half-baked attempt at > > > redoing CrConfig.json -- cannot possibly be of use to anyone in its > > > current state > > > - the ATS config generation endpoints -- Rob actually proposed to > > > remove these from 1.x due to danger of being out of sync w/ cache-side > > > config generator > > > - /hwinfo/dtdata -- AFAIK the "dtdata" endpoints were only > > > used/required by the old Perl Mojolicious UI, no valid reason to be > > > using this as an external API client. In fact, I'm not even sure _how_ > > > this endpoint can be used outside of the Mojolicious framework/UI. It > > > calls the Mojolicious `render` function, which usually means rendering > > > an embedded Perl HTML template given some data. This is one I would > > > consider completely unreasonable to rewrite to Go. > > > > > > Can we get on board with leaving at least these 1.x endpoints behind > > > in the Perl? Doing a cursory skim over the remaining 1.x endpoints to > > > rewrite, it _seems_ like everything else would fall into the "simpler > > > to just rewrite rather than gain consensus to remove" category. > > > > > > - Rawlin > > > > > > On Tue, Nov 19, 2019 at 9:19 AM Robert O Butts <[email protected]> wrote: > > > > > > > > >Option 1 - Graduate current API from 1.x to 2.x - allowing us to leave > > > > behind routes in 1.x that are dangerous or deprecated > > > > >Option 2 - Rewrite all endpoints from Perl to Golang in current > > version > > > as > > > > we are doing now. > > > > > > > > +1 on either Option 1 or Option 2, without a strong preference. Option > > 1 > > > > has advantages of letting us also safely remove the Cache Config > > > endpoints > > > > and implement Layered Profiles. But Option 2 also has the advantage of > > > not > > > > requiring users to upgrade. > > > > > > > > > Option 3 - Only rewrite endpoints from Perl to Golang in current > > > version > > > > that we agree are not dangerous or should deprecate > > > > > > > > -1. As before, it breaks users unpredictably. > > > > > > > > >It will literally outright break clients (*all* of them, not just > > > *some*) > > > > > > > > This is technically correct: technically, new versions 'break' all > > users. > > > > The difference is, it does it in a predictable, safe way that users can > > > > handle, inspect, and safely upgrade. > > > > > > > > >Can someone please explain to me why (A) artificially graduating the > > > > current 1.x API to 2.x, and removing the deprecated endpoints from 1.x, > > > is > > > > better for API clients overall than (B) just removing deprecated > > > endpoints > > > > from 1.x? Existing 1.x clients would gain absolutely nothing from being > > > > forced to update to 2.x like this. > > > > > > > > >for zero gain in value > > > > > > > > Users would gain something. There is a very specific gain. > > > > > > > > The gain is that the breakage is predictable. People frequently upgrade > > > > minor versions automatically, either running commands without thinking, > > > or > > > > even automating the entire process. > > > > > > > > For TC, this applies to both CDN operators, and their end-users. > > > > > > > > When the minor versions contain breaking changes, that automatic > > upgrade > > > > then breaks. In this case, any scripts or tools they have using the > > > removed > > > > endpoints will break. If those scripts are important, this could bring > > > down > > > > their CDN, or break their Delivery Service, or otherwise cause an > > outage. > > > > > > > > Contrawise, users expect breaking changes in Major versions, and don't > > > > typically automatically upgrade them. Rather, they'll manually inspect > > > the > > > > update, do testing, and check the changelog for breaking changes. > > > > > > > > It's not a technical thing, it's a human thing. Humans automatically > > > > upgrade minor versions, and write scripts to do so, and dependency > > files > > > to > > > > allow it, because the software industry taught people that minor > > versions > > > > shouldn't break anything. > > > > > > > > If we start breaking minor versions, initially users will experience > > > > breakage. Not all users, and not all catastrophic. But some users will > > > > experience catastrophic outages. And it will be our fault as a project. > > > And > > > > after enough times, they'll fork TC, or find another product, or write > > > > their own; and stop participating in the community, or never start. > > > > > > > > Then, once they've learned that we can't be trusted not to break > > things, > > > > they'll experience the 2nd problem: where nothing can be safely > > upgraded, > > > > and the smallest change requires long and expensive testing to verify a > > > > single minor version or even patch upgrade, and still risk missing > > > > something and bringing their CDN down. To be honest, we're pretty much > > > > already here. It already costs our dev and ops teams unbelievable > > amounts > > > > of time and person-hours of testing for every minor upgrade. That time > > > far > > > > exceeds the development time to just rewrite the small endpoints we > > don't > > > > like anymore (and tangentially, also the time to write the Integration > > > and > > > > E2E tests TC desperately needs). > > > > > > > > ATC has so much potential. There aren't many OSS CDNs out there, and > > > > there's certainly the demand. We have the potential to become a huge > > > > project, which for us existing users, translates into potentially an > > > > enormous amount of free development. But we'll never get there if we > > keep > > > > breaking things. > > > > > > > > And for my particular organization (Comcast), we have a hard enough > > time > > > > upgrading. Our upgrades, especially Traffic Ops, are extremely painful, > > > and > > > > require enormous amounts of manual testing. Even then, we frequently > > have > > > > issues that require rollback, sometimes even an outage; even after > > those > > > > unbelievably amounts of developer and operator time testing. We > > > desperately > > > > need a system that is safer and easier to upgrade. Breaking changes are > > > > always going to require more testing; but we can at least make minor > > > > versions safe to upgrade. > > > > > > > > > I know I am changing my initial stance but I am +1 on rewriting all > > the > > > > Perl endpoints to Golang and not breaking the API contract we have in > > > > place. Most of the API routes to rewrite are rather simple to port and > > I > > > > think would be faster than getting consensus on actually deprecating it > > > > from the mailing list. > > > > > > > > I'm inclined to agree with this. The things I've seen marked > > "deprecated" > > > > seem to mostly be small, and look like they'd be pretty easy to > > rewrite. > > > > But I haven't seen the whole list. Is there a list somewhere, so the > > > > community can see it to better decide? > > > > > > > > > > > > On Tue, Nov 19, 2019 at 7:24 AM Hoppal, Michael < > > > [email protected]> > > > > wrote: > > > > > > > > > I would love to drive this to some type of consensus on rewrite > > > strategy > > > > > as we have a good amount of resources (and growing) working on the > > > rewrite > > > > > and I want to keep that momentum. > > > > > > > > > > Sounds like there is three options from the email thread (please add > > > any I > > > > > missed) > > > > > > > > > > Option 1 - Graduate current API from 1.x to 2.x - allowing us to > > leave > > > > > behind routes in 1.x that are dangerous or deprecated > > > > > Option 2 - Rewrite all endpoints from Perl to Golang in current > > > version as > > > > > we are doing now. > > > > > Option 3 - Only rewrite endpoints from Perl to Golang in current > > > version > > > > > that we agree are not dangerous or should deprecate > > > > > > > > > > I am +1 for option 2 (which I know is different from my original > > > stance). > > > > > > > > > > Which direction would others like to go? > > > > > > > > > > Thanks, > > > > > > > > > > Michael > > > > > > > > > > > > > > > I know I am changing my initial stance but I am +1 on rewriting all > > the > > > > > Perl endpoints to Golang and not breaking the API contract we have in > > > > > place. Most of the API routes to rewrite are rather simple to port > > and > > > I > > > > > think would be faster than getting consensus on actually deprecating > > it > > > > > from the mailing list. > > > > > > > > > > On 11/16/19, 3:31 PM, "Rawlin Peters" <[email protected]> wrote: > > > > > > > > > > > Moving to 2.x means that it's my fault as a consumer of the API > > > if I > > > > > was still using the deprecated endpoint (and that change should be > > > part of > > > > > a larger upgrade plan). Simply removing the deprecated routes, > > > > > retroactively, means I can't rely on having a transition period to > > get > > > to > > > > > 2.0 outside of a large completely down upgrade window. > > > > > > > > > > There would be a transition period either way. Endpoints would be > > > > > deprecated in ATC 4.0 and removed in 5.0. The release notes for > > 4.0 > > > > > would clearly list out the deprecated endpoints. If they have > > > > > replacements, there would be instructions on how to transition to > > > the > > > > > replacement. Deprecated endpoints would return alerts. After > > > upgrading > > > > > to 4.0, operators should be well aware that certain endpoints are > > > > > being removed in 5.0, and it's on them to start using new > > > endpoints, > > > > > change workflows, etc. > > > > > > > > > > By forcing operators to upgrade all of their clients to start > > using > > > > > API 2.x, it just adds to their burden of upgrading when that time > > > > > comes. Rather than surgically fixing the clients of the 2% of > > > > > deprecated API endpoints before upgrading to ATC 5.0, operators > > > would > > > > > have to fix every single client they have to use API 2.x before > > > > > upgrading to 5.0. Is that exercise really worth it? Most people > > > would > > > > > probably be blissfully unaware that the "POST > > > /api/1.1/capabilities" > > > > > endpoint disappeared (among other deprecated endpoints). Less > > > footguns > > > > > in the system, and they didn't even have to update their clients. > > > > > > > > > > Anyways, circling back to Michael's original question, it sounds > > > like > > > > > either option we choose, we at least have consensus on this > > > > > deprecation strategy: > > > > > - The route IS NOT rewritten from Perl to Golang and a > > deprecation > > > > > notice is added to the alert response in the Perl handler > > > > > > > > > > If we're content to force all API clients to update their clients > > > to > > > > > 2.x for zero gain in value, at least we won't be wasting work by > > > > > rewriting deprecated Perl endpoints to Go. > > > > > > > > > > - Rawlin > > > > > > > > > > On Fri, Nov 15, 2019 at 7:06 PM Gray, Jonathan > > > > > <[email protected]> wrote: > > > > > > > > > > > > Moving to 2.x means that it's my fault as a consumer of the API > > > if I > > > > > was still using the deprecated endpoint (and that change should be > > > part of > > > > > a larger upgrade plan). Simply removing the deprecated routes, > > > > > retroactively, means I can't rely on having a transition period to > > get > > > to > > > > > 2.0 outside of a large completely down upgrade window. Depending on > > > how > > > > > the API is used, even in our own components such as TM, that could > > > prove > > > > > disastrous. For the older minor revs like 1.1/1.2, I could live with > > > > > dropping those as part of a major project release (including upgrade > > > > > instructions outlining what releases you must be on beforehand). > > > Currently > > > > > 1.3 I view as stable enough to code to or use a native client library > > > from, > > > > > and 1.4 is my risk/overhead to take for new stuff as it's still > > > evolving > > > > > and improving. Going to 1.5 or 2.0 moves that forward. The path of > > > least > > > > > surprise for humans is to expect big shifts on those major revs. > > > Removal > > > > > falls under that umbrella and gets as a consumer not to strictly > > focus > > > on > > > > > endpoint->endpoint moves, but rather workflow shifts and maybe now I > > > have > > > > > to use the API a different way than before. I believe getting this > > > right > > > > > is something that would greatly help us understand does X component > > at > > > > > version 1 work with Y component at version 2, because now you're > > > asking do > > > > > components X and Y both speak protocol A. With major releases we can > > > work > > > > > through that each time, but it's error prone and requires experts in > > > all > > > > > products to agree upon. If you do nothing but consume major > > releases, > > > and > > > > > then conduct your own impact evaluation having read the upgrade > > > > > instructions thoroughly, maybe you'd be able to smoothly transition. > > > > > > > > > > > > Jonathan G > > > > > > > > > > > > > > > > > > On 11/15/19, 5:24 PM, "Rawlin Peters" <[email protected]> > > wrote: > > > > > > > > > > > > Can someone please explain to me why (A) artificially > > > graduating > > > > > the > > > > > > current 1.x API to 2.x, and removing the deprecated > > endpoints > > > > > from > > > > > > 1.x, is better for API clients overall than (B) just > > removing > > > > > > deprecated endpoints from 1.x? Existing 1.x clients would > > > gain > > > > > > absolutely nothing from being forced to update to 2.x like > > > this. > > > > > > > > > > > > The way I see the two choices: > > > > > > A) break every single API client out there > > > > > > B) break only clients of APIs that we've decided upon as a > > > > > community > > > > > > to be deprecated > > > > > > > > > > > > IMO, when we have to break things, we should try break the > > > least > > > > > > amount of clients possible. All I'm looking for is a > > valuable > > > > > reason > > > > > > to choose option A over option B, that isn't just "because > > > > > SemVer". > > > > > > > > > > > > - Rawlin > > > > > > > > > > > > On Wed, Nov 13, 2019 at 3:24 PM Schmidt, Andrew > > (Contractor) > > > > > > <[email protected]> wrote: > > > > > > > > > > > > > > If we decide to get rid of the API versioning it makes > > > sense > > > > > to me that we would do that on a major version upgrade, probably to > > > the ATC > > > > > version. But I think it would be too early to do it on the 2.0 API > > > version. > > > > > I like this compromise +1. I was wondering why we were stuck on the > > > idea > > > > > that 2.0 had to be an API rewrite. That seems like it’s a long way > > > out. 2.0 > > > > > = deprecations and fixes. 3.0 or 5.0 = api rewrite and maybe version > > > > > convergence. > > > > > > > > > > > > > > On 11/13/19, 11:52 AM, "[email protected]" < > > > > > [email protected]> wrote: > > > > > > > > > > > > > > Lemme lead with: making breaking changes on a major > > > > > version boundary > > > > > > > gets +1 from me - no reason the API needs to be fully > > > > > redesigned for > > > > > > > any particular number. > > > > > > > > > > > > > > But I just gotta bring this up again: if we want to > > be > > > > > able to > > > > > > > deprecate an endpoint for a major release and remove > > > it in > > > > > the next > > > > > > > without needing to deal with the messiness of API > > > > > versions, we could > > > > > > > always just get rid of API versions. Major ATC > > > revisions > > > > > should be > > > > > > > easier to come by with our new release strategy. > > > > > > > > > > > > > > On Wed, 2019-11-13 at 10:23 -0700, Rawlin Peters > > wrote: > > > > > > > > > It is a compromise - it's an inconvenience to > > > users, > > > > > forcing them > > > > > > > > > to fix > > > > > > > > scripts to upgrade. Which isn't good for user > > > experience > > > > > either; but > > > > > > > > it > > > > > > > > doesn't outright break people. It's a compromise > > I'm > > > > > willing to live > > > > > > > > with. > > > > > > > > > > > > > > > > It will literally outright break clients (*all* of > > > them, > > > > > not just > > > > > > > > *some*). We have been down this road before, so > > I'll > > > > > just repeat what > > > > > > > > I said when this exact conversation came up months > > > > > ago[1]: > > > > > > > > > > > > > > > > "So it basically boils down to: > > > > > > > > A) promote Go 1.x routes to 2.x and eventually > > break > > > ALL > > > > > users of the > > > > > > > > 1.x API when the time comes to remove 1.x > > > > > > > > B) deprecate and remove some 1.x Perl routes and > > > > > POSSIBLY break SOME > > > > > > > > users" > > > > > > > > > > > > > > > > Either way we slice and dice it with versioning, > > > > > removing an endpoint > > > > > > > > means that clients of that endpoint will break. By > > > > > graduating only > > > > > > > > the > > > > > > > > "good stuff" from 1.x to 2.x, and removing > > endpoints > > > > > from 1.x, we > > > > > > > > will > > > > > > > > unnecessarily break all 1.x clients when the time > > > comes > > > > > to remove the > > > > > > > > rest of 1.x. Instead, we can choose to break only > > > > > clients of 1.x APIs > > > > > > > > that we already deemed dangerous, unnecessary, > > > useless, > > > > > etc. enough > > > > > > > > to > > > > > > > > deprecate. > > > > > > > > > > > > > > > > Why break 100% of our API clients when we can > > choose > > > to > > > > > break only a > > > > > > > > small number of clients that happen to use > > > > > > > > dangerous/unnecessary/useless APIs? Why would we > > > break > > > > > our API > > > > > > > > clients > > > > > > > > unnecessarily for no added value? There is no sense > > > in > > > > > blindly > > > > > > > > adhering to Semantic Versioning just for the sake > > of > > > > > adhering to > > > > > > > > Semantic Versioning. Instead, use it as a > > guideline, > > > and > > > > > break the > > > > > > > > rules where it makes the most sense to do so. > > > > > > > > > > > > > > > > - Rawlin > > > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > https://urldefense.com/v3/__https://lists.apache.org/thread.html/15c6bfcdce0ac00a3a8612a96c3d668a8a5a16c6ca9401de9473d31c@*3Cdev.trafficcontrol.apache.org*3E__;JSU!rx_L75ITgOQ!V_b3E9iEYqNlZus18f-sVuofwuyxapCthg3LdO4NxyMCU39b7mdf4klP5KNR7XZ7d3Uc$ > > > > > > > > > > > > > > > > On Wed, Nov 13, 2019 at 9:43 AM Robert O Butts < > > > > > [email protected]> > > > > > > > > wrote: > > > > > > > > > > But if it's really valuable to remove these > > > > > endpoints, maybe it's > > > > > > > > > appropriate to bump the major version and ask > > > > > everybody to update > > > > > > > > > their > > > > > > > > > clients? > > > > > > > > > > > > > > > > > > +1 > > > > > > > > > > > > > > > > > > This sounds like the best compromise to me. It > > > > > addresses both sides > > > > > > > > > - it > > > > > > > > > lets us remove endpoints, and it also prevents > > > > > breaking users. > > > > > > > > > > > > > > > > > > It is a compromise - it's an inconvenience to > > > users, > > > > > forcing them > > > > > > > > > to fix > > > > > > > > > scripts to upgrade. Which isn't good for user > > > > > experience either; > > > > > > > > > but it > > > > > > > > > doesn't outright break people. It's a compromise > > > I'm > > > > > willing to > > > > > > > > > live with. > > > > > > > > > > > > > > > > > > This actually lets us do at least 3 things: > > > > > > > > > 1. Remove "deprecated" endpoints. > > > > > > > > > 2. Remove the unmaintained dangerously-large > > Cache > > > > > Config > > > > > > > > > endpoints. > > > > > > > > > 3. Lets us do Layered Profiles, changing Servers > > > and > > > > > DSes to have > > > > > > > > > arrays of > > > > > > > > > Profiles instead of a single Profile, without > > hacky > > > > > backward- > > > > > > > > > compatibility > > > > > > > > > attempts. > > > > > > > > > > > > > > > > > > To clarify, `/api/2.0` would be essentially the > > > > > current API, just > > > > > > > > > with the > > > > > > > > > above breaking changes (possibly others) which > > > require > > > > > a SemVer > > > > > > > > > major > > > > > > > > > version increase. And the big API > > rewrite/redesign > > > > > we've been > > > > > > > > > calling "2.0" > > > > > > > > > would become `/api/3.0` or some future version. > > > > > > > > > > > > > > > > > > It's not perfect, but it seems to address the > > most > > > > > concerns on all > > > > > > > > > sides. > > > > > > > > > Thoughts? Objections? > > > > > > > > > > > > > > > > > > > > > > > > > > > On Wed, Nov 13, 2019 at 8:45 AM Chris Lemmons < > > > > > [email protected]> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > For the endpoints, I agree in part and disagree > > > in > > > > > part. > > > > > > > > > > > > > > > > > > > > Agree for these two: > > > > > > > > > > - the endpoints are dangerous and pose a risk > > to > > > the > > > > > overall > > > > > > > > > > Traffic > > > > > > > > > > Control system > > > > > > > > > > - the task of rewriting the endpoints from Perl > > > to > > > > > Go represents > > > > > > > > > > an > > > > > > > > > > unreasonable amount of work for the project > > > > > > > > > > > > > > > > > > > > And I would add: > > > > > > > > > > - the endpoint does not serve its intended > > > purpose > > > > > and cannot > > > > > > > > > > reasonably > > > > > > > > > > be used > > > > > > > > > > - the endpoint has been made unsafe or unusable > > > by > > > > > new features > > > > > > > > > > added to TC > > > > > > > > > > > > > > > > > > > > Disagree for these two: > > > > > > > > > > - the endpoints have been obsoleted by other > > > > > endpoints > > > > > > > > > > - the endpoints don't seem to serve a true > > > purpose > > > > > or don't seem > > > > > > > > > > to be > > > > > > > > > > valuable to the project anymore > > > > > > > > > > > > > > > > > > > > For the most part, obsoleted endpoints can be > > > > > maintained with a > > > > > > > > > > shim > > > > > > > > > > into the new code. If they would require "an > > > > > unreasonable amount > > > > > > > > > > of > > > > > > > > > > work", it might make sense to remove them, > > then. > > > > > > > > > > > > > > > > > > > > Things that don't seem valuable usually fall > > into > > > > > another > > > > > > > > > > category > > > > > > > > > > like "cannot reasonably be used", but if the > > only > > > > > reason for > > > > > > > > > > deprecation is that the feature is going away, > > we > > > > > should really > > > > > > > > > > support it according to our promise. > > > > > > > > > > > > > > > > > > > > But if it's really valuable to remove these > > > > > endpoints, maybe it's > > > > > > > > > > appropriate to bump the major version and ask > > > > > everybody to update > > > > > > > > > > their clients? > > > > > > > > > > > > > > > > > > > > On Tue, Nov 12, 2019 at 5:27 PM Rawlin Peters < > > > > > [email protected]> > > > > > > > > > > wrote: > > > > > > > > > > > The real crux of the issue here is whether or > > > not > > > > > we agree as a > > > > > > > > > > > project that we can move forward with this > > plan > > > > > that I believe > > > > > > > > > > > to be a > > > > > > > > > > > good compromise that meets most concerns but > > > > > "breaks the > > > > > > > > > > > Semantic > > > > > > > > > > > Versioning promise": > > > > > > > > > > > 1) don't rewrite deprecated endpoints from > > > Perl to > > > > > Go > > > > > > > > > > > 2) in the Perl handler, add a deprecation > > > notice > > > > > to the > > > > > > > > > > > "alerts" > > > > > > > > > > > section of the response > > > > > > > > > > > 3) once a deprecated API has been deprecated > > > for > > > > > one major > > > > > > > > > > > release, > > > > > > > > > > > delete it from the API in the subsequent > > major > > > > > release (e.g. if > > > > > > > > > > > deprecation notices are added and released in > > > ATC > > > > > 4.x, we can > > > > > > > > > > > remove > > > > > > > > > > > the deprecated endpoint in ATC 5.x) > > > > > > > > > > > > > > > > > > > > > > Rob, from your latest proposal to remove the > > > ATS > > > > > config API > > > > > > > > > > > endpoints > > > > > > > > > > > from 1.x, I thought we were past this issue > > > > > already and could > > > > > > > > > > > finally > > > > > > > > > > > agree to remove deprecated endpoints from 1.x > > > > > (following steps > > > > > > > > > > > 1-3 > > > > > > > > > > > above) without requiring an all-new 2.x API. > > > Are > > > > > we not > > > > > > > > > > > actually past > > > > > > > > > > > that? You came up with valid reasons for > > > applying > > > > > these steps > > > > > > > > > > > to the > > > > > > > > > > > ATS config API endpoints, and I think it's > > > fair to > > > > > say that we > > > > > > > > > > > can > > > > > > > > > > > come up with valid reasons for applying these > > > > > steps to other > > > > > > > > > > > deprecated API endpoints as well. > > > > > > > > > > > > > > > > > > > > > > It seems like the real problem is actually > > > > > agreeing on which > > > > > > > > > > > _specific_ endpoints we should be able to > > > follow > > > > > steps 1-3 for. > > > > > > > > > > > IMO I > > > > > > > > > > > think it should apply to any 1.x endpoints > > > that we > > > > > have valid > > > > > > > > > > > reasons > > > > > > > > > > > for not carrying forward to Go, including but > > > not > > > > > limited to: > > > > > > > > > > > - the endpoints are dangerous and pose a risk > > > to > > > > > the overall > > > > > > > > > > > Traffic > > > > > > > > > > > Control system > > > > > > > > > > > - the endpoints have been obsoleted by other > > > > > endpoints > > > > > > > > > > > - the endpoints don't seem to serve a true > > > purpose > > > > > or don't > > > > > > > > > > > seem to be > > > > > > > > > > > valuable to the project anymore > > > > > > > > > > > - the task of rewriting the endpoints from > > > Perl to > > > > > Go > > > > > > > > > > > represents an > > > > > > > > > > > unreasonable amount of work for the project > > > > > > > > > > > > > > > > > > > > > > Obviously most of these reasons are > > > subjective, so > > > > > we'd have to > > > > > > > > > > > come > > > > > > > > > > > to some level of consensus on deprecating > > > > > particular endpoints > > > > > > > > > > > first, > > > > > > > > > > > but we should be able to follow steps 1-3 for > > > > > whatever > > > > > > > > > > > endpoints we > > > > > > > > > > > decide. > > > > > > > > > > > > > > > > > > > > > > Making breaking changes shouldn't be taken > > > > > lightly, but we need > > > > > > > > > > > a path > > > > > > > > > > > forward that allows some calculated, breaking > > > > > changes where > > > > > > > > > > > necessary. > > > > > > > > > > > Otherwise, we are unnecessarily holding the > > > > > project back. > > > > > > > > > > > > > > > > > > > > > > - Rawlin > > > > > > > > > > > > > > > > > > > > > > On Tue, Nov 12, 2019 at 11:44 AM Jeremy > > > Mitchell < > > > > > > > > > > > [email protected]> > > > > > > > > > > wrote: > > > > > > > > > > > > Yeah, technically a "rewrite" of 1.x means > > > > > shifting the > > > > > > > > > > > > implementation > > > > > > > > > > from > > > > > > > > > > > > Perl to Go for ALL of 1.x. If you leave a > > > subset > > > > > of 1.x in > > > > > > > > > > > > Perl, then > > > > > > > > > > you > > > > > > > > > > > > have to call it a "partial rewrite". > > > > > > > > > > > > > > > > > > > > > > > > So, assuming we want to do a full rewrite > > (no > > > > > more perl for > > > > > > > > > > > > the api > > > > > > > > > > > > whatsoever), I think that means every 1.x > > api > > > > > endpoint (good > > > > > > > > > > > > or bad) > > > > > > > > > > needs > > > > > > > > > > > > to be rewritten to Go so we can remove the > > > > > dependency on > > > > > > > > > > > > perl. > > > > > > > > > > > > > > > > > > > > > > > > However, I still like marking many as > > > > > "deprecated" so we > > > > > > > > > > > > train users > > > > > > > > > > to use > > > > > > > > > > > > "better" endpoints that we envision will > > > exist > > > > > in 2.x and > > > > > > > > > > > > beyond. > > > > > > > > > > > > > > > > > > > > > > > > Jeremy > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Nov 12, 2019 at 11:22 AM Hoppal, > > > Michael > > > > > < > > > > > > > > > > [email protected]> > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > We do not have a lot of deprecated routes > > > as > > > > > of now as we > > > > > > > > > > > > > focused on > > > > > > > > > > > > > routes that are more used first. > > > > > > > > > > > > > > > > > > > > > > > > > > I know on a lot of the remaining rewrites > > > > > there has been > > > > > > > > > > > > > discussion > > > > > > > > > > on the > > > > > > > > > > > > > matching Github issues on potential > > > > > deprecation (I cannot > > > > > > > > > > > > > think of > > > > > > > > > > the > > > > > > > > > > > > > exact count but enough to warrant this > > > email > > > > > __) > > > > > > > > > > > > > > > > > > > > > > > > > > It is key to note that I am not arguing > > for > > > > > removal of > > > > > > > > > > > > > endpoints > > > > > > > > > > within > > > > > > > > > > > > > the current API version but if there is > > an > > > > > endpoint we > > > > > > > > > > > > > agree should > > > > > > > > > > be > > > > > > > > > > > > > deprecated going forward to put a message > > > > > within the > > > > > > > > > > > > > response and on > > > > > > > > > > the > > > > > > > > > > > > > docs. > > > > > > > > > > > > > > > > > > > > > > > > > > And thanks for the input sounds like you > > > are > > > > > +1 on > > > > > > > > > > > > > rewriting all > > > > > > > > > > routes to > > > > > > > > > > > > > Golang as Brennan is. > > > > > > > > > > > > > > > > > > > > > > > > > > On 11/12/19, 9:06 AM, "Robert O Butts" < > > > > > [email protected]> > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > Whether we rewrite a route in Go is > > an > > > > > implementation > > > > > > > > > > > > > detail. To > > > > > > > > > > the > > > > > > > > > > > > > interface, to users, it doesn't > > matter > > > > > whether a route > > > > > > > > > > > > > is > > > > > > > > > > rewritten or > > > > > > > > > > > > > not. > > > > > > > > > > > > > > > > > > > > > > > > > > But our API follows Semantic > > > Versioning, > > > > > in order to > > > > > > > > > > > > > not break > > > > > > > > > > users. > > > > > > > > > > > > > We > > > > > > > > > > > > > can't just remove endpoints that some > > > of > > > > > us don't use, > > > > > > > > > > > > > and > > > > > > > > > > assume other > > > > > > > > > > > > > people don't, maybe people who never > > > speak > > > > > up on the > > > > > > > > > > > > > mailing > > > > > > > > > > list. > > > > > > > > > > > > > We'll > > > > > > > > > > > > > never gain a big userbase if we keep > > > > > breaking users. > > > > > > > > > > > > > > > > > > > > > > > > > > Per the _project_ SemVer, once we > > have > > > API > > > > > 2.0, we can > > > > > > > > > > > > > deprecate > > > > > > > > > > API > > > > > > > > > > > > > 1.x, > > > > > > > > > > > > > and in the next major _project_ > > > release, > > > > > remove API > > > > > > > > > > > > > 1.x. > > > > > > > > > > Irrespective > > > > > > > > > > > > > of > > > > > > > > > > > > > Perl or Go. > > > > > > > > > > > > > > > > > > > > > > > > > > My big concern is, API 2.0 is a big > > > > > project. How long > > > > > > > > > > > > > has the > > > > > > > > > > rewrite > > > > > > > > > > > > > to Go > > > > > > > > > > > > > taken? Do we really believe designing > > > and > > > > > implementing > > > > > > > > > > > > > a > > > > > > > > > > completely > > > > > > > > > > > > > new API > > > > > > > > > > > > > will be any less time? > > > > > > > > > > > > > > > > > > > > > > > > > > I don't want killing Perl to have to > > > wait > > > > > on that. > > > > > > > > > > > > > > > > > > > > > > > > > > I know it feels like a waste to > > rewrite > > > > > routes that you > > > > > > > > > > > > > don't > > > > > > > > > > use, and > > > > > > > > > > > > > probably few people do. But that's > > the > > > > > cost of a stable > > > > > > > > > > > > > project. > > > > > > > > > > How > > > > > > > > > > > > > many > > > > > > > > > > > > > "deprecated" routes are there? If it > > > comes > > > > > down to > > > > > > > > > > > > > taking the > > > > > > > > > > > > > development > > > > > > > > > > > > > time to rewrite them so we can kill > > > Perl > > > > > faster, or > > > > > > > > > > > > > leaving Perl > > > > > > > > > > > > > around, I > > > > > > > > > > > > > vote we just do the work and kill > > Perl. > > > > > > > > > > > > > > > > > > > > > > > > > > >If we don't rewrite them, then Perl > > > will > > > > > last until > > > > > > > > > > > > > API 2.0 has > > > > > > > > > > been > > > > > > > > > > > > > designed, released and then *another > > > full > > > > > major release > > > > > > > > > > > > > cycle*. > > > > > > > > > > That's > > > > > > > > > > > > > way > > > > > > > > > > > > > too long to have two codebases for > > the > > > > > same component, > > > > > > > > > > > > > IMO, > > > > > > > > > > especially > > > > > > > > > > > > > since the rewrite is already 50% > > > complete. > > > > > > > > > > > > > > > > > > > > > > > > > > +1 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Nov 12, 2019 at 8:54 AM ocket > > > 8888 > > > > > < > > > > > > > > > > > > > [email protected]> > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > I vote that by and large we DO > > > rewrite > > > > > them, with > > > > > > > > > > > > > exceptions > > > > > > > > > > for > > > > > > > > > > > > > routes > > > > > > > > > > > > > > that just plain don't work, even in > > > > > Perl. Those are > > > > > > > > > > > > > few, > > > > > > > > > > though. > > > > > > > > > > > > > > > > > > > > > > > > > > > > If we don't rewrite them, then Perl > > > will > > > > > last until > > > > > > > > > > > > > API 2.0 > > > > > > > > > > has been > > > > > > > > > > > > > > designed, released and then > > *another > > > > > full major > > > > > > > > > > > > > release cycle*. > > > > > > > > > > > > > That's way > > > > > > > > > > > > > > too long to have two codebases for > > > the > > > > > same > > > > > > > > > > > > > component, IMO, > > > > > > > > > > > > > especially > > > > > > > > > > > > > > since the rewrite is already 50% > > > > > complete. > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Nov 12, 2019 at 8:43 AM > > > Hoppal, > > > > > Michael < > > > > > > > > > > > > > > [email protected]> > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > As the Traffic Ops API is being > > > > > rewritten from Perl > > > > > > > > > > > > > to Golang > > > > > > > > > > > > > there has > > > > > > > > > > > > > > > been several routes that have > > been > > > > > deprecated and > > > > > > > > > > > > > probably > > > > > > > > > > more to > > > > > > > > > > > > > come. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > In the deprecation efforts I have > > > seen > > > > > two > > > > > > > > > > > > > strategies: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > * The route IS NOT rewritten > > > from > > > > > Perl to > > > > > > > > > > > > > Golang and a > > > > > > > > > > > > > deprecation > > > > > > > > > > > > > > > notice is added to the alert > > > response > > > > > in the Perl > > > > > > > > > > > > > handler > > > > > > > > > > > > > > > * The route IS rewritten from > > > Perl > > > > > to Golang > > > > > > > > > > > > > and a > > > > > > > > > > deprecation > > > > > > > > > > > > > notice > > > > > > > > > > > > > > > is added to the alert response in > > > the > > > > > Golang > > > > > > > > > > > > > handler > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I think we should have > > consistency > > > in > > > > > our approach > > > > > > > > > > > > > and > > > > > > > > > > wanted to > > > > > > > > > > > > > get > > > > > > > > > > > > > > > people’s thoughts. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I would vote that we do not > > > rewrite a > > > > > deprecated > > > > > > > > > > > > > route from > > > > > > > > > > Perl to > > > > > > > > > > > > > > Golang. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Michael > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
