@Ryan, I think its better to have conversations on the dev list than a wiki page...
On Tue, Apr 11, 2017 at 9:01 AM, Durfey, Ryan <[email protected]> wrote: > Started a new wiki page to discuss this here https://cwiki.apache.org/ > confluence/display/TC/Configuration+Management > > I will do my best to summarize the discussion below later today. > > Ryan M | 303-524-5099 > > > -----Original Message----- > From: Eric Friedrich (efriedri) [mailto:[email protected]] > Sent: Tuesday, April 11, 2017 8:55 AM > To: [email protected] > Subject: Re: Proposal for CDN definition file based configuration > management > > A few questions/thoughts, apologies for not in-lining: > > 1) If we move away from individually queued updates, we give up the > ability to make changes and then selectively deploy them. How often do TC > operations teams make config changes but do not immediately queue updates. > (I personally think that we currently have a bit of a tricky situation > where queuing updates much later can push down an unknowingly large config > change to a cache- i.e. many new DS added/removed since last time updates > were queued maybe months earlier). I wouldn't be sad to see queue updates > go away, but don't want to cause hardship on operators using that feature. > > 2) If we move away from individually queued updates, how does that affect > the implicit "config state machine"? Specifically, how will edges know when > their parents have been configured and are ready for service? Today we > don't config an edge cache with a new DS unless the mid is ready to handle > traffic as well. > > 3) If we move away from individually queued updates, how do we do things > like unassign a delivery service from a cache? Today we have to snapshot > CRConfig first to stop redirects to the cache before we queue the update. > If updates are immediately applied and snapshot is still separate, how do > we get TR to stop sending traffic to a cache that no longer has the remap > rule? > > 4) Also along the lines of the config state machine, we never really > closed on if we would make any changes to the queue update/snapshot > CRConfig flow. If we are looking at redoing how we generate config files, > it would be great to have consensus on an approach (if not an > implementation) to remove the need to sequence queue updates and snapshot > CRConfig. I think the requirement here would be to have Traffic Control > figure out on its own when to activate/deactivate routing to a cache from > TR. > > 5) I like the suggestion of cache-based config file generation. > - Caches only retrieve relevant information, so scale proportional to > number of caches/DSs in the CDN is much better > - We could modify TR/TM to use the same approach, rather than > snapshotting a CRConfig. > - Cache/TR/TM-based config could play a greater role in config state > machine, rather than having Traffic Ops build static configuration ahead of > time. > > Downsides > - Versioning is still possible, but more work than maintaining snapshots > of a config file > - Have to be very careful with API changes, any breakage now impacts > cache updates. > > -Eric > > > On Apr 10, 2017, at 9:45 PM, Gelinas, Derek <[email protected]> > wrote: > > > > Thanks Rob. To your point about scalability: I think that this is more > scaleable than the current crconfig implementation due to the caching. > However that is a very valid point and one that has been considered. I've > started looking into the problem from that angle and hope to have some more > solid data soon. I still believe that this is ultimately more scaleable > than current config implementation, even with the scope caching, but the > proof will be in the data. > > > > Derek > > > >> On Apr 10, 2017, at 9:23 PM, Robert Butts <[email protected]> > wrote: > >> > >> I'd propose: > >> * Instead of storing the JSON as blob, use > >> https://www.postgresql.org/doc s/9.2/static/datatype-json.html > >> * Instead of version-then-file request, use a "latest" endpoint with > >> `If-Modified-Since` > >> (https://tools.ietf.org/html/rfc7232#section-3.3). We can also serve > >> each version at endpoints, but `If-Modified-Since` lets us determine > >> whether there's a new snapshot and get it in a single request, both > >> efficiently and using a standard. (We should do the same for the > CRConfig). > >> > >> Also for cache-side config generation, consider > >> https://github.com/apache/incubator-trafficcontrol/pull/151 . It's a > >> prototype and needs work to bring it to production, but the basic > >> functionality is there. Go is safer and faster to develop than Perl, > >> and this is already encapsulated in a library, with both CLI and HTTP > >> microservice examples. I'm certainly willing to help bring it to > production. > >> > >> > >> "a single definition file for each CDN which will contain all the > >> information required for any server within that CDN to generate its > >> own configs" > >> > >> Also, long-term, that doesn't scale, nor does the CRConfig. As > >> Traffic Control is deployed with larger and larger CDNs, the CRConfig > >> grows uncontrollably. It's already 5-7mb for us, which takes an > >> approaching-unreasonable amount of time for Traffic Monitor and > >> Router to fetch. This isn't an immediate concern, but long-term, we > >> need to develop a scalable solution, something that says "only give > >> me the data modified since this timestamp". > >> > >> Again, this isn't an immediate crisis. I only mention it now because, > >> if a scalable solution is about the same amount of work, now sounds > >> like a good time. If it's relevantly more work, no worries. > >> > >> > >> But otherwise, +1. We've long needed to Separate our Concerns of > >> Traffic Ops and the cache application. > >> > >> > >> On Mon, Apr 10, 2017 at 5:05 PM, Gelinas, Derek > >> <[email protected]> > >> wrote: > >> > >>> I would like to propose a new method for ATS config file generation, > >>> in which a single definition file for each CDN which will contain > >>> all the information required for any server within that CDN to > >>> generate its own configs, rather than requesting them from traffic > >>> ops. This would be a version-controlled json file that, when > >>> generated, would be stored in a new table in the traffic ops > >>> database as a blob type. This will satisfy high-availability > >>> requirements and allow several versions of the configuration to be > >>> retained for rollback, as well as "freezing" the config at that > >>> moment in time. Combined with cache support coming in 2.1, this file > would only need be generated once per traffic ops server instance. > >>> Instead of queueing servers to update their configurations, the > >>> configuration would be snapshotted similar to the crconfig file and > >>> downloaded by each cache according to their set interval checks - > >>> rather than performing a syncds and checking that the server has > >>> been queued for update, the version number would simply be checked > >>> and compared against the currently active version on the cache > >>> itself. Should a difference be found the server would request the > >>> definition file and begin generating configuration files for itself > using the data in the definition file. > >>> > >>> I would like feedback from the community regarding this proposal, > >>> and any suggestions or comments you may have. > >>> > >>> Thanks! > >>> Derek > >>> > >>> Derek Gelinas > >>> IPCDN Engineering > >>> [email protected]<mailto:[email protected] > >>> om> > >>> 603.812.5379 > >>> > >>> > > >
