Thanks Derek for the clarification. So the definition file is a global file for the CDN. Does it contain the information of which server has which DS? Does it hold all CDN's DSs configuration together? On a single DS change, will all servers in the CDN download the entire file for every DS change?
What I'm practically asking is, if it is not already your intention, "can the definition file hold only the information of which server holds which DS (and configuration version when we add it), and the DS configuration be held and pulled separately on a DS level granularity?" When discussing "self-service" we would like to decouple the operations of the different users / content providers. Ultimately, when a DS is changed, the change should be deployed immediately to the CDN - with no dependency with other DSs, and possibly with "no buffering" by the operator deploying batch of DS changes together. This allows to improve the user experience and independence when working on the CDN. Following the change you are suggesting, will the DS configuration deployment coupling get tighter? We prefer not to have the need to "finish your work and not start additional work before the queued run has completed". Another requirement is to be able to rollback changes on a DS level and not only on a CDN level, as it is not desired to rollback changes of user "A" because of errors of user "B". If I understand correctly, the definition file does not support that. Last, using the definition file, must all servers in a CDN work with the same set of DSs? One of the reasons we consider "DS versioning" is to allow deployment of a change in a DS only for a subset of the server for canary testing. Thanks, Nir On Wed, Apr 12, 2017 at 3:00 AM, Dewayne Richardson <[email protected]> wrote: > +1 I was just about to formulate that response. The "dev" list is our > discussion forum. > > On Tue, Apr 11, 2017 at 9:35 AM, Dave Neuman <[email protected]> wrote: > > > @Ryan, I think its better to have conversations on the dev list than a > wiki > > page... > > > > On Tue, Apr 11, 2017 at 9:01 AM, Durfey, Ryan <[email protected]> > > wrote: > > > > > Started a new wiki page to discuss this here https://cwiki.apache.org/ > > > confluence/display/TC/Configuration+Management > > > > > > I will do my best to summarize the discussion below later today. > > > > > > Ryan M | 303-524-5099 > > > > > > > > > -----Original Message----- > > > From: Eric Friedrich (efriedri) [mailto:[email protected]] > > > Sent: Tuesday, April 11, 2017 8:55 AM > > > To: [email protected] > > > Subject: Re: Proposal for CDN definition file based configuration > > > management > > > > > > A few questions/thoughts, apologies for not in-lining: > > > > > > 1) If we move away from individually queued updates, we give up the > > > ability to make changes and then selectively deploy them. How often do > TC > > > operations teams make config changes but do not immediately queue > > updates. > > > (I personally think that we currently have a bit of a tricky situation > > > where queuing updates much later can push down an unknowingly large > > config > > > change to a cache- i.e. many new DS added/removed since last time > updates > > > were queued maybe months earlier). I wouldn't be sad to see queue > updates > > > go away, but don't want to cause hardship on operators using that > > feature. > > > > > > 2) If we move away from individually queued updates, how does that > affect > > > the implicit "config state machine"? Specifically, how will edges know > > when > > > their parents have been configured and are ready for service? Today we > > > don't config an edge cache with a new DS unless the mid is ready to > > handle > > > traffic as well. > > > > > > 3) If we move away from individually queued updates, how do we do > things > > > like unassign a delivery service from a cache? Today we have to > snapshot > > > CRConfig first to stop redirects to the cache before we queue the > update. > > > If updates are immediately applied and snapshot is still separate, how > do > > > we get TR to stop sending traffic to a cache that no longer has the > remap > > > rule? > > > > > > 4) Also along the lines of the config state machine, we never really > > > closed on if we would make any changes to the queue update/snapshot > > > CRConfig flow. If we are looking at redoing how we generate config > files, > > > it would be great to have consensus on an approach (if not an > > > implementation) to remove the need to sequence queue updates and > snapshot > > > CRConfig. I think the requirement here would be to have Traffic Control > > > figure out on its own when to activate/deactivate routing to a cache > from > > > TR. > > > > > > 5) I like the suggestion of cache-based config file generation. > > > - Caches only retrieve relevant information, so scale proportional to > > > number of caches/DSs in the CDN is much better > > > - We could modify TR/TM to use the same approach, rather than > > > snapshotting a CRConfig. > > > - Cache/TR/TM-based config could play a greater role in config state > > > machine, rather than having Traffic Ops build static configuration > ahead > > of > > > time. > > > > > > Downsides > > > - Versioning is still possible, but more work than maintaining > > snapshots > > > of a config file > > > - Have to be very careful with API changes, any breakage now impacts > > > cache updates. > > > > > > -Eric > > > > > > > On Apr 10, 2017, at 9:45 PM, Gelinas, Derek < > [email protected] > > > > > > wrote: > > > > > > > > Thanks Rob. To your point about scalability: I think that this is > more > > > scaleable than the current crconfig implementation due to the caching. > > > However that is a very valid point and one that has been considered. > I've > > > started looking into the problem from that angle and hope to have some > > more > > > solid data soon. I still believe that this is ultimately more > scaleable > > > than current config implementation, even with the scope caching, but > the > > > proof will be in the data. > > > > > > > > Derek > > > > > > > >> On Apr 10, 2017, at 9:23 PM, Robert Butts <[email protected] > > > > > wrote: > > > >> > > > >> I'd propose: > > > >> * Instead of storing the JSON as blob, use > > > >> https://www.postgresql.org/doc s/9.2/static/datatype-json.html > > > >> * Instead of version-then-file request, use a "latest" endpoint with > > > >> `If-Modified-Since` > > > >> (https://tools.ietf.org/html/rfc7232#section-3.3). We can also > serve > > > >> each version at endpoints, but `If-Modified-Since` lets us determine > > > >> whether there's a new snapshot and get it in a single request, both > > > >> efficiently and using a standard. (We should do the same for the > > > CRConfig). > > > >> > > > >> Also for cache-side config generation, consider > > > >> https://github.com/apache/incubator-trafficcontrol/pull/151 . It's > a > > > >> prototype and needs work to bring it to production, but the basic > > > >> functionality is there. Go is safer and faster to develop than Perl, > > > >> and this is already encapsulated in a library, with both CLI and > HTTP > > > >> microservice examples. I'm certainly willing to help bring it to > > > production. > > > >> > > > >> > > > >> "a single definition file for each CDN which will contain all the > > > >> information required for any server within that CDN to generate its > > > >> own configs" > > > >> > > > >> Also, long-term, that doesn't scale, nor does the CRConfig. As > > > >> Traffic Control is deployed with larger and larger CDNs, the > CRConfig > > > >> grows uncontrollably. It's already 5-7mb for us, which takes an > > > >> approaching-unreasonable amount of time for Traffic Monitor and > > > >> Router to fetch. This isn't an immediate concern, but long-term, we > > > >> need to develop a scalable solution, something that says "only give > > > >> me the data modified since this timestamp". > > > >> > > > >> Again, this isn't an immediate crisis. I only mention it now > because, > > > >> if a scalable solution is about the same amount of work, now sounds > > > >> like a good time. If it's relevantly more work, no worries. > > > >> > > > >> > > > >> But otherwise, +1. We've long needed to Separate our Concerns of > > > >> Traffic Ops and the cache application. > > > >> > > > >> > > > >> On Mon, Apr 10, 2017 at 5:05 PM, Gelinas, Derek > > > >> <[email protected]> > > > >> wrote: > > > >> > > > >>> I would like to propose a new method for ATS config file > generation, > > > >>> in which a single definition file for each CDN which will contain > > > >>> all the information required for any server within that CDN to > > > >>> generate its own configs, rather than requesting them from traffic > > > >>> ops. This would be a version-controlled json file that, when > > > >>> generated, would be stored in a new table in the traffic ops > > > >>> database as a blob type. This will satisfy high-availability > > > >>> requirements and allow several versions of the configuration to be > > > >>> retained for rollback, as well as "freezing" the config at that > > > >>> moment in time. Combined with cache support coming in 2.1, this > file > > > would only need be generated once per traffic ops server instance. > > > >>> Instead of queueing servers to update their configurations, the > > > >>> configuration would be snapshotted similar to the crconfig file and > > > >>> downloaded by each cache according to their set interval checks - > > > >>> rather than performing a syncds and checking that the server has > > > >>> been queued for update, the version number would simply be checked > > > >>> and compared against the currently active version on the cache > > > >>> itself. Should a difference be found the server would request the > > > >>> definition file and begin generating configuration files for itself > > > using the data in the definition file. > > > >>> > > > >>> I would like feedback from the community regarding this proposal, > > > >>> and any suggestions or comments you may have. > > > >>> > > > >>> Thanks! > > > >>> Derek > > > >>> > > > >>> Derek Gelinas > > > >>> IPCDN Engineering > > > >>> [email protected]<mailto:Derek_Gelinas@ > cable.comcast.c > > > >>> om> > > > >>> 603.812.5379 > > > >>> > > > >>> > > > > > > > > > > > >
