Thanks to all of those that attended and contributed. Below are my raw notes.
I plan to summarize these further in the wiki and kick off new email threads
for key topics that remain open. I would love to get some presentations
organized for the Atlanta meeting on these topics.
I am also working on how to host future virtual international meetings with
25-50 heads. I would love any feedback or ideas people have on what works best.
Ryan
Discussion Notes
1. Configuration of Multiple Caching Engines – how do we setup for this long
term?
* In agreement, moving towards multiple caching proxies
* ATS & NGINX are both definitely on the table, any other specific
caches?
i. None
specifically mentioned but there may be others like Varnish
1. Configuration Format & Storage – database tables vs. config files/json
* Moving away from database as storage for configs? No.
* LUA Engine – discussed json blob, but talked out of this. Use a
structured database for this.
2. State Machine
* How do we push/remove configs in correct order without breaking the
CDN?
i. It’s
possible if we cut time to seconds to roll out changes that temporary conflicts
are not that great of a concern, but this is definitely based on the fact that
the config completes without issue and does so very rapidly.
* Push out configs without snapshot or queue, with specific order, in a
way that does not affect others
* Discussing from traffic ops standpoint. Also affects Traffic Router
and live traffic on the CDN. Must consider the other components and order of
operation. Must make updates in parallel. Need to architect in parallel.
* Breaking crconfig into separate components is another element.
* State machine will be separate from ORT, lean towards implementation
inTraffic Ops
i. Must be
used by Traffic Portal
ii. Should be
integrated with TO? Decision for down the road.
iii. Make it
more generic on TO, to generate configs. All the other components that use
crconfig. Won’t use LUA for those. We would need a translation layer. Use a
templated approach on TO side, then would be modified as needed.
iv. Next Gen
ORT client is taking shape.
v. Need to
separate DS, Traffic Router, component configs. DS configs must be autonomous.
Will need a translation layer.
vi. Translation
layer for each component, rather than have TO put out different formats, it
puts out one format and each component must be able to interpret. This makes
configs generic, puts work on each component.
vii. Will this
limit what TO outputs? Pushes changes into the components.
1. LUA Rules Engine – creating greater options for customized service
* Rules engine would allow users to input generic rules which would be
interpreted behind the scenes and implemented via LUA plugin
i. Example
of a generic rule would be something like “on this trigger, perform this
action”. Something like on requests to origin, add this special header for
access.
* Many existing plugins could be implemented in LUA, it is much safer
and much less likely to crash server since LUA coroutines are non-blocking
i. Not all
plugins can be replaced, by LUA, but many can.
* Supporting LUA helps with cross caching engine support.
* LUA is not the same across ATS / NGINX, you would need to translate,
but it was written by same person.
* Extension of CDN without Code, benefits of not having to update
traffic ops
* How does this affect non-technical users? How can they still build
services?
i. The goal
is to make simple rules available to end users. They don’t have to configure a
script they build a simple rule.
ii. Restrict
raw scripting to CDN Dev/Ops, but simplify rules access for everyone
iii. How do we
prevent end users from having too much access and potentially manipulating
system?
1. Delivery Service Configuration Versioning (DSCV)
* Allows rapid deployment, testing, and roll back of configs
* Decouple changing config from deployment
* Allows users to change config, deployment decisions can be made
separately and can be regulated
* Changes can be compared
* Two types
i. Simple –
Single delivery service deployed version 5 to version 7 (replace in all caches)
ii. Multiple
Versions – Allow multiple versions to be deployed simultaneously
* Start with Simple for now, move to multiple versions in future
* Tell caches delivery service and version
* Concerns
i. If we
store versions in memory, that could cause issues in Traffic Monitor/Router
ii. Notion of
an instant roll back is difficult.
* DNSSEC zone signing is an issue
* State machine issues for ordering / coordinating of changes
iii. Example
change a property on a service, sometimes traffic monitor may need to know
first. Must take into account the coordination between components
* DSCV provides an ID to every configuration
1. Time Stamping Config Versions – How to track versioning
* By having a deployed version table with history, you don’t remove
lines, you just mark it as removed. You can then see the version history by
date in the table.
* You can say the ID is the creation date, but may not simplify things.
This is a trade off requiring further discussion.
* Sounds like we would like to have an ID field to keep things simple
(ie deploy config 7 vs. config 5 instead of deploy config 08/31/2017 18:05:36
vs. 08/31/2017 18:15:36)
* We can have both an ID and a timestamp and either could be used to
stipulate a particular version
2. ORT – What is the future?
* Rewrite in new golang traffic API?
* Derek working on this, in Go using TO client
* ORT doesn’t care what it talks to
* Does ORT work as we want it to? Not yet. Change from script to a
service with active communication
* Need to start a thread on this topic
* Not the highest priority yet
3. Self Service
* Tenancy – restrictions by tenant, by Nir, about to go into production
* Roles & Capabilities – need to have ops role to create a service.
Naama worked on this. Need to enforce.
* These are the first two steps for self-service ^^^
4. Generic Configuration Model
* We can make the configuration parameters for a CDN service all generic
(ie not specific to a caching engine)
* Each caching engine implemented would need a layer written to read the
generic config and translate to appropriate settings for the cache updates.
* This forces complexity where it belongs on the cache owner to
interpret and translate instead of on the TC developers who would need to know
how to configure all types of caches.
* This also fits with generic instructions in Rules Engine where each
cache would need an interpreter for generic rules
* Vendor Specific Configuration
i. Nginx
caches know how to deal with certain configs. May need new parameters.
ii. Delivery
service profile needs to be taken into consideration.
iii. Profiles
could go back to being a server profile only
iv. Delivery
service profiles lack structure currently?
Wiki: https://cwiki.apache.org/confluence/display/TC/Configuration+Management
Ryan Durfey
Sr. Product Manager - CDN | Comcast Technology Solutions
1899 Wynkoop Ste. 550 | Denver, CO 80202
M | 303-524-5099
[email protected]<mailto:[email protected]>
CDN Support (24x7): 866-405-2993 or
[email protected]<mailto:[email protected]>
