Re: Delivery-Service Configuration Versioning

[Nir Sopher]

Thanks Ryan & Eric for the feedback.
Answers inline.
Thanks again,
Nir


On Thu, May 4, 2017 at 3:59 PM, Eric Friedrich (efriedri) <
efrie...@cisco.com> wrote:

> Thanks Nir-
> Comments inline
> > On May 1, 2017, at 1:12 PM, Nir Sopher  wrote:
> >
> > Dear all,
> >
> > Planning the efforts toward "self-service", we are considering
> > "delivery-service configuration versioning" (DSCV) as one of our next
> > steps.
> > In a very high level, by DSCV we refer to the ability to hold multiple
> > configuration versions/revisions per delivery-service, and actively
> choose
> > which version should be deployed.
> >
> > A significant portion of the value we would like to bring when working
> > toward "self-service" can be achieved using the initial step of
> > configuration versioning:
> >
> >   1. As the amount of delivery-services handled by TC is increasing,
> >   denying the "non dev-ops" user from changing delivery-services
> >   configuration by himself, and require a "dev-ops" user to actually
> make the
> >   changes in the DB, put an increasing load on the operations team.
> >   Via DSCV the operator may allow the users to really push configurations
> >   into the DB, as it separates the provisioning phase from the
> deployment.
> >   Once commited, the CDN's "dev-ops" user is able to examine the changes
> >   and choose which version should be deployed, subject to the operator's
> >   acceptance policy.
> EF> How do we get from DSCV to the ultimate self-service goals where the
> CDN operator is no longer in the critical path for deploying DS changes?
>
> NS> Indeed Eric, at this stage the deployment itself is still in the hands
of the operator: The operator has to change the deployed DS version, Queue
Update and Cr-Config snapshot.
Allowing the DS owner to "change the DS version to deploy" is a "process"
issue, as it should be subject to the different operators changes
acceptance policy. We will need to model it and add flexible, probably
plugins supporting, building block in the future.
Allowing the DS owner to actually deploy the changes (Cr-Config snapshot
and Queue-Update / future push mechanism) cannot be done as long as the
different DSs configuration is coupled in the sames files (remap.config &
cr-config) and processes.
Once decoupled, the deploy operations can be done on a DS granularity and
therefore the operator can delegate the control to the "DS owner"


> >   2. DSCV brings improved auditing and troubleshooting capabilities,
> which
> >   is important for supporting TC deployment growth, as well as allow
> users to
> >   be more independent.
> >   It allows to investigate issues using versions associated log records,
> >   as well as the data in the DB itself: Examining the delivery-service
> >   versions, their meta data (e.g. "deployed dates") as well as use tools
> for
> >   versions comparisons.
> >   3. DSCV allows a simple delivery service configuration rollback, which
> >   provides a quick remedy for configuration errors issues.
> >
> > Moreover, we suggest to allow the deployment of multiple versions of the
> > same delivery service simultaneously, on the same caches. Doing so, and
> > allowing the operator to orchestrate the usage of the different
> > versions (for example, via "steering"), the below become available:
> EF> This feature will extend to both caches and TR, right? Lots of
> DS-specific policy is evaluated by the TR.
>
NS> There are few ways to implement the feature, but as I currently see it
there is no real need to change the data-plane for supporting this feature.
Please let me know if you think I'm missing something here.
One option is to deploy the different versions of the same delivery service
as if they are entirely different delivery-services. With "ids" and
"host-regexes" which include the version. No changes in Cr-Config
structure, remap.config, etc. Therefore, all changes need to be done are in
traffic-ops, simulating the different version towards the rest of the
system.
Let now improve this solution.
First improvement is giving the different components the ability to
understand a the concept of "DS and version". For example adding the
version field to the cr-config and adjust the different component. For
example, this may allow traffic-stats to show the reports about different
DS version separately, as well as aggregated for the DS. More changes will
probably be required, but as far as I currently see, these changes are all
in the control plane, and not the data plane. Caches are effectively
unaware of the change, and Router just have to support the version as part
of the DS id.
And still this is not good enough, dealing with https for example, we
cannot really use "version based host regex" - it will require a new
certificate for each version. There are a few ways to solve this issue. I
personally believe that allowing different delivery-services to have the
same "host-regex" and be differentiated according to the "path" (as
suggested in TC-55 

Re: Delivery-Service Configuration Versioning

[Eric Friedrich (efriedri)]

Thanks Nir-
Comments inline
> On May 1, 2017, at 1:12 PM, Nir Sopher  wrote:
> 
> Dear all,
> 
> Planning the efforts toward "self-service", we are considering
> "delivery-service configuration versioning" (DSCV) as one of our next
> steps.
> In a very high level, by DSCV we refer to the ability to hold multiple
> configuration versions/revisions per delivery-service, and actively choose
> which version should be deployed.
> 
> A significant portion of the value we would like to bring when working
> toward "self-service" can be achieved using the initial step of
> configuration versioning:
> 
>   1. As the amount of delivery-services handled by TC is increasing,
>   denying the "non dev-ops" user from changing delivery-services
>   configuration by himself, and require a "dev-ops" user to actually make the
>   changes in the DB, put an increasing load on the operations team.
>   Via DSCV the operator may allow the users to really push configurations
>   into the DB, as it separates the provisioning phase from the deployment.
>   Once commited, the CDN's "dev-ops" user is able to examine the changes
>   and choose which version should be deployed, subject to the operator's
>   acceptance policy.
EF> How do we get from DSCV to the ultimate self-service goals where the CDN 
operator is no longer in the critical path for deploying DS changes?

>   2. DSCV brings improved auditing and troubleshooting capabilities, which
>   is important for supporting TC deployment growth, as well as allow users to
>   be more independent.
>   It allows to investigate issues using versions associated log records,
>   as well as the data in the DB itself: Examining the delivery-service
>   versions, their meta data (e.g. "deployed dates") as well as use tools for
>   versions comparisons.
>   3. DSCV allows a simple delivery service configuration rollback, which
>   provides a quick remedy for configuration errors issues.
> 
> Moreover, we suggest to allow the deployment of multiple versions of the
> same delivery service simultaneously, on the same caches. Doing so, and
> allowing the operator to orchestrate the usage of the different
> versions (for example, via "steering"), the below become available:
EF> This feature will extend to both caches and TR, right? Lots of DS-specific 
policy is evaluated by the TR.

> 
>   1. Manual testing of a new delivery-service configuration, via dedicated
>   URL or using request headers.
>   2. Staging / Canary testing of new versions, applying them only for a
>   specific content path, or filtering base on source IP.
>   3. Gradual transition between the different configuration versions.
>   4. Configuration versions A/B testing (assuming the reporting/stats also
>   becomes "version aware").
>   5. Immediate (no CRON wait, cr-config change only) delivery-service
>   version"switch", and specifically immediate rollback capabilities.
EF> Does #5 imply that it will be the TR choosing between the versions of DS’ 
deployed on the caches? How will this modify the format of requests to 
TrafficServer?
This will have impacts to log analysis, HTTPS, DNSSEC, and many other aspects 
of the system. 

> 
> Note that, engineering wise, one may consider DSCV as a building block for
> other "self-service" steps. It allows the system to identify what
> configuration is deployed on which server, as well as allows the servers to
> identify configuration changes with DS granularity. Therefore, it can help
> to decouple the individual delivery services deployment as well as reduce
> the load derived from the caches update process.
> We would greatly appreciate community input on the subject.
> 
> Many thanks,
> Nir

Re: Access Control - Limiting Roles / Capabilities Tenant Admins can Assign to Users

[Eric Friedrich (efriedri)]

Could we further differentiate the user creation capabilities to:
- Create CDN Admin user
- Create CDN Ops user
- Create CDN Viewer user
- Create Tenant Admin user
- Create Tenant Ops user
- Create Tenant Viewer user

Then only the CDN-Admin role would have the capability to create a cdn admin 
user. Would be good to see the capabilities assigned at a granularity below API 
endpoint in this case.

As for creation of new roles, I like #2 and #3. Users should not be able to 
level-up anyone’s capabilities beyond their own. Further, capabilities are 
enforced by code, so we should not allow creation of new capabilities by API

- - Eric



On May 3, 2017, at 9:44 AM, Durfey, Ryan 
mailto:ryan_dur...@comcast.com>> wrote:

Moving this active debate into the mailing list.
-Jeremy makes a good point.  We need a method for making restricting roles and 
capabilities for lower tier staff that can create new users.  Jeremy has 
suggested a point system or a hierarchy.  I think either of these would work if 
applied correctly.   I am open to any approach that works.

My thoughts:
1. We need to limit which users can build new roles from capabilities or new 
capabilities from APIs.  This could be limited to a master role like “CDN 
Admin”.  Otherwise other admins could circumvent the system by matching APIs to 
lower tier roles.
2. Another simple approach may be to only allow non-CDN Admins to assign roles 
to users which they have access.  Basically you can’t give anyone more rights 
than you have.
3. Perhaps with this approach we allow non-CDN Admins to build roles from 
existing capabilities to which they have access, but not create capabilities 
from APIs.  Then they can build new roles and assign any capabilities or roles 
to which they already have access.



From: Jeremy Mitchell

I like this model of a user has a role which has capabilities which map to API 
endpoints, however, there seems to be one flaw or at least one unaccounted for 
use case.
Let's look at the roles listed above:

  *   CDN-Admin
  *   CDN-Ops
  *   CDN-Viewer
  *   Tenant-Admin
  *   Tenant-Ops
  *   Tenant-Viewer

Jeremy is a CDN-Admin which has the user-create capability (among others) so he 
creates Bob, a Tenant-Admin. Being a Tenant-Admin, Bob also has user-create so 
he creates Sally and he can give her ANY role so he decides to give Sally the 
CDN-Admin rolewhoops, we don't want that...
Bob should be limited to creating users with role=Tenant-Admin (his role), 
Tenant-Ops or Tenant-Viewer...but how do we correlate one role with another? 
Currently, we have "privilege level" attached to a role. So I guess we could 
use that like so:

  *   CDN-Admin (100)
  *   CDN-Ops (50)
  *   CDN-Viewer (40)
  *   Tenant-Admin (30)
  *   Tenant-Ops (20)
  *   Tenant-Viewer (10)

Now, being a Tenant-Admin with the user-create capability, Bob can only create 
users where role.priv_level is 30 or below. I feel like this might be the 
easiest solution.
Thoughts?


...
Now, being a Tenant-Admin with the user-create capability, Bob can only create 
users where role.priv_level is 30 or below. I feel like this might be the 
easiest solution.
Or...you could make roles hierarchical the way that tenants are hierarchical
-CDN-Admin
--CDN-Ops
--CDN-Viewer
--Tenant-Admin
---Tenant-Ops
---Tenant-Viewer
And in this scenario, if you have the user-create capability you can create 
users with your role or a child of your role...
Thoughts?


Ryan Durfey
Sr. Product Manager - CDN | Comcast Technology Solutions
1899 Wynkoop Ste. 550 | Denver, CO 8020
M | 303-524-5099
ryan_dur...@comcast.com
24x7 CDN Support: 866-405-2993  or 
cdn_supp...@comcast.com