Re: Reducing toil in resource quota bumping (Andrew Feller)
There are a lot of possible ways to automate or improve this. Part of the reason we have hesitated to add more complexity was because of the divergence of use cases. It usually boils down to asking “who” or “why”. One option for empowering specific trusted individuals is to write a quick loop to identify when someone is given access to a project, or just do it via their membership in a group. I’ve seen a few people build things like this, but by design most of them are very simple and use case driven. > On Sep 4, 2018, at 12:37 PM, Walters, Todd wrote: > > This is a similar option we’d like to have a solution to. We have ‘tech > leads’ for projects that are in openshift role that can provision projects. > These are typically done via Jenkins job. It would be nice to be able to > apply custom project template for each project type so that smaller project > may have small project template with smaller quota and could have med and > large quotas. > > Currently we deploy the default project that has quota/limits set. If an lead > or team need additional resources they submit request to my team to adjust. > > Thanks, > Todd > > > Today's Topics: > > 1. Re: Reducing toil in resource quota bumping (Andrew Feller) > > >-- > >Message: 1 >Date: Tue, 4 Sep 2018 09:08:41 -0400 >From: Andrew Feller > To: users@lists.openshift.redhat.com >Subject: Re: Reducing toil in resource quota bumping >Message-ID: > >Content-Type: text/plain; charset="utf-8" > >Yeah, we have heterogenous trust levels between production and >non-production environments. This discussion was focused solely on >non-production environment where project default template is minimal >footprint and system administrators are in line for increasing project >resource quotes per developer per project. Since some projects require >more resources than others, we want to prevent non-administrators from >blowing up the cluster with wasteful resource usage but not so confined >they need someone holding their hands on a regular basis. > > > > > The information contained in this message, and any attachments thereto, > is intended solely for the use of the addressee(s) and may contain > confidential and/or privileged material. Any review, retransmission, > dissemination, copying, or other use of the transmitted information is > prohibited. If you received this in error, please contact the sender > and delete the material from any computer. UNIGROUP.COM > > ___ users mailing list users@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/users
Re: Reducing toil in resource quota bumping (Andrew Feller)
This is a similar option we’d like to have a solution to. We have ‘tech leads’ for projects that are in openshift role that can provision projects. These are typically done via Jenkins job. It would be nice to be able to apply custom project template for each project type so that smaller project may have small project template with smaller quota and could have med and large quotas. Currently we deploy the default project that has quota/limits set. If an lead or team need additional resources they submit request to my team to adjust. Thanks, Todd Today's Topics: 1. Re: Reducing toil in resource quota bumping (Andrew Feller) -- Message: 1 Date: Tue, 4 Sep 2018 09:08:41 -0400 From: Andrew Feller To: users@lists.openshift.redhat.com Subject: Re: Reducing toil in resource quota bumping Message-ID: Content-Type: text/plain; charset="utf-8" Yeah, we have heterogenous trust levels between production and non-production environments. This discussion was focused solely on non-production environment where project default template is minimal footprint and system administrators are in line for increasing project resource quotes per developer per project. Since some projects require more resources than others, we want to prevent non-administrators from blowing up the cluster with wasteful resource usage but not so confined they need someone holding their hands on a regular basis. The information contained in this message, and any attachments thereto, is intended solely for the use of the addressee(s) and may contain confidential and/or privileged material. Any review, retransmission, dissemination, copying, or other use of the transmitted information is prohibited. If you received this in error, please contact the sender and delete the material from any computer. UNIGROUP.COM ___ users mailing list users@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/users
Re: Reducing toil in resource quota bumping
Yeah, we have heterogenous trust levels between production and non-production environments. This discussion was focused solely on non-production environment where project default template is minimal footprint and system administrators are in line for increasing project resource quotes per developer per project. Since some projects require more resources than others, we want to prevent non-administrators from blowing up the cluster with wasteful resource usage but not so confined they need someone holding their hands on a regular basis. On Tue, Sep 4, 2018 at 8:54 AM Clayton Coleman wrote: > > > On Sep 4, 2018, at 8:30 AM, Andrew Feller wrote: > > While that is a fair assessment of the situations where this pain point > arises, there should be a better option for facilitating #3 without > allowing #1 or #2 as this is a common problem disrupting both sides of the > situation. > > Knee jerk thought would be to modify the ProjectRequest API to allow > requestor to specify anything the ClusterResourceQuota object can control. > This would allow new projects being created to sized right until some > situation to support resizing existing projects can be developed. > > > If you trust your developers to resize, create a role that you bind to the > namespace / account when you trust them. Or just add it to edit and they’d > have it by default. > > The complicated scenarios are usually when your trust domains are > heterogeneous - it didn’t sound like that was your case. > > > On Thu, Aug 30, 2018 at 2:46 PM Clayton Coleman > wrote: > >> Ultimately you need to ask what you are trying to prevent: >> >> 1. a user from accidentally blowing up the cluster >> 2. malicious users >> 3. an application breaking at runtime because it needs more resources >> than it is allotted >> >> The second one is more what we've been discussing here - being draconian >> up front. 1 is usually where you'd have two quotas - initial, and >> generous, and then just swap them out as needed, possibly via some >> automation. 3 is what most people are most afraid of (failing to meet your >> SLA because you didn't allocate resources). >> >> >> >> >> >> On Thu, Aug 30, 2018 at 2:17 PM Andrew Feller >> wrote: >> >>> Thanks for the feedback Jessica! >>> >>> Limiting # of projects users can create is definitely one of the things >>> expected, however the question was mostly focused on reducing toil due to >>> changing resource quotas for projects. The idea with option #1 was >>> restricting devs to 1 project with heftier resources allocated whereas the >>> hope with option #2 was ClusterResourceQuota per developer might relax >>> other options for developers to modify project resource quotas without >>> waiting on system administrators. >>> >>> On Thu, Aug 30, 2018 at 10:14 AM Jessica Forrester >>> wrote: >>> On Thu, Aug 30, 2018 at 8:18 AM Andrew Feller wrote: > Has anyone found an effective way to minimize toil between developers > and system administrators regarding project resource quotas *without > resorting to letting people do whatever they want unrestrained*? > > There are only 2 ideas I can see to address this issue: > >1. Removing self-provisioning access, provisioning a single >project per developer, and giving them admin access to it. > > You can limit the number of self-provisioned projects they can have https://docs.openshift.com/container-platform/3.10/admin_guide/managing_projects.html#limit-projects-per-user > >1. Create ClusterResourceQuota per developer restricting total >resources allowed > > I don't know how ClusterResourceQuota handle a system administrator > increasing a project's quotas for a user who is already met their total. > If either a cluster resource quota or a resource quota has been exceeded, then you you've exceeded quota for that resource and can't make more. > > Any feedback is welcomed and appreciated! > -- > > [image: BandwidthMaroon.png] > > Andy Feller • Sr DevOps Engineer > > 900 Main Campus Drive, Suite 500, Raleigh, NC 27606 > > > e: afel...@bandwidth.com > ___ > users mailing list > users@lists.openshift.redhat.com > http://lists.openshift.redhat.com/openshiftmm/listinfo/users > >>> >>> -- >>> >>> [image: BandwidthMaroon.png] >>> >>> Andy Feller • Sr DevOps Engineer >>> >>> 900 Main Campus Drive, Suite 500, Raleigh, NC 27606 >>> >>> >>> e: afel...@bandwidth.com >>> ___ >>> users mailing list >>> users@lists.openshift.redhat.com >>> http://lists.openshift.redhat.com/openshiftmm/listinfo/users >>> >> > > -- > > [image: BandwidthMaroon.png] > > Andy Feller • Sr DevOps Engineer > > 900 Main Campus Drive, Suite 500, Raleigh, NC 27606 > > > e: afel...@bandwidth.com > >
Re: Reducing toil in resource quota bumping
On Sep 4, 2018, at 8:30 AM, Andrew Feller wrote: While that is a fair assessment of the situations where this pain point arises, there should be a better option for facilitating #3 without allowing #1 or #2 as this is a common problem disrupting both sides of the situation. Knee jerk thought would be to modify the ProjectRequest API to allow requestor to specify anything the ClusterResourceQuota object can control. This would allow new projects being created to sized right until some situation to support resizing existing projects can be developed. If you trust your developers to resize, create a role that you bind to the namespace / account when you trust them. Or just add it to edit and they’d have it by default. The complicated scenarios are usually when your trust domains are heterogeneous - it didn’t sound like that was your case. On Thu, Aug 30, 2018 at 2:46 PM Clayton Coleman wrote: > Ultimately you need to ask what you are trying to prevent: > > 1. a user from accidentally blowing up the cluster > 2. malicious users > 3. an application breaking at runtime because it needs more resources than > it is allotted > > The second one is more what we've been discussing here - being draconian > up front. 1 is usually where you'd have two quotas - initial, and > generous, and then just swap them out as needed, possibly via some > automation. 3 is what most people are most afraid of (failing to meet your > SLA because you didn't allocate resources). > > > > > > On Thu, Aug 30, 2018 at 2:17 PM Andrew Feller > wrote: > >> Thanks for the feedback Jessica! >> >> Limiting # of projects users can create is definitely one of the things >> expected, however the question was mostly focused on reducing toil due to >> changing resource quotas for projects. The idea with option #1 was >> restricting devs to 1 project with heftier resources allocated whereas the >> hope with option #2 was ClusterResourceQuota per developer might relax >> other options for developers to modify project resource quotas without >> waiting on system administrators. >> >> On Thu, Aug 30, 2018 at 10:14 AM Jessica Forrester >> wrote: >> >>> >>> >>> On Thu, Aug 30, 2018 at 8:18 AM Andrew Feller >>> wrote: >>> Has anyone found an effective way to minimize toil between developers and system administrators regarding project resource quotas *without resorting to letting people do whatever they want unrestrained*? There are only 2 ideas I can see to address this issue: 1. Removing self-provisioning access, provisioning a single project per developer, and giving them admin access to it. >>> You can limit the number of self-provisioned projects they can have >>> >>> https://docs.openshift.com/container-platform/3.10/admin_guide/managing_projects.html#limit-projects-per-user >>> >>> 1. Create ClusterResourceQuota per developer restricting total resources allowed I don't know how ClusterResourceQuota handle a system administrator increasing a project's quotas for a user who is already met their total. >>> >>> If either a cluster resource quota or a resource quota has been >>> exceeded, then you you've exceeded quota for that resource and can't make >>> more. >>> >>> Any feedback is welcomed and appreciated! -- [image: BandwidthMaroon.png] Andy Feller • Sr DevOps Engineer 900 Main Campus Drive, Suite 500, Raleigh, NC 27606 e: afel...@bandwidth.com ___ users mailing list users@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/users >>> >> >> -- >> >> [image: BandwidthMaroon.png] >> >> Andy Feller • Sr DevOps Engineer >> >> 900 Main Campus Drive, Suite 500, Raleigh, NC 27606 >> >> >> e: afel...@bandwidth.com >> ___ >> users mailing list >> users@lists.openshift.redhat.com >> http://lists.openshift.redhat.com/openshiftmm/listinfo/users >> > -- [image: BandwidthMaroon.png] Andy Feller • Sr DevOps Engineer 900 Main Campus Drive, Suite 500, Raleigh, NC 27606 e: afel...@bandwidth.com ___ users mailing list users@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/users ___ users mailing list users@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/users
Re: Reducing toil in resource quota bumping
While that is a fair assessment of the situations where this pain point arises, there should be a better option for facilitating #3 without allowing #1 or #2 as this is a common problem disrupting both sides of the situation. Knee jerk thought would be to modify the ProjectRequest API to allow requestor to specify anything the ClusterResourceQuota object can control. This would allow new projects being created to sized right until some situation to support resizing existing projects can be developed. On Thu, Aug 30, 2018 at 2:46 PM Clayton Coleman wrote: > Ultimately you need to ask what you are trying to prevent: > > 1. a user from accidentally blowing up the cluster > 2. malicious users > 3. an application breaking at runtime because it needs more resources than > it is allotted > > The second one is more what we've been discussing here - being draconian > up front. 1 is usually where you'd have two quotas - initial, and > generous, and then just swap them out as needed, possibly via some > automation. 3 is what most people are most afraid of (failing to meet your > SLA because you didn't allocate resources). > > > > > > On Thu, Aug 30, 2018 at 2:17 PM Andrew Feller > wrote: > >> Thanks for the feedback Jessica! >> >> Limiting # of projects users can create is definitely one of the things >> expected, however the question was mostly focused on reducing toil due to >> changing resource quotas for projects. The idea with option #1 was >> restricting devs to 1 project with heftier resources allocated whereas the >> hope with option #2 was ClusterResourceQuota per developer might relax >> other options for developers to modify project resource quotas without >> waiting on system administrators. >> >> On Thu, Aug 30, 2018 at 10:14 AM Jessica Forrester >> wrote: >> >>> >>> >>> On Thu, Aug 30, 2018 at 8:18 AM Andrew Feller >>> wrote: >>> Has anyone found an effective way to minimize toil between developers and system administrators regarding project resource quotas *without resorting to letting people do whatever they want unrestrained*? There are only 2 ideas I can see to address this issue: 1. Removing self-provisioning access, provisioning a single project per developer, and giving them admin access to it. >>> You can limit the number of self-provisioned projects they can have >>> >>> https://docs.openshift.com/container-platform/3.10/admin_guide/managing_projects.html#limit-projects-per-user >>> >>> 1. Create ClusterResourceQuota per developer restricting total resources allowed I don't know how ClusterResourceQuota handle a system administrator increasing a project's quotas for a user who is already met their total. >>> >>> If either a cluster resource quota or a resource quota has been >>> exceeded, then you you've exceeded quota for that resource and can't make >>> more. >>> >>> Any feedback is welcomed and appreciated! -- [image: BandwidthMaroon.png] Andy Feller • Sr DevOps Engineer 900 Main Campus Drive, Suite 500, Raleigh, NC 27606 e: afel...@bandwidth.com ___ users mailing list users@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/users >>> >> >> -- >> >> [image: BandwidthMaroon.png] >> >> Andy Feller • Sr DevOps Engineer >> >> 900 Main Campus Drive, Suite 500, Raleigh, NC 27606 >> >> >> e: afel...@bandwidth.com >> ___ >> users mailing list >> users@lists.openshift.redhat.com >> http://lists.openshift.redhat.com/openshiftmm/listinfo/users >> > -- [image: BandwidthMaroon.png] Andy Feller • Sr DevOps Engineer 900 Main Campus Drive, Suite 500, Raleigh, NC 27606 e: afel...@bandwidth.com ___ users mailing list users@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/users
Re: Reducing toil in resource quota bumping
Ultimately you need to ask what you are trying to prevent: 1. a user from accidentally blowing up the cluster 2. malicious users 3. an application breaking at runtime because it needs more resources than it is allotted The second one is more what we've been discussing here - being draconian up front. 1 is usually where you'd have two quotas - initial, and generous, and then just swap them out as needed, possibly via some automation. 3 is what most people are most afraid of (failing to meet your SLA because you didn't allocate resources). On Thu, Aug 30, 2018 at 2:17 PM Andrew Feller wrote: > Thanks for the feedback Jessica! > > Limiting # of projects users can create is definitely one of the things > expected, however the question was mostly focused on reducing toil due to > changing resource quotas for projects. The idea with option #1 was > restricting devs to 1 project with heftier resources allocated whereas the > hope with option #2 was ClusterResourceQuota per developer might relax > other options for developers to modify project resource quotas without > waiting on system administrators. > > On Thu, Aug 30, 2018 at 10:14 AM Jessica Forrester > wrote: > >> >> >> On Thu, Aug 30, 2018 at 8:18 AM Andrew Feller >> wrote: >> >>> Has anyone found an effective way to minimize toil between developers >>> and system administrators regarding project resource quotas *without >>> resorting to letting people do whatever they want unrestrained*? >>> >>> There are only 2 ideas I can see to address this issue: >>> >>>1. Removing self-provisioning access, provisioning a single project >>>per developer, and giving them admin access to it. >>> >>> >> You can limit the number of self-provisioned projects they can have >> >> https://docs.openshift.com/container-platform/3.10/admin_guide/managing_projects.html#limit-projects-per-user >> >> >>> >>>1. Create ClusterResourceQuota per developer restricting total >>>resources allowed >>> >>> I don't know how ClusterResourceQuota handle a system administrator >>> increasing a project's quotas for a user who is already met their total. >>> >> >> If either a cluster resource quota or a resource quota has been exceeded, >> then you you've exceeded quota for that resource and can't make more. >> >> >>> >>> Any feedback is welcomed and appreciated! >>> -- >>> >>> [image: BandwidthMaroon.png] >>> >>> Andy Feller • Sr DevOps Engineer >>> >>> 900 Main Campus Drive, Suite 500, Raleigh, NC 27606 >>> >>> >>> e: afel...@bandwidth.com >>> ___ >>> users mailing list >>> users@lists.openshift.redhat.com >>> http://lists.openshift.redhat.com/openshiftmm/listinfo/users >>> >> > > -- > > [image: BandwidthMaroon.png] > > Andy Feller • Sr DevOps Engineer > > 900 Main Campus Drive, Suite 500, Raleigh, NC 27606 > > > e: afel...@bandwidth.com > ___ > users mailing list > users@lists.openshift.redhat.com > http://lists.openshift.redhat.com/openshiftmm/listinfo/users > ___ users mailing list users@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/users
Re: Reducing toil in resource quota bumping
Thanks for the feedback Jessica! Limiting # of projects users can create is definitely one of the things expected, however the question was mostly focused on reducing toil due to changing resource quotas for projects. The idea with option #1 was restricting devs to 1 project with heftier resources allocated whereas the hope with option #2 was ClusterResourceQuota per developer might relax other options for developers to modify project resource quotas without waiting on system administrators. On Thu, Aug 30, 2018 at 10:14 AM Jessica Forrester wrote: > > > On Thu, Aug 30, 2018 at 8:18 AM Andrew Feller > wrote: > >> Has anyone found an effective way to minimize toil between developers and >> system administrators regarding project resource quotas *without >> resorting to letting people do whatever they want unrestrained*? >> >> There are only 2 ideas I can see to address this issue: >> >>1. Removing self-provisioning access, provisioning a single project >>per developer, and giving them admin access to it. >> >> > You can limit the number of self-provisioned projects they can have > > https://docs.openshift.com/container-platform/3.10/admin_guide/managing_projects.html#limit-projects-per-user > > >> >>1. Create ClusterResourceQuota per developer restricting total >>resources allowed >> >> I don't know how ClusterResourceQuota handle a system administrator >> increasing a project's quotas for a user who is already met their total. >> > > If either a cluster resource quota or a resource quota has been exceeded, > then you you've exceeded quota for that resource and can't make more. > > >> >> Any feedback is welcomed and appreciated! >> -- >> >> [image: BandwidthMaroon.png] >> >> Andy Feller • Sr DevOps Engineer >> >> 900 Main Campus Drive, Suite 500, Raleigh, NC 27606 >> >> >> e: afel...@bandwidth.com >> ___ >> users mailing list >> users@lists.openshift.redhat.com >> http://lists.openshift.redhat.com/openshiftmm/listinfo/users >> > -- [image: BandwidthMaroon.png] Andy Feller • Sr DevOps Engineer 900 Main Campus Drive, Suite 500, Raleigh, NC 27606 e: afel...@bandwidth.com ___ users mailing list users@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/users
Re: Reducing toil in resource quota bumping
On Thu, Aug 30, 2018 at 8:18 AM Andrew Feller wrote: > Has anyone found an effective way to minimize toil between developers and > system administrators regarding project resource quotas *without > resorting to letting people do whatever they want unrestrained*? > > There are only 2 ideas I can see to address this issue: > >1. Removing self-provisioning access, provisioning a single project >per developer, and giving them admin access to it. > > You can limit the number of self-provisioned projects they can have https://docs.openshift.com/container-platform/3.10/admin_guide/managing_projects.html#limit-projects-per-user > >1. Create ClusterResourceQuota per developer restricting total >resources allowed > > I don't know how ClusterResourceQuota handle a system administrator > increasing a project's quotas for a user who is already met their total. > If either a cluster resource quota or a resource quota has been exceeded, then you you've exceeded quota for that resource and can't make more. > > Any feedback is welcomed and appreciated! > -- > > [image: BandwidthMaroon.png] > > Andy Feller • Sr DevOps Engineer > > 900 Main Campus Drive, Suite 500, Raleigh, NC 27606 > > > e: afel...@bandwidth.com > ___ > users mailing list > users@lists.openshift.redhat.com > http://lists.openshift.redhat.com/openshiftmm/listinfo/users > ___ users mailing list users@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/users
Reducing toil in resource quota bumping
Has anyone found an effective way to minimize toil between developers and system administrators regarding project resource quotas *without resorting to letting people do whatever they want unrestrained*? There are only 2 ideas I can see to address this issue: 1. Removing self-provisioning access, provisioning a single project per developer, and giving them admin access to it. 2. Create ClusterResourceQuota per developer restricting total resources allowed I don't know how ClusterResourceQuota handle a system administrator increasing a project's quotas for a user who is already met their total. Any feedback is welcomed and appreciated! -- [image: BandwidthMaroon.png] Andy Feller • Sr DevOps Engineer 900 Main Campus Drive, Suite 500, Raleigh, NC 27606 e: afel...@bandwidth.com ___ users mailing list users@lists.openshift.redhat.com http://lists.openshift.redhat.com/openshiftmm/listinfo/users
