I agree with Pedros points. We've got a test-system running at http://jenkins.mxnet-ci-dev.amazon-ml.com and all changes should be tested there before they are deployed to prod. For now, we can move forward as-is and I would kindly ask the committers to check back with me if any changes are required. Please don't forget that the whole setup is configured as infrastructure-as-code, this means that any changes you make without checking back will be lost after a redeployment, so please sync up.
After successful review, I've enabled SSO via GitHub on our prod system at http://jenkins.mxnet-ci.amazon-ml.com. Next time you press 'log in' on the top right corner, you will be redirected to a GitHub login page which will look like the following: https://imgur.com/Bg8LGWD You can safely press 'Authorize marcoabreu'. The section about the organizations can be safely ignored, do NOT press 'grant' besides an organization as the CI system does not require access to the private data of your organizations. This permission is only required in order to identify which organizations you're in. Jenkins GitHub SSO requires this information in order to allow organization-based access rights. Although we are not using this feature, it can not be disabled unfortunately - the system will crash if the read-org-permission is not granted. For now, I've added the following people as administrators: - Meghna Baijal - Gautam Kumar - Kellen Sunderland - Pedro Larroy - Marco de Abreu - Chris Olivier Also, I've tried to add all committers listed at http://incubator.apache.org/projects/mxnet.html. Please drop me quick message if it doesn't work for you, it might have been possible that I have used the wrong account name. It seems like that the Apache alias sometimes does not match the GitHub alias. Also, I haven't been able to link Jian Zhangs alias 'jianzhangzju'. Please try to log in and let me know if any issues arise. Best regards, Marco On Mon, Jan 8, 2018 at 12:41 PM, Pedro Larroy <[email protected]> wrote: > Regarding the proposed permissions, I would like stricter permissions. > I think a committer should be able to stop-start-cancel jobs. But I > think only admins should be able to create new jobs, otherwise we run > the risk of the CI becoming a mess of jobs that nobody owns and > maintains. Please let's only have the jobs that are needed, supported, > maintained and have an owner on CI and agreed on the mailing list. > > Pedro. > > On Fri, Jan 5, 2018 at 11:54 PM, Marco de Abreu > <[email protected]> wrote: > > I'm proposing following permissions: https://i.imgur.com/uiFBtuW.png. > The > > meaning of every permission is explained at https://wiki.jenkins.io/ > > display/JENKINS/Matrix-based+security. > > > > Any objections? > > > > On Fri, Jan 5, 2018 at 11:03 PM, Marco de Abreu < > > [email protected]> wrote: > > > >> I'm currently working on a prototype of SSO based on GitHub and a few > >> issues arose: > >> > >> We are not able to use the permission strategy which determines the > access > >> rights based on the read/write permission to a project as the > >> Jenkins-plugin is not able to resolve the link between Jenkins-jobs and > >> GitHub-repositories. Instead I would propose to use a role-based > approach > >> using https://wiki.jenkins.io/display/JENKINS/Role+Strategy+Plugin. In > >> this case we would have three roles: Anonymous, Administrator and > >> Committer. While everybody would authenticate using their regular GitHub > >> account, the role assignment would have to happen manually. Considering > >> that the amount of administrators and committers doesn't change that > >> frequently, this shouldn't be too much of an issue - auto populating the > >> status is not possible unfortunately. > >> > >> Reason for splitting Administrators and Committers into two separate > roles > >> has a security reason. At the moment, we're using Chris Oliviers GitHub > >> credentials to populate the commit status. If all committers would gain > >> full admin rights, they would have access to these credentials. Chris is > >> not fine with this approach and would like to limit the amount of people > >> with access to his credentials as much as possible. > >> > >> In order to address his concerns, I propose to add Chris to the > committer > >> as well as to the admin role, while all other committers will only > receive > >> the committer role without read access to the credentials. In a later > >> email, I will make a proposal for the detailed committer role rights. > You > >> can check all available options at https://wiki.jenkins.io/ > >> display/JENKINS/Matrix-based+security. > >> > >> All people who have access to the underlying AWS account would be > granted > >> the Administrator role with full access. At the moment, this would be > >> Meghna Baijal, Gautam Kumar and myself. > >> > >> An alternative solution would be to create a bot account specifically > for > >> MXNet CI and use its credentials instead of Chris'. This account > requires > >> write permission to the repository, but would give us the advantage that > >> these credentials would be shared within the committers and thus making > the > >> restrictions regarding credentials obsolete (and Chris would be happy > not > >> the see his face within every single PR :P ). I've asked around and > >> received the feedback from multiple people that Apache Infra does not > want > >> to grant bot accounts write permission to a repository, but I would > like to > >> confirm back considering that AppVeyor, for example, has a bot account > with > >> write permission. I would like to check back with a mentor and create an > >> Apache Infra ticket to request details and permission. > >> > >> I would propose to take both approaches at the same time, meaning we can > >> start with Chris in the committer AND admin role while trying to get > >> permission for a bot account in the meantime. > >> > >> wdyt? > >> > >> On Fri, Jan 5, 2018 at 8:21 PM, Chris Olivier <[email protected]> > >> wrote: > >> > >>> I am fine without a vote unless a vote is required? Any objections, > >>> anyone? You're sort of adding functionality here, not changing or > >>> restricting... We can always change to Apache later. > >>> > >>> On Fri, Jan 5, 2018 at 11:18 AM, Marco de Abreu < > >>> [email protected]> wrote: > >>> > >>> > I'd be in favour of GitHub. Shall we open a vote or would you like > me to > >>> > create a POC with GitHub first and afterwards we can check if that's > >>> > enough? > >>> > > >>> > -Marco > >>> > > >>> > On Fri, Jan 5, 2018 at 8:13 PM, Chris Olivier <[email protected] > > > >>> > wrote: > >>> > > >>> > > Apparently Apache supports OATH, so I am open to either. > >>> > > Good idea for the docker thing. > >>> > > > >>> > > On Fri, Jan 5, 2018 at 11:02 AM, Marco de Abreu < > >>> > > [email protected]> wrote: > >>> > > > >>> > > > GitHub SSO allows the neat feature that login and permission can > be > >>> > > > selected depending on the access rights a user has to a project. > >>> > Somebody > >>> > > > with write access (committers) would be get different permissions > >>> than > >>> > > > somebody with only read access. > >>> > > > > >>> > > > We could check back with Apache for SSO, but this would involve > >>> Apache > >>> > > > infra. We could put it up to a vote whether to use GitHub or > Apache > >>> > SSO. > >>> > > > > >>> > > > In order to reproduce a build failure we have been thinking about > >>> > > changing > >>> > > > the ci_build.sh in such a way that it can be run manually without > >>> > > Jenkins. > >>> > > > The setup I took over binds the Jenkins work directory into the > >>> docker > >>> > > > containers and uses a few hacks which are hard to reproduce > >>> locally. We > >>> > > > plan to reengineer this script to make it easier to run manually. > >>> > > > But making the AMI public is a good idea! We plan to make the > whole > >>> > > > infrastructure code (based on Terraform) completely public - at > the > >>> > > moment > >>> > > > it's in a private repository as it contains credentials, but they > >>> will > >>> > be > >>> > > > moved to KMS soon. It would definitely be a good approach to just > >>> > supply > >>> > > > the AMI so everybody could recreate the environment in their own > >>> > account. > >>> > > > > >>> > > > -Marco > >>> > > > > >>> > > > Am 05.01.2018 7:51 nachm. schrieb "Chris Olivier" < > >>> > [email protected] > >>> > > >: > >>> > > > > >>> > > > Well, login to the Jenkins server, I would imagine. > >>> > > > > >>> > > > github or Apache SSO (does Apache support OAUTH?) seems like a > good > >>> > idea > >>> > > as > >>> > > > long as there's a way to not let everyone with a github account > log > >>> in. > >>> > > > > >>> > > > Access to actual slave machines could be more restricted, I > imagine. > >>> > > > > >>> > > > Eventually, a public current AMI for a build slave would be good > in > >>> > order > >>> > > > to reproduce build or test problems that can't be reproduced > >>> locally. > >>> > > > > >>> > > > wdyt? > >>> > > > > >>> > > > > >>> > > > > >>> > > > On Fri, Jan 5, 2018 at 10:41 AM, Marco de Abreu < > >>> > > > [email protected]> wrote: > >>> > > > > >>> > > > > Would it be an acceptable solution if we add SSO or do you also > >>> want > >>> > > > access > >>> > > > > to the actual AWS account and all machines? > >>> > > > > > >>> > > > > Yes, the build jobs are automatically getting created for new > >>> > branches. > >>> > > > > > >>> > > > > -Marco > >>> > > > > > >>> > > > > Am 05.01.2018 7:35 nachm. schrieb "Marco de Abreu" < > >>> > > > > [email protected]>: > >>> > > > > > >>> > > > > I totally agree, this is not the way it should work in an > Apache > >>> > > Project. > >>> > > > > It's running on an isengard account, meaning it is only > accessible > >>> > for > >>> > > > > Amazon employees. The problem is that a compromised account > could > >>> > cause > >>> > > > > damage up to 170,000$ per day. There are alarms in place to > notice > >>> > > those > >>> > > > > cases, but we still have to be very careful. These high limits > >>> have > >>> > > been > >>> > > > > chosen due to auto scaling being added within the next week's. > >>> > > > > > >>> > > > > I'd be happy to introduce a committer into the CI process and > all > >>> the > >>> > > > > necessary steps as well as granting them permission. The only > >>> > > restriction > >>> > > > > being that it has to be and Amazon employee and access to > console, > >>> > > master > >>> > > > > and slave only being possible from the Corp network. > >>> > > > > > >>> > > > > There is no open ticket. What would you like to request? > >>> > > > > > >>> > > > > -Marco > >>> > > > > > >>> > > > > > >>> > > > > Am 05.01.2018 7:22 nachm. schrieb "Chris Olivier" < > >>> > > [email protected] > >>> > > > >: > >>> > > > > > >>> > > > > Like John and other mentors were saying, it's not proper for > CI to > >>> > be a > >>> > > > > closed/inaccessible environment. Is it running on an Isengard > >>> > account > >>> > > or > >>> > > > > in PROD or CORP or just generic EC2? I think that we should > >>> remedy > >>> > > this. > >>> > > > > It's very strange that no committers have access at all. Is > >>> there a > >>> > > > ticket > >>> > > > > open to IPSEC? > >>> > > > > > >>> > > > > On Fri, Jan 5, 2018 at 10:17 AM, Marco de Abreu < > >>> > > > > [email protected]> wrote: > >>> > > > > > >>> > > > > > Hello Chris, > >>> > > > > > > >>> > > > > > At the moment this is not possible due Amazon AppSec > >>> (Application > >>> > > > > security) > >>> > > > > > restrictions which does not permit user data and credentials > on > >>> > these > >>> > > > > > machines. > >>> > > > > > > >>> > > > > > I have been thinking about adding single sign on bound to > >>> GitHub, > >>> > but > >>> > > > we > >>> > > > > > would have to check back with AppSec. > >>> > > > > > > >>> > > > > > Is the reason for your request still the ability to start and > >>> stop > >>> > > > > running > >>> > > > > > builds? > >>> > > > > > > >>> > > > > > Best regards, > >>> > > > > > Marco > >>> > > > > > > >>> > > > > > Am 05.01.2018 7:11 nachm. schrieb "Chris Olivier" < > >>> > > > [email protected] > >>> > > > > >: > >>> > > > > > > >>> > > > > > Marco, > >>> > > > > > > >>> > > > > > Are all committers able to get login access to the Jenkins > >>> Server? > >>> > > If > >>> > > > > not, > >>> > > > > > why? > >>> > > > > > > >>> > > > > > -Chris > >>> > > > > > > >>> > > > > > >>> > > > > >>> > > > >>> > > >>> > >> > >> >
