Random question - can the CI be split such that the Apache CI is doing a basic set of checks on that hardware, and is hooked to a PR, while there is a larger "Is trunk good for release?" test that is running periodically rather than on every PR?
ie: do we need each PR to be run on varied hardware, or can we have this two tier approach? Hen On Fri, Oct 20, 2017 at 1:01 PM, sandeep krishnamurthy < sandeep.krishn...@gmail.com> wrote: > Hello all, > > I am hereby opening up a discussion thread on how we can stabilize Apache > MXNet CI build system. > > Problems: > > ======== > > Recently, we have seen following issues with Apache MXNet CI build systems: > > 1. Apache Jenkins master is overloaded and we see issues like - unable > to trigger builds, difficult to load and view the blue ocean and other > Jenkins build status page. > 2. We are generating too many request/interaction on Apache Infra team. > 1. Addition/deletion of new slave: Caused from scaling activity, > recycling, troubleshooting or any actions leading to change of slave > machines. > 2. Plugins / other Jenkins Master configurations. > 3. Experimentation on CI pipelines. > 3. Harder to debug and resolve issues - Since access to master and slave > is not with the same community, it requires Infra and community to dive > deep together on all action items. > > Possible Solutions: > > ============== > > 1. Can we set up a separate Jenkins CI build system for Apache MXNet > outside Apache Infra? > 2. Can we have a separate Jenkins Master in Apache Infra for MXNet? > 3. Review design of current setup, refine and fill the gaps. > > @ Mentors/Infra team/Community: > > ========================== > > Please provide your suggestions on how we can proceed further and work on > stabilizing the CI build systems for MXNet. > > Also, if the community decides on separate Jenkins CI build system, what > important points should be taken care of apart from the below: > > 1. Community being able to access the build page for build statuses. > 2. Committers being able to login with apache credentials. > 3. Hook setup from apache/incubator-mxnet repo to Jenkins master. > > > Irrespective of the solution we come up, I think we should initiate a > technical design discussion on how to setup the CI build system. Probably 1 > or 2 pager documents with the architecture and review with Infra and > community members. > > ***There were few proposal and discussion on the slack channel, to reach > wider community members, moving that discussion formally to this list. > > > My Proposal: Option 1 - Set up separate Jenkins CI build system. > > Thanks, > > Sandeep > > > > -- > Sandeep Krishnamurthy >
