+1
It seems that the Apache CI is quite overloaded these days, and MXNet's CI pipeline is too complex to run there. In addition, we may need to add more devices, e.g. macpro and rasbperry pi, into the server, and more tasks such as pip build. It means a lot of requests to the Infra team. We can reuse our previous Jenkins server at http://ci.mxnet.io/. But we probably need a dedicate developer to maintain it. On Fri, Oct 20, 2017 at 1:01 PM, sandeep krishnamurthy < [email protected]> wrote: > Hello all, > > I am hereby opening up a discussion thread on how we can stabilize Apache > MXNet CI build system. > > Problems: > > ======== > > Recently, we have seen following issues with Apache MXNet CI build systems: > > 1. Apache Jenkins master is overloaded and we see issues like - unable > to trigger builds, difficult to load and view the blue ocean and other > Jenkins build status page. > 2. We are generating too many request/interaction on Apache Infra team. > 1. Addition/deletion of new slave: Caused from scaling activity, > recycling, troubleshooting or any actions leading to change of slave > machines. > 2. Plugins / other Jenkins Master configurations. > 3. Experimentation on CI pipelines. > 3. Harder to debug and resolve issues - Since access to master and slave > is not with the same community, it requires Infra and community to dive > deep together on all action items. > > Possible Solutions: > > ============== > > 1. Can we set up a separate Jenkins CI build system for Apache MXNet > outside Apache Infra? > 2. Can we have a separate Jenkins Master in Apache Infra for MXNet? > 3. Review design of current setup, refine and fill the gaps. > > @ Mentors/Infra team/Community: > > ========================== > > Please provide your suggestions on how we can proceed further and work on > stabilizing the CI build systems for MXNet. > > Also, if the community decides on separate Jenkins CI build system, what > important points should be taken care of apart from the below: > > 1. Community being able to access the build page for build statuses. > 2. Committers being able to login with apache credentials. > 3. Hook setup from apache/incubator-mxnet repo to Jenkins master. > > > Irrespective of the solution we come up, I think we should initiate a > technical design discussion on how to setup the CI build system. Probably 1 > or 2 pager documents with the architecture and review with Infra and > community members. > > ***There were few proposal and discussion on the slack channel, to reach > wider community members, moving that discussion formally to this list. > > > My Proposal: Option 1 - Set up separate Jenkins CI build system. > > Thanks, > > Sandeep > > > > -- > Sandeep Krishnamurthy >
