aaronmarkham opened a new pull request #10485: [MXNET-304][RFC] Jenkins docs build URL: https://github.com/apache/incubator-mxnet/pull/10485 ## Description ## Adapted updated build scripts to the new Jenkins CI setup. This enables nightly docs builds. **Preview:** http://107.21.90.138/ ## Checklist ## ### Essentials ### Please feel free to remove inapplicable items for your PR. - [x] The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant [JIRA issue](https://issues.apache.org/jira/projects/MXNET/issues) created (except PRs with tiny changes) - [x] Changes are complete (i.e. I finished coding on this PR) - [x] To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change ### Changes ### - [x] Trigger via Jenkins CI (currently pointing to my forks) 1. Use [Pipeline website build - test pipeline](http://jenkins.mxnet-ci.amazon-ml.com/job/website%20build%20-%20test%20pipeline/): this uses parameters from the UI so you can select which tag(s) to build, which tags should appear in the versions dropdown, which tag to be default (now master), and the domain (to facilitate dev server testing). 2. Then run [Project website build - test publish](http://jenkins.mxnet-ci.amazon-ml.com/job/website%20build%20-%20test%20publish/): this is mostly a clone from the previous site deploy Jenkins job; it deletes everything and grabs the artifacts from the build job and pushes those new files as a commit to the asf-site branch of mxnet-incubator-site. - [x] Updated [dependencies](https://github.com/aaronmarkham/incubator-mxnet/blob/1c08ec9b3b93a0a108e62ac4c21dfec9c7002274/ci/docker/install/ubuntu_docs.sh) for the docs Docker image - [x] Uses a [Jenkinsfile](https://github.com/aaronmarkham/incubator-mxnet/blob/1c08ec9b3b93a0a108e62ac4c21dfec9c7002274/docs/Jenkinsfile) and [runtime_functions.sh](https://github.com/aaronmarkham/incubator-mxnet/blob/1c08ec9b3b93a0a108e62ac4c21dfec9c7002274/ci/docker/runtime_functions.sh) to call the [build_version_doc/build_all_version.sh](https://github.com/aaronmarkham/incubator-mxnet/blob/1c08ec9b3b93a0a108e62ac4c21dfec9c7002274/docs/build_version_doc/build_all_version.sh) to build the desired tags, then calls [update_all_version.sh](https://github.com/aaronmarkham/incubator-mxnet/blob/1c08ec9b3b93a0a108e62ac4c21dfec9c7002274/docs/build_version_doc/update_all_version.sh) to inject a variety of changes to the site versions. These are then saved as [artifacts.tgz](http://jenkins.mxnet-ci.amazon-ml.com/job/website%20build%20-%20test%20pipeline/lastSuccessfulBuild/artifact/). - [x] `update_all_version.sh` and `build_all_version.sh` now use IFS to delimit the tags by `;` since Jenkins has a hard time passing quotes downstream to the shell scripts. * This means usage for dev builds will change like this to use semicolons: `./update_all_version.sh master;1.1.0;1.0.0 master http://localhost` ## Comments ## - The site deploy CI job should be converted to use Jenkinsfile instead of having script logic in the job itself. So that's a future TODO. - The publish job shouldn't delete the whole site repo. You should be able to just run master for a docs build and replace just master on the site. Another future TODO. - You should be able to run these scripts in the context of your current fork. At the moment they're hardcoded to checkout the project's main fork. This means that you can build the docs locally still with `make docs`, but you won't get to see all of the customizations that happen with the `update_all_version.sh` script. Another future TODO. - I'm seeing hardcoded problems: for current tag as default and the domain. These should be ferreted out as we test this new build structure. The site should work on a dev server perfectly, without directing to a hardcoded url. (Too often during testing you realize you're not on your dev server anymore, but on production.) - Artifacts could be handled better. We should investigate different plugins and artifacts strategies, so that we build smarter. - There's some waste on the Docker deps - these should probably be baked into an image and loaded on docker hub so that the jobs aren't reinstalling this stuff every time. - Jumping from the CI groovy script to shell is not the right pattern from what I can tell. Seems like these things should be refactored to be all in groovy and using built-in Jenkins functions. Another TODO. ## Notes ## * Make use of the [Pipeline Syntax link](http://jenkins.mxnet-ci.amazon-ml.com/job/website%20build%20-%20test%20pipeline/pipeline-syntax/) at the bottom of the Jenkins configure screen for the jobs. Many buried features in there to help build your job.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
