ThomasDelteil opened a new pull request #15883: [WIP] New Website: New Pipeline [3/3] URL: https://github.com/apache/incubator-mxnet/pull/15883 - [ ] https://github.com/ThomasDelteil/incubator-mxnet/pull/6 Review the step from [2/3] to [3/3] - [x] #15884 merged - [ ] #15885 merged Website available here: https://mxnet-beta.staged.apache.org/ ## Description ## Instructions on using these updates are on the wiki: https://cwiki.apache.org/confluence/display/MXNET/Building+the+New+Website **Break out each docs package generation to separate pipelines.** ✅Python - porting the mx-theme to be local to the repo to facilitate modification as part of this project's PRs; this is from the "beta site" ✅R - using the same theme as Python; also from the "beta site"; found to be not reproducible or current, so disabled for now. ✅C/CPP - currently co-mingled Doxygen microsite output ✅Julia - julia docs output to microsite ✅Scala - scaladocs output to microsite ✅Java - javadocs output to microsite ✅Clojure - lein output to microsite ✅Jekyll website for main pages Legend: ✅Artifacts generating in CI ❌WIP ## Jenkins Updates * [New docs folder pipelines to generate artifacts for each docs package](http://jenkins.mxnet-ci.amazon-ml.com/job/docs-pipelines/) - also has an MXNet binary build pipeline that can be used to share the latest binary as an archived artifact. * [Publish job](http://jenkins.mxnet-ci.amazon-ml.com/job/test-website-publish/) - takes artifacts from the docs pipelines, collates the website and publishes it to incubator-mxnet-site. * using the last successful build URLs for each pipeline assures that the website gets published even if one docs package is failing * Dockerfile updates - to speed up testing, each docs package has a custom dockerfile that only installs dependencies required for that specific package rather than installing everything (as is done now). * New Python and R sites use Conda, so miniconda Python deps are installed with conda now. * New support for Jekyll: Docker container and related runtime functions ## Preview The publishing job uses CI artifacts generated from my fork of mxnet and pushes to my fork of mxnet-site, and you can see the output on a dev server here: http://54.211.76.74/ ## TODO when this PR is approved * Add notifications for when docs package builds fail. * Convert test pipelines to production website deployment flows. * Upload old version artifacts. * New PR triggering of a website test * Switch to GPU builds for the Python docs, so the outputs are in the tutorials ## Comments * Stash operations vs. archiving - recommendations in the docs suggest that large artifacts should be archived; stash is super slow; archived artifacts seems to be faster and can be used between pipelines. This is helpful for the MXNet binary and for the Scala package, both of which are used by various other docs packages. However, there's an implication with the master server. Archived artifacts are stored there, so if the pipeline is related to PR validation, this would be unwieldy. If related to publishing final artifacts for specific versions, well, that's probably ok. * It seems that efficiency in development and testing can be gained by checkpointing the docker containers after the dependencies are installed. * A version/branch parameter would be useful for the Jenkins pipelines for generating docs artifacts from different branches. * Publishing scripts seem to need a security refactor, or we don't bother offering stand-alone access to them; running local versus on Jenkins. * A bug bash is needed on the new Python docs - I turned off "warnings as errors" just so I could get this thing working, but that needs to be turned back on.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
