aaronmarkham opened a new pull request #15668: [WIP] Separate Jenkins docs pipelines per language URL: https://github.com/apache/incubator-mxnet/pull/15668 ## Description ## **Break out each docs package generation to separate pipelines.** ✅Python - porting the mx-theme to be local to the repo to facilitate modification as part of this project's PRs; this is from the "beta site" ❌R - using the same theme as Python; also from the "beta site"; found to be not reproducible or current, so disabled for now. ✅C/CPP - currently co-mingled Doxygen microsite output ✅Julia - julia docs output to microsite ✅Scala - scaladocs output to microsite ✅Java - javadocs output to microsite ✅Clojure - lein output to microsite ✅Jekyll website for main pages (currently on Thomas's repo, but will be folded into the docs folder) Legend: ✅Artifacts generating in CI ❌WIP ## Jenkins Updates * [New docs folder pipelines to generate artifacts for each docs package](http://jenkins.mxnet-ci.amazon-ml.com/job/docs-pipelines/) - also has an MXNet binary build pipeline that can be used to share the latest binary as an archived artifact. * [Publish job](http://jenkins.mxnet-ci.amazon-ml.com/job/test-website-publish/) - takes artifacts from the docs pipelines, collates the website and publishes it to incubator-mxnet-site. * using the last successful build URLs for each pipeline assures that the website gets published even if one docs package is failing * Dockerfile updates - to speed up testing, each docs package has a custom dockerfile that only installs dependencies required for that specific package rather than installing everything (as is done now). * New Python and R sites use Conda, so miniconda Python deps are installed with conda now. * New support for Jekyll: Docker container and related runtime functions ## Preview The publishing job uses CI artifacts generated from my fork of mxnet and pushes to my fork of mxnet, and you can see the output on a dev server here: http://54.211.76.74/ This isn't up-to-date on the front-end as I don't have automatic publishing enabled yet. You can see that update here: https://thomasdelteil.github.io/mxnet.io-v2/ ## TODO * It seems that efficiency in development and testing can be gained by checkpointing the docker containers after the dependencies are installed. * Installations of package managers like `conda` and `bundle` should be updated to best practices * Local use of the Docker containers should allow a user to interact with the package managers. * A version/branch parameter would be useful for the Jenkins pipelines for generating docs artifacts from different branches. * Pipelines for PR validation for "website" could be updated. Artifact strategy might need to be different if previews are desired. * Add notifications for when docs package builds fail. * Convert test pipelines to production website deployment flows. * Publishing scripts seem to need a security refactor, or we don't bother offering stand-alone access to them; running local versus on Jenkins. * Upload old version artifacts. ## Comments * Stash operations vs. archiving - recommendations in the docs suggest that large artifacts should be archived; stash is super slow; archived artifacts seems to be faster and can be used between pipelines. This is helpful for the MXNet binary and for the Scala package, both of which are used by various other docs packages. However, there's an implication with the master server. Archived artifacts are stored there, so if the pipeline is related to PR validation, this would be unwieldy. If related to publishing final artifacts for specific versions, well, that's probably ok.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
