aaronmarkham opened a new pull request #15668: [WIP] Separate Jenkins docs
pipelines per language
URL: https://github.com/apache/incubator-mxnet/pull/15668
## Description ##
**Break out each docs package generation to separate pipelines.**
✅Python - porting the mx-theme to be local to the repo to facilitate
modification as part of this project's PRs; this is from the "beta site"
❌R - using the same theme as Python; also from the "beta site"; found to
be not reproducible or current, so disabled for now.
✅C/CPP - currently co-mingled Doxygen microsite output
✅Julia - julia docs output to microsite
✅Scala - scaladocs output to microsite
✅Java - javadocs output to microsite
✅Clojure - lein output to microsite
✅Jekyll website for main pages (currently on Thomas's repo, but will be
folded into the docs folder)
Legend:
✅Artifacts generating in CI
❌WIP
## Jenkins Updates
* [New docs folder pipelines to generate artifacts for each docs
package](http://jenkins.mxnet-ci.amazon-ml.com/job/docs-pipelines/) - also has
an MXNet binary build pipeline that can be used to share the latest binary as
an archived artifact.
* [Publish
job](http://jenkins.mxnet-ci.amazon-ml.com/job/test-website-publish/) - takes
artifacts from the docs pipelines, collates the website and publishes it to
incubator-mxnet-site.
* using the last successful build URLs for each pipeline assures that the
website gets published even if one docs package is failing
* Dockerfile updates - to speed up testing, each docs package has a custom
dockerfile that only installs dependencies required for that specific package
rather than installing everything (as is done now).
* New Python and R sites use Conda, so miniconda Python deps are installed
with conda now.
* New support for Jekyll: Docker container and related runtime functions
## Preview
The publishing job uses CI artifacts generated from my fork of mxnet and
pushes to my fork of mxnet, and you can see the output on a dev server here:
http://54.211.76.74/
This isn't up-to-date on the front-end as I don't have automatic publishing
enabled yet.
You can see that update here:
https://thomasdelteil.github.io/mxnet.io-v2/
## TODO
* It seems that efficiency in development and testing can be gained by
checkpointing the docker containers after the dependencies are installed.
* Installations of package managers like `conda` and `bundle` should be
updated to best practices
* Local use of the Docker containers should allow a user to interact
with the package managers.
* A version/branch parameter would be useful for the Jenkins pipelines for
generating docs artifacts from different branches.
* Pipelines for PR validation for "website" could be updated. Artifact
strategy might need to be different if previews are desired.
* Add notifications for when docs package builds fail.
* Convert test pipelines to production website deployment flows.
* Publishing scripts seem to need a security refactor, or we don't bother
offering stand-alone access to them; running local versus on Jenkins.
* Upload old version artifacts.
## Comments
* Stash operations vs. archiving - recommendations in the docs suggest that
large artifacts should be archived; stash is super slow; archived artifacts
seems to be faster and can be used between pipelines. This is helpful for the
MXNet binary and for the Scala package, both of which are used by various other
docs packages. However, there's an implication with the master server. Archived
artifacts are stored there, so if the pipeline is related to PR validation,
this would be unwieldy. If related to publishing final artifacts for specific
versions, well, that's probably ok.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
