This PR rolls up https://github.com/apache/beam/pull/5809 and runs website builds and tests inside a Docker container.
Description of #5809: This adds a new pre-commit job for the website sources in the apache/beam repo. This is a step in the migration effort to move website sources into the main repo and improve automation reliability. See: https://s.apache.org/beam-site-automation Note that this change doesn't cover automatic staging of the repository. I'll work on that in a follow-up PR. Running the jobs in a container has these advantages: - Consistency of environment: minimal differences between local dev machine and Jenkins. - Using an existing Ruby container minimizes automation effort and build time. - No need to install Jekyll and other website dependencies locally. I investigated the following ways of running precommit and postcommit website jobs using Docker. 1. https://plugins.jenkins.io/docker-custom-build-environment This plugin has been installed in builds.apache.org's Jenkins service. I couldn't test it locally however, and I wouldn't expect developers working on Jenkins jobs to go through the hassle of Installing and configuring Jenkins on their local machine or VM. I tried using the dockerized-jenkins (in this repo), and while I managed to configure and launch the jobs with the plugin, I still came across the issue of running [docker-in-docker](https://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-docker-for-ci/). I opted for the solution of running jobs as sibling containers (of the Jenkins container), but the plugin wants to bind mount the repo and /tmp directories to the job container and that doesn't seem to work. 1. https://github.com/bmuschko/gradle-docker-plugin I tried this plugin hoping to make managing Docker containers simpler, but it seems to be aimed more at building and publishing images. For instance I was missing the -u flag for docker create. 1. Calling docker CLI directly. This is what I settled on. It should work very similarly on dev and Jenkins machines. ------------------------ Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). It will help us expedite review of your Pull Request if you tag someone (e.g. `@username`) to look at it. Post-Commit Tests Status (on master branch) ------------------------------------------------------------------------------------------------ Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) </br> [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | --- | --- | --- | --- [ Full content available at: https://github.com/apache/beam/pull/6282 ] This message was relayed via gitbox.apache.org for [email protected]
