Hi All,
Below is a summary of the improvements made to Beam’s infrastructure in the
last few months. The completion of this work is thanks to Damian (damgad)
and Tobiasz (TobKed) from Polidea. Thank you!!!
-
Jenkins tests are not always triggered / Switch to new jenkins infra
-
Beam now has a dedicated Jenkins master.
-
Beam now has 4x the Jenkins agents which essentially eliminated build
queues.
-
Jenkins /tmp directory periodic cleanup
-
Moved to the inventory job for better transparency [tmp-cleanup].
-
Jenkins post commit tests does not clearly point to failing tests
-
installed Test Result History Plugin [test-results-plugin,
screenshot] to start measuring flakiness. This plugin produces a similar
matrix of test results as the Grafana Chart but it is clickable and has
aggregation.
-
BEAM-3105 Cancel Jenkins builds when an update to a PR makes a prior
build irrelevant
-
BEAM-3105 Cancel Jenkins builds when an update to a PR makes a prior
build irrelevant
-
BEAM-9388 Github Actions instead of Travis CI as a build framework
(Wheel Files)
-
BEAM-9271 Fix "beam_PostCommit_Py_ValCont failure due to timeout"
-
BEAM-4709 Javadoc build only tested during release
-
BEAM-8665 Run 2 (one for Java, one for Python) new test suites in
Windows environment.
-
BEAM-10281 Jenkins Load Sharing Improvements (stackdriver agents &
extend nb of executors)
-
BEAM-10542 Investigate a possible Nexmark performance regression around
06/16
-
INFRA-20649 Beam Jenkins not triggering jobs from phrases
-
BEAM-10831 Fix broken Beam Dependency Check Report
-
BEAM-10837 Remove unused beam_PerformanceTests_Analysis Jenkins Job
-
BEAM-10835 Improve Github Actions cancelling duplicated runs
-
BEAM-10807 Performance test weekly email sent to dev@ list automatically
[perf-email]
-
Apply more strict credentials bindings to Jenkins jobs
-
Configure seed & dependency check jobs to be triggered only by committers
There are plans to continue to invest into Beam’s infra in the future. A
few examples include,
-
Test Flakiness tracking improvements (e.g. increase build history
retention)
-
Reduce flakiness with cross-job caching (maven, pypi, gradle, build
cache, licenses, …)
-
More Github Actions (java artifacts, docker images, flaky test
quarantine like Apache Ariflow)
If you have additional infra related tasks, or pain points, please create a
Jira issue so we can track them. Contributions are always welcome. Could
someone create an ‘infra’ component in Jira (I don’t have access)?
-Tyson
[tmp-cleanup]:
https://github.com/apache/beam/commit/50366bae1c86136331be097f0f0250bd619dc7f8
[test-results-plugin]: https://plugins.jenkins.io/test-results-analyzer/
[test-results-screenshot]: https://photos.app.goo.gl/E4cGpp5WqxYhwtTAA
[perf-email]:
https://lists.apache.org/thread.html/rc0054933a3e40c35f09641786a873391988caf6529899ec8bba67772%40%3Cdev.beam.apache.org%3E
