Thank you Tyson for the update. Thank you Damian, Tobiasz for the improvements.
On Mon, Sep 14, 2020 at 3:29 PM Kenneth Knowles <[email protected]> wrote: > I think the big bucket makes sense and is pretty self-documenting. I think > we'll hit some name conflicts between our infra component and ASF INFRA. > They just took the best name :-) > > I think "testing" is a worse name, really. It is documented as "Testing: > General Infrastructure" but someone not reading that could put all sorts of > things in the "testing" component. > > Since you can tag as many components as you want, I think having a bunch > of fine-grained components could be fine. On the other hand, I don't want > to invent a component for "email report about our dependencies". > > I'm happy to create an "infra" component but is there a synonym, or a > small collection of other terms, we could use? > > Kenn > > On Fri, Sep 11, 2020 at 3:39 PM Tyson Hamilton <[email protected]> wrote: > >> >> >> On Fri, Sep 11, 2020 at 3:29 PM Kyle Weaver <[email protected]> wrote: >> >>> > Could someone create an ‘infra’ component in Jira (I don’t have >>> access)? >>> >>> >>> Could these issues go under the "testing" component? >>> >> >> That is an option. There are things beyond testing as well like Github >> Actions, Jenkins inventory jobs, email reports, artifact publishing, etc. >> so IMO having a general 'infra' would be handy. Another option is to have >> dedicated components for each of these things but the proliferation may >> become a problem. >> >> >>> >>> On Fri, Sep 11, 2020 at 3:11 PM Tyson Hamilton <[email protected]> >>> wrote: >>> >>>> Hi All, >>>> >>>> Below is a summary of the improvements made to Beam’s infrastructure in >>>> the last few months. The completion of this work is thanks to Damian >>>> (damgad) and Tobiasz (TobKed) from Polidea. Thank you!!! >>>> >>>> >>>> - >>>> >>>> Jenkins tests are not always triggered / Switch to new jenkins infra >>>> - >>>> >>>> Beam now has a dedicated Jenkins master. >>>> - >>>> >>>> Beam now has 4x the Jenkins agents which essentially eliminated >>>> build queues. >>>> - >>>> >>>> Jenkins /tmp directory periodic cleanup >>>> - >>>> >>>> Moved to the inventory job for better transparency [tmp-cleanup]. >>>> - >>>> >>>> Jenkins post commit tests does not clearly point to failing tests >>>> - >>>> >>>> installed Test Result History Plugin [test-results-plugin, >>>> screenshot] to start measuring flakiness. This plugin produces a >>>> similar >>>> matrix of test results as the Grafana Chart but it is clickable and >>>> has >>>> aggregation. >>>> - >>>> >>>> BEAM-3105 Cancel Jenkins builds when an update to a PR makes a >>>> prior build irrelevant >>>> - >>>> >>>> BEAM-3105 Cancel Jenkins builds when an update to a PR makes a >>>> prior build irrelevant >>>> - >>>> >>>> BEAM-9388 Github Actions instead of Travis CI as a build framework >>>> (Wheel Files) >>>> - >>>> >>>> BEAM-9271 Fix "beam_PostCommit_Py_ValCont failure due to timeout" >>>> - >>>> >>>> BEAM-4709 Javadoc build only tested during release >>>> - >>>> >>>> BEAM-8665 Run 2 (one for Java, one for Python) new test suites in >>>> Windows environment. >>>> - >>>> >>>> BEAM-10281 Jenkins Load Sharing Improvements (stackdriver agents & >>>> extend nb of executors) >>>> - >>>> >>>> BEAM-10542 Investigate a possible Nexmark performance regression >>>> around 06/16 >>>> - >>>> >>>> INFRA-20649 Beam Jenkins not triggering jobs from phrases >>>> - >>>> >>>> BEAM-10831 Fix broken Beam Dependency Check Report >>>> - >>>> >>>> BEAM-10837 Remove unused beam_PerformanceTests_Analysis Jenkins Job >>>> - >>>> >>>> BEAM-10835 Improve Github Actions cancelling duplicated runs >>>> - >>>> >>>> BEAM-10807 Performance test weekly email sent to dev@ list >>>> automatically [perf-email] >>>> - >>>> >>>> Apply more strict credentials bindings to Jenkins jobs >>>> - >>>> >>>> Configure seed & dependency check jobs to be triggered only by >>>> committers >>>> >>>> >>>> There are plans to continue to invest into Beam’s infra in the future. >>>> A few examples include, >>>> >>>> - >>>> >>>> Test Flakiness tracking improvements (e.g. increase build history >>>> retention) >>>> - >>>> >>>> Reduce flakiness with cross-job caching (maven, pypi, gradle, build >>>> cache, licenses, …) >>>> - >>>> >>>> More Github Actions (java artifacts, docker images, flaky test >>>> quarantine like Apache Ariflow) >>>> >>>> >>>> If you have additional infra related tasks, or pain points, please >>>> create a Jira issue so we can track them. Contributions are always welcome. >>>> >>> +1 to this request. Identifying the largest productivity blockers to the contributors would greatly help with the future of this work track. Please report any infra related frictions to your own workflows. > Could someone create an ‘infra’ component in Jira (I don’t have access)? >>>> >>>> >>>> -Tyson >>>> >>>> [tmp-cleanup]: >>>> https://github.com/apache/beam/commit/50366bae1c86136331be097f0f0250bd619dc7f8 >>>> >>>> [test-results-plugin]: >>>> https://plugins.jenkins.io/test-results-analyzer/ >>>> >>>> [test-results-screenshot]: https://photos.app.goo.gl/E4cGpp5WqxYhwtTAA >>>> >>>> [perf-email]: >>>> https://lists.apache.org/thread.html/rc0054933a3e40c35f09641786a873391988caf6529899ec8bba67772%40%3Cdev.beam.apache.org%3E >>>> >>>>
