Sergey, We are working on a full automated build pipeline that will test against a wide range of configurations. Ideally, the pipeline will work as follows:
1. PR is submitted -> the PR is built, unit tested, and run through static analysis. Smoke tests are run on a matrix of configurations 2. The PR is reviewed by two or more committers 3. When a PR commit receives two or more LGTMs and no -1s, it is enqueued to be merged to the target branch 4. When it’s PR’s turn to be merged, it merged against the target release branch and rebuilt (unit tests + static analysis), packaged, and tested (smoke test + a set of component tests) on a matrix of configurations 5. If the PR needs to be forward merged, it is enqueued for the next branch and Step 4 is re-executed In this system, only one PR would be actively merged at a time. Steps 4 and 5 are repeated until the PR is merged to master. This process not only ensures that the PR is adequately reviewed and tested, and it also detects introduced by merges of adjacent PRs. blueorganatan is first step in the process which provides a way for indicate that a PR is ready for the next step. We are working towards such a system, but, obviously, there are a lot of pieces we need to refine/develop and integrate. I am working up a release management roadmap to capture these ideas in more detail which I hope publish in the near future, Thanks, -John > john.burw...@shapeblue.com www.shapeblue.com 53 Chandos Place, Covent Garden, London VA WC2N 4HSUK @shapeblue On Sep 12, 2016, at 3:29 PM, Sergey Levitskiy <sergey.levits...@autodesk.com> wrote: > > @swill @jburwell From my, non-committer, point of view introducing fully > automated test system integrated with PR submission and leveraging complete > list of supported hypervisors is the key to stability. I really like > blueorangutain idea and if this can be brought back and also cover full set > of integration tests along with packaging that would be awesome. When we > develop PR unfortunately we can only test it only with a certain set of > hypervisors and network configs so although giving test LGTM it doesn’t cover > all possible integration points. I believe most other test results people > execute manually and post on PRs are the same. > > > On 9/12/16, 11:38 AM, "jburwell" <g...@git.apache.org> wrote: > Even > Github user jburwell commented on the issue: > > https://github.com/apache/cloudstack/pull/1658 > > @swill where hardware are being varies by PR. In some cases, we have > people running them in their labs and reporting results. Other cases, it's > blueorangatan going through ShapeBlue Jenkins + trillian. But yes, PRs are > getting tested on hardware. > > (Sadly, we are still recovering from a dead storage array, so > blueorangatan has been out of commission for a little bit), > > > --- > If your project is set up for it, you can reply to this email and have your > reply appear on GitHub as well. If your project does not have this feature > enabled and wishes so, or if the feature is enabled but not working, please > contact infrastructure at infrastruct...@apache.org or file a JIRA ticket > with INFRA. > --- > > >