On Tue, Nov 7, 2017 at 12:29 AM, Lukas Zapletal <[email protected]> wrote:
> We should start tracking test breakages in time, if we have a test > that did not break for a year, lets delete it. As simple as that. It's > added value is almost zero. > The only time tests should be deleted is when the functionality they verify has been removed. Test fragility, long execution time, and complexity are not the reasons to get rid of them, but the signs of badly factored code, wrong level of testing, or both. > > This is not out of my head, I've heard that on a talk somewhere. > Instead of deleting, we can move it to archive - run them on a nightly > basis, if we find this too much. Frankly, I prefer deletion. > > Also, let's ditch unit tests in favor of integration tests. We have a > lot of areas covered with both and this is extra work and slowing down > things. > > Let's not. Integration and higher-up tests give big gains in test coverage quickly, but that comes at the price of fragility, slow execution, and inability to pinpoint failures. Unit tests, in our case it will be model and controller tests, should be the main suite of tests. -d > LZ > > On Tue, Nov 7, 2017 at 9:19 AM, Ohad Levy <[email protected]> wrote: > > Hi, > > > > The challenge: > > > > As a developer, I feel under utilized waiting for tests, the current test > > suite for foreman core takes over an hour, multiply it by number of > really > > simple code fixes based on comment and it takes up a good chunk of a day. > > > > I would like to suggest a short and long term solution, and would > appreciate > > your feedback. > > > > Short-term: > > > > Break down test matrix into more specific groups, this will allow to > > re-trigger just a smaller part of the tests vs. the entire suit today, > for > > example: > > * API tests > > * DB migrations > > * UI Integration tests > > * ... > > > > Long term: > > > > Break down core into smaller repositories - This is probably a bigger > topic > > then just tests, but its worth raising in this context as well.. the > > benefits i see are: > > > > * smaller repos - faster tests per repo (we would still need a plugin > matrix > > kind of tests done at some point) > > * better review process - smaller repos would mean the maintainers of the > > repo actually care for all/most of the pr's - today its not the case - > many > > PR's are handled by sub groups (e.g. anything under webpack folder is > > "someone else") > > * potentially better API between parts - e.g. we can create a schema > > specific repo (not saying its a good idea) - which will always work with > a > > dummy rails app - in the long run - this will ensure our schema is > resilient > > to upgrades and model changes in core. > > > > I would even go further and enable auto merge after review + successful > > tests, e.g. > > 1. PR is sent > > 2. repo tests are executed and pass > > 3. reviewer approve the change > > 4. larger test matrix is executed and pass > > 5. code get auto merged. > > > > -- > > You received this message because you are subscribed to the Google Groups > > "foreman-dev" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [email protected]. > > For more options, visit https://groups.google.com/d/optout. > > > > -- > Later, > Lukas @lzap Zapletal > > -- > You received this message because you are subscribed to the Google Groups > "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
