Nicely written. > Ruby 2.2 -- postgresql: 48 minutes > Ruby 2.2 -- mysql: 30 minutes > Ruby 2.2 -- sqlite3: 26 minutes > Ruby 2.3 -- postgresql: 56 minutes > Ruby 2.4 -- postgresql: 57 minutes
Fun fact, the newer Ruby version the slower time. > Setup: 12 minutes This is one major pain we have we need to solve. > Let's remove tests not > because they don't break, but because they aren't providing any real value. That's what I tried to articulate, not my idea at all. I've seen some Bob talks. > Let's also focus on where the bottlenecks lie. If we remove 20 unit tests > because they are old and don't break, but run in 100 ms each, we've only > shaved 2 seconds off of a 45 minute test run. Jenkins output for a given PR > gives a view of what the slowest test suites are. I'd encourage developers > to start there and tackle the bottlenecks [3]. Yeah very relevant point. > 2) Re-examine Parallel of Testing For foreman-core tests with discovery, I am able to pull this down to less than 4 minutes on 8 core CPU, but I see about 60 failures mostly JavaScript. Most of these errors are: TypeError: undefined is not an object (evaluating 'tfm.nav.init') If you want to give this a try, it is as easy as: be rake parallel:create be rake parallel:migrate be rake parallel:test For more details visit: https://github.com/theforeman/foreman/pull/2299 > 3) "Static" Environments > > This comes from both a trend in the Jenkins world and a question from Timo > to move towards essentially Docker containers with pre-populated > environments. The idea would be to ensure these are routinely updated with > baseline dependencies installed so that setup time is reduced by having most > assets (e.g. gems and npm modules) already on the container. I don't think we necessary need containers for that, I think putting those modules and gems into git (e.g. via git annex) or to some side-repo could do the trick. Also migrations could be solved using similar technique. I think introducing containers can be counter-productive. You need to build those containers in the end, that costs time and resources. > 4) Quick to Fail > > This is the idea that we have the test suite bail out on the first failure, > freeing up Jenkins and giving feedback sooner that the PR has issues. I am > not a huge fan of this as it has some downsides for developers who rely on > Jenkins to tell them what issues their work has versus running tests locally > first. I actually like this. Although I also sometimes send small patch without running tests and rely on Jenkins, I have no problems running them locally when it blows up. > 6) Reduce Support Matrix Please let's do this. I think that maximum two Ruby versions is enough (oldest and newest supported) and we could skip MySQL for PR tests (but keep it in full test suite in master/develop branches). -- Later, Lukas @lzap Zapletal -- You received this message because you are subscribed to the Google Groups "foreman-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
