I like what you're saying Adam, you pretty much describe our environment and our solution.
We are working to move more features into unit tests, increase use of stubs within features and to change the architecture to reduce close-coupling between components to get our test time down and reduce our crippling dependencies. In the meantime we have hit it with pragmatic and unsophisticated brute force - we have broken our builds up into many smaller builds (broken up by application or context) and build using a feature branch on a build server which kicks off all necessary builds in parallel on different servers. Each server is completely independent and deterministic, creating the database from scratch and running bundle install before every build. We then work on any broken builds in isolation locally and re-submit when ready. It's not cheap (we have 15 VM's) but it's very effective and devs that were once browsing the web while 'raking' can now be productive while waiting for relatively fast feedback. I still think that if you need to maintain the ability to refactor aggressively and confidently you need excellent coverage at the feature or integration level. Unit tests with mocked or stubbed dependencies will not really help you find regressions when you are refactoring across many classes and relying on manual testing to find regressions is not an option if you want to run continuous delivery so i think build time is always likely to be an issue and simple parallelisation solutions will always be necessary for big projects. On Monday, November 26, 2012 10:28:38 AM UTC+11, Adam wrote: > > I know this opinion is probably going to be unpopular, but I just want to > put out it there: > > If you need to parallellize your build, then your problems getting > parallel building working are certainly not your biggest problem. > > Even if you get a parallel build working, and it is certainly doable. > Where is that going to go? > > Step 1 people will stop doing a pre-commit build - The fact that you need > a parallel build means that it is almost certainly prohibitively time > consuming for a dev to do one. > > Step 2, people start caring less about build efficiency. Why care, you > can't run it locally anyway. The net effect of this is that your parallel > build will eventually start taking as long or longer than your old serial > one. > > Step 3 big refactorings become super scary, prohibitively time consuming > or simply not doable. This is because you can't quickly re-run your specs > anymore. A core refactor to be tested will have to go back and forth to the > build server to be verified (probably many times). That means you are going > to need another set of parallel builders focussing on another branch so > that you don't go screwing up the deployable master. > > Step 4, the super smart guy/gal who built your parallel build system > leaves the team and now you are in a world of pain :-) But seriously, a lot > of effort will go into understanding, maintaining and expanding your build. > > Every project I have worked on that required a parallel build had 3 common > threads: > > > 1. Lots of Factory based (database dependent) unit-level specs > 2. Large amounts of what should really be orthogonal, unit level specs > being done in acceptance tests > 3. A large application with more than one candidate application > extraction > > > I have been privileged to work with some super smart people, and the > projects suffering from the problems above were no exception, so it wasn't > that they were poor developers or bad designers. Generally early > extractions don't get done when there are tight deadlines. By the time you > are feeling that your build is too slow and it is starting to get in the > way of productivity, your application is already probably too big. At this > point I recommend trying to identify groups of functionality that are > candidates for an extraction. If they can operate completely independently > and be deployed completely independently then that is great. Worst case > scenario look to extract large pieces out as mountable engines. > > There are certainly non-trivial issues with maintaining multiple, > interdependent apps, but all-in-all I think teams will maintain greater > flexibility and development speed if effort is put into this kind of > modularisation rather than ever more complicated build structures designed > to stop the team needing to address the design issue. > > > Adam Boas > e: [email protected] <javascript:> > m: +61 457 741 117 > > > > On 26/11/2012, at 9:37 AM, Thomas Egret <[email protected] <javascript:>> > wrote: > > I have been using parallel_tests as well and it's very awesome however > sometimes the tests suite just run smoothly and sometimes just fails so we > had to deactivate it for the moment (wonder if you use it as well on > Travis). > > Wish some people could give some hints regarding how to ensure my tests to > pass using parallel_tests. > > > 2012/11/26 James Healy <[email protected] <javascript:>> > >> A previous project I worked on used Jenkins with a vagrant plugin to run >> concurrent tests in isolated VMs. >> >> It took a bit of setting up, but once it got going it was amazing and >> dropped our build time from >20 minutes to <5 minutes. >> >> The downside was that it became tricker to understand the "state of the >> build" at a glance. Instead of checking 1 project, you had to check 6 (unit >> tests, acceptance tests 1, acceptance tests 2, etc). >> >> We also tried parallel_tests for a while and it mostly worked, but we ran >> into a few issues with parallel acceptance specs stepping on each others >> toes. >> >> James >> >> >> >> On 22 November 2012 13:12, Sebastian Porto <[email protected]<javascript:> >> > wrote: >> >>> Hi Guys >>> >>> There was a discussion the other day about CI server and it was mention >>> several times how good it is to run the tests in parallel. I am intrigued >>> by this and will like to implement this. But I don't know where to start. >>> >>> What is your approach for running CI test in parallel? We are using >>> TeamCity at the moment, were using Jenkins before. And Rspec. >>> >>> Thanks >>> Sebastian >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Ruby or Rails Oceania" group. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msg/rails-oceania/-/HqhReVKyNx0J. >>> To post to this group, send email to [email protected]<javascript:> >>> . >>> To unsubscribe from this group, send email to >>> [email protected] <javascript:>. >>> For more options, visit this group at >>> http://groups.google.com/group/rails-oceania?hl=en. >>> >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Ruby or Rails Oceania" group. >> To post to this group, send email to [email protected]<javascript:> >> . >> To unsubscribe from this group, send email to >> [email protected] <javascript:>. >> For more options, visit this group at >> http://groups.google.com/group/rails-oceania?hl=en. >> > > > -- > You received this message because you are subscribed to the Google Groups > "Ruby or Rails Oceania" group. > To post to this group, send email to [email protected]<javascript:> > . > To unsubscribe from this group, send email to > [email protected] <javascript:>. > For more options, visit this group at > http://groups.google.com/group/rails-oceania?hl=en. > > > -- You received this message because you are subscribed to the Google Groups "Ruby or Rails Oceania" group. To view this discussion on the web visit https://groups.google.com/d/msg/rails-oceania/-/9G7UoLydrrYJ. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/rails-oceania?hl=en.
