I'm not Sean, but at my place down at that Notonthehighstreet.com (known to 
those within as "NOTHS"), we've been doing continuous deployment for at least 
3-4 years, and it's now on its 2nd or 3rd generation/iteration.

So I'll answer these questions based on what we do, not on what Sean does, 
obviously. :-)


On Tue, Feb 6, 2018 at 11:32 AM Ian Moss wrote:

> Were there pre-conditions as part of the auto deploy? i.e. all tests, or some 
> particular tests had to pass... did it run the full suit, or was there there 
> some detections over which particular tests covered this push to master?


At NOTHS we do GitHub flow with web hook integrations into various tools.

When a branch is pushed and a pull request is created, we create a test job 
named after the branch automatically in Jenkins. This will run the full suite 
of tests, which for our main monolith requires 20+ spot EC2 instances and takes 
25-30 minutes. We have a lot of tests (about 85% coverage), and it's a ~70k 
SLOC Rails application. Most people will not need to do as complex a dance as 
ours on that step.

Normally, whilst you're doing that, you'll be deploying your branch to QA and 
doing some checks anyway. Most stories have a manual QA check, some do not.

We also have some tasks for Rubocop checks and the like. We don't enforce style 
across the whole app, just the files you've touched in your PR - otherwise we'd 
never ship anything.

We also normally require peer review, so each PR is thrown into a Slack channel 
and people ask for a review from another dev who can read/write code in the 
same language (we have services in Ruby, Java, Go and a smattering of NodeJS 
and Python).

If all the tests are green and the PR is approved, it can be merged. If it's 
red for any reason, a github org admin (typically a senior dev or tech lead), 
can merge despite the red build. We would do this when we have a P1 and we're 
not interested in waiting for the green build (Hail Mary deploys!), or perhaps 
the Rubocop "fails" are ones we're happy to ignore on this iteration.

On merging to master, the tests run again, because we don't want to force 
rebases but there may be regressions occurring when lots of PR merges happen at 
once. So, about once every 60 seconds, whatever is new on master gets the test 
suite run, final rubocop checks, etc. and if all is good, we kick off a Jenkins 
task to deploy to production.

That task builds the Docker container, pushes it, and then shouts at the 
schedulers to deploy it, and they then roll the instances over to the new 
version. You're then deployed.

We would ideally build the docker container first, then run tests against that, 
so the build artefact we're deploying is the one that was tested, however there 
are problems with that approach I won't go into now. If you can do that 
however, do that: it's a cleaner approach and will pick up any oddness in 
Docker we just assume always magically works without fail (TBF, we've never 
really had a problem with the images).

In short, any green build of master will deploy, 24x7x52 night and day. We 
occasionally suspend deploys because a) we have a problem in the pipeline or b) 
we are under high load, and a deploy has to kill instances and bring new ones 
up, and losing capacity under exceptionally high load is never great. 

We have build screens up around the tech team and when a master build goes red, 
we're meant to go "all stop" and we figure out what just happened and triage it 
until it goes back to green. In reality, we isolate that to a small group of 
devs who actively triage. That group is a rotation which is pseudo-randomly 
assigned each sprint, and they also get to handle BAU and directing bug tickets 
to the relevant teams. :-)

Suspending CD is a manual measure taken by seniors in conversation with 
management, and often resisted on philosophical, technical or commercial 
grounds: the question is not "should we have CD turned on?" but rather "why the 
flip would you ever, ever not want to have CD? Are you crazy?".

If you do the maths, you might spot each feature takes about an hour to get to 
production, which is about right, but it's wall clock time not developer time: 
the dev is moving onto the next feature, and so there's no loss of 
productivity, and it's rare that we care about a feature landing at 13:07 
instead of 14:07.

Some might do more maths and realise this means we only do about 8 deploys/day 
to prod. 

That's true for the monolith, but increasingly new features and services are 
being built in micro-services and those have much smaller/faster test suites.

We're still standardising on a pattern around those to have consistency (to 
date it's been each team for themselves), but it's likely to be Jenkins files 
with tweaks when needed for anything that involves a docker container, and we 
have AWS deploy services being used for Lambdas et al. That'll evolve a lot 
over the next year.
 
> I presume if there's a regular commit going on, then running the full suite 
> of tests would not be  possible.


It is, you just queue them up, one after the other. They'll go green/red 
sequentially and accordingly.

> Often thought it'd be great to have a tool, that given a particular 
> class/method within a project can identify all the tests using it. Maybe 
> there's something already out there for this? Only 1 coffee in to the day, so 
> unsure.

So unit tests, you can probably do that. A well written unit test is only going 
to test the behaviour of a single class.

However, integration tests really need to test full integration. We try and do 
the "pyramid of testing" where we have a large number of unit tests, a smaller 
number of integration tests and then a tiny number at the top of full-stack 
tests checking a user journey that might involve multiple services which we run 
relatively rarely, and normally for performance profiling reasons.

For deploying a service we assume all unit tests and integration tests need to 
run for that service, but we don't run the full-stack tests, because hopefully 
any such regressions will have been spotted by manual QA anyway.

I'm currently working in a team on a new set of tools that mean we get to go 
more event-sourced as an architecture, and I'm pushing for canary deploys to be 
something we do more of. That would give us a lot more options and confidence 
when deploying new versions of services.

P.S. if anybody wants to come and work at ^^, working and living in London is 
not as horrific as you'd think, especially in leafy/sleepy Richmond in the 
extremities of West London where NOTHS calls home. We have a career page and a 
LinkedIn page. Tell them I sent you. :-)

-- 
You received this message because you are subscribed to the Google Groups 
"North West Ruby User Group (NWRUG)" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to nwrug-members+unsubscr...@googlegroups.com.
To post to this group, send an email to nwrug-members@googlegroups.com.
Visit this group at https://groups.google.com/group/nwrug-members.
For more options, visit https://groups.google.com/d/optout.

Reply via email to