It looks like someone was able to spare me a trip back to Manchester, and
the discourse was good.
I'd like to weigh in and propose a question… "why do we do CI/CD?"
I've seen (and am currently enjoying the spoils of) companies pour hundreds
of thousands of pounds into CI/CD setups on Docker or K8s, with or without
service meshes, with monitoring, and alerting, and dashboards and status
pages and strictly regulated pipelines and etc, etc, etc..... and I wonder
why we do it.
There is for sure an element of cargo-culting. For anyone unfamiliar with
the term “.... describes software development organizations that *attempt
to emulate more successful development houses, either by slavishly
following a software development process without understanding the
reasoning behind it*, or by attempting to emulate a commitment-oriented
development approach by mandating the long hours and unpaid overti.....” (
I think there's a general assumption that because without CI/CD you end up
with "risky deploys" that non-technical stake holders
that "never [explicitly] deploying" must be better... (also they tend to
glaze over their eyes and let developers do whatever we want, if we even
bother to ask).
I'm presently reading "Principles of product development flow
this book is* way* too expensive, and it's written in a pretty hard to
follow way, but it makes sane links between manufacturing, and the origins
of the "JIT" (just in time) delivery (aka agile) and drawing parallels
between many physical production processes which successfully map to the
intangible assets we produce when writing code.
One of the main things that stood out for me was that the author explicitly
draws on a concept of "WIP", e.g the unfinished car that currently is being
pushed from stations one, then two then three on the factory floor (the
ticket, being pushed from staging to qa, and eventually production) as
being a huge area where "invisible" losses are made in manufacturing. The
book happens to mention "agile" manufacturing, where on a car production
line, when you deliver parts to each station "just in time" you no longer
have to have parts-storage colocated with each station, so your stations
can be smaller, and closer together leading to less time pushing partly
finished widgets from one station to the next, which is actually where the
benefits of MVP, JIT and CI/CD start to become apparent *in the physical
I think the same applies to knowledge work, we still produce things that
"rot" that occurs if, say your "station 1" and "station 2" are separated by
large geographies, and you have a non trivial travel time, so, if you need
a batch of widgets from station one for station two you have to preempt the
demand by overproducing, or risk that station two can't produce anything,
if you don't have good quality widgets coming out of station one then the
half-life of bad quality parts is very high until QA, etc.
Fortunately because we "lease" nearly everything we produce to end-users
via short-lived rentals over the internet for a dozen milliseconds at a
time, if we introduce a low quality part into a product, it's easily
replaced retrospectively, a luxury that Boeing engineers don't enjoy.
He also floats a nice "waterfall vs agile" example of how to setup weight
distribution in plane manufacture, if you are Boening and you don't know
how to allocate things up-front, if you setup the wrong constraints (each
team gets 10% of allowance) you over and under incentivise people to waste
or engineer around weight/limits. In other words the product owner produced
a low quality decision (low quality parts) into the manufacturing cycle.
This would be roughly equivalent to putting a bad chassis in all the cars
you produce, or choosing the wrong architectural paradigm.
Thinking of what the PO/PM outputs as "consumables" for the manufacturing
line is a bit of a stretch and off-topic for this, but I think it helps to
keep in mind. In the Boeing example if the PM produces a low quality
decision on distribution the costs are astronomical and unfixable. (there's
some ways to address that, by setting up a 2nd market for "weight" and
allowing teams to trade/barter, etc, but that's not relevant here, but it's
a nice example of a PM kicking a decision down the road by setting up an
environment that produces the desired outcomes, rather than having to
proscribe them up-front).
I'm at risk of waffling, but I think the CI/CD thing is either:
- an attempt to minimize the WIP/in-flight work, if it gets to
production quicker, we can get feedback quicker, and that leads to...
- ...or re-orientate (educate?) the perspective of the PO/PM to embrace
the fact that we can be "agile" and constantly ship tiny updates after the
fact to customers in near realtime might be the goal.
So maybe those two bullet points are two sides of the same coin, and it's
all about faster feedback as proscribed in agile methodologies. I wish I
I don't know how much of this I believe, but I'm not sure you can put a
financial value on CI/CD and claim that it causes a cost saving. I kind of
feel like it's a simple cost-centre, and not an investment, at least I lack
the business chops to adequately value it.
Aside from CI/CD and branch based deployment or blue/green and/or
microservices vs. monoliths, I'd like to invite everyone on the list to
look into service meshes such as linkerd and Envoy. If you make some
concessions (forwarding of headers from client to within micro-services on
the backend) they allow you to do things like running *one* version of your
app in production, saving the cost of a staging environment and deploying a
single microservice *along side* the currently active production one, in
the same infrastructure, and selectively routing a percentage, or a
specific user's traffic. (e.g Sean sets "X-Routing-For-SeanµService:
sean-v2' in his browser and it it honoured by the routing mesh and he can
test his service live, in-situ against real traffic and data)
Either way, thoughts are welcome, sorry for the wall of text :) Happy
weekend one and all.
+49 (0) 170 298 5667
On 9 February 2018 at 16:16, doug livesey <biot...@gmail.com> wrote:
> That's all rather inspiring, thank you!
> I'll look forwards to your posts, Sean (and sorry again for trying to
> volunteer you!), if you get chance to make them.
> And thanks for that in-depth coverage, Paul, I shall go back and read it
> more slowly, now, with my mind's digestive juices switched on.
> Immediate takeaway: I'm deploying from the stone age! :)
> On 8 February 2018 at 15:48, John Cleary <j...@createk.io> wrote:
>> On Sunday, 4 February 2018 14:15:38 UTC, doug livesey wrote:
>> > I'm really sorry, I can never remember anybody's name.
>> > There was a chap at the last NWRUG who was telling a bunch of us about
>> how his workplace had recently shifted to auto-deploying anything committed
>> to master.
>> > It was a system organised with kubernetes, and there was CI and tests
>> running on the live stack and everything.
>> > Anyway, it was really interesting and I thought it might make a cool
>> > I wasn't going to try to nominate anybody else until I could suggest an
>> idea for a talk I'd do, too, but I'm coming up with nothing for me so far!
>> > Anyway, seeing as I can never remember anyone's name, the gentleman in
>> question can always plead ignorance if he reads this and doesn't feel
>> inclined to sharing it with the group. :)
>> Whilst we don't use Kubernetes, we are using a fairly unusual deployment
>> approach at Createk for our client which is inspired by Kanban and giving
>> control back to the Product Owner. We have built some of our own tools
>> (open source of course) which support our approach and I was going to give
>> a talk at some point when the tools were in a more ready state.
>> However, I would be happy enough to give a talk based on what we have
>> now, plus where we are planning to go with our CI and Deployment Pipeline.
>> It sounds like many of you have some interesting and diverse perspectives
>> and experiences on the matter, so if I left some "hooks" in the structure
>> of the talk to allow everyone to contribute ideas I think we could end up
>> with a really interesting discussion.
>> You received this message because you are subscribed to the Google Groups
>> "North West Ruby User Group (NWRUG)" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to nwrug-members+unsubscr...@googlegroups.com.
>> To post to this group, send an email to email@example.com.
>> Visit this group at https://groups.google.com/group/nwrug-members.
>> For more options, visit https://groups.google.com/d/optout.
> You received this message because you are subscribed to the Google Groups
> "North West Ruby User Group (NWRUG)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to nwrug-members+unsubscr...@googlegroups.com.
> To post to this group, send email to firstname.lastname@example.org.
> Visit this group at https://groups.google.com/group/nwrug-members.
> For more options, visit https://groups.google.com/d/optout.
You received this message because you are subscribed to the Google Groups
"North West Ruby User Group (NWRUG)" group.
To unsubscribe from this group and stop receiving emails from it, send an email
To post to this group, send an email to email@example.com.
Visit this group at https://groups.google.com/group/nwrug-members.
For more options, visit https://groups.google.com/d/optout.