Re: Gradual installation of debian packages

2016-08-06 Thread Amos Shapira
I see. Valid points. Whenever you break a production site - do you try to add a test which simulates the parameters of the breakage? It sounds to me like some sort of an image versioning could still help here, that way you can really "roll back" (actually boot to a previous version of the image)

Re: Gradual installation of debian packages

2016-08-06 Thread Elazar Leibovich
It's radio antenna. It is of course tested before to some extent, in a "staging" environment. But since the physical environment varies, and sometimes antenna related parameters change between releases (e.g., duration of receive time), it is not easy to know you're not breaking something for

Re: Gradual installation of debian packages

2016-08-05 Thread Amos Shapira
What kind of hardware is this that's connected to the servers, and what does the software do that you can't test before installing on production servers? On 6 August 2016 at 02:14, Elazar Leibovich wrote: > All real servers, with custom hardware attached, geographically >

Re: Gradual installation of debian packages

2016-08-05 Thread Elazar Leibovich
All real servers, with custom hardware attached, geographically distributed across the planet. Real people actually use the hardware attached to this computers, and it's not obvious to test whether or not it failed. The strategy therefor is, deploy randomly to small percentage of the machines,

Re: Gradual installation of debian packages

2016-08-05 Thread Amos Shapira
What provisioning tools do you use to manage these servers? Please tell me you aren't doing all of this manually. Also what's your environment? All hardware servers? Any virtualisation involved? Cloud servers? Reading your question it feels like you are setting yourself up to fail instead of

Re: Gradual installation of debian packages

2016-08-03 Thread Elazar Leibovich
How exactly you connect to the server is not in the scope of the discussion, and I agree that ansible is a sensible solution. But what you're proposing is to manually update the package on a small percent of the machines. Manual solution is fine, but I would like to hear experience of people who

Re: Gradual installation of debian packages

2016-08-03 Thread Evgeniy Ginzburg
Hello. I'm assuming that you have paswordless ssh to the servers in question as root. Also I assume that you don't use central management/deployment software (ansible/puppet/chef) In similar cases I usully use parallel-ssh (gnu-parallel is another alternative). First stage install the package