Re: [Nix-dev] hotswappable self managing services in nix

[Tomasz Czyż]

Stewart:
check my comment for configuration reload, if you want this on service
level, https://github.com/NixOS/nixpkgs/issues/1988#issuecomment-247779639
could be helpful for you (at least I'm implementing such things that way).



2016-11-28 10:49 GMT+00:00 zimbatm :

> For process-level graceful restarts see https://github.com/zimbatm/
> socketmaster and https://github.com/pusher/crank . Those could be
> integrated into the activation script.
>
> On Mon, 28 Nov 2016 at 09:33 zimbatm  wrote:
>
>> Hi Stewart,
>>
>> In a HA setup availability is generally achieved on a network level
>> instead of system level. Typically you would have two hotswappable
>> load-balancers that distribute the traffic to multiple instances of your
>> service boxes. In that context is doesn't matter how processes are being
>> restarted because the load-balancer will automatically detect unresponsive
>> machines and route the traffic accordingly. It's also handy because it
>> allows to restart the machines in the event where the kernel needs an
>> upgrade. In that setup I suppose you can think of each machine as being one
>> Erlang OTP "process" and the network the "message-passing".
>>
>> One responsibility of the service in that setup is to shutdown properly
>> to avoid unnecessary disruption of service. Mainly when the process gets
>> the SIGTERM signal it should close the listening socket (so the
>> load-balancer can route new incoming connections to a different machine)
>> and then drain the existing client connection gracefully. It shouldn't stop
>> all at once but let the clients disconnect when they are done with their
>> sessions (and optionally signal them to go away if the protocol supports
>> it).
>>
>> A last thing regarding this approach: generally you need a way to control
>> the deploys; if all the service boxes are being upgraded at the same time
>> then the load-balancer doesn't have anywhere to route the traffic to. It's
>> also something desirable to have to do blue/green deployments.
>>
>> I need to stop there for now but I also have a similar design answer on
>> the system level where processes get replaced gracefully.
>>
>> Cheers,
>> z
>>
>> On Sun, 27 Nov 2016 at 04:33 stewart mackenzie 
>> wrote:
>>
>> 9 9s not unheard of in these circles, Google uptimes are a joke not
>> worthy of mention.
>>
>> There are systems that have been running for some 40 odd years in
>> production that factor in changes to legal banking regulations, hardware,
>> business logic etc. Erlang has a system called the Ericsson AXD301 which
>> has achieved this time frame.
>>
>> Just because Nixos hasn't been around that long doesn't mean it can't
>> have the primitives to allow for such feats. Its these primitives I'm
>> enquiring about.
>>
>> So let's use a new, less controversial figure of 5 9s and keep on topic.
>>
>> The thing is, we're designing this system so that its governed by nix
>> don't necessarily have to depend heavily on the runtime - I really don't
>> want to go down the imperative route, by introducing imperative language
>> concepts into our declarative language which is managed by another
>> declarative language (nix). Besides just bringing in a single component
>> with an OS Dependency demands we manage this change from nix level.
>>
>> We currently have a hack in place, that will resolve dependencies and
>> give us a path to load a correctly compiled shared object into memory:
>> https://github.com/fractalide/fractalide/blob/master/
>> components/nucleus/find/component/src/lib.rs#L43 nasty and cringe worthy
>> I know.
>>
>> Thanks for your pointer, I'll take a look at these activation scripts.
>>
>> Maybe this hack is the answer, and confine the dynamism to an ssh login
>> al a Erlang style...
>> ___
>> nix-dev mailing list
>> nix-dev@lists.science.uu.nl
>> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>>
>>
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
>


-- 
Tomasz Czyż
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] hotswappable self managing services in nix

[zimbatm]

For process-level graceful restarts see
https://github.com/zimbatm/socketmaster and https://github.com/pusher/crank
. Those could be integrated into the activation script.

On Mon, 28 Nov 2016 at 09:33 zimbatm  wrote:

> Hi Stewart,
>
> In a HA setup availability is generally achieved on a network level
> instead of system level. Typically you would have two hotswappable
> load-balancers that distribute the traffic to multiple instances of your
> service boxes. In that context is doesn't matter how processes are being
> restarted because the load-balancer will automatically detect unresponsive
> machines and route the traffic accordingly. It's also handy because it
> allows to restart the machines in the event where the kernel needs an
> upgrade. In that setup I suppose you can think of each machine as being one
> Erlang OTP "process" and the network the "message-passing".
>
> One responsibility of the service in that setup is to shutdown properly to
> avoid unnecessary disruption of service. Mainly when the process gets the
> SIGTERM signal it should close the listening socket (so the load-balancer
> can route new incoming connections to a different machine) and then drain
> the existing client connection gracefully. It shouldn't stop all at once
> but let the clients disconnect when they are done with their sessions (and
> optionally signal them to go away if the protocol supports it).
>
> A last thing regarding this approach: generally you need a way to control
> the deploys; if all the service boxes are being upgraded at the same time
> then the load-balancer doesn't have anywhere to route the traffic to. It's
> also something desirable to have to do blue/green deployments.
>
> I need to stop there for now but I also have a similar design answer on
> the system level where processes get replaced gracefully.
>
> Cheers,
> z
>
> On Sun, 27 Nov 2016 at 04:33 stewart mackenzie  wrote:
>
> 9 9s not unheard of in these circles, Google uptimes are a joke not worthy
> of mention.
>
> There are systems that have been running for some 40 odd years in
> production that factor in changes to legal banking regulations, hardware,
> business logic etc. Erlang has a system called the Ericsson AXD301 which
> has achieved this time frame.
>
> Just because Nixos hasn't been around that long doesn't mean it can't have
> the primitives to allow for such feats. Its these primitives I'm enquiring
> about.
>
> So let's use a new, less controversial figure of 5 9s and keep on topic.
>
> The thing is, we're designing this system so that its governed by nix
> don't necessarily have to depend heavily on the runtime - I really don't
> want to go down the imperative route, by introducing imperative language
> concepts into our declarative language which is managed by another
> declarative language (nix). Besides just bringing in a single component
> with an OS Dependency demands we manage this change from nix level.
>
> We currently have a hack in place, that will resolve dependencies and give
> us a path to load a correctly compiled shared object into memory:
> https://github.com/fractalide/fractalide/blob/master/components/nucleus/find/component/src/lib.rs#L43
> nasty and cringe worthy I know.
>
> Thanks for your pointer, I'll take a look at these activation scripts.
>
> Maybe this hack is the answer, and confine the dynamism to an ssh login al
> a Erlang style...
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] hotswappable self managing services in nix

[Graham Christensen]

First: 9 9s means that for all 13(?) years NixOS has existed, there would
be just over 4 seconds of down time.

Any real system would use many redundant machines to accomplish this.

Google strives for 99.95%.

Second: I think more generally what you're looking for is graceful
restarting. NixOS has ~some~ support for this, but I believe it isn't
implemented too thoroughly and you would have to carefully read the
activation scripts to find it.

You lose the some of the determinism NixOS provides, since you are now
depending on erlang and all to implement it correctly.

I believe corner cases exist, like when the version of Erlang changes

Graham

On Sat, Nov 26, 2016 at 10:23 PM stewart mackenzie 
wrote:

> Let me rephrase the question.
>
> Is it possible to build a highly available system on nix such that SLA
> uptimes of 9 9s. (99.999 % uptime) are achievable?
>
> If so what primitives does nix offer to allow this to happen?
>
> Such primitives should be: the ability for nix to invoke the
> hotswapping capabilities of a live running system.
>
> Now just connecting to Erlang and making changes isn't good enough.
> Why? - nix cannot see those "imperative" changes. Git vs svn, git
> takes the global view and svn the file view. Git got it right, you
> need to factor in the entire repo's changes. Nix gets it right too by
> looking at this from an entire system perspective, so I'm seeking a
> way to introduce change to a live fractalide system from a nix level,
> though it mustn't be a "connect to fractalide via ssh and change it
> there" approach. Making changes to Erlang via ssh under Nix management
> is like having a declarative system then introducing imperative
> mechanisms.
>
> I need a formalized method to ensure global determinism yet be able to
> make changes to a live running system from nix.
>
> Are there any Erlangers here who achieve huge uptimes on nix? If so
> how do you do it? How do you make use of Erlang's hotswapping
> mechanisms yet retain the lovely system wide determinism nix offers.
>
> kr/sjm
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
http://lists.science.uu.nl/mailman/listinfo/nix-dev