Re: Pattern for multiple subservices and dynamic discovery i.e. VPN
On Thu, Aug 18, 2022 at 02:39:36PM +0200, Oliver Schad wrote: > > - It creates the appropriate service directory for this particular > > VPN config (possibly based on a template) > > - It calls s6-svlink or the like to register the new service, which > > automatically also notifies s6-svscan that a new service has been > > added > > That would just move 3 components to another level but they are > still needed: scanning existing service directories, diffing between > desired and current state and applying - so creating or removing > directories. Right. And unfortunately that scanning/comparing of current and desired states can quickly destroy the elegance of the original idea. But I believe there is a way to make this work without explicitly scanning (except for the necessary minimum) and without comparing or keeping internal state. Various features and behavior details that Laurent already added to s6 make this very doable. Please take a look at: https://github.com/docelic/s6-dyn It is a working demo I put together in the last couple hours to demonstrate the proposed model. The README covers a runnable demo / test case. After that, you could read the comments in `manager/manager.rb` to get a better description of the design decisions and alternative behaviors that are possible. (The manager is in Ruby since I saw you using it. It could be any laguage.) If you, Laurent, or the rest of the list have any comments or suggestions for improvements, please share. (The manager is slightly customized to your VPN example, but it can easily be extended and made generic.) Thanks, Davor
Re: Pattern for multiple subservices and dynamic discovery i.e. VPN
That would just move 3 components to another level but they are still needed: scanning existing service directories, diffing between desired and current state and applying - so creating or removing directories. So, diffing between desired and current state, and applying the modifications are components of a *service manager*, not a supervision suite, and it is important to maintain the distinction in order to avoid scope creep in s6. Even when a service is *not* instanced, these components are somewhat needed; it's just not noticed because their implementation over a single supervised service is trivial. But it is important to remember that the job of a supervision suite is to maintain the service in its current state (up or down), *not* to manage the wanted state or apply it. (Of course, it does provide tools to perform state transitions for longruns, but it comes with no policy on when to call these tools.) The components you want definitely have their place in s6-rc; but in the meantime, they can also be scripted on top of regular s6 if you have a good modelization for implementing instances, which I will add in the near future. I see there a problem with multiple dynamic services. I'm not sure about concurrency behaviour of updating processes in the service directory. Maybe Laurent can explain problems in that area, if they exist. s6 manages processes and every supervised process needs its own service directory. There will be as many service directories as they are instances. (Some components of a template service directory can of course be reused.) So there's no concurrency issue; however, the instance management tool I'm thinking of could adopt various updating methods depending on what you want. Best effort? Clean shutdown, service replacement, then firing up of the new service's instances? Rolling upgrade across the instances? These policies all have their uses. I'm not sure how complex the supervision itself is - however I would love to solve the problem without doing supervision on my own. I thought about your approach as well but it really depends how resilient an update process is. It will definitely be resilient, but there are several ways to implement it, see above. -- Laurent
Re: Pattern for multiple subservices and dynamic discovery i.e. VPN
On Thu, 18 Aug 2022 06:04:25 -0400 Davor Ocelic wrote: > On Thu, Aug 18, 2022 at 11:32:30AM +0200, Oliver Schad wrote: > How about an approach that would be more tightly integrated > with s6? For example: > > > - we need an scanning component for the desired state of running > > instances (something like 'find /etc/openvpn -name "*conf"') > > Right, the scanning component would be the key part. Ideally it > would use inotify so that it doesn't have to poll and also it would > get the type of event automatically (file created or file deleted). > > After it sees that a change in a directory has occurred (let's say a > file was added), then it does the following: > > - It creates the appropriate service directory for this particular > VPN config (possibly based on a template) > - It calls s6-svlink or the like to register the new service, which > automatically also notifies s6-svscan that a new service has been > added That would just move 3 components to another level but they are still needed: scanning existing service directories, diffing between desired and current state and applying - so creating or removing directories. I see there a problem with multiple dynamic services. I'm not sure about concurrency behaviour of updating processes in the service directory. Maybe Laurent can explain problems in that area, if they exist. I'm not sure how complex the supervision itself is - however I would love to solve the problem without doing supervision on my own. I thought about your approach as well but it really depends how resilient an update process is. Best Regards Oli -- Automatic-Server AG • Oliver Schad Geschäftsführer Hardstr. 46 9434 Au | Schweiz www.automatic-server.com | oliver.sc...@automatic-server.com Tel: +41 71 511 31 11 | Mobile: +41 76 330 03 47 pgpObylnQ62PO.pgp Description: OpenPGP digital signature
Re: Pattern for multiple subservices and dynamic discovery i.e. VPN
- we need an scanning component for the desired state of running instances (something like 'find /etc/openvpn -name "*conf"') - we need an scanning component for the current state in process list - we need a diffing component - we need a state applier component That sounds very much like what is planned for s6-rc v1, so I think you will like it when it drops - but it won't be in the near future. However, I have some ideas for new s6 tools that wouldn't follow this model directly but would make it easy for users to create and delete new instance models, and add/remove instances - so your components could be implemented over these tools by simple shell scripts. I'll try to work on that soon. -- Laurent
Re: Pattern for multiple subservices and dynamic discovery i.e. VPN
On Thu, Aug 18, 2022 at 11:32:30AM +0200, Oliver Schad wrote: Hey, How about an approach that would be more tightly integrated with s6? For example: > - we need an scanning component for the desired state of running > instances (something like 'find /etc/openvpn -name "*conf"') Right, the scanning component would be the key part. Ideally it would use inotify so that it doesn't have to poll and also it would get the type of event automatically (file created or file deleted). After it sees that a change in a directory has occurred (let's say a file was added), then it does the following: - It creates the appropriate service directory for this particular VPN config (possibly based on a template) - It calls s6-svlink or the like to register the new service, which automatically also notifies s6-svscan that a new service has been added > - we need an scanning component for the current state in process list > - we need a diffing component > - we need a state applier component I hope these steps would not be necessary, since each .conf file would be represented as a proper service, and would be managed by s6 natively. Later, if/when a conf file is removed, the vpn-manager component would do the opposite and call s6-svunlink. As a final safeguard, in the `finish` script for such a service, the script would check whether its original VPN config file is still present. If it is not, it would trigger a permanent failure event for the service. > A process tree would look like /usr/bin/s6-svscan s6-supervise vpn-manager s6-supervise openvpn foo1.conf s6-supervise openvpn foo2.conf (The individual openvpn services could also have a dependency on vpn-manager.) Thanks, Best regards, Davor > On Wed, 17 Aug 2022 11:04:50 + > "Laurent Bercot" wrote: > > > > > > >I'm looking for a pattern to solve a problem, where you have to > > >discover dynamically the services you have to start. > > > > > >Examples could be VPN configurations, where you discover the > > >configuration files and start for every file an instance of the VPN > > >service. > > > > Hi Oliver, > > > > Dynamic instantiation is a real pain point - it's an often requested > > feature, but it's surprisingly hard to make it work correctly and > > safely in a supervision scheme. Supervision works very well in static > > environments, but dynamic discovery is at odds with the architecture. > > > > I have a few ideas to mitigate that and help people create instanced > > services. Instantiation is a planned feature of the future s6-rc v1 > > but it's still a ways away; I am also thinking of adding tools to help > > people handle instances with regular s6, amd they may come in the near > > future, but there are currently no such helpers, sorry. > > > > -- > > Laurent > > > > > > -- > Automatic-Server AG • > Oliver Schad > Geschäftsführer > Hardstr. 46 > 9434 Au | Schweiz > > www.automatic-server.com | oliver.sc...@automatic-server.com > Tel: +41 71 511 31 11 | Mobile: +41 76 330 03 47
Re: Pattern for multiple subservices and dynamic discovery i.e. VPN
Dear Laurent, thanks a lot for your quick response. I understand, that dynamic discovery needs some thoughts to implement in an easy and flexible way. Just to understand the requirements, if we want to implement that today: - we need an scanning component for the desired state of running instances (something like 'find /etc/openvpn -name "*conf"') - we need an scanning component for the current state in process list - we need a diffing component - we need a state applier component A component could be a shell function. Applying state would mean to fork and exec. A process tree would look like /usr/bin/s6-svscan s6-supervise vpn-manager vpn-manager openvpn foo1.conf openvpn foo2.conf Best Regards Oli On Wed, 17 Aug 2022 11:04:50 + "Laurent Bercot" wrote: > > > >I'm looking for a pattern to solve a problem, where you have to > >discover dynamically the services you have to start. > > > >Examples could be VPN configurations, where you discover the > >configuration files and start for every file an instance of the VPN > >service. > > Hi Oliver, > > Dynamic instantiation is a real pain point - it's an often requested > feature, but it's surprisingly hard to make it work correctly and > safely in a supervision scheme. Supervision works very well in static > environments, but dynamic discovery is at odds with the architecture. > > I have a few ideas to mitigate that and help people create instanced > services. Instantiation is a planned feature of the future s6-rc v1 > but it's still a ways away; I am also thinking of adding tools to help > people handle instances with regular s6, amd they may come in the near > future, but there are currently no such helpers, sorry. > > -- > Laurent > -- Automatic-Server AG • Oliver Schad Geschäftsführer Hardstr. 46 9434 Au | Schweiz www.automatic-server.com | oliver.sc...@automatic-server.com Tel: +41 71 511 31 11 | Mobile: +41 76 330 03 47 pgpdOkW5Fkkak.pgp Description: OpenPGP digital signature
Re: Pattern for multiple subservices and dynamic discovery i.e. VPN
I'm looking for a pattern to solve a problem, where you have to discover dynamically the services you have to start. Examples could be VPN configurations, where you discover the configuration files and start for every file an instance of the VPN service. Hi Oliver, Dynamic instantiation is a real pain point - it's an often requested feature, but it's surprisingly hard to make it work correctly and safely in a supervision scheme. Supervision works very well in static environments, but dynamic discovery is at odds with the architecture. I have a few ideas to mitigate that and help people create instanced services. Instantiation is a planned feature of the future s6-rc v1 but it's still a ways away; I am also thinking of adding tools to help people handle instances with regular s6, amd they may come in the near future, but there are currently no such helpers, sorry. -- Laurent
