[systemd-devel] systemd and process migration
Using CRIU I have been migrating processes from one system to another for the last few months (even years). I am now interested in migrating processes under systemd's control. Before starting to look how to make it work I wanted to know if there has been any discussion if and how systemd and CRIU can work together? Dumping a process under systemd's control should be no problem. After criu has dumped the process (and killed it) systemd should know that it does not need to restart the process, but even if the process is restarted by systemd it does not hurt the process migration. The interesting part happens on the system where the process wants to be migrated to. After the process has been dumped and transfered from the source system to the destination system it needs to be restarted. So far this works with different tested processes (postgresql is my current test process). If I want to restore the process as a systemd child process I have the problem that the process would have to be re-parented to systemd after restarting which is not possible in Linux. Therefore I need the help of systemd. My plan now would be to transfer the process to the destination system and tell systemd I want to restore a process which should be under systemd's control after the restore has completed. Therefore systemd needs to run criu with the option to restore the new process as a criu sibling. Thus systemd would be the correct parent process. Is this something which would be useful to integrate into systemd? I have some ideas how this could be implemented and how the user interfaces could look like. Before going in too much detail I want to find out if this is a direction which makes sense. Adrian ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] systemd and process migration
On Mon, 20.04.15 14:33, Jóhann B. Guðmundsson (johan...@gmail.com) wrote: On 04/20/2015 01:58 PM, Lennart Poettering wrote: systemd has no provisions for anything like this and I have doubts it really should have. I mean, normal programs retain ccess to the cgroup tree, so you could readd whatever you restore back in the cgroups, but that's hardly sufficient to make systemd aware of a process like this properly. Arguably systemd should have this functionality ( along with quite few others like HA etc ) built in ( not using CRIU ) since you want to live migrate at least native containers between hosts. I thought fancy enterprise features like this was on hold until networkd was ready ? This is not an enterprise feature. It's a promise one cannot keep. We will not add code to systemd that works often but not always, and CRIU is certainly of that kind. Sorry, Lennart -- Lennart Poettering, Red Hat ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] systemd and process migration
On 04/20/2015 02:35 PM, Lennart Poettering wrote: On Mon, 20.04.15 14:33, Jóhann B. Guðmundsson (johan...@gmail.com) wrote: On 04/20/2015 01:58 PM, Lennart Poettering wrote: systemd has no provisions for anything like this and I have doubts it really should have. I mean, normal programs retain ccess to the cgroup tree, so you could readd whatever you restore back in the cgroups, but that's hardly sufficient to make systemd aware of a process like this properly. Arguably systemd should have this functionality ( along with quite few others like HA etc ) built in ( not using CRIU ) since you want to live migrate at least native containers between hosts. I thought fancy enterprise features like this was on hold until networkd was ready ? This is not an enterprise feature. It's a promise one cannot keep. We will not add code to systemd that works often but not always, and CRIU is certainly of that kind. We will not add code to systemd that works often but not always you need to explain this a bit further since we might have different perception on this since from my perception such code has been added to systemd in more then one occasion anyway I was not advocating for the use/inclusion of CRIU but something built in. We should be able to support non live migration out of the box if live migration is a pandora's box or are you opposed to that as well? JBG ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] systemd and process migration
On Mon, 20.04.15 14:59, Adrian Reber (adr...@lisas.de) wrote: Using CRIU I have been migrating processes from one system to another for the last few months (even years). I am now interested in migrating processes under systemd's control. Before starting to look how to make it work I wanted to know if there has been any discussion if and how systemd and CRIU can work together? Not that I was aware of. Personally I do not really believe in the idea of CRIU. I mean, I can see that people want the functionality, but given the amount of context processes maintain about the system I find it quite illusionary that this can work for any but the most simple programs. Dumping a process under systemd's control should be no problem. After criu has dumped the process (and killed it) systemd should know that it does not need to restart the process, but even if the process is restarted by systemd it does not hurt the process migration. The interesting part happens on the system where the process wants to be migrated to. After the process has been dumped and transfered from the source system to the destination system it needs to be restarted. So far this works with different tested processes (postgresql is my current test process). If I want to restore the process as a systemd child process I have the problem that the process would have to be re-parented to systemd after restarting which is not possible in Linux. Therefore I need the help of systemd. My plan now would be to transfer the process to the destination system and tell systemd I want to restore a process which should be under systemd's control after the restore has completed. Therefore systemd needs to run criu with the option to restore the new process as a criu sibling. Thus systemd would be the correct parent process. Is this something which would be useful to integrate into systemd? I have some ideas how this could be implemented and how the user interfaces could look like. Before going in too much detail I want to find out if this is a direction which makes sense. systemd has no provisions for anything like this and I have doubts it really should have. I mean, normal programs retain ccess to the cgroup tree, so you could readd whatever you restore back in the cgroups, but that's hardly sufficient to make systemd aware of a process like this properly. Lennart -- Lennart Poettering, Red Hat ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] systemd and process migration
On 04/20/2015 02:49 PM, Lennart Poettering wrote: On Mon, 20.04.15 14:43, Jóhann B. Guðmundsson (johan...@gmail.com) wrote: I thought fancy enterprise features like this was on hold until networkd was ready ? This is not an enterprise feature. It's a promise one cannot keep. We will not add code to systemd that works often but not always, and CRIU is certainly of that kind. We will not add code to systemd that works often but not always you need to explain this a bit further since we might have different perception on this since from my perception such code has been added to systemd in more then one occasion anyway I was not advocating for the use/inclusion of CRIU but something built in. We should be able to support non live migration out of the box if live migration is a pandora's box or are you opposed to that as well? Well, sure, it would be nice, but I also believe it is not realistic to support. For example, if you have network protocols you can probably still live with their connections being severed during migration, but this is really not the case for local IPC connections, like bus connections. Restoring the state for that is an immense amount of work, and I am pretty sure that it is not desirable to add code for this to all IPC systems like this just because of CRIU. Also, I doubt this really is such an important thing to have. I mean, containers are not VMs, they are easily instantiated and shut down. You can socket actviate them, and have them exit-on-idle. If you accept that containers are a much more volatile, dynamic thing that VMs then live migration becomes much less attractive. There are valid migration cases beside live migration ones ( for example reallocation to a different host(s) due to resources etc ) Think in terms of shutting down and disable container on host A, send relevant units ( including any network related ones ) along with the underlying filesystem snapshot, in the case of btrfs it would be using btrfs send/receive feature to host B and start it there. What I'm wondering if you would be opposed to supporting those use cases ? JBG ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] systemd and process migration
On Mon, 20.04.15 14:43, Jóhann B. Guðmundsson (johan...@gmail.com) wrote: I thought fancy enterprise features like this was on hold until networkd was ready ? This is not an enterprise feature. It's a promise one cannot keep. We will not add code to systemd that works often but not always, and CRIU is certainly of that kind. We will not add code to systemd that works often but not always you need to explain this a bit further since we might have different perception on this since from my perception such code has been added to systemd in more then one occasion anyway I was not advocating for the use/inclusion of CRIU but something built in. We should be able to support non live migration out of the box if live migration is a pandora's box or are you opposed to that as well? Well, sure, it would be nice, but I also believe it is not realistic to support. For example, if you have network protocols you can probably still live with their connections being severed during migration, but this is really not the case for local IPC connections, like bus connections. Restoring the state for that is an immense amount of work, and I am pretty sure that it is not desirable to add code for this to all IPC systems like this just because of CRIU. Also, I doubt this really is such an important thing to have. I mean, containers are not VMs, they are easily instantiated and shut down. You can socket actviate them, and have them exit-on-idle. If you accept that containers are a much more volatile, dynamic thing that VMs then live migration becomes much less attractive. I am pretty sure you can make CRIU work for a lot of cases. But it's obvious that it will fail to save/restore a large number of cases, including any that involve dbus or any other non-trivial IPC (X11, PA, mysql connections, ...), hence it's really a promise you cannot keep, and I will not support something with systemd that is pretty obviously unsupportable. Lennart -- Lennart Poettering, Red Hat ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] systemd and process migration
On 04/20/2015 01:58 PM, Lennart Poettering wrote: systemd has no provisions for anything like this and I have doubts it really should have. I mean, normal programs retain ccess to the cgroup tree, so you could readd whatever you restore back in the cgroups, but that's hardly sufficient to make systemd aware of a process like this properly. Arguably systemd should have this functionality ( along with quite few others like HA etc ) built in ( not using CRIU ) since you want to live migrate at least native containers between hosts. I thought fancy enterprise features like this was on hold until networkd was ready ? JBG ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] systemd and process migration
On Mon, 20.04.15 15:10, Jóhann B. Guðmundsson (johan...@gmail.com) wrote: On 04/20/2015 02:49 PM, Lennart Poettering wrote: On Mon, 20.04.15 14:43, Jóhann B. Guðmundsson (johan...@gmail.com) wrote: I thought fancy enterprise features like this was on hold until networkd was ready ? This is not an enterprise feature. It's a promise one cannot keep. We will not add code to systemd that works often but not always, and CRIU is certainly of that kind. We will not add code to systemd that works often but not always you need to explain this a bit further since we might have different perception on this since from my perception such code has been added to systemd in more then one occasion anyway I was not advocating for the use/inclusion of CRIU but something built in. We should be able to support non live migration out of the box if live migration is a pandora's box or are you opposed to that as well? Well, sure, it would be nice, but I also believe it is not realistic to support. For example, if you have network protocols you can probably still live with their connections being severed during migration, but this is really not the case for local IPC connections, like bus connections. Restoring the state for that is an immense amount of work, and I am pretty sure that it is not desirable to add code for this to all IPC systems like this just because of CRIU. Also, I doubt this really is such an important thing to have. I mean, containers are not VMs, they are easily instantiated and shut down. You can socket actviate them, and have them exit-on-idle. If you accept that containers are a much more volatile, dynamic thing that VMs then live migration becomes much less attractive. There are valid migration cases beside live migration ones ( for example reallocation to a different host(s) due to resources etc ) Think in terms of shutting down and disable container on host A, send relevant units ( including any network related ones ) along with the underlying filesystem snapshot, in the case of btrfs it would be using btrfs send/receive feature to host B and start it there. What I'm wondering if you would be opposed to supporting those use cases ? Well, but non-live migration is easy: you simply stop the container, transfer the container tree to the destination, and start it there again. We support migrating offline containers like that already to a certain degree with machinectl export-tar and machinectl import-tar. And we probably could even add more support for this, for example I think a machinectl migrate command would make a lot of sense that basically connects a local machinectl export-tar with a machinectl import-tar on another host via ssh. I'd be totally on board with adding more support for things like that. It's really just the *live* migration part I have issues with, because I don't trust we could deliver the promise it makes. Lennart -- Lennart Poettering, Red Hat ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
Re: [systemd-devel] systemd and process migration
Heya, On 20 April 2015 at 06:59, Adrian Reber adr...@lisas.de wrote: Using CRIU I have been migrating processes from one system to another for the last few months (even years). I am now interested in migrating processes under systemd's control. Before starting to look how to make it work I wanted to know if there has been any discussion if and how systemd and CRIU can work together? Dumping a process under systemd's control should be no problem. After criu has dumped the process (and killed it) systemd should know that it does not need to restart the process, but even if the process is restarted by systemd it does not hurt the process migration. The interesting part happens on the system where the process wants to be migrated to. After the process has been dumped and transfered from the source system to the destination system it needs to be restarted. So far this works with different tested processes (postgresql is my current test process). If I want to restore the process as a systemd child process I have the problem that the process would have to be re-parented to systemd after restarting which is not possible in Linux. Therefore I need the help of systemd. My plan now would be to transfer the process to the destination system and tell systemd I want to restore a process which should be under systemd's control after the restore has completed. Therefore systemd needs to run criu with the option to restore the new process as a criu sibling. Thus systemd would be the correct parent process. Is this something which would be useful to integrate into systemd? I have some ideas how this could be implemented and how the user interfaces could look like. Before going in too much detail I want to find out if this is a direction which makes sense. So systemd has re-exec support. I would go down the route of trying to serialise the systemd state of the process as well on the orginal host, and then deserialise it on the target. I haven't done anything with re-exec serialisation on systemd yet, so no idea how sufficient that would be to side-load a process like that. It will be fragile. Going down the route of machinectl api / containers sounds more interesting. As there is more support for importing those. E.g. instead of importing a tarball and execing, it, one unfreezes it. -- Regards, Dimitri. Pura Vida! https://clearlinux.org Open Source Technology Center Intel Corporation (UK) Ltd. - Co. Reg. #1134945 - Pipers Way, Swindon SN3 1RJ. ___ systemd-devel mailing list systemd-devel@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/systemd-devel
