Re: [Xen-devel] Upstream QEMU based stubdom and rump kernel
po...@iki.fi said: Where do you propose to inject that faking out (and what does it even mean)? Someone at Berkeley decided that socket drivers should be globally enumerated, and PF_UNIX leads to exactly one handler. Just hacking hooks as local patches into the PF_UNIX driver is against the whole point of having unmodified, tested drivers from upstream. We do not want to hack hooks as local patches into the PF_UNIX driver. Rather, we'd like to develop an entirely new driver (nothing wrong with that?), which would mimic PF_UNIX semantics but talk to hyperspace instead. See below for the purpose we want to use it for. So, if you want your bus to appear as a socket to userspace, I don't see any shortcut to not going via NetBSD. If you're happy with something else than a socket, that's another story. Especially if the interface doesn't matter too much for whatever purpose you plan to use it for, it's silly to specify the interface so that the implementation process is as convoluted as possible ;) By faking out Anil means a shim to get existing applications which currently use PF_UNIX (and possibly PF_INET, though that will be harder to fake) to use the hypervisor bus to talk to another colocated unikernel instead. The motivations for this are: - Taking the TCP stack out of the picture entirely for intra-unikernel comms (eg. PHP unikernel - MySQL unikernel). Both of those could be thus be linked without the PF_INET component. - This means that you do not need to set up and manage a TCP network in your infrastructure for intra-unikernel comms, which is a huge advantage from an operations point of view. - It also means that unikernels which should not be talking TCP to anywhere, ever, can't do that. Anil, have I missed anything? So, the interface does matter in the sense that it should be as simple as possible to take an existing application and get it to use the new bus. This could be as simple as linking your unikernel against -lrumpnet_hyper instead of -lrumpnet_local. Taking a longer-term view, I do think that there is a wider case for PF_HYPER and I will be happy to sell it to NetBSD (or whoever) once we are ready to make that case. In my mind the semantics of PF_HYPER from an application PoV are pretty clear: exactly the same as PF_UNIX except that you substitute filesystem path for hyperspace path, with the exact semantics of hyperspace path left up to your hypervisor. The application need not care, as long as you can tell it to e.g. use vchan/mysql instead of /tmp/mysql.sock when doing bind(). Martin ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Upstream QEMU based stubdom and rump kernel
On Tue, 2015-03-17 at 15:15 +, Anthony PERARD wrote: On Tue, Mar 17, 2015 at 02:29:07PM +, Wei Liu wrote: I've now successfully built QEMU upstream with rump kernel. However to make it fully functional as a stubdom, there are some missing pieces to be added in. 1. The ability to access QMP socket (a unix socket) from Dom0. That will be used to issue command to QEMU. The QMP socket does not needs to be a unix socket. It can be any of those (from qemu --help): Character device options: -chardev null,id=id[,mux=on|off] How much flexibility/modularity is there on the qemu side for adding new chardev types? Could we for example add -chardev vchan,path=path without too much trouble? To save a QEMU state (write), we do use a filename. But I guest we could expand the QMP command (xen-save-devices-state) to use something else, if it's easier. Like, perhaps, an arbitrary chardev? Ian. ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Upstream QEMU based stubdom and rump kernel
On 19 Mar 2015, at 09:35, Antti Kantee po...@iki.fi wrote: On 19/03/15 08:48, Martin Lucina wrote: By faking out Anil means a shim to get existing applications which currently use PF_UNIX (and possibly PF_INET, though that will be harder to fake) to use the hypervisor bus to talk to another colocated unikernel instead. The motivations for this are: - Taking the TCP stack out of the picture entirely for intra-unikernel comms (eg. PHP unikernel - MySQL unikernel). Both of those could be thus be linked without the PF_INET component. - This means that you do not need to set up and manage a TCP network in your infrastructure for intra-unikernel comms, which is a huge advantage from an operations point of view. - It also means that unikernels which should not be talking TCP to anywhere, ever, can't do that. Aah, ic, you want to do what rumpnet_sockin does, except use the hypervisor bus instead of an external sockets-like networking facility like sockin does. rumpnet_sockin was indeed originally developed so that you wouldn't need to include the full TCP/IP stack in a rump kernel, which is nice for scenarios where you want to do networking without configuring anything for each guest instance; running the kernel NFS client in userspace and using the host's network was the original use case. Yea, that'll just work on the rump kernel side for PF_INET/PF_INET6 (though you might have to do a bit more handling in your fake driver). Not sure what doing the same for PF_UNIX would entail, if anything special, but only one way to find out. That's right -- the primary motivation from my end is to short-circuit all the unnecessary network stack serialisation and configuration, and end up with a very simple data path such as shared memory rings and/or vchan. The challenge is figuring out where to hook in the dynamic lookups required, and what form they would take on the coordination bus (XenStore). One slight hitch with using XenStore for this is that its permissions model isn't quite good enough to build a full Plan9-like interface (where every listen is published in a per-VM path and can be written to by a connecting VM). Dave Scott had some thoughts on how to extend XS with this, but it wouldn't be a short-term solution for working with existing toolstacks. One workaround is to have a trusted arbiter VM running that would coordinate the establishment of connections and hand them off. -anil ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Upstream QEMU based stubdom and rump kernel
po...@rumpkernel.org said: etfs isn't a file system, e.g. it doesn't allow listing files or removing them, but it does give you complete control of what happens when data is read or written for /some/path. But based on the other posts, sounds like it might be enough for what you need. See: http://man.netbsd.org/cgi-bin/man-cgi?rump_etfs++NetBSD-current They'd still need to implement the rumphyper/Mini-OS backend to get etfs to talk over vchan to the dom0, right? That's not really a problem, though I do want to limit the amount of interface we claim to support with rump kernels. For example, ISTR you mentioned on irc you'd like to use minios wait.h. It would be better to use pthread synchronization instead of minios synchronization. That way, if we do have a need to change the underlying threading in the future, you won't run into trouble. So, we should just determine what is actually needed and expose those bits by default. +1 Martin ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Upstream QEMU based stubdom and rump kernel
ian.campb...@citrix.com said: On Tue, 2015-03-17 at 15:27 +, Wei Liu wrote: This looks most interesting as it implies we can easily pipe a console to it. BTW, rather than rawe consoles we should probably consider using the channel extension: http://xenbits.xen.org/docs/unstable/misc/channel.txt What would be the advantage/rationale for using channels rather than vchan? (See my other reply to this thread) Martin ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Upstream QEMU based stubdom and rump kernel
(Adding some of the Mirage folks to Cc:) wei.l...@citrix.com said: Hi all I'm now working on upstream QEMU stubdom, and rump kernel seems to be a good fit for this purpose. A bit background information. A stubdom is a service domain. With QEMU stubdom we are able to run QEMU device emulation code in a separate domain so that bugs in QEMU don't affect Dom0 (the controlling domain). Xen currently has a QEMU stubdom, but it's based on our fork of ancient QEMU (plus some other libraries and mini-os). Eventually we would like to use upstream QEMU in stubdom. I've now successfully built QEMU upstream with rump kernel. However to make it fully functional as a stubdom, there are some missing pieces to be added in. 1. The ability to access QMP socket (a unix socket) from Dom0. That will be used to issue command to QEMU. 2. The ability to access files in Dom0. That will be used to write to / read from QEMU state file. As I understand from Stefano's and Anthony's replies in this thread, both of the above can be implemented using an AF_UNIX or AF_INET socket on the QEMU end. Such an implementation would not require anything special done in QEMU, just telling it which socket to use using existing mechanisms. So, let's step back a bit: What we need is a trusted communication channel from a Rump Kernel domU to dom0, using existing socket or socket-like[*] APIs at both the domU and dom0 ends. This fits in with a couple of things I hope to make time to work on in the next couple of months: 1. Introspection of Rump Kernel domUs for ops purposes, i.e. get some basic ps, top, vmstat-like information about what the domU is doing from the dom0. 2. Connecting up multiple Rump Kernel domUs and/or Mirage domUs. The general idea here is that you can have e.g. a Mirage domU running a HTTP+TLS frontend, communicating with a Rump Kernel domU running PHP + FastCGI. The Mirage folks are already doing something similar in their Jitsu work, using a protocol called Conduit which runs over vchan. Now, both of the above require exactly the same underlying mechanism. Point 2. will further require implementing support in the Rump Kernel, either for a shim which would proxy AF_UNIX / AF_INET transparently using vchan, or possibly later implementing a separate socket family (AF_VCHAN / AF_HYPER?). Once that is done you should be able to just drop it in to QEMU on Rump. [*] Aside: What I mean by socket-like is that the implementation does not need to be in the dom0 kernel, it can just be a user-space library. For example, see the nanomsg or ZeroMQ APIs, which I have worked on extensively in the past. 3. The building process requires mini-os headers. That will be used to build libxc (the controlling library). As Antti already suggested, if you can use POSIX interfaces rather than mini-os ones in QEMU, then that would be a better approach. One of my lessons learned from the existing stubdom stuffs is that I should work with upstream and produce maintainable code. So before I do anything for real I'd better consult the community. My gut feeling is that the first two requirements are not really Xen specific. Let me know what you guys plan and think. Thanks for getting in touch. I think this is an important discussion! Martin ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Upstream QEMU based stubdom and rump kernel
On Wed, 2015-03-18 at 12:24 +0100, Martin Lucina wrote: ian.campb...@citrix.com said: On Tue, 2015-03-17 at 15:27 +, Wei Liu wrote: This looks most interesting as it implies we can easily pipe a console to it. BTW, rather than rawe consoles we should probably consider using the channel extension: http://xenbits.xen.org/docs/unstable/misc/channel.txt What would be the advantage/rationale for using channels rather than vchan? (See my other reply to this thread) Not much really. About the only relevant difference between vchan and channels(/consoles) is that there is an existing backend running on most xen systems (xenconsoled) which can be leveraged in some cases for channels, whereas vchan would need a specific backend writing for each case. Apart from that implementation convenience vchan is probably going to be better in terms of proper integration for the other end. But iff the decision goes the way of consoles then using channels in preference to raw consoles makes sense. Ian. ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Upstream QEMU based stubdom and rump kernel
On Wed, 18 Mar 2015, Ian Campbell wrote: On Wed, 2015-03-18 at 12:24 +0100, Martin Lucina wrote: ian.campb...@citrix.com said: On Tue, 2015-03-17 at 15:27 +, Wei Liu wrote: This looks most interesting as it implies we can easily pipe a console to it. BTW, rather than rawe consoles we should probably consider using the channel extension: http://xenbits.xen.org/docs/unstable/misc/channel.txt What would be the advantage/rationale for using channels rather than vchan? (See my other reply to this thread) Not much really. About the only relevant difference between vchan and channels(/consoles) is that there is an existing backend running on most xen systems (xenconsoled) which can be leveraged in some cases for channels, whereas vchan would need a specific backend writing for each case. Apart from that implementation convenience vchan is probably going to be better in terms of proper integration for the other end. But iff the decision goes the way of consoles then using channels in preference to raw consoles makes sense. I think that for simplicity's sake and to limit dependencies on the system, using consoles for low bandwidth channels, such as QMP, is preferable. ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Upstream QEMU based stubdom and rump kernel
On 18/03/15 11:22, Martin Lucina wrote:
po...@rumpkernel.org said:
etfs isn't a file system, e.g. it doesn't allow listing files or
removing them, but it does give you complete control of what happens
when data is read or written for /some/path. But based on the other
posts, sounds like it might be enough for what you need.
See:
http://man.netbsd.org/cgi-bin/man-cgi?rump_etfs++NetBSD-current
They'd still need to implement the rumphyper/Mini-OS backend to get etfs to
talk over vchan to the dom0, right?
Strictly speaking, they'd have to implement the iov{read,write}
hypercalls to do that. But, no, etfs doesn't do magic. IOW, they'd
have to define what host path means.
It occurred to me that I wrote that manpage, umm, 5 years ago when rump
kernels ran only in userspace and host path was a better defined term.
Pile that manpage on the neverending heap of documentation which broke
while the code kept working.
___
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
Re: [Xen-devel] Upstream QEMU based stubdom and rump kernel
On Wed, 2015-03-18 at 12:45 +, Stefano Stabellini wrote: On Wed, 18 Mar 2015, Ian Campbell wrote: On Wed, 2015-03-18 at 12:24 +0100, Martin Lucina wrote: ian.campb...@citrix.com said: On Tue, 2015-03-17 at 15:27 +, Wei Liu wrote: This looks most interesting as it implies we can easily pipe a console to it. BTW, rather than rawe consoles we should probably consider using the channel extension: http://xenbits.xen.org/docs/unstable/misc/channel.txt What would be the advantage/rationale for using channels rather than vchan? (See my other reply to this thread) Not much really. About the only relevant difference between vchan and channels(/consoles) is that there is an existing backend running on most xen systems (xenconsoled) which can be leveraged in some cases for channels, whereas vchan would need a specific backend writing for each case. Apart from that implementation convenience vchan is probably going to be better in terms of proper integration for the other end. But iff the decision goes the way of consoles then using channels in preference to raw consoles makes sense. I think that for simplicity's sake and to limit dependencies on the system, using consoles for low bandwidth channels, such as QMP, is preferable. s/consoles/channels/, please ;-) That said, a having libxl be a user of libvchan to slurp the data in/out of qemu directly (perhaps using the datacopier infrastructure) might be nicer from a design point of view, since it would mean libxl could read/write things directly instead of via a temp file and it takes xenconsoled out of that path, which might be nice. Ian, ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Upstream QEMU based stubdom and rump kernel
Hello, Wei Liu, le Tue 17 Mar 2015 14:29:07 +, a écrit : One of my lessons learned from the existing stubdom stuffs is that I should work with upstream and produce maintainable code. Not only maintainable, but really make sure to have the time to stick with upstream on the long run, first until it gets integrated in the upstream QEMU release process and then still to maintain it there on the long run. The old work on mini-os qemu stubdomain wasn't too bad, but without actual integration in the QEMU process, and nobody to update the fork, it was deemed to fall behind. Samuel ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Upstream QEMU based stubdom and rump kernel
On 18 Mar 2015, at 11:20, Martin Lucina mar...@lucina.net wrote: A bit background information. A stubdom is a service domain. With QEMU stubdom we are able to run QEMU device emulation code in a separate domain so that bugs in QEMU don't affect Dom0 (the controlling domain). Xen currently has a QEMU stubdom, but it's based on our fork of ancient QEMU (plus some other libraries and mini-os). Eventually we would like to use upstream QEMU in stubdom. I've now successfully built QEMU upstream with rump kernel. However to make it fully functional as a stubdom, there are some missing pieces to be added in. 1. The ability to access QMP socket (a unix socket) from Dom0. That will be used to issue command to QEMU. 2. The ability to access files in Dom0. That will be used to write to / read from QEMU state file. As I understand from Stefano's and Anthony's replies in this thread, both of the above can be implemented using an AF_UNIX or AF_INET socket on the QEMU end. Such an implementation would not require anything special done in QEMU, just telling it which socket to use using existing mechanisms. So, let's step back a bit: What we need is a trusted communication channel from a Rump Kernel domU to dom0, using existing socket or socket-like[*] APIs at both the domU and dom0 ends. This fits in with a couple of things I hope to make time to work on in the next couple of months: 1. Introspection of Rump Kernel domUs for ops purposes, i.e. get some basic ps, top, vmstat-like information about what the domU is doing from the dom0. 2. Connecting up multiple Rump Kernel domUs and/or Mirage domUs. The general idea here is that you can have e.g. a Mirage domU running a HTTP+TLS frontend, communicating with a Rump Kernel domU running PHP + FastCGI. The Mirage folks are already doing something similar in their Jitsu work, using a protocol called Conduit which runs over vchan. Yeah, this is currently requiring a couple of things: - kicking the tires with Vchan and its associated machinery, which has taken some time. Dave Scott has built a complementary system for the xentropyd which simply sets up a console ring instead of vchan. This has the drawback of being a single fixed page, but far simpler. - A XenStore protocol for setting up stream connections. This could indeed quite easily turn into a AF_VCHAN that could be transparently used by rump/Mirage/HaLVM and normal domUs for VM-VM comms. Now, both of the above require exactly the same underlying mechanism. Point 2. will further require implementing support in the Rump Kernel, either for a shim which would proxy AF_UNIX / AF_INET transparently using vchan, or possibly later implementing a separate socket family (AF_VCHAN / AF_HYPER?). Once that is done you should be able to just drop it in to QEMU on Rump. I'm a little wary of point 2) asking for filesystem access to dom0. What exactly is the qemu state API? Does it need arbitrary file access, or is there a slightly higher level set of operations that could be marshalled along the socket? In fact, why doesn't qemu privilege separate and use a QMP socket for its host filesystem operations as well? [*] Aside: What I mean by socket-like is that the implementation does not need to be in the dom0 kernel, it can just be a user-space library. For example, see the nanomsg or ZeroMQ APIs, which I have worked on extensively in the past. 3. The building process requires mini-os headers. That will be used to build libxc (the controlling library). As Antti already suggested, if you can use POSIX interfaces rather than mini-os ones in QEMU, then that would be a better approach. One of my lessons learned from the existing stubdom stuffs is that I should work with upstream and produce maintainable code. So before I do anything for real I'd better consult the community. My gut feeling is that the first two requirements are not really Xen specific. Let me know what you guys plan and think. Thanks for getting in touch. I think this is an important discussion! Very much so -- the time is definitely right to establish some unikernel interop standards. I'm also looking forward to better rump-MirageOS comms in particular (shiny new protocol stacks working alongside existing applications in separate VM containers). -anil ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Upstream QEMU based stubdom and rump kernel
a...@recoil.org said: Point 2. will further require implementing support in the Rump Kernel, either for a shim which would proxy AF_UNIX / AF_INET transparently using vchan, or possibly later implementing a separate socket family (AF_VCHAN / AF_HYPER?). Once that is done you should be able to just drop it in to QEMU on Rump. I'm a little wary of point 2) asking for filesystem access to dom0. What exactly is the qemu state API? Does it need arbitrary file access, or is there a slightly higher level set of operations that could be marshalled along the socket? In fact, why doesn't qemu privilege separate and use a QMP socket for its host filesystem operations as well? Email thread context confusion here. I meant point 2 that I wrote (connecting up Rump-Mirage domUs where the Rump application is unmodified and listens on what it believes is AF_INET/AF_UNIX). Regarding the qemu state API, as I understood from others replies to the full thread (http://www.freelists.org/post/rumpkernel-users/Upstream-QEMU-based-stubdom-and-rump-kernel) the state API does not require filesystem access and can be made to work over a socket. Martin ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Upstream QEMU based stubdom and rump kernel
On 18/03/15 21:21, Anil Madhavapeddy wrote: This is not an argument for or against; if you want to expose AF_WHATEVER to applications running on a rump kernel, you need to sell AF_WHATEVER to NetBSD, not to rumpkernel-users. Well, preferably you need to sell it to everyone implementing sockets and running on some sort of hypervisor, but of course gotta start from somewhere. Given that most of the uses of this will be in userspace code, just faking out AF_UNIX in Rump does seem a lot easier. It doesn't matter to MirageOS either way -- we just need a well-defined XenStore/ring protocol to obey to do connection setup on the other side. Where do you propose to inject that faking out (and what does it even mean)? Someone at Berkeley decided that socket drivers should be globally enumerated, and PF_UNIX leads to exactly one handler. Just hacking hooks as local patches into the PF_UNIX driver is against the whole point of having unmodified, tested drivers from upstream. So, if you want your bus to appear as a socket to userspace, I don't see any shortcut to not going via NetBSD. If you're happy with something else than a socket, that's another story. Especially if the interface doesn't matter too much for whatever purpose you plan to use it for, it's silly to specify the interface so that the implementation process is as convoluted as possible ;) ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Upstream QEMU based stubdom and rump kernel
On 18 Mar 2015, at 20:23, Antti Kantee po...@iki.fi wrote: On 18/03/15 19:05, Anil Madhavapeddy wrote: This fits in with a couple of things I hope to make time to work on in the next couple of months: 1. Introspection of Rump Kernel domUs for ops purposes, i.e. get some basic ps, top, vmstat-like information about what the domU is doing from the dom0. 2. Connecting up multiple Rump Kernel domUs and/or Mirage domUs. The general idea here is that you can have e.g. a Mirage domU running a HTTP+TLS frontend, communicating with a Rump Kernel domU running PHP + FastCGI. The Mirage folks are already doing something similar in their Jitsu work, using a protocol called Conduit which runs over vchan. Yeah, this is currently requiring a couple of things: - kicking the tires with Vchan and its associated machinery, which has taken some time. Dave Scott has built a complementary system for the xentropyd which simply sets up a console ring instead of vchan. This has the drawback of being a single fixed page, but far simpler. - A XenStore protocol for setting up stream connections. This could indeed quite easily turn into a AF_VCHAN that could be transparently used by rump/Mirage/HaLVM and normal domUs for VM-VM comms. This is not an argument for or against; if you want to expose AF_WHATEVER to applications running on a rump kernel, you need to sell AF_WHATEVER to NetBSD, not to rumpkernel-users. Well, preferably you need to sell it to everyone implementing sockets and running on some sort of hypervisor, but of course gotta start from somewhere. Given that most of the uses of this will be in userspace code, just faking out AF_UNIX in Rump does seem a lot easier. It doesn't matter to MirageOS either way -- we just need a well-defined XenStore/ring protocol to obey to do connection setup on the other side. -anil ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Upstream QEMU based stubdom and rump kernel
On Tue, Mar 17, 2015 at 02:29:07PM +, Wei Liu wrote: I've now successfully built QEMU upstream with rump kernel. However to make it fully functional as a stubdom, there are some missing pieces to be added in. 1. The ability to access QMP socket (a unix socket) from Dom0. That will be used to issue command to QEMU. The QMP socket does not needs to be a unix socket. It can be any of those (from qemu --help): Character device options: -chardev null,id=id[,mux=on|off] -chardev socket,id=id[,host=host],port=port[,to=to][,ipv4][,ipv6][,nodelay][,reconnect=seconds] [,server][,nowait][,telnet][,reconnect=seconds][,mux=on|off] (tcp) -chardev socket,id=id,path=path[,server][,nowait][,telnet][,reconnect=seconds][,mux=on|off] (unix) -chardev udp,id=id[,host=host],port=port[,localaddr=localaddr] [,localport=localport][,ipv4][,ipv6][,mux=on|off] -chardev msmouse,id=id[,mux=on|off] -chardev vc,id=id[[,width=width][,height=height]][[,cols=cols][,rows=rows]] [,mux=on|off] -chardev ringbuf,id=id[,size=size] -chardev file,id=id,path=path[,mux=on|off] -chardev pipe,id=id,path=path[,mux=on|off] -chardev pty,id=id[,mux=on|off] -chardev stdio,id=id[,mux=on|off][,signal=on|off] -chardev serial,id=id,path=path[,mux=on|off] -chardev tty,id=id,path=path[,mux=on|off] -chardev parallel,id=id,path=path[,mux=on|off] -chardev parport,id=id,path=path[,mux=on|off] -chardev spicevmc,id=id,name=name[,debug=debug] -chardev spiceport,id=id,name=name[,debug=debug] 2. The ability to access files in Dom0. That will be used to write to / read from QEMU state file. To save a QEMU state (write), we do use a filename. But I guest we could expand the QMP command (xen-save-devices-state) to use something else, if it's easier. To restore, we provide a file descriptor from libxl to QEMU, with the fd on the file that contain the state we want to restore. But there are a few other way to load a state (from qemu.git/docs/migration.txt): - tcp migration: do the migration using tcp sockets - unix migration: do the migration using unix sockets - exec migration: do the migration using the stdin/stdout through a process. - fd migration: do the migration using an file descriptor that is passed to QEMU. QEMU doesn't care how this file descriptor is opened. -- Anthony PERARD ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Upstream QEMU based stubdom and rump kernel
On Tue, Mar 17, 2015 at 02:54:09PM +, Ian Campbell wrote: On Tue, 2015-03-17 at 14:29 +, Wei Liu wrote: 2. The ability to access files in Dom0. That will be used to write to / read from QEMU state file. This requirement is not as broad as you make it sound. Yes. You're right. All which is really required is the ability to slurp in or write out a blob of bytes to a service running in a control domain, not actual This is more accurate. ability to read/write files in dom0 (which would need careful security consideration!). For the old qemu-traditional stubdom for example this is implemented as a pair of console devices (one r/o for restore + one w/o for save) which are setup by the toolstack at start of day and pre-plumbed into two temporary files. Unfortunately I don't think that hack in mini-os is upstreamable in rump kernel. Wei. Ian. ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Upstream QEMU based stubdom and rump kernel
On Tue, Mar 17, 2015 at 03:15:17PM +, Anthony PERARD wrote: On Tue, Mar 17, 2015 at 02:29:07PM +, Wei Liu wrote: I've now successfully built QEMU upstream with rump kernel. However to make it fully functional as a stubdom, there are some missing pieces to be added in. 1. The ability to access QMP socket (a unix socket) from Dom0. That will be used to issue command to QEMU. The QMP socket does not needs to be a unix socket. It can be any of those (from qemu --help): Character device options: -chardev null,id=id[,mux=on|off] -chardev socket,id=id[,host=host],port=port[,to=to][,ipv4][,ipv6][,nodelay][,reconnect=seconds] [,server][,nowait][,telnet][,reconnect=seconds][,mux=on|off] (tcp) -chardev socket,id=id,path=path[,server][,nowait][,telnet][,reconnect=seconds][,mux=on|off] (unix) -chardev udp,id=id[,host=host],port=port[,localaddr=localaddr] [,localport=localport][,ipv4][,ipv6][,mux=on|off] -chardev msmouse,id=id[,mux=on|off] -chardev vc,id=id[[,width=width][,height=height]][[,cols=cols][,rows=rows]] [,mux=on|off] -chardev ringbuf,id=id[,size=size] -chardev file,id=id,path=path[,mux=on|off] -chardev pipe,id=id,path=path[,mux=on|off] -chardev pty,id=id[,mux=on|off] -chardev stdio,id=id[,mux=on|off][,signal=on|off] -chardev serial,id=id,path=path[,mux=on|off] -chardev tty,id=id,path=path[,mux=on|off] -chardev parallel,id=id,path=path[,mux=on|off] -chardev parport,id=id,path=path[,mux=on|off] -chardev spicevmc,id=id,name=name[,debug=debug] -chardev spiceport,id=id,name=name[,debug=debug] Ha, thanks for the list. My brain was too locked in to the current implementation. So yes, we now have an array of possible transports at our disposal. 2. The ability to access files in Dom0. That will be used to write to / read from QEMU state file. To save a QEMU state (write), we do use a filename. But I guest we could expand the QMP command (xen-save-devices-state) to use something else, if it's easier. That's also an option. To restore, we provide a file descriptor from libxl to QEMU, with the fd on the file that contain the state we want to restore. But there are a few other way to load a state (from qemu.git/docs/migration.txt): - tcp migration: do the migration using tcp sockets - unix migration: do the migration using unix sockets - exec migration: do the migration using the stdin/stdout through a process. This looks most interesting as it implies we can easily pipe a console to it. Wei. - fd migration: do the migration using an file descriptor that is passed to QEMU. QEMU doesn't care how this file descriptor is opened. -- Anthony PERARD ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Upstream QEMU based stubdom and rump kernel
On Tue, 2015-03-17 at 15:27 +, Wei Liu wrote: This looks most interesting as it implies we can easily pipe a console to it. BTW, rather than rawe consoles we should probably consider using the channel extension: http://xenbits.xen.org/docs/unstable/misc/channel.txt Ian. ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Upstream QEMU based stubdom and rump kernel
On Tue, 2015-03-17 at 14:29 +, Wei Liu wrote: 2. The ability to access files in Dom0. That will be used to write to / read from QEMU state file. This requirement is not as broad as you make it sound. All which is really required is the ability to slurp in or write out a blob of bytes to a service running in a control domain, not actual ability to read/write files in dom0 (which would need careful security consideration!). For the old qemu-traditional stubdom for example this is implemented as a pair of console devices (one r/o for restore + one w/o for save) which are setup by the toolstack at start of day and pre-plumbed into two temporary files. Ian. ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Upstream QEMU based stubdom and rump kernel
On Tue, 17 Mar 2015, Anthony PERARD wrote: On Tue, Mar 17, 2015 at 02:29:07PM +, Wei Liu wrote: I've now successfully built QEMU upstream with rump kernel. However to make it fully functional as a stubdom, there are some missing pieces to be added in. 1. The ability to access QMP socket (a unix socket) from Dom0. That will be used to issue command to QEMU. The QMP socket does not needs to be a unix socket. It can be any of those (from qemu --help): Character device options: -chardev null,id=id[,mux=on|off] -chardev socket,id=id[,host=host],port=port[,to=to][,ipv4][,ipv6][,nodelay][,reconnect=seconds] [,server][,nowait][,telnet][,reconnect=seconds][,mux=on|off] (tcp) -chardev socket,id=id,path=path[,server][,nowait][,telnet][,reconnect=seconds][,mux=on|off] (unix) -chardev udp,id=id[,host=host],port=port[,localaddr=localaddr] [,localport=localport][,ipv4][,ipv6][,mux=on|off] -chardev msmouse,id=id[,mux=on|off] -chardev vc,id=id[[,width=width][,height=height]][[,cols=cols][,rows=rows]] [,mux=on|off] -chardev ringbuf,id=id[,size=size] -chardev file,id=id,path=path[,mux=on|off] -chardev pipe,id=id,path=path[,mux=on|off] -chardev pty,id=id[,mux=on|off] -chardev stdio,id=id[,mux=on|off][,signal=on|off] -chardev serial,id=id,path=path[,mux=on|off] -chardev tty,id=id,path=path[,mux=on|off] -chardev parallel,id=id,path=path[,mux=on|off] -chardev parport,id=id,path=path[,mux=on|off] -chardev spicevmc,id=id,name=name[,debug=debug] -chardev spiceport,id=id,name=name[,debug=debug] 2. The ability to access files in Dom0. That will be used to write to / read from QEMU state file. To save a QEMU state (write), we do use a filename. But I guest we could expand the QMP command (xen-save-devices-state) to use something else, if it's easier. To restore, we provide a file descriptor from libxl to QEMU, with the fd on the file that contain the state we want to restore. But there are a few other way to load a state (from qemu.git/docs/migration.txt): - tcp migration: do the migration using tcp sockets - unix migration: do the migration using unix sockets - exec migration: do the migration using the stdin/stdout through a process. - fd migration: do the migration using an file descriptor that is passed to QEMU. QEMU doesn't care how this file descriptor is opened. QEMU would definitely be happy if we started using fds instead of files to save/restore the state on Xen. ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
Re: [Xen-devel] Upstream QEMU based stubdom and rump kernel
On 17/03/15 14:29, Wei Liu wrote: I've now successfully built QEMU upstream with rump kernel. However to make it fully functional as a stubdom, there are some missing pieces to be added in. 1. The ability to access QMP socket (a unix socket) from Dom0. That will be used to issue command to QEMU. 2. The ability to access files in Dom0. That will be used to write to / read from QEMU state file. There's a way to map file access to rump kernel hypercalls with a facility called etfs (extra-terrestrial file system). In fact, the current implementation for accessing the Xen block device from the rump kernel is done using etfs (... historical reasons, I'd have to go back 5+ years to explain why it doesn't attach as a regular block device). etfs isn't a file system, e.g. it doesn't allow listing files or removing them, but it does give you complete control of what happens when data is read or written for /some/path. But based on the other posts, sounds like it might be enough for what you need. See: http://man.netbsd.org/cgi-bin/man-cgi?rump_etfs++NetBSD-current 3. The building process requires mini-os headers. That will be used to build libxc (the controlling library). That's not really a problem, though I do want to limit the amount of interface we claim to support with rump kernels. For example, ISTR you mentioned on irc you'd like to use minios wait.h. It would be better to use pthread synchronization instead of minios synchronization. That way, if we do have a need to change the underlying threading in the future, you won't run into trouble. So, we should just determine what is actually needed and expose those bits by default. One of my lessons learned from the existing stubdom stuffs is that I should work with upstream and produce maintainable code. So before I do anything for real I'd better consult the community. My gut feeling is that the first two requirements are not really Xen specific. Let me know what you guys plan and think. Yes, please. If there's something silly going on, it's most likely due to: 1) we didn't get that far in our experiments and weren't aware of it 2) we were aware, but some bits were even sillier, taking priority Either way, a real need is a definite reason to expedite fixing. - antti ___ Xen-devel mailing list Xen-devel@lists.xen.org http://lists.xen.org/xen-devel
