Thanks for the writeup Zhou Zheng!  This is a very nice explanation of the
possible approaches that are on the table.  I would like to add my thoughts on
each approach inline below.

On Tue, Nov 27, 2012 at 05:22:20PM +0800, Zhou Zheng Sheng wrote:
> Hi all,
> For now in there is no agreement on the remote guest console
> solution, so I decide to do some investigation continue the
> discussion.
> Our goal
>   VM serial console remote access in CLI mode. That means the client
> runs without X environment.
> There are several proposals.
> 1. Sandboxed sshd
>   VDSM runs a new host sshd instance in virtual machine/sandbox and
> redirects the virtio console to it.
> 2. Third-party sshd
>   VDSM runs third-party sshd library/implementation and redirects
> virtio console to it.
> 3. Spice
>   Extend spice to support console and implement a client to be run
> without GUI environment
> 4. oVirt shell -> Engine -> libvirt
>   The user connects to Engine via oVirt CLI, then issues a
> "serial-console" command, then Engine locates the host and connect
> to the guest console. Currently there is a workaround, it invokes
> "virsh -c qemu+tls://host/qemu console vmid" from Engine side.
> 5. VDSM console streaming API
>   VDSM exposes getConsoleReadStream() and getConsoleWriteStream()
> via XMLRPC binding. Then implement the related client in vdsClient
> and Engine
> Detailed discussion
> 1. Sandboxes
> Solution 1 and 2 allow users connect to console using their favorite
> ssh client. The login name is vmid, the password is set by
> setVmTicket() call of VDSM. The connection will be lost during
> migration. This is similar to VNC in oVirt.
> I take a look at several sandbox technologies, including
> libvirt-sandbox, lxc and selinux.
> a) libvirt-sandbox boots a VM using host kernel and initramfs, then
> passthru the host file system to the VM in read only mode. We can
> also add extra binding to the guest file system. It's very easy to
> use. To run shell in a VM, one can just issues
> virt-sandbox -c qemu:///session  /bin/sh
> Then the VM will be ready in several seconds.
> However it will trigger some selinux violations. Currently there is
> no official support for selinux policy configuration from this
> project. In the project page this is put in the todo list.
> b) lxc utilize Linux container to run a process in sandbox. It needs
> to be configured properly. I find in the package lxc-templates there
> is an example configuration file for running sshd in lxc.
> c) sandbox command in the package policycoreutils-python makes use
> of selinux to run a process in sandbox, but there is no official or
> example policy files for sshd.
> In a word, for sandbox technologies, we have to configure the
> policies/file system binding/network carefully and test the
> compatibility with popular sshd implementations (openssh-server).
> When those sshd upgrade, the policy must be upgraded by us at the
> same time. Since the policies are not maintained by who implements
> sshd, this is a burden for us.
> Work to do
>   Write and maintain the policies.
>   Find ways for auth callback and redirecting data to openssh-server.
> pros
>   Re-use existing pieces and technologies (host sshd, sandbox).
>   User friendly, they can use existing ssh clients.
> cons
>   Connection is lost in migration, this is not a big problem because
> 1) VNC connection share the same problem, 2) the user can reconnect
> manually.
>   It's not easy to maintain the sandbox policies/file system
> binding/network for compatibility with sshd.

I find all of these sandbox techniques to be far too cumbersome to be useful.
In each case, the dependencies on the base operating system (selinux, etc.) are
too great to make this a maintainable option going forward.

> 2. Third-party sshd implementations
> Almost the same as solution 1 but with better flexibility. VDSM can
> import a third-party sshd library and let that library deal with
> auth and transport. VDSM just have to implement the data forwarding.
> Many people consider this is insecure but I think the ticket
> solution for VNC is even not as secure as this. Currently most of us
> only trust openssh-server and think the quality of third-party sshd
> is low. I searched for a while and found twisted.conch from the
> popular twisted project. I'm not familiar with twisted.conch, but I
> still put it in this mail to collect opinions from potential
> twisted.conch experts.
> In a word, I prefer sandbox technologies to third-party sshd
> implementations unless there is a implementation as good as
> openssh-server.
> Work to do
>   Integrate twisted.conch into VDSM
> pros
>   Very flexible. If library provide auth callback to VDSM, then VDSM
> can just compares the login password to the VM ticket without
> knowing SSH detials.
> cons
>   Third party implementations are not as secure and carefully
> maintained as sshd in the host (probably openssh-server).

As others have said previously, the security implications of relying on a
third-party ssh implementation makes this idea a non-starter for me.
> 3. Extend Spice to support console
> Is it possible to implement a spice client can be run in pure text
> mode without GUI environment? If we extend the protocol to support
> console stream but the client must be run in GUI, it will be less
> useful.
> pros
>   No new VMs and server process, easy for maintenance.
> cons
>   Must wait for Spice developers to commit the support.
>   Need special client program in CLI, the user may prefer existing
> client program like ssh. It not a big problem because this feature
> can be put in to oVirt shell.

Can someone familiar with spice weigh in on whether a console connection as
described here could survive a live migration?  In general, I really like this
approach if it can be done cleanly.  Spice is already oVirt's primary end-user
application so in a deployed environment, we'd expect users to already have this
program.  If a scripted interface is required, I am sure that I/O redirection
could be added either to the existing spice client or as part of a new
spice-console program.  This approach also works with a vdsm that is connected
to ovirt-engine or running in standalone mode.

This seems like the best approach to me so long as the spice team agrees that it
can and should be done.

> 4. oVirt shell -> Engine -> libvirtd
> This is the current workaround described in
> The design is good but I do not like Engine talking to libvirtd
> directly, thus comes the VDSM console streaming API below.
> Work to do
>   Provide console streaming API from Engine to be invoked in oVirt shell.
>   Implement the "serial-console" command in oVirt shell.
> pros
>   Support migration. Engine can reconnect to the guest automatically
> after migration while keeping the connection from oVirt shell.
>   Fit well in the current oVirt architecture: no new server process
> introduced, no new VM introduced, easy to maintain and manage.
> cons
>   Engine talking to libvirtd directly breaks the encapsulation of VDSM.
>   Users only can get the console stream from Engine, they can not
> directly connect to the host as VNC and the above two sshd solutions
> do.

I agree that this is a layering violation and should not be persued as the
long-term solution.  We do not want to expose the libvirt connection outside of
the host.

> 5. VDSM console streaming API
> Implement new APIs in VDSM to forward the raw data from console. It
> exposes getConsoleReadStream() and getConsoleWriteStream() via
> XMLRPC binding. Then Engine can get the console data stream via API
> instead of directly connecting to libvirtd. Other things will be the
> same as solution 4.
> Work to do
>   Implement getConsoleReadStream() and getConsoleWriteStream() in VDSM.
>   Provide console streaming API from Engine to be invoked in oVirt shell.
>   Implement the "serial-console" command in oVirt shell.
>   Optional: Implement a client program in vdsClient to consume the
> stream API.
> pros
>   Same as solution 4
> cons
>   We can not allow ordinary user directly connect to VDSM and invoke
> the stream API, because there is no ACL in VDSM, once a client cert
> is setup for the ordinary user, he can call all the APIs in VDSM and
> get total control. So the ordinary user can only get the stream from
> Engine, and we leave Engine to do the ACL.

One issue that was raised is console buffering.  What happens if a client does
not call getConsoleReadStream() fast enough?  Will characters be dropped?  This
could create a reliability problem and would make scripting against this
interface risky at best.

> I like solution 4 best.

I will note again for others that you mentioned you like #5 (console streaming
API) best.  I think the spice approach is best based on weighing the following

1. Simple and easy to maintain
2. Can access via the host or ovirt-engine
3. Scripting mode is possible
4. Reliable

Adam Litke <>
IBM Linux Technology Center

vdsm-devel mailing list

Reply via email to