Re: [Qemu-devel] [PATCH] block/curl: Add support for Secure Shell (ssh/sftp) block device
On Fri, Mar 22, 2013 at 01:04:55PM +, Richard W.M. Jones wrote: I got it working with Curl, patch attached. However there are multiple issues (these are mainly notes for myself): (1) libcurl cannot read the size of the file. I had to hard-code this. This is probably just a shortcoming of libcurl (libssh2/sftp itself can read the size of files). Will try to work on a patch for upstream. After my holiday and in the cold of day I've had a long look at the SFTP implementation in libcurl. https://github.com/bagder/curl/blob/master/lib/ssh.c It's implemented as a huge state machine and simply implementing (1) above is problematic (I believe we would have to reopen the connection after our call to curl_easy_perform). The larger issue is that the qemu curl block driver doesn't support writes. Now these could in theory be added. Indeed curl does support random access writes, although AFAICT you have to open a new connection each time you want to seek backwards, and you can't read and write over the same connection (so you'd need = 1 connections for reading and another = 1 connections for writing). I think I will continue with the pure libssh2-based block driver, making it support all the missing features discussed earlier: http://www.mail-archive.com/qemu-devel@nongnu.org/msg161997.html plus of course non-blocking AIO. I think this way we'll end up with a much more robust, reliable and easier to debug ssh implementation in qemu. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones libguestfs lets you edit virtual machines. Supports shell scripting, bindings from many languages. http://libguestfs.org
Re: [Qemu-devel] [PATCH] block/curl: Add support for Secure Shell (ssh/sftp) block device
On Mon, Mar 25, 2013 at 1:32 PM, Richard W.M. Jones rjo...@redhat.com wrote: On Fri, Mar 22, 2013 at 01:04:55PM +, Richard W.M. Jones wrote: I got it working with Curl, patch attached. However there are multiple issues (these are mainly notes for myself): (1) libcurl cannot read the size of the file. I had to hard-code this. This is probably just a shortcoming of libcurl (libssh2/sftp itself can read the size of files). Will try to work on a patch for upstream. After my holiday and in the cold of day I've had a long look at the SFTP implementation in libcurl. https://github.com/bagder/curl/blob/master/lib/ssh.c It's implemented as a huge state machine and simply implementing (1) above is problematic (I believe we would have to reopen the connection after our call to curl_easy_perform). The larger issue is that the qemu curl block driver doesn't support writes. Now these could in theory be added. Indeed curl does support random access writes, although AFAICT you have to open a new connection each time you want to seek backwards, and you can't read and write over the same connection (so you'd need = 1 connections for reading and another = 1 connections for writing). I think I will continue with the pure libssh2-based block driver, making it support all the missing features discussed earlier: http://www.mail-archive.com/qemu-devel@nongnu.org/msg161997.html plus of course non-blocking AIO. I think this way we'll end up with a much more robust, reliable and easier to debug ssh implementation in qemu. Fair enough. Stefan
Re: [Qemu-devel] [PATCH] block/curl: Add support for Secure Shell (ssh/sftp) block device
On Fri, Mar 22, 2013 at 01:04:55PM +, Richard W.M. Jones wrote: I got it working with Curl, patch attached. However there are multiple issues (these are mainly notes for myself): (1) libcurl cannot read the size of the file. I had to hard-code this. This is probably just a shortcoming of libcurl (libssh2/sftp itself can read the size of files). Will try to work on a patch for upstream. (2) Fedora's curl (which is heavily patched) is broken in some way and deadlocks itself. Upstream curl from git works better. I haven't yet identified which patch/commit is responsible. (3) ssh-agent authentication doesn't work. It appears that either ssh-agent itself doesn't like multiple connections from a single process (qemu), or libcurl/libssh2 is having a problem with making multiple connections out to ssh-agent. If I disable ssh-agent auth, it works. Still investigating this. (4) You must specify a user@ in the URL, else libcurl tries to authenticate as user . I will see if I can send a fix for this upstream. (5) Although it gets much of the way through a boot of a guest, it eventually segfaults. Still investigating this. (6) There are several more issues marked by XXX's in the code. Thank you for improving libcurl! You're making it better for everybody. A lot of people go back to NIH when they hit limitations in existing software. Stefan
