Re: [Libguestfs] [PATCH nbdkit 2/2] curl: Fallback to GET if HEAD not supported

2023-06-06 Thread Richard W.M. Jones
Michael Henriksen pointed out an issue with this approach. If the web server is actually generating the content on the fly then it may send it as chunked encoding, and in HTTP/1.1 it's not required that the Content-Length field is present (since it may not be known when the server begins

Re: [Libguestfs] [PATCH nbdkit 2/2] curl: Fallback to GET if HEAD not supported

2023-06-06 Thread Richard W.M. Jones
On Tue, Jun 06, 2023 at 06:09:09PM +0200, Laszlo Ersek wrote: > only superficial comments: > > On 6/6/23 13:22, Richard W.M. Jones wrote: [...] > > diff --git a/tests/test-curl-head-forbidden.c > > b/tests/test-curl-head-forbidden.c > > new file mode 100644 > > index 0..16b1f0533 > > ---

Re: [Libguestfs] [PATCH nbdkit 2/2] curl: Fallback to GET if HEAD not supported

2023-06-06 Thread Laszlo Ersek
only superficial comments: On 6/6/23 13:22, Richard W.M. Jones wrote: > Some servers do not support HEAD for requesting the headers. If the > HEAD request fails, fallback to using the GET method, abandoning the > transfer as soon as possible after the headers have been received. > > Fixes:

Re: [Libguestfs] [PATCH nbdkit 1/2] tests/web-server.c: Ignore SIGPIPE

2023-06-06 Thread Laszlo Ersek
On 6/6/23 13:22, Richard W.M. Jones wrote: > If the client (curl plugin) disconnects early then the whole test can > fail with SIGPIPE, and it can be unclear why the test failed -- you > just get the mysterious error "FAIL test (exit status: 141)". We > always check the return code from write(2)

Re: [Libguestfs] [PATCH nbdkit 0/2] curl: Fallback to GET if HEAD not supported

2023-06-06 Thread Richard W.M. Jones
On Tue, Jun 06, 2023 at 12:22:12PM +0100, Richard W.M. Jones wrote: > I haven't been able to test it against a real server, but I modified > our internal test web server so it can behave like this and added a > test. > > Fixes: https://github.com/kubevirt/containerized-data-importer/issues/2737

Re: [Libguestfs] [PATCH libguestfs 1/2] ocaml/implicit_close test: collect all currently unreachable blocks

2023-06-06 Thread Richard W.M. Jones
On Sat, May 27, 2023 at 03:32:36PM +0200, Jürgen Hötzel wrote: > Fixes failing implice_close test on OCaml 5. > --- > ocaml/t/guestfs_065_implicit_close.ml | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) > > diff --git a/ocaml/t/guestfs_065_implicit_close.ml >

[Libguestfs] [PATCH nbdkit 1/2] tests/web-server.c: Ignore SIGPIPE

2023-06-06 Thread Richard W.M. Jones
If the client (curl plugin) disconnects early then the whole test can fail with SIGPIPE, and it can be unclear why the test failed -- you just get the mysterious error "FAIL test (exit status: 141)". We always check the return code from write(2) so just report EPIPE failures through that. In

[Libguestfs] [PATCH nbdkit 0/2] curl: Fallback to GET if HEAD not supported

2023-06-06 Thread Richard W.M. Jones
AWS S3 servers are mad. You can do an unclear "pre-signing" operation on them, but only for a single method (eg. just GET). This means that the way we get the size of the export, by first querying the headers with HEAD and then using GET for partial data transfers, does not work. In theory we

[Libguestfs] [PATCH nbdkit 2/2] curl: Fallback to GET if HEAD not supported

2023-06-06 Thread Richard W.M. Jones
Some servers do not support HEAD for requesting the headers. If the HEAD request fails, fallback to using the GET method, abandoning the transfer as soon as possible after the headers have been received. Fixes: https://github.com/kubevirt/containerized-data-importer/issues/2737 ---