Script 'mail_helper' called by obssrc Hello community, here is the log from the commit of package s3backer for openSUSE:Factory checked in at 2023-10-24 20:08:07 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Comparing /work/SRC/openSUSE:Factory/s3backer (Old) and /work/SRC/openSUSE:Factory/.s3backer.new.24901 (New) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Package is "s3backer" Tue Oct 24 20:08:07 2023 rev:28 rq:1119747 version:2.1.1 Changes: -------- --- /work/SRC/openSUSE:Factory/s3backer/s3backer.changes 2023-09-25 21:02:21.494372956 +0200 +++ /work/SRC/openSUSE:Factory/.s3backer.new.24901/s3backer.changes 2023-10-24 20:08:15.616313109 +0200 @@ -1,0 +2,10 @@ +Mon Oct 23 20:55:49 UTC 2023 - Archie Cobbs <[email protected]> + +- Upgrade to release 2.1.1 + + Fixed bug with bulk delete HTTP requests (issue #211) + + Fixed empty payload bug when retrying an upload (issue #212) + + Check for errors returned by curl_easy_setopt() + + Added --noCurlCache flag to facilitate debugging + + Bump wait time for nbdkit to start from 1s to 5s + +------------------------------------------------------------------- Old: ---- s3backer-2.1.0.tar.gz New: ---- s3backer-2.1.1.tar.gz ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Other differences: ------------------ ++++++ s3backer.spec ++++++ --- /var/tmp/diff_new_pack.92QSuf/_old 2023-10-24 20:08:16.268336814 +0200 +++ /var/tmp/diff_new_pack.92QSuf/_new 2023-10-24 20:08:16.268336814 +0200 @@ -25,7 +25,7 @@ %endif Name: s3backer -Version: 2.1.0 +Version: 2.1.1 Release: 0 Summary: FUSE and NBD single file backing store via Amazon S3 License: GPL-2.0-or-later ++++++ s3backer-2.1.0.tar.gz -> s3backer-2.1.1.tar.gz ++++++ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/s3backer-2.1.0/CHANGES new/s3backer-2.1.1/CHANGES --- old/s3backer-2.1.0/CHANGES 2023-09-23 17:28:38.000000000 +0200 +++ new/s3backer-2.1.1/CHANGES 2023-10-23 22:52:02.000000000 +0200 @@ -1,3 +1,11 @@ +Version 2.1.1 released October 23, 2023 + + - Fixed bug with bulk delete HTTP requests (issue #211) + - Fixed empty payload bug when retrying an upload (issue #212) + - Check for errors returned by curl_easy_setopt() + - Added --noCurlCache flag to facilitate debugging + - Bump wait time for nbdkit to start from 1s to 5s + Version 2.1.0 released September 23, 2023 - Avoid sending an empty Accept-Encoding header (issue #208) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/s3backer-2.1.0/Makefile.am new/s3backer-2.1.1/Makefile.am --- old/s3backer-2.1.0/Makefile.am 2022-07-17 18:56:09.000000000 +0200 +++ new/s3backer-2.1.1/Makefile.am 2023-10-08 19:15:57.000000000 +0200 @@ -2,7 +2,7 @@ # # s3backer - FUSE-based single file backing store via Amazon S3 # -# Copyright 2008-2020 Archie L. Cobbs <[email protected]> +# Copyright 2008-2023 Archie L. Cobbs <[email protected]> # # This program is free software; you can redistribute it and/or # modify it under the terms of the GNU General Public License diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/s3backer-2.1.0/Makefile.in new/s3backer-2.1.1/Makefile.in --- old/s3backer-2.1.0/Makefile.in 2023-09-23 17:35:00.000000000 +0200 +++ new/s3backer-2.1.1/Makefile.in 2023-10-23 22:53:15.000000000 +0200 @@ -17,7 +17,7 @@ # # s3backer - FUSE-based single file backing store via Amazon S3 # -# Copyright 2008-2020 Archie L. Cobbs <[email protected]> +# Copyright 2008-2023 Archie L. Cobbs <[email protected]> # # This program is free software; you can redistribute it and/or # modify it under the terms of the GNU General Public License diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/s3backer-2.1.0/block_cache.c new/s3backer-2.1.1/block_cache.c --- old/s3backer-2.1.0/block_cache.c 2023-02-07 21:13:27.000000000 +0100 +++ new/s3backer-2.1.1/block_cache.c 2023-10-08 19:15:18.000000000 +0200 @@ -2,7 +2,7 @@ /* * s3backer - FUSE-based single file backing store via Amazon S3 * - * Copyright 2008-2020 Archie L. Cobbs <[email protected]> + * Copyright 2008-2023 Archie L. Cobbs <[email protected]> * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/s3backer-2.1.0/block_cache.h new/s3backer-2.1.1/block_cache.h --- old/s3backer-2.1.0/block_cache.h 2022-04-29 19:04:41.000000000 +0200 +++ new/s3backer-2.1.1/block_cache.h 2023-10-08 19:15:18.000000000 +0200 @@ -2,7 +2,7 @@ /* * s3backer - FUSE-based single file backing store via Amazon S3 * - * Copyright 2008-2020 Archie L. Cobbs <[email protected]> + * Copyright 2008-2023 Archie L. Cobbs <[email protected]> * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/s3backer-2.1.0/block_part.c new/s3backer-2.1.1/block_part.c --- old/s3backer-2.1.0/block_part.c 2023-02-07 21:13:27.000000000 +0100 +++ new/s3backer-2.1.1/block_part.c 2023-10-08 19:15:18.000000000 +0200 @@ -2,7 +2,7 @@ /* * s3backer - FUSE-based single file backing store via Amazon S3 * - * Copyright 2008-2020 Archie L. Cobbs <[email protected]> + * Copyright 2008-2023 Archie L. Cobbs <[email protected]> * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/s3backer-2.1.0/block_part.h new/s3backer-2.1.1/block_part.h --- old/s3backer-2.1.0/block_part.h 2022-05-21 21:11:32.000000000 +0200 +++ new/s3backer-2.1.1/block_part.h 2023-10-08 19:15:18.000000000 +0200 @@ -2,7 +2,7 @@ /* * s3backer - FUSE-based single file backing store via Amazon S3 * - * Copyright 2008-2020 Archie L. Cobbs <[email protected]> + * Copyright 2008-2023 Archie L. Cobbs <[email protected]> * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/s3backer-2.1.0/compress.c new/s3backer-2.1.1/compress.c --- old/s3backer-2.1.0/compress.c 2023-02-07 21:13:27.000000000 +0100 +++ new/s3backer-2.1.1/compress.c 2023-10-08 19:15:18.000000000 +0200 @@ -2,7 +2,7 @@ /* * s3backer - FUSE-based single file backing store via Amazon S3 * - * Copyright 2008-2020 Archie L. Cobbs <[email protected]> + * Copyright 2008-2023 Archie L. Cobbs <[email protected]> * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/s3backer-2.1.0/compress.h new/s3backer-2.1.1/compress.h --- old/s3backer-2.1.0/compress.h 2022-05-20 18:37:31.000000000 +0200 +++ new/s3backer-2.1.1/compress.h 2023-10-08 19:15:18.000000000 +0200 @@ -2,7 +2,7 @@ /* * s3backer - FUSE-based single file backing store via Amazon S3 * - * Copyright 2008-2020 Archie L. Cobbs <[email protected]> + * Copyright 2008-2023 Archie L. Cobbs <[email protected]> * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/s3backer-2.1.0/configure new/s3backer-2.1.1/configure --- old/s3backer-2.1.0/configure 2023-09-23 17:35:00.000000000 +0200 +++ new/s3backer-2.1.1/configure 2023-10-23 22:53:14.000000000 +0200 @@ -1,6 +1,6 @@ #! /bin/sh # Guess values for system-dependent variables and create Makefiles. -# Generated by GNU Autoconf 2.69 for s3backer FUSE filesystem backed by Amazon S3 2.1.0. +# Generated by GNU Autoconf 2.69 for s3backer FUSE filesystem backed by Amazon S3 2.1.1. # # Report bugs to <https://github.com/archiecobbs/s3backer>. # @@ -590,8 +590,8 @@ # Identity of this package. PACKAGE_NAME='s3backer FUSE filesystem backed by Amazon S3' PACKAGE_TARNAME='s3backer' -PACKAGE_VERSION='2.1.0' -PACKAGE_STRING='s3backer FUSE filesystem backed by Amazon S3 2.1.0' +PACKAGE_VERSION='2.1.1' +PACKAGE_STRING='s3backer FUSE filesystem backed by Amazon S3 2.1.1' PACKAGE_BUGREPORT='https://github.com/archiecobbs/s3backer' PACKAGE_URL='' @@ -1345,7 +1345,7 @@ # Omit some internal or obsolete options to make the list less imposing. # This message is too long to be a string in the A/UX 3.1 sh. cat <<_ACEOF -\`configure' configures s3backer FUSE filesystem backed by Amazon S3 2.1.0 to adapt to many kinds of systems. +\`configure' configures s3backer FUSE filesystem backed by Amazon S3 2.1.1 to adapt to many kinds of systems. Usage: $0 [OPTION]... [VAR=VALUE]... @@ -1415,7 +1415,7 @@ if test -n "$ac_init_help"; then case $ac_init_help in - short | recursive ) echo "Configuration of s3backer FUSE filesystem backed by Amazon S3 2.1.0:";; + short | recursive ) echo "Configuration of s3backer FUSE filesystem backed by Amazon S3 2.1.1:";; esac cat <<\_ACEOF @@ -1545,7 +1545,7 @@ test -n "$ac_init_help" && exit $ac_status if $ac_init_version; then cat <<\_ACEOF -s3backer FUSE filesystem backed by Amazon S3 configure 2.1.0 +s3backer FUSE filesystem backed by Amazon S3 configure 2.1.1 generated by GNU Autoconf 2.69 Copyright (C) 2012 Free Software Foundation, Inc. @@ -1960,7 +1960,7 @@ This file contains any messages produced by compilers while running configure, to aid debugging if configure makes a mistake. -It was created by s3backer FUSE filesystem backed by Amazon S3 $as_me 2.1.0, which was +It was created by s3backer FUSE filesystem backed by Amazon S3 $as_me 2.1.1, which was generated by GNU Autoconf 2.69. Invocation command line was $ $0 $@ @@ -2823,7 +2823,7 @@ # Define the identity of the package. PACKAGE='s3backer' - VERSION='2.1.0' + VERSION='2.1.1' cat >>confdefs.h <<_ACEOF @@ -14405,7 +14405,7 @@ # report actual input values of CONFIG_FILES etc. instead of their # values after options handling. ac_log=" -This file was extended by s3backer FUSE filesystem backed by Amazon S3 $as_me 2.1.0, which was +This file was extended by s3backer FUSE filesystem backed by Amazon S3 $as_me 2.1.1, which was generated by GNU Autoconf 2.69. Invocation command line was CONFIG_FILES = $CONFIG_FILES @@ -14471,7 +14471,7 @@ cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1 ac_cs_config="`$as_echo "$ac_configure_args" | sed 's/^ //; s/[\\""\`\$]/\\\\&/g'`" ac_cs_version="\\ -s3backer FUSE filesystem backed by Amazon S3 config.status 2.1.0 +s3backer FUSE filesystem backed by Amazon S3 config.status 2.1.1 configured by $0, generated by GNU Autoconf 2.69, with options \\"\$ac_cs_config\\" diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/s3backer-2.1.0/configure.ac new/s3backer-2.1.1/configure.ac --- old/s3backer-2.1.0/configure.ac 2023-09-23 17:28:51.000000000 +0200 +++ new/s3backer-2.1.1/configure.ac 2023-10-23 22:52:09.000000000 +0200 @@ -1,7 +1,7 @@ # # s3backer - FUSE-based single file backing store via Amazon S3 # -# Copyright 2008-2020 Archie L. Cobbs <[email protected]> +# Copyright 2008-2023 Archie L. Cobbs <[email protected]> # # This program is free software; you can redistribute it and/or # modify it under the terms of the GNU General Public License @@ -32,7 +32,7 @@ # this exception statement from all source files in the program, then # also delete it here. -AC_INIT([s3backer FUSE filesystem backed by Amazon S3],[2.1.0],[https://github.com/archiecobbs/s3backer],[s3backer]) +AC_INIT([s3backer FUSE filesystem backed by Amazon S3],[2.1.1],[https://github.com/archiecobbs/s3backer],[s3backer]) AC_CONFIG_AUX_DIR(scripts) AM_INIT_AUTOMAKE(foreign) LT_INIT() diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/s3backer-2.1.0/dcache.c new/s3backer-2.1.1/dcache.c --- old/s3backer-2.1.0/dcache.c 2023-02-07 21:15:55.000000000 +0100 +++ new/s3backer-2.1.1/dcache.c 2023-10-08 19:15:18.000000000 +0200 @@ -2,7 +2,7 @@ /* * s3backer - FUSE-based single file backing store via Amazon S3 * - * Copyright 2008-2020 Archie L. Cobbs <[email protected]> + * Copyright 2008-2023 Archie L. Cobbs <[email protected]> * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/s3backer-2.1.0/dcache.h new/s3backer-2.1.1/dcache.h --- old/s3backer-2.1.0/dcache.h 2022-04-29 19:04:41.000000000 +0200 +++ new/s3backer-2.1.1/dcache.h 2023-10-08 19:15:18.000000000 +0200 @@ -2,7 +2,7 @@ /* * s3backer - FUSE-based single file backing store via Amazon S3 * - * Copyright 2008-2020 Archie L. Cobbs <[email protected]> + * Copyright 2008-2023 Archie L. Cobbs <[email protected]> * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/s3backer-2.1.0/ec_protect.c new/s3backer-2.1.1/ec_protect.c --- old/s3backer-2.1.0/ec_protect.c 2023-02-07 21:13:27.000000000 +0100 +++ new/s3backer-2.1.1/ec_protect.c 2023-10-08 19:15:18.000000000 +0200 @@ -2,7 +2,7 @@ /* * s3backer - FUSE-based single file backing store via Amazon S3 * - * Copyright 2008-2020 Archie L. Cobbs <[email protected]> + * Copyright 2008-2023 Archie L. Cobbs <[email protected]> * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/s3backer-2.1.0/ec_protect.h new/s3backer-2.1.1/ec_protect.h --- old/s3backer-2.1.0/ec_protect.h 2022-04-28 04:09:59.000000000 +0200 +++ new/s3backer-2.1.1/ec_protect.h 2023-10-08 19:15:18.000000000 +0200 @@ -2,7 +2,7 @@ /* * s3backer - FUSE-based single file backing store via Amazon S3 * - * Copyright 2008-2020 Archie L. Cobbs <[email protected]> + * Copyright 2008-2023 Archie L. Cobbs <[email protected]> * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/s3backer-2.1.0/erase.c new/s3backer-2.1.1/erase.c --- old/s3backer-2.1.0/erase.c 2023-02-07 21:13:27.000000000 +0100 +++ new/s3backer-2.1.1/erase.c 2023-10-08 19:15:18.000000000 +0200 @@ -2,7 +2,7 @@ /* * s3backer - FUSE-based single file backing store via Amazon S3 * - * Copyright 2008-2020 Archie L. Cobbs <[email protected]> + * Copyright 2008-2023 Archie L. Cobbs <[email protected]> * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/s3backer-2.1.0/erase.h new/s3backer-2.1.1/erase.h --- old/s3backer-2.1.0/erase.h 2022-04-28 04:09:59.000000000 +0200 +++ new/s3backer-2.1.1/erase.h 2023-10-08 19:15:18.000000000 +0200 @@ -2,7 +2,7 @@ /* * s3backer - FUSE-based single file backing store via Amazon S3 * - * Copyright 2008-2020 Archie L. Cobbs <[email protected]> + * Copyright 2008-2023 Archie L. Cobbs <[email protected]> * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/s3backer-2.1.0/fuse_ops.c new/s3backer-2.1.1/fuse_ops.c --- old/s3backer-2.1.0/fuse_ops.c 2023-02-07 21:13:27.000000000 +0100 +++ new/s3backer-2.1.1/fuse_ops.c 2023-10-08 19:15:18.000000000 +0200 @@ -2,7 +2,7 @@ /* * s3backer - FUSE-based single file backing store via Amazon S3 * - * Copyright 2008-2020 Archie L. Cobbs <[email protected]> + * Copyright 2008-2023 Archie L. Cobbs <[email protected]> * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/s3backer-2.1.0/fuse_ops.h new/s3backer-2.1.1/fuse_ops.h --- old/s3backer-2.1.0/fuse_ops.h 2022-05-21 21:26:46.000000000 +0200 +++ new/s3backer-2.1.1/fuse_ops.h 2023-10-08 19:15:18.000000000 +0200 @@ -2,7 +2,7 @@ /* * s3backer - FUSE-based single file backing store via Amazon S3 * - * Copyright 2008-2020 Archie L. Cobbs <[email protected]> + * Copyright 2008-2023 Archie L. Cobbs <[email protected]> * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/s3backer-2.1.0/gitrev.c new/s3backer-2.1.1/gitrev.c --- old/s3backer-2.1.0/gitrev.c 2023-09-23 17:35:05.000000000 +0200 +++ new/s3backer-2.1.1/gitrev.c 2023-10-23 22:53:21.000000000 +0200 @@ -1 +1 @@ -const char *const s3backer_version = "2.1.0"; +const char *const s3backer_version = "2.1.1"; diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/s3backer-2.1.0/hash.c new/s3backer-2.1.1/hash.c --- old/s3backer-2.1.0/hash.c 2023-02-07 21:13:27.000000000 +0100 +++ new/s3backer-2.1.1/hash.c 2023-10-08 19:15:18.000000000 +0200 @@ -2,7 +2,7 @@ /* * s3backer - FUSE-based single file backing store via Amazon S3 * - * Copyright 2008-2020 Archie L. Cobbs <[email protected]> + * Copyright 2008-2023 Archie L. Cobbs <[email protected]> * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/s3backer-2.1.0/hash.h new/s3backer-2.1.1/hash.h --- old/s3backer-2.1.0/hash.h 2022-04-28 04:09:59.000000000 +0200 +++ new/s3backer-2.1.1/hash.h 2023-10-08 19:15:18.000000000 +0200 @@ -2,7 +2,7 @@ /* * s3backer - FUSE-based single file backing store via Amazon S3 * - * Copyright 2008-2020 Archie L. Cobbs <[email protected]> + * Copyright 2008-2023 Archie L. Cobbs <[email protected]> * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/s3backer-2.1.0/http_io.c new/s3backer-2.1.1/http_io.c --- old/s3backer-2.1.0/http_io.c 2023-09-19 22:34:31.000000000 +0200 +++ new/s3backer-2.1.1/http_io.c 2023-10-19 17:24:10.000000000 +0200 @@ -2,7 +2,7 @@ /* * s3backer - FUSE-based single file backing store via Amazon S3 * - * Copyright 2008-2020 Archie L. Cobbs <[email protected]> + * Copyright 2008-2023 Archie L. Cobbs <[email protected]> * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License @@ -125,8 +125,8 @@ #define LIST_BLOCKS_CHUNK 1000 #define DELETE_BLOCKS_CHUNK 1000 -// Maximum error payload size in bytes -#define MAX_ERROR_PAYLOAD_SIZE 0x100000 +// Maximum payload size in bytes to capture for debug +#define MAX_DEBUG_PAYLOAD_SIZE 100000 // PBKDF2 key generation iterations #define PBKDF2_ITERATIONS 5000 @@ -217,10 +217,10 @@ // I/O buffers struct http_io_bufs { - size_t rdremain; - size_t wrremain; - char *rddata; - const char *wrdata; + size_t rdremain; // number of bytes left in download buffer + size_t wrremain; // number of bytes left in upload buffer + char *rddata; // download buffer + const char *wrdata; // upload buffer }; // I/O state when reading/writing a block @@ -281,8 +281,8 @@ size_t error_payload_len; // error response length }; -// CURL prepper function type -typedef void http_io_curl_prepper_t(CURL *curl, struct http_io *io); +// CURL prepper function type - returns 1 on succes, 0 on error +typedef int http_io_curl_prepper_t(struct http_io_private *const priv, CURL *curl, struct http_io *io); // s3backer_store functions static int http_io_create_threads(struct s3backer_store *s3b); @@ -313,6 +313,7 @@ static int http_io_add_auth4(struct http_io_private *priv, struct http_io *io, time_t now, const void *payload, size_t plen); static size_t url_encode(const char *src, size_t len, char *dst, int buflen, int encode_slash); static void digest_url_encoded(EVP_MD_CTX* hash_ctx, const char *data, size_t len, int encode_slash); +static char *canonicalize_query_string(const char *query, size_t len); // EC2 IAM thread static void *update_iam_credentials_main(void *arg); @@ -374,6 +375,9 @@ static void http_io_init_io(struct http_io_private *priv, struct http_io *io, const char *method, const char *url); static void http_io_curl_header_reset(struct http_io *const io); static int http_io_verify_etag_provided(struct http_io *io); +static int http_io_curl_setopt_long(struct http_io_private *priv, CURL *curl, CURLoption option, long value); +static int http_io_curl_setopt_ptr(struct http_io_private *priv, CURL *curl, CURLoption option, const void *ptr); +static int http_io_curl_setopt_off(struct http_io_private *priv, CURL *curl, CURLoption option, curl_off_t offset); // Internal variables static pthread_mutex_t *openssl_locks; @@ -869,18 +873,25 @@ return r; } -static void -http_io_xml_prepper(CURL *curl, struct http_io *io) +static int +http_io_xml_prepper(struct http_io_private *const priv, CURL *curl, struct http_io *io) { if (io->bufs.wrdata != NULL) { - curl_easy_setopt(curl, CURLOPT_READFUNCTION, http_io_curl_writer); - curl_easy_setopt(curl, CURLOPT_READDATA, io); + if (!http_io_curl_setopt_ptr(priv, curl, CURLOPT_READFUNCTION, http_io_curl_writer) + || !http_io_curl_setopt_ptr(priv, curl, CURLOPT_READDATA, io) + || !http_io_curl_setopt_long(priv, curl, CURLOPT_UPLOAD, 1) + || !http_io_curl_setopt_long(priv, curl, CURLOPT_POST, 1) + || !http_io_curl_setopt_off(priv, curl, CURLOPT_POSTFIELDSIZE_LARGE, (curl_off_t)io->bufs.wrremain)) + return 0; } - curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, http_io_curl_xml_reader); - curl_easy_setopt(curl, CURLOPT_WRITEDATA, io); - curl_easy_setopt(curl, CURLOPT_HTTPHEADER, io->headers); - curl_easy_setopt(curl, CURLOPT_ENCODING, ""); - curl_easy_setopt(curl, CURLOPT_HTTP_CONTENT_DECODING, (long)1); + if (!http_io_curl_setopt_ptr(priv, curl, CURLOPT_WRITEFUNCTION, http_io_curl_xml_reader) + || !http_io_curl_setopt_ptr(priv, curl, CURLOPT_WRITEDATA, io) + || !http_io_curl_setopt_ptr(priv, curl, CURLOPT_HTTPHEADER, io->headers) + || !http_io_curl_setopt_ptr(priv, curl, CURLOPT_ACCEPT_ENCODING, "") + || !http_io_curl_setopt_long(priv, curl, CURLOPT_HTTP_CONTENT_DECODING, 1)) + return 0; + http_io_curl_header_reset(io); + return 1; } static size_t @@ -1207,17 +1218,19 @@ return r; } -static void -http_io_head_prepper(CURL *curl, struct http_io *io) +static int +http_io_head_prepper(struct http_io_private *const priv, CURL *curl, struct http_io *io) { memset(&io->bufs, 0, sizeof(io->bufs)); - curl_easy_setopt(curl, CURLOPT_NOBODY, 1); - curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, http_io_curl_reader); - curl_easy_setopt(curl, CURLOPT_WRITEDATA, io); - curl_easy_setopt(curl, CURLOPT_HEADERFUNCTION, http_io_curl_header); - curl_easy_setopt(curl, CURLOPT_HEADERDATA, io); - curl_easy_setopt(curl, CURLOPT_HTTPHEADER, io->headers); + if (!http_io_curl_setopt_long(priv, curl, CURLOPT_NOBODY, 1) + || !http_io_curl_setopt_ptr(priv, curl, CURLOPT_WRITEFUNCTION, http_io_curl_reader) + || !http_io_curl_setopt_ptr(priv, curl, CURLOPT_WRITEDATA, io) + || !http_io_curl_setopt_ptr(priv, curl, CURLOPT_HEADERFUNCTION, http_io_curl_header) + || !http_io_curl_setopt_ptr(priv, curl, CURLOPT_HEADERDATA, io) + || !http_io_curl_setopt_ptr(priv, curl, CURLOPT_HTTPHEADER, io->headers)) + return 0; http_io_curl_header_reset(io); + return 1; } static int @@ -1460,17 +1473,19 @@ return value; } -static void -http_io_iamcreds_prepper(CURL *curl, struct http_io *io) +static int +http_io_iamcreds_prepper(struct http_io_private *const priv, CURL *curl, struct http_io *io) { memset(&io->bufs, 0, sizeof(io->bufs)); io->bufs.rdremain = io->buf_size; io->bufs.rddata = io->dest; - curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, http_io_curl_reader); - curl_easy_setopt(curl, CURLOPT_WRITEDATA, io); - curl_easy_setopt(curl, CURLOPT_MAXFILESIZE_LARGE, (curl_off_t)io->buf_size); - curl_easy_setopt(curl, CURLOPT_ENCODING, ""); - curl_easy_setopt(curl, CURLOPT_HTTP_CONTENT_DECODING, (long)0); + if (!http_io_curl_setopt_ptr(priv, curl, CURLOPT_WRITEFUNCTION, http_io_curl_reader) + || !http_io_curl_setopt_ptr(priv, curl, CURLOPT_WRITEDATA, io) + || !http_io_curl_setopt_off(priv, curl, CURLOPT_MAXFILESIZE_LARGE, (curl_off_t)io->buf_size) + || !http_io_curl_setopt_ptr(priv, curl, CURLOPT_ACCEPT_ENCODING, "") + || !http_io_curl_setopt_long(priv, curl, CURLOPT_HTTP_CONTENT_DECODING, 0)) + return 0; + return 1; } static int @@ -1769,21 +1784,23 @@ return r; } -static void -http_io_read_prepper(CURL *curl, struct http_io *io) +static int +http_io_read_prepper(struct http_io_private *const priv, CURL *curl, struct http_io *io) { memset(&io->bufs, 0, sizeof(io->bufs)); io->bufs.rdremain = io->buf_size; io->bufs.rddata = io->dest; - curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, http_io_curl_reader); - curl_easy_setopt(curl, CURLOPT_WRITEDATA, io); - curl_easy_setopt(curl, CURLOPT_MAXFILESIZE_LARGE, (curl_off_t)io->buf_size); - curl_easy_setopt(curl, CURLOPT_HTTPHEADER, io->headers); - curl_easy_setopt(curl, CURLOPT_HEADERFUNCTION, http_io_curl_header); - curl_easy_setopt(curl, CURLOPT_HEADERDATA, io); - curl_easy_setopt(curl, CURLOPT_ENCODING, ""); - curl_easy_setopt(curl, CURLOPT_HTTP_CONTENT_DECODING, (long)0); + if (!http_io_curl_setopt_ptr(priv, curl, CURLOPT_WRITEFUNCTION, http_io_curl_reader) + || !http_io_curl_setopt_ptr(priv, curl, CURLOPT_WRITEDATA, io) + || !http_io_curl_setopt_off(priv, curl, CURLOPT_MAXFILESIZE_LARGE, (curl_off_t)io->buf_size) + || !http_io_curl_setopt_ptr(priv, curl, CURLOPT_HTTPHEADER, io->headers) + || !http_io_curl_setopt_ptr(priv, curl, CURLOPT_HEADERFUNCTION, http_io_curl_header) + || !http_io_curl_setopt_ptr(priv, curl, CURLOPT_HEADERDATA, io) + || !http_io_curl_setopt_ptr(priv, curl, CURLOPT_ACCEPT_ENCODING, "") + || !http_io_curl_setopt_long(priv, curl, CURLOPT_HTTP_CONTENT_DECODING, 0)) + return 0; http_io_curl_header_reset(io); + return 1; } /* @@ -1998,27 +2015,30 @@ return 0; } -static void -http_io_write_prepper(CURL *curl, struct http_io *io) +static int +http_io_write_prepper(struct http_io_private *const priv, CURL *curl, struct http_io *io) { memset(&io->bufs, 0, sizeof(io->bufs)); if (io->src != NULL) { io->bufs.wrremain = io->buf_size; io->bufs.wrdata = io->src; } - curl_easy_setopt(curl, CURLOPT_READFUNCTION, http_io_curl_writer); - curl_easy_setopt(curl, CURLOPT_READDATA, io); - curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, http_io_curl_reader); - curl_easy_setopt(curl, CURLOPT_WRITEDATA, io); - curl_easy_setopt(curl, CURLOPT_HEADERFUNCTION, http_io_curl_header); - curl_easy_setopt(curl, CURLOPT_HEADERDATA, io); + if (!http_io_curl_setopt_ptr(priv, curl, CURLOPT_READFUNCTION, http_io_curl_writer) + || !http_io_curl_setopt_ptr(priv, curl, CURLOPT_READDATA, io) + || !http_io_curl_setopt_ptr(priv, curl, CURLOPT_WRITEFUNCTION, http_io_curl_reader) + || !http_io_curl_setopt_ptr(priv, curl, CURLOPT_WRITEDATA, io) + || !http_io_curl_setopt_ptr(priv, curl, CURLOPT_HEADERFUNCTION, http_io_curl_header) + || !http_io_curl_setopt_ptr(priv, curl, CURLOPT_HEADERDATA, io) + || !http_io_curl_setopt_ptr(priv, curl, CURLOPT_CUSTOMREQUEST, io->method) + || !http_io_curl_setopt_ptr(priv, curl, CURLOPT_HTTPHEADER, io->headers)) + return 0; if (io->src != NULL) { - curl_easy_setopt(curl, CURLOPT_UPLOAD, 1); - curl_easy_setopt(curl, CURLOPT_INFILESIZE_LARGE, (curl_off_t)io->buf_size); + if (!http_io_curl_setopt_long(priv, curl, CURLOPT_UPLOAD, 1) + || !http_io_curl_setopt_off(priv, curl, CURLOPT_INFILESIZE_LARGE, (curl_off_t)io->buf_size)) + return 0; } - curl_easy_setopt(curl, CURLOPT_CUSTOMREQUEST, io->method); - curl_easy_setopt(curl, CURLOPT_HTTPHEADER, io->headers); http_io_curl_header_reset(io); + return 1; } static int @@ -2207,11 +2227,25 @@ XML_SetElementHandler(io->xml, http_io_xml_elem_start, end_handler); XML_SetCharacterDataHandler(io->xml, http_io_xml_text); + // Add headers for payload (if needed) + if (io->bufs.wrdata != NULL) { + u_char md5[MD5_DIGEST_LENGTH]; + char md5buf[MD5_DIGEST_LENGTH * 2 + 1]; + + // Add Content-MD5 header + md5_quick(io->bufs.wrdata, io->bufs.wrremain, md5); + http_io_base64_encode(md5buf, sizeof(md5buf), md5, MD5_DIGEST_LENGTH); + io->headers = http_io_add_header(priv, io->headers, "%s: %s", MD5_HEADER, md5buf); + + // Add Content-Type header + io->headers = http_io_add_header(priv, io->headers, "%s: %s", CTYPE_HEADER, "application/xml"); + } + // Add Date header http_io_add_date(priv, io, now); // Add Authorization header - if ((r = http_io_add_auth(priv, io, now, NULL, 0)) != 0) + if ((r = http_io_add_auth(priv, io, now, io->bufs.wrdata, io->bufs.wrremain)) != 0) return r; // Perform operation @@ -2267,8 +2301,10 @@ http_io_perform_io(struct http_io_private *priv, struct http_io *io, http_io_curl_prepper_t *prepper) { struct http_io_conf *const config = priv->config; + struct http_io_bufs obufs; struct timespec delay; CURLcode curl_code; + int last_error = EIO; u_int retry_pause = 0; u_int total_pause; long http_code; @@ -2278,16 +2314,35 @@ CURL *curl; // Debug - if (config->debug) + if (config->debug) { (*config->log)(LOG_DEBUG, "%s %s", io->method, io->url); + if (io->bufs.wrremain > 0) { + size_t chars_to_print = io->bufs.wrremain; + + if (chars_to_print > MAX_DEBUG_PAYLOAD_SIZE) + chars_to_print = MAX_DEBUG_PAYLOAD_SIZE; + (*config->log)(LOG_DEBUG, "HTTP %s request payload:\n%.*s", + io->method, (int)chars_to_print, io->bufs.wrdata); + } + } + + // If we are uploading something, snapshot it now so we can reset on retry + memcpy(&obufs, &io->bufs, sizeof(io->bufs)); // Make attempts for (attempt = 0, total_pause = 0; 1; attempt++, total_pause += retry_pause) { + // Reset upload payload on retry + if (attempt > 0) { + io->bufs.wrdata = obufs.wrdata; + io->bufs.wrremain = obufs.wrremain; + } + // Acquire and initialize CURL instance if ((curl = http_io_acquire_curl(priv, io)) == NULL) return EIO; - (*prepper)(curl, io); + if (!(*prepper)(priv, curl, io)) + return EIO; // Reset error payload capture io->http_status = 0; @@ -2394,6 +2449,7 @@ pthread_mutex_lock(&priv->mutex); priv->stats.curl_timeouts++; CHECK_RETURN(pthread_mutex_unlock(&priv->mutex)); + last_error = ETIMEDOUT; break; case CURLE_HTTP_RETURNED_ERROR: // special handling for some specific HTTP codes switch (http_code) { @@ -2408,21 +2464,22 @@ priv->stats.http_unauthorized++; CHECK_RETURN(pthread_mutex_unlock(&priv->mutex)); http_io_log_error_payload(io); - http_io_free_error_payload(io); - return EACCES; + last_error = EACCES; + break; case HTTP_FORBIDDEN: (*config->log)(LOG_ERR, "rec'd %ld response: %s %s", http_code, io->method, io->url); pthread_mutex_lock(&priv->mutex); priv->stats.http_forbidden++; CHECK_RETURN(pthread_mutex_unlock(&priv->mutex)); http_io_log_error_payload(io); - http_io_free_error_payload(io); - return EPERM; + last_error = EPERM; + break; case HTTP_PRECONDITION_FAILED: (*config->log)(LOG_INFO, "rec'd stale content: %s %s", io->method, io->url); pthread_mutex_lock(&priv->mutex); priv->stats.http_stale++; CHECK_RETURN(pthread_mutex_unlock(&priv->mutex)); + last_error = ESTALE; break; case HTTP_MOVED_PERMANENTLY: case HTTP_FOUND: @@ -2433,6 +2490,7 @@ pthread_mutex_lock(&priv->mutex); priv->stats.http_redirect++; CHECK_RETURN(pthread_mutex_unlock(&priv->mutex)); + last_error = ELOOP; break; case HTTP_NOT_MODIFIED: if (io->expect_304) { @@ -2460,6 +2518,7 @@ } CHECK_RETURN(pthread_mutex_unlock(&priv->mutex)); http_io_log_error_payload(io); + last_error = EIO; break; } break; @@ -2470,15 +2529,19 @@ switch (curl_code) { case CURLE_OUT_OF_MEMORY: priv->stats.curl_out_of_memory++; + last_error = ENOMEM; break; case CURLE_COULDNT_CONNECT: priv->stats.curl_connect_failed++; + last_error = ENXIO; break; case CURLE_COULDNT_RESOLVE_HOST: priv->stats.curl_host_unknown++; + last_error = ENXIO; break; default: priv->stats.curl_other_error++; + last_error = EIO; break; } CHECK_RETURN(pthread_mutex_unlock(&priv->mutex)); @@ -2507,7 +2570,7 @@ // Give up (*config->log)(LOG_ERR, "giving up on: %s %s", io->method, io->url); - return EIO; + return last_error; } /* @@ -2682,6 +2745,7 @@ char access_id[128]; char access_key[128]; char *iam_token = NULL; + char *cquery; struct tm tm; char *p; int r; @@ -2779,11 +2843,16 @@ #endif // Canonical query string - EVP_DigestUpdate(hash_ctx, (const u_char *)query_params, query_params_len); + if ((cquery = canonicalize_query_string(query_params, query_params_len)) == NULL) { + r = errno; + goto fail; + } + EVP_DigestUpdate(hash_ctx, (const u_char *)cquery, strlen(cquery)); EVP_DigestUpdate(hash_ctx, (const u_char *)"\n", 1); #if DEBUG_AUTHENTICATION - snvprintf(sigbuf + strlen(sigbuf), sizeof(sigbuf) - strlen(sigbuf), "%.*s\n", (int)query_params_len, query_params); + snvprintf(sigbuf + strlen(sigbuf), sizeof(sigbuf) - strlen(sigbuf), "%s\n", cquery); #endif + free(cquery); // Canonical headers header_names_length = 0; @@ -2954,6 +3023,46 @@ } /* + * Reformat a query string for digest, adding any missing equals signs. + * NOTE: This assumes that the query string parameters are already sorted. + * Caller must free the result. + */ +static char * +canonicalize_query_string(const char *query, size_t qlen) +{ + char *buf; + char *bp; + size_t i; + int saweq; + + if ((buf = malloc(qlen * 2 + 1)) == NULL) + return NULL; + bp = buf; + saweq = 0; + for (i = 0; i < qlen; i++) { + char c = query[i]; + switch (c) { + case '&': + if (!saweq) + *bp++ = '='; + saweq = 0; + break; + case '=': + if (!saweq) + saweq = 1; + break; + default: + break; + } + *bp++ = c; + } + if (qlen > 0 && !saweq) + *bp++ = '='; + *bp++ = '\0'; + return buf; +} + +/* * Add data to digest, but in URL-encoded form. */ static void @@ -3065,6 +3174,7 @@ struct curl_holder *holder; CURL *curl; + // Get a CURL instance pthread_mutex_lock(&priv->mutex); if ((holder = LIST_FIRST(&priv->curls)) != NULL) { curl = holder->curl; @@ -3085,31 +3195,47 @@ return NULL; } } - curl_easy_setopt(curl, CURLOPT_URL, io->url); - curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1); - curl_easy_setopt(curl, CURLOPT_NOSIGNAL, (long)1); - curl_easy_setopt(curl, CURLOPT_TCP_KEEPALIVE, (long)1); - curl_easy_setopt(curl, CURLOPT_TCP_KEEPIDLE, (long)TCP_KEEP_ALIVE_IDLE); - curl_easy_setopt(curl, CURLOPT_TCP_KEEPINTVL, (long)TCP_KEEP_ALIVE_INTERVAL); - curl_easy_setopt(curl, CURLOPT_TIMEOUT, (long)config->timeout); - curl_easy_setopt(curl, CURLOPT_NOPROGRESS, 1); - curl_easy_setopt(curl, CURLOPT_USERAGENT, config->user_agent); - curl_easy_setopt(curl, CURLOPT_SOCKOPTFUNCTION, http_io_sockopt_callback); - if (config->max_speed[HTTP_UPLOAD] != 0) - curl_easy_setopt(curl, CURLOPT_MAX_SEND_SPEED_LARGE, (curl_off_t)(config->max_speed[HTTP_UPLOAD] / 8)); - if (config->max_speed[HTTP_DOWNLOAD] != 0) - curl_easy_setopt(curl, CURLOPT_MAX_RECV_SPEED_LARGE, (curl_off_t)(config->max_speed[HTTP_DOWNLOAD] / 8)); + + // Set various options + if (!http_io_curl_setopt_ptr(priv, curl, CURLOPT_URL, io->url) + || !http_io_curl_setopt_long(priv, curl, CURLOPT_FOLLOWLOCATION, 1) + || !http_io_curl_setopt_long(priv, curl, CURLOPT_NOSIGNAL, 1) + || !http_io_curl_setopt_long(priv, curl, CURLOPT_TCP_KEEPALIVE, 1) + || !http_io_curl_setopt_long(priv, curl, CURLOPT_TCP_KEEPIDLE, TCP_KEEP_ALIVE_IDLE) + || !http_io_curl_setopt_long(priv, curl, CURLOPT_TCP_KEEPINTVL, TCP_KEEP_ALIVE_INTERVAL) + || !http_io_curl_setopt_long(priv, curl, CURLOPT_TIMEOUT, config->timeout) + || !http_io_curl_setopt_long(priv, curl, CURLOPT_NOPROGRESS, 1) + || !http_io_curl_setopt_ptr(priv, curl, CURLOPT_USERAGENT, config->user_agent) + || !http_io_curl_setopt_ptr(priv, curl, CURLOPT_SOCKOPTFUNCTION, http_io_sockopt_callback)) + goto optfail; + if (config->max_speed[HTTP_UPLOAD] != 0 + && !http_io_curl_setopt_off(priv, curl, CURLOPT_MAX_SEND_SPEED_LARGE, (curl_off_t)(config->max_speed[HTTP_UPLOAD] / 8))) + goto optfail; + if (config->max_speed[HTTP_DOWNLOAD] != 0 + && !http_io_curl_setopt_off(priv, curl, CURLOPT_MAX_RECV_SPEED_LARGE, (curl_off_t)(config->max_speed[HTTP_DOWNLOAD] / 8))) + goto optfail; if (strncmp(io->url, "https", 5) == 0) { - if (config->insecure) - curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, (long)0); - if (config->cacert != NULL) - curl_easy_setopt(curl, CURLOPT_CAINFO, config->cacert); - } - if (config->debug_http) - curl_easy_setopt(curl, CURLOPT_VERBOSE, 1); - if (config->http_11) - curl_easy_setopt(curl, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_1); + if (config->insecure + && !http_io_curl_setopt_long(priv, curl, CURLOPT_SSL_VERIFYPEER, 0)) + goto optfail; + if (config->cacert != NULL + && !http_io_curl_setopt_ptr(priv, curl, CURLOPT_CAINFO, config->cacert)) + goto optfail; + } + if (config->debug_http + && !http_io_curl_setopt_long(priv, curl, CURLOPT_VERBOSE, 1)) + goto optfail; + if (config->http_11 + && !http_io_curl_setopt_long(priv, curl, CURLOPT_HTTP_VERSION, (long)CURL_HTTP_VERSION_1_1)) + goto optfail; + if (strcmp(io->method, HTTP_POST) != 0 + && !http_io_curl_setopt_long(priv, curl, CURLOPT_POST, 0)) + goto optfail; return curl; + +optfail: + http_io_release_curl(priv, &curl, 0); + return NULL; } static size_t @@ -3212,6 +3338,46 @@ } static int +http_io_curl_setopt_long(struct http_io_private *priv, CURL *curl, CURLoption option, long value) +{ + CURLcode curl_code; + + assert(option / 10000 == CURLOPTTYPE_LONG / 10000); + if ((curl_code = curl_easy_setopt(curl, option, value)) != CURLE_OK) { + (*priv->config->log)(LOG_ERR, "curl_easy_setopt(%d): %s", (int)option, curl_easy_strerror(curl_code)); + return 0; + } + return 1; +} + +static int +http_io_curl_setopt_ptr(struct http_io_private *priv, CURL *curl, CURLoption option, const void *ptr) +{ + CURLcode curl_code; + + assert(option / 10000 == CURLOPTTYPE_OBJECTPOINT / 10000 + || option / 10000 == CURLOPTTYPE_FUNCTIONPOINT / 10000); + if ((curl_code = curl_easy_setopt(curl, option, ptr)) != CURLE_OK) { + (*priv->config->log)(LOG_ERR, "curl_easy_setopt(%d): %s", (int)option, curl_easy_strerror(curl_code)); + return 0; + } + return 1; +} + +static int +http_io_curl_setopt_off(struct http_io_private *priv, CURL *curl, CURLoption option, curl_off_t offset) +{ + CURLcode curl_code; + + assert(option / 10000 == CURLOPTTYPE_OFF_T / 10000); + if ((curl_code = curl_easy_setopt(curl, option, offset)) != CURLE_OK) { + (*priv->config->log)(LOG_ERR, "curl_easy_setopt(%d): %s", (int)option, curl_easy_strerror(curl_code)); + return 0; + } + return 1; +} + +static int http_io_sockopt_callback(void *cookie, curl_socket_t fd, curlsocktype purpose) { (void)fcntl(fd, F_SETFD, FD_CLOEXEC); @@ -3233,12 +3399,13 @@ static void http_io_release_curl(struct http_io_private *priv, CURL **curlp, int may_cache) { + struct http_io_conf *const config = priv->config; struct curl_holder *holder; CURL *const curl = *curlp; *curlp = NULL; assert(curl != NULL); - if (!may_cache) { + if (config->no_curl_cache || !may_cache) { curl_easy_cleanup(curl); return; } @@ -3275,8 +3442,8 @@ return 1; // Impose limit on how much error payload we'll remember - if (io->error_payload_len + len > MAX_ERROR_PAYLOAD_SIZE) - len = MAX_ERROR_PAYLOAD_SIZE - io->error_payload_len; + if (io->error_payload_len + len > MAX_DEBUG_PAYLOAD_SIZE) + len = MAX_DEBUG_PAYLOAD_SIZE - io->error_payload_len; // Capture the error payload in the error buffer if ((new_error_payload = realloc(io->error_payload, io->error_payload_len + len)) == NULL) diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/s3backer-2.1.0/http_io.h new/s3backer-2.1.1/http_io.h --- old/s3backer-2.1.0/http_io.h 2022-04-28 04:09:59.000000000 +0200 +++ new/s3backer-2.1.1/http_io.h 2023-10-14 16:07:40.000000000 +0200 @@ -2,7 +2,7 @@ /* * s3backer - FUSE-based single file backing store via Amazon S3 * - * Copyright 2008-2020 Archie L. Cobbs <[email protected]> + * Copyright 2008-2023 Archie L. Cobbs <[email protected]> * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License @@ -80,6 +80,7 @@ int debug_http; int http_11; // restrict to HTTP 1.1 int quiet; + int no_curl_cache; // don't cache cURL handles const struct comp_alg *compress_alg; // compression algorithm, or NULL for none void *compress_level; // compression level info int vhost; // use virtual host style URL diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/s3backer-2.1.0/main.c new/s3backer-2.1.1/main.c --- old/s3backer-2.1.0/main.c 2023-02-07 21:13:27.000000000 +0100 +++ new/s3backer-2.1.1/main.c 2023-10-16 17:04:21.000000000 +0200 @@ -2,7 +2,7 @@ /* * s3backer - FUSE-based single file backing store via Amazon S3 * - * Copyright 2008-2020 Archie L. Cobbs <[email protected]> + * Copyright 2008-2023 Archie L. Cobbs <[email protected]> * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License @@ -52,7 +52,7 @@ // Some definitions #define NBD_CLIENT_BLOCK_SIZE 4096 #define NBDKIT_STARTUP_WAIT_PAUSE (long)50 -#define MAX_NBDKIT_STARTUP_WAIT_MILLIS (long)1000 +#define MAX_NBDKIT_STARTUP_WAIT_MILLIS (long)5000 #define NBD_MODULE_NAME "nbd" // Internal state diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/s3backer-2.1.0/nbdkit.c new/s3backer-2.1.1/nbdkit.c --- old/s3backer-2.1.0/nbdkit.c 2023-02-07 21:13:27.000000000 +0100 +++ new/s3backer-2.1.1/nbdkit.c 2023-10-19 20:59:12.000000000 +0200 @@ -342,13 +342,13 @@ // Calculate what bits to read, then read them calculate_boundary_info(&info, config->block_size, buf, size, offset); if (info.header.length > 0 && (r = block_part_read_block_part(fuse_priv->s3b, fuse_priv->block_part, &info.header)) != 0) { - nbdkit_error("error reading block %0*jx: %m", S3B_BLOCK_NUM_DIGITS, (uintmax_t)info.header.block); + nbdkit_error("error reading block %0*jx: %s", S3B_BLOCK_NUM_DIGITS, (uintmax_t)info.header.block, strerror(r)); nbdkit_set_error(r); return -1; } while (info.mid_block_count-- > 0) { if ((r = (*fuse_priv->s3b->read_block)(fuse_priv->s3b, info.mid_block_start, info.mid_data, NULL, NULL, 0)) != 0) { - nbdkit_error("error reading block %0*jx: %m", S3B_BLOCK_NUM_DIGITS, (uintmax_t)info.mid_block_start); + nbdkit_error("error reading block %0*jx: %s", S3B_BLOCK_NUM_DIGITS, (uintmax_t)info.mid_block_start, strerror(r)); nbdkit_set_error(r); return -1; } @@ -356,7 +356,7 @@ info.mid_data += config->block_size; } if (info.footer.length > 0 && (r = block_part_read_block_part(fuse_priv->s3b, fuse_priv->block_part, &info.footer)) != 0) { - nbdkit_error("error reading block %0*jx: %m", S3B_BLOCK_NUM_DIGITS, (uintmax_t)info.footer.block); + nbdkit_error("error reading block %0*jx: %s", S3B_BLOCK_NUM_DIGITS, (uintmax_t)info.footer.block, strerror(r)); nbdkit_set_error(r); return -1; } @@ -375,20 +375,20 @@ // Calculate what bits to write, then write them calculate_boundary_info(&info, config->block_size, buf, size, offset); if (info.header.length > 0 && (r = block_part_write_block_part(fuse_priv->s3b, fuse_priv->block_part, &info.header)) != 0) { - nbdkit_error("error writing block %0*jx: %m", S3B_BLOCK_NUM_DIGITS, (uintmax_t)info.header.block); + nbdkit_error("error writing block %0*jx: %s", S3B_BLOCK_NUM_DIGITS, (uintmax_t)info.header.block, strerror(r)); goto fail; } for (i = 0; i < info.mid_block_count; i++) { const s3b_block_t block_num = info.mid_block_start + i; if ((r = (*fuse_priv->s3b->write_block)(fuse_priv->s3b, block_num, info.mid_data, NULL, NULL, NULL)) != 0) { - nbdkit_error("error writing block %0*jx: %m", S3B_BLOCK_NUM_DIGITS, (uintmax_t)block_num); + nbdkit_error("error writing block %0*jx: %s", S3B_BLOCK_NUM_DIGITS, (uintmax_t)block_num, strerror(r)); goto fail; } info.mid_data += config->block_size; } if (info.footer.length > 0 && (r = block_part_write_block_part(fuse_priv->s3b, fuse_priv->block_part, &info.footer)) != 0) { - nbdkit_error("error writing block %0*jx: %m", S3B_BLOCK_NUM_DIGITS, (uintmax_t)info.footer.block); + nbdkit_error("error writing block %0*jx: %s", S3B_BLOCK_NUM_DIGITS, (uintmax_t)info.footer.block, strerror(r)); goto fail; } @@ -414,7 +414,7 @@ // Calculate what bits to trim, then trim them calculate_boundary_info(&info, config->block_size, NULL, size, offset); if (info.header.length > 0 && (r = block_part_write_block_part(fuse_priv->s3b, fuse_priv->block_part, &info.header)) != 0) { - nbdkit_error("error writing block %0*jx: %m", S3B_BLOCK_NUM_DIGITS, (uintmax_t)info.header.block); + nbdkit_error("error writing block %0*jx: %s", S3B_BLOCK_NUM_DIGITS, (uintmax_t)info.header.block, strerror(r)); goto fail; } if (info.mid_block_count > 0) { @@ -432,13 +432,13 @@ r = (*fuse_priv->s3b->bulk_zero)(fuse_priv->s3b, block_nums, info.mid_block_count); free(block_nums); if (r != 0) { - nbdkit_error("error zeroing %jd block(s) starting at %0*jx: %m", - (uintmax_t)info.mid_block_count, S3B_BLOCK_NUM_DIGITS, (uintmax_t)info.mid_block_start); + nbdkit_error("error zeroing %jd block(s) starting at %0*jx: %s", + (uintmax_t)info.mid_block_count, S3B_BLOCK_NUM_DIGITS, (uintmax_t)info.mid_block_start, strerror(r)); goto fail; } } if (info.footer.length > 0 && (r = block_part_write_block_part(fuse_priv->s3b, fuse_priv->block_part, &info.footer)) != 0) { - nbdkit_error("error writing block %0*jx: %m", S3B_BLOCK_NUM_DIGITS, (uintmax_t)info.footer.block); + nbdkit_error("error writing block %0*jx: %s", S3B_BLOCK_NUM_DIGITS, (uintmax_t)info.footer.block, strerror(r)); goto fail; } diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/s3backer-2.1.0/nbdkit.h new/s3backer-2.1.1/nbdkit.h --- old/s3backer-2.1.0/nbdkit.h 2022-05-20 18:37:31.000000000 +0200 +++ new/s3backer-2.1.1/nbdkit.h 2023-10-08 19:15:18.000000000 +0200 @@ -2,7 +2,7 @@ /* * s3backer - FUSE-based single file backing store via Amazon S3 * - * Copyright 2008-2020 Archie L. Cobbs <[email protected]> + * Copyright 2008-2023 Archie L. Cobbs <[email protected]> * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/s3backer-2.1.0/reset.c new/s3backer-2.1.1/reset.c --- old/s3backer-2.1.0/reset.c 2023-02-07 21:13:27.000000000 +0100 +++ new/s3backer-2.1.1/reset.c 2023-10-08 19:15:18.000000000 +0200 @@ -2,7 +2,7 @@ /* * s3backer - FUSE-based single file backing store via Amazon S3 * - * Copyright 2008-2020 Archie L. Cobbs <[email protected]> + * Copyright 2008-2023 Archie L. Cobbs <[email protected]> * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/s3backer-2.1.0/reset.h new/s3backer-2.1.1/reset.h --- old/s3backer-2.1.0/reset.h 2022-04-28 04:09:59.000000000 +0200 +++ new/s3backer-2.1.1/reset.h 2023-10-08 19:15:18.000000000 +0200 @@ -2,7 +2,7 @@ /* * s3backer - FUSE-based single file backing store via Amazon S3 * - * Copyright 2008-2020 Archie L. Cobbs <[email protected]> + * Copyright 2008-2023 Archie L. Cobbs <[email protected]> * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/s3backer-2.1.0/s3b_config.c new/s3backer-2.1.1/s3b_config.c --- old/s3backer-2.1.0/s3b_config.c 2023-02-07 21:13:27.000000000 +0100 +++ new/s3backer-2.1.1/s3b_config.c 2023-10-14 16:11:18.000000000 +0200 @@ -2,7 +2,7 @@ /* * s3backer - FUSE-based single file backing store via Amazon S3 * - * Copyright 2008-2020 Archie L. Cobbs <[email protected]> + * Copyright 2008-2023 Archie L. Cobbs <[email protected]> * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License @@ -362,6 +362,11 @@ .value= 1 }, { + .templ= "--noCurlCache", + .offset= offsetof(struct s3b_config, http_io.no_curl_cache), + .value= 1 + }, + { .templ= "--quiet", .offset= offsetof(struct s3b_config, quiet), .value= 1 @@ -1077,7 +1082,7 @@ // Version if (strcmp(arg, "--version") == 0 || strcmp(arg, "-v") == 0) { fprintf(stderr, "%s version %s (%s)\n", PACKAGE, VERSION, s3backer_version); - fprintf(stderr, "Copyright (C) 2008-2020 Archie L. Cobbs.\n"); + fprintf(stderr, "Copyright (C) 2008-2023 Archie L. Cobbs.\n"); fprintf(stderr, "This is free software; see the source for copying conditions. There is NO\n"); fprintf(stderr, "warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.\n"); exit(0); @@ -1930,6 +1935,7 @@ c->max_speed_str[HTTP_DOWNLOAD] != NULL ? c->max_speed_str[HTTP_DOWNLOAD] : "-", c->http_io.max_speed[HTTP_DOWNLOAD]); (*c->log)(LOG_DEBUG, "%24s: %s", "http_11", c->http_io.http_11 ? "true" : "false"); + (*c->log)(LOG_DEBUG, "%24s: %s", "noCurlCache", c->http_io.no_curl_cache ? "true" : "false"); (*c->log)(LOG_DEBUG, "%24s: %us", "timeout", c->http_io.timeout); (*c->log)(LOG_DEBUG, "%24s: \"%s\"", "sse", c->http_io.sse); (*c->log)(LOG_DEBUG, "%24s: \"%s\"", "sse-key-id", c->http_io.sse_key_id); @@ -2012,6 +2018,7 @@ fprintf(stderr, "\t--%-27s %s\n", "force", "Ignore different auto-detected block and file sizes"); fprintf(stderr, "\t--%-27s %s\n", "help", "Show this information and exit"); fprintf(stderr, "\t--%-27s %s\n", "http11", "Restrict to HTTP version 1.1"); + fprintf(stderr, "\t--%-27s %s\n", "noCurlCache", "Disable caching of cURL handles"); fprintf(stderr, "\t--%-27s %s\n", "initialRetryPause=MILLIS", "Initial retry pause after stale data or server error"); fprintf(stderr, "\t--%-27s %s\n", "insecure", "Don't verify SSL server identity"); fprintf(stderr, "\t--%-27s %s\n", "keyLength", "Override generated cipher key length"); diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/s3backer-2.1.0/s3b_config.h new/s3backer-2.1.1/s3b_config.h --- old/s3backer-2.1.0/s3b_config.h 2022-07-08 22:33:42.000000000 +0200 +++ new/s3backer-2.1.1/s3b_config.h 2023-10-08 19:15:18.000000000 +0200 @@ -2,7 +2,7 @@ /* * s3backer - FUSE-based single file backing store via Amazon S3 * - * Copyright 2008-2020 Archie L. Cobbs <[email protected]> + * Copyright 2008-2023 Archie L. Cobbs <[email protected]> * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/s3backer-2.1.0/s3backer.1 new/s3backer-2.1.1/s3backer.1 --- old/s3backer-2.1.0/s3backer.1 2023-09-23 17:35:03.000000000 +0200 +++ new/s3backer-2.1.1/s3backer.1 2023-10-23 22:53:19.000000000 +0200 @@ -2,7 +2,7 @@ .\" .\" s3backer - FUSE-based single file backing store via Amazon S3 .\" -.\" Copyright 2008-2020 Archie L. Cobbs <[email protected]> +.\" Copyright 2008-2023 Archie L. Cobbs <[email protected]> .\" .\" This program is free software; you can redistribute it and/or .\" modify it under the terms of the GNU General Public License @@ -918,6 +918,9 @@ If this flag is given, then the block size defaults to 4096 and the .Fl \-size flag is required. +.It Fl \-noCurlCache +Disable caching of cURL handles. +This flag will reduce performance; it exists to help in certain debugging situations. .It Fl \-password=PASSWORD Supply the password for encryption and authentication as a command-line parameter. .It Fl \-passwordFile=FILE diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/s3backer-2.1.0/s3backer.1.in new/s3backer-2.1.1/s3backer.1.in --- old/s3backer-2.1.0/s3backer.1.in 2023-01-19 18:31:06.000000000 +0100 +++ new/s3backer-2.1.1/s3backer.1.in 2023-10-14 16:13:59.000000000 +0200 @@ -2,7 +2,7 @@ .\" .\" s3backer - FUSE-based single file backing store via Amazon S3 .\" -.\" Copyright 2008-2020 Archie L. Cobbs <[email protected]> +.\" Copyright 2008-2023 Archie L. Cobbs <[email protected]> .\" .\" This program is free software; you can redistribute it and/or .\" modify it under the terms of the GNU General Public License @@ -918,6 +918,9 @@ If this flag is given, then the block size defaults to 4096 and the .Fl \-size flag is required. +.It Fl \-noCurlCache +Disable caching of cURL handles. +This flag will reduce performance; it exists to help in certain debugging situations. .It Fl \-password=PASSWORD Supply the password for encryption and authentication as a command-line parameter. .It Fl \-passwordFile=FILE diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/s3backer-2.1.0/s3backer.h new/s3backer-2.1.1/s3backer.h --- old/s3backer-2.1.0/s3backer.h 2023-03-20 15:49:47.000000000 +0100 +++ new/s3backer-2.1.1/s3backer.h 2023-10-08 19:15:18.000000000 +0200 @@ -2,7 +2,7 @@ /* * s3backer - FUSE-based single file backing store via Amazon S3 * - * Copyright 2008-2020 Archie L. Cobbs <[email protected]> + * Copyright 2008-2023 Archie L. Cobbs <[email protected]> * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/s3backer-2.1.0/sslcompat.c new/s3backer-2.1.1/sslcompat.c --- old/s3backer-2.1.0/sslcompat.c 2023-02-07 21:13:27.000000000 +0100 +++ new/s3backer-2.1.1/sslcompat.c 2023-10-08 19:15:18.000000000 +0200 @@ -1,7 +1,7 @@ /* * s3backer - FUSE-based single file backing store via Amazon S3 * - * Copyright 2008-2020 Archie L. Cobbs <[email protected]> + * Copyright 2008-2023 Archie L. Cobbs <[email protected]> * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/s3backer-2.1.0/test_io.c new/s3backer-2.1.1/test_io.c --- old/s3backer-2.1.0/test_io.c 2023-03-17 22:35:05.000000000 +0100 +++ new/s3backer-2.1.1/test_io.c 2023-10-08 19:15:18.000000000 +0200 @@ -2,7 +2,7 @@ /* * s3backer - FUSE-based single file backing store via Amazon S3 * - * Copyright 2008-2020 Archie L. Cobbs <[email protected]> + * Copyright 2008-2023 Archie L. Cobbs <[email protected]> * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/s3backer-2.1.0/test_io.h new/s3backer-2.1.1/test_io.h --- old/s3backer-2.1.0/test_io.h 2022-04-28 04:09:59.000000000 +0200 +++ new/s3backer-2.1.1/test_io.h 2023-10-08 19:15:18.000000000 +0200 @@ -2,7 +2,7 @@ /* * s3backer - FUSE-based single file backing store via Amazon S3 * - * Copyright 2008-2020 Archie L. Cobbs <[email protected]> + * Copyright 2008-2023 Archie L. Cobbs <[email protected]> * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/s3backer-2.1.0/tester.c new/s3backer-2.1.1/tester.c --- old/s3backer-2.1.0/tester.c 2023-02-07 21:13:27.000000000 +0100 +++ new/s3backer-2.1.1/tester.c 2023-10-08 19:15:18.000000000 +0200 @@ -2,7 +2,7 @@ /* * s3backer - FUSE-based single file backing store via Amazon S3 * - * Copyright 2008-2020 Archie L. Cobbs <[email protected]> + * Copyright 2008-2023 Archie L. Cobbs <[email protected]> * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/s3backer-2.1.0/util.c new/s3backer-2.1.1/util.c --- old/s3backer-2.1.0/util.c 2023-09-23 17:33:02.000000000 +0200 +++ new/s3backer-2.1.1/util.c 2023-10-08 19:15:18.000000000 +0200 @@ -2,7 +2,7 @@ /* * s3backer - FUSE-based single file backing store via Amazon S3 * - * Copyright 2008-2020 Archie L. Cobbs <[email protected]> + * Copyright 2008-2023 Archie L. Cobbs <[email protected]> * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/s3backer-2.1.0/util.h new/s3backer-2.1.1/util.h --- old/s3backer-2.1.0/util.h 2023-03-20 15:57:55.000000000 +0100 +++ new/s3backer-2.1.1/util.h 2023-10-08 19:15:18.000000000 +0200 @@ -2,7 +2,7 @@ /* * s3backer - FUSE-based single file backing store via Amazon S3 * - * Copyright 2008-2020 Archie L. Cobbs <[email protected]> + * Copyright 2008-2023 Archie L. Cobbs <[email protected]> * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/s3backer-2.1.0/zero_cache.c new/s3backer-2.1.1/zero_cache.c --- old/s3backer-2.1.0/zero_cache.c 2023-02-07 21:13:27.000000000 +0100 +++ new/s3backer-2.1.1/zero_cache.c 2023-10-08 19:15:18.000000000 +0200 @@ -2,7 +2,7 @@ /* * s3backer - FUSE-based single file backing store via Amazon S3 * - * Copyright 2008-2020 Archie L. Cobbs <[email protected]> + * Copyright 2008-2023 Archie L. Cobbs <[email protected]> * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/s3backer-2.1.0/zero_cache.h new/s3backer-2.1.1/zero_cache.h --- old/s3backer-2.1.0/zero_cache.h 2022-04-28 04:09:59.000000000 +0200 +++ new/s3backer-2.1.1/zero_cache.h 2023-10-08 19:15:18.000000000 +0200 @@ -2,7 +2,7 @@ /* * s3backer - FUSE-based single file backing store via Amazon S3 * - * Copyright 2008-2020 Archie L. Cobbs <[email protected]> + * Copyright 2008-2023 Archie L. Cobbs <[email protected]> * * This program is free software; you can redistribute it and/or * modify it under the terms of the GNU General Public License
