Package: release.debian.org Control: affects -1 + src:curl X-Debbugs-Cc: [email protected] User: [email protected] Usertags: pu Tags: bookworm Severity: normal
[ Reason ] I would like to backport wcurl into stable, wcurl is a script which we ship as part of the curl package on unstable and testing. wcurl is a command line tool which lets you download URLs without having to remember any parameters. https://samueloph.dev/blog/announcing-wcurl-a-curl-wrapper-to-download-files/ https://curl.se/wcurl/ [ Impact ] Users need to use wcurl from bookworm-backports or wait until the next stable release. [ Tests ] wcurl has unit testing and is shipped and tested in other OSes. [ Risks ] wcurl is a POSIX-compliant shell script with 318 lines of code including comments. There's a risk here if the user already installed something at /usr/bin/wcurl, but if that's the case, it's very likely they manually installed wcurl itself. [ Checklist ] [x] *all* changes are documented in the d/changelog [x] I reviewed all changes and I approve them [x] attach debdiff against the package in (old)stable [x] the issue is verified as fixed in unstable [ Changes ] Install wcurl and wcurl.1 on the curl package. [ Other info ] This is not a bugfix, so I will understand if the release team rejects this. I wanted to at least create the request as I believe it's a good change for stable users. -- Samuel Henrique <samueloph>
diff -Nru curl-7.88.1/debian/changelog curl-7.88.1/debian/changelog --- curl-7.88.1/debian/changelog 2024-09-17 21:29:24.000000000 +0200 +++ curl-7.88.1/debian/changelog 2024-12-29 16:49:11.000000000 +0100 @@ -1,3 +1,9 @@ +curl (7.88.1-10+deb12u9) bookworm; urgency=medium + + * Install wcurl on curl package + + -- Samuel Henrique <[email protected]> Sun, 29 Dec 2024 16:49:11 +0100 + curl (7.88.1-10+deb12u8) bookworm; urgency=medium * Team upload. diff -Nru curl-7.88.1/debian/curl.install curl-7.88.1/debian/curl.install --- curl-7.88.1/debian/curl.install 2024-09-17 21:29:24.000000000 +0200 +++ curl-7.88.1/debian/curl.install 2024-12-29 16:49:11.000000000 +0100 @@ -1,3 +1,4 @@ #!/usr/bin/dh-exec usr/bin/curl <!cross> usr/share/zsh/* +../wcurl/wcurl usr/bin diff -Nru curl-7.88.1/debian/curl.manpages curl-7.88.1/debian/curl.manpages --- curl-7.88.1/debian/curl.manpages 2024-09-17 21:29:24.000000000 +0200 +++ curl-7.88.1/debian/curl.manpages 2024-12-29 16:49:11.000000000 +0100 @@ -1 +1,2 @@ docs/curl.1 +debian/wcurl/wcurl.1 diff -Nru curl-7.88.1/debian/wcurl/wcurl curl-7.88.1/debian/wcurl/wcurl --- curl-7.88.1/debian/wcurl/wcurl 1970-01-01 01:00:00.000000000 +0100 +++ curl-7.88.1/debian/wcurl/wcurl 2024-12-29 16:49:11.000000000 +0100 @@ -0,0 +1,318 @@ +#!/bin/sh + +# wcurl - a simple wrapper around curl to easily download files. +# +# Requires curl >= 7.46.0 (2015) +# +# Copyright (C) Samuel Henrique <[email protected]>, Sergio Durigan +# Junior <[email protected]> and many contributors, see the AUTHORS +# file. +# +# Permission to use, copy, modify, and distribute this software for any purpose +# with or without fee is hereby granted, provided that the above copyright +# notice and this permission notice appear in all copies. +# +# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OF THIRD PARTY RIGHTS. IN +# NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, +# DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR +# OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE +# OR OTHER DEALINGS IN THE SOFTWARE. +# +# Except as contained in this notice, the name of a copyright holder shall not be +# used in advertising or otherwise to promote the sale, use or other dealings in +# this Software without prior written authorization of the copyright holder. +# +# SPDX-License-Identifier: curl + +# Stop on errors and on usage of unset variables. +set -eu + +VERSION="2024.12.08" + +PROGRAM_NAME="$(basename "$0")" +readonly PROGRAM_NAME + +# Display the version. +print_version() +{ + cat << _EOF_ +${VERSION} +_EOF_ +} + +# Display the program usage. +usage() +{ + cat << _EOF_ +${PROGRAM_NAME} -- a simple wrapper around curl to easily download files. + +Usage: ${PROGRAM_NAME} <URL>... + ${PROGRAM_NAME} [--curl-options <CURL_OPTIONS>]... [--no-decode-filename] [-o|-O|--output <PATH>] [--dry-run] [--] <URL>... + ${PROGRAM_NAME} [--curl-options=<CURL_OPTIONS>]... [--no-decode-filename] [--output=<PATH>] [--dry-run] [--] <URL>... + ${PROGRAM_NAME} -h|--help + ${PROGRAM_NAME} -V|--version + +Options: + + --curl-options <CURL_OPTIONS>: Specify extra options to be passed when invoking curl. May be + specified more than once. + + -o, -O, --output <PATH>: Use the provided output path instead of getting it from the URL. If + multiple URLs are provided, all files will have the same name with a + number appended to the end (curl >= 7.83.0). If this option is provided + multiple times, only the last value is considered. + + --no-decode-filename: Don't percent-decode the output filename, even if the percent-encoding in + the URL was done by wcurl, e.g.: The URL contained whitespaces. + + --dry-run: Don't actually execute curl, just print what would be invoked. + + -V, --version: Print version information. + + -h, --help: Print this usage message. + + <URL>: The URL to be downloaded. May be specified more than once. + + <CURL_OPTIONS>: Any option supported by curl can be set here. This is not used by wcurl; it's + instead forwarded to the curl invocation. + + <URL>: The URL to be downloaded. May be specified more than once. +_EOF_ +} + +# Display an error message and bail out. +error() +{ + printf "%s\n" "$*" > /dev/stderr + exit 1 +} + +# Extra curl options provided by the user. +# This will be set per-URL for every URL provided. +# Some options are global, but we are erroring on the side of needlesly setting +# them multiple times instead of causing issues with parameters that needs to +# be set per-URL. +CURL_OPTIONS="" + +# The URLs to be downloaded. +URLS="" + +# Will be set to the percent-decoded filename parsed from the URL, unless +# --output or --no-decode-filename are used. +OUTPUT_PATH="" +HAS_USER_SET_OUTPUT="false" + +# The parameters that will be passed per-URL to curl. +readonly PER_URL_PARAMETERS="\ + --fail \ + --globoff \ + --location \ + --proto-default https \ + --remote-time \ + --retry 10 \ + --retry-max-time 10 " + +# Whether to invoke curl or not. +DRY_RUN="false" + +# Sanitize parameters. +sanitize() +{ + if [ -z "${URLS}" ]; then + error "You must provide at least one URL to download." + fi + + readonly CURL_OPTIONS URLS DRY_RUN HAS_USER_SET_OUTPUT +} + +# Indicate via exit code whether the string given in the first parameter +# consists solely of characters from the string given in the second parameter. +# In other words, it returns 0 if the first parameter only contains characters +# from the second parameter, e.g.: Are $1 characters a subset of $2 characters? +is_subset_of() +{ + case "${1}" in + *[!${2}]*|'') return 1;; + esac +} + +# Print the given string percent-decoded. +percent_decode() +{ + # Encodings of control characters (00-1F) are passed through without decoding. + # Iterate on the input character-by-character, decoding it. + printf "%s\n" "${1}" | fold -w1 | while IFS= read -r decode_out; do + # If character is a "%", read the next character as decode_hex1. + if [ "${decode_out}" = % ] && IFS= read -r decode_hex1; then + decode_out="${decode_out}${decode_hex1}" + # If there's one more character, read it as decode_hex2. + if IFS= read -r decode_hex2; then + decode_out="${decode_out}${decode_hex2}" + # Skip decoding if this is a control character (00-1F). + # Skip decoding if DECODE_FILENAME is not "true". + if is_subset_of "${decode_hex1}" "23456789abcdefABCDEF" && \ + is_subset_of "${decode_hex2}" "0123456789abcdefABCDEF" && \ + [ "${DECODE_FILENAME}" = "true" ]; then + # Use printf to decode it into octal and then decode it to the final format. + decode_out="$(printf "%b" "\\$(printf %o "0x${decode_hex1}${decode_hex2}")")" + fi + fi + fi + printf %s "${decode_out}" + done +} + +# Print the percent-decoded filename portion of the given URL. +get_url_filename() +{ + # Remove protocol and query string if present. + hostname_and_path="$(printf %s "${1}" | sed -e 's,^[^/]*//,,' -e 's,?.*$,,')" + # If what remains contains a slash, there's a path; return it percent-decoded. + case "${hostname_and_path}" in + # sed to remove everything preceeding the last '/', e.g.: "example/something" becomes "something" + */*) percent_decode "$(printf %s "${hostname_and_path}" | sed -e 's,^.*/,,')";; + esac + # No slash means there was just a hostname and no path; return empty string. +} + +# Execute curl with the list of URLs provided by the user. +exec_curl() +{ + CMD="curl " + + # Store version to check if it supports --no-clobber and --parallel. + curl_version=$($CMD --version | cut -f2 -d' ' | head -n1) + curl_version_major=$(echo "$curl_version" | cut -f1 -d.) + curl_version_minor=$(echo "$curl_version" | cut -f2 -d.) + + CURL_HAS_NO_CLOBBER="" + CURL_HAS_PARALLEL="" + # --no-clobber is only supported since 7.83.0. + # --parallel is only supported since 7.66.0. + if [ "${curl_version_major}" -ge 8 ]; then + CURL_HAS_NO_CLOBBER="--no-clobber" + CURL_HAS_PARALLEL="--parallel" + elif [ "${curl_version_major}" -eq 7 ];then + if [ "${curl_version_minor}" -ge 83 ]; then + CURL_HAS_NO_CLOBBER="--no-clobber" + fi + if [ "${curl_version_minor}" -ge 66 ]; then + CURL_HAS_PARALLEL="--parallel" + fi + fi + + # Detecting whether we need --parallel. It's easier to rely on + # the shell's argument parsing. + # shellcheck disable=SC2086 + set -- $URLS + + if [ "$#" -gt 1 ]; then + CURL_PARALLEL="$CURL_HAS_PARALLEL" + else + CURL_PARALLEL="" + fi + + # Start assembling the command. + # + # We use 'set --' here (again) because (a) we don't have arrays on + # POSIX shell, and (b) we need better control over the way we + # split arguments. + # + # shellcheck disable=SC2086 + set -- ${CMD} ${CURL_PARALLEL} + + NEXT_PARAMETER="" + for url in ${URLS}; do + # If the user did not provide an output path, define one. + if [ "${HAS_USER_SET_OUTPUT}" = "false" ]; then + OUTPUT_PATH="$(get_url_filename "${url}")" + # If we could not get a path from the URL, use the default: index.html. + [ -z "${OUTPUT_PATH}" ] && OUTPUT_PATH=index.html + fi + # shellcheck disable=SC2086 + set -- "$@" ${NEXT_PARAMETER} ${PER_URL_PARAMETERS} ${CURL_HAS_NO_CLOBBER} ${CURL_OPTIONS} --output "${OUTPUT_PATH}" "${url}" + NEXT_PARAMETER="--next" + done + + if [ "${DRY_RUN}" = "false" ]; then + exec "$@" + else + printf "%s\n" "$@" + fi +} + +# Default to decoding the output filename +DECODE_FILENAME="true" + +# Use "${1-}" in order to avoid errors because of 'set -u'. +while [ -n "${1-}" ]; do + case "${1}" in + --curl-options=*) + opt=$(printf "%s\n" "${1}" | sed 's/^--curl-options=//') + CURL_OPTIONS="${CURL_OPTIONS} ${opt}" + ;; + + --curl-options) + shift + CURL_OPTIONS="${CURL_OPTIONS} ${1}" + ;; + + --dry-run) + DRY_RUN="true" + ;; + + --output=*) + opt=$(printf "%s\n" "${1}" | sed 's/^--output=//') + HAS_USER_SET_OUTPUT="true" + OUTPUT_PATH="${opt}" + ;; + + -o|-O|--output) + shift + HAS_USER_SET_OUTPUT="true" + OUTPUT_PATH="${1}" + ;; + + --no-decode-filename) + DECODE_FILENAME="false" + ;; + + -h|--help) + usage + exit 0 + ;; + + -V|--version) + print_version + exit 0 + ;; + + --) + # This is the start of the list of URLs. + shift + for url in "$@"; do + # Encode whitespaces into %20, since wget supports those URLs. + newurl=$(printf "%s\n" "${url}" | sed 's/ /%20/g') + URLS="${URLS} ${newurl}" + done + break + ;; + + -*) + error "Unknown option: '$1'." + ;; + + *) + # This must be a URL. + # Encode whitespaces into %20, since wget supports those URLs. + newurl=$(printf "%s\n" "${1}" | sed 's/ /%20/g') + URLS="${URLS} ${newurl}" + ;; + esac + shift +done + +sanitize +exec_curl diff -Nru curl-7.88.1/debian/wcurl/wcurl.1 curl-7.88.1/debian/wcurl/wcurl.1 --- curl-7.88.1/debian/wcurl/wcurl.1 1970-01-01 01:00:00.000000000 +0100 +++ curl-7.88.1/debian/wcurl/wcurl.1 2024-12-29 16:49:11.000000000 +0100 @@ -0,0 +1,127 @@ +.\" ************************************************************************** +.\" * _ _ ____ _ +.\" * Project ___| | | | _ \| | +.\" * / __| | | | |_) | | +.\" * | (__| |_| | _ <| |___ +.\" * \___|\___/|_| \_\_____| +.\" * +.\" * Copyright (C) Samuel Henrique <[email protected]>, et al. +.\" * +.\" * This software is licensed as described in the file COPYING, which +.\" * you should have received as part of this distribution. The terms +.\" * are also available at https://curl.se/docs/copyright.html. +.\" * +.\" * You may opt to use, copy, modify, merge, publish, distribute and/or sell +.\" * copies of the Software, and permit persons to whom the Software is +.\" * furnished to do so, under the terms of the COPYING file. +.\" * +.\" * This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF ANY +.\" * KIND, either express or implied. +.\" * +.\" * SPDX-License-Identifier: curl +.\" * +.\" ************************************************************************** +.TH wcurl "1" "2024.12.08" "wcurl" "User Commands" +.SH NAME +.B wcurl +- a simple wrapper around curl to easily download files. +.SH SYNOPSIS +.nf +\fBwcurl \fI<URL>\fP...\fR +\fBwcurl [\-\-curl\-options \fI<CURL_OPTIONS>\fP]... [\-\-dry\-run] [\-\-no\-decode\-filename] [\-o|\-O|\-\-output <PATH>] [\-\-] \fI<URL>\fP...\fR +\fBwcurl [\-\-curl\-options=\fI<CURL_OPTIONS>\fP]... [\-\-dry\-run] [\-\-no\-decode\-filename] [\-\-output=<PATH>] [\-\-] \fI<URL>\fP...\fR +\fBwcurl \-V|\-\-version\fR +\fBwcurl \-h|\-\-help\fR +.fi +.SH DESCRIPTION +\fBwcurl\fR is a simple curl wrapper which lets you use curl to download files +without having to remember any parameters. +.PP +Simply call \fBwcurl\fR with a list of URLs you want to download and \fBwcurl\fR will pick +sane defaults. +.PP +If you need anything more complex, you can provide any of curl's supported +parameters via the \fB\-\-curl\-options\fR option. Just beware that you likely +should be using curl directly if your use case is not covered. +.PP +.TP +By default, \fBwcurl\fR will: +.br +\[bu] Percent-encode whitespaces in URLs; +.br +\[bu] Download multiple URLs in parallel if the installed curl's version is >= 7.66.0; +.br +\[bu] Follow redirects; +.br +\[bu] Automatically choose a filename as output; +.br +\[bu] Avoid overwriting files if the installed curl's version is >= 7.83.0 (--no-clobber); +.br +\[bu] Perform retries; +.br +\[bu] Set the downloaded file timestamp to the value provided by the server, if available; +.br +\[bu] Default to the protocol used as https if the URL doesn't contain any; +.br +\[bu] Disable \fBcurl\fR's URL globbing parser so \fB{}\fR and \fB[]\fR characters in URLs are not treated specially. +.br +\[bu] Percent-decode the resulting filename. +.br +\[bu] Use "index.html" as default filename if there's none in the URL. +.SH OPTIONS +.TP +\fB\-\-curl\-options, \-\-curl\-options=\fI<CURL_OPTIONS>\fR...\fR +Specify extra options to be passed when invoking curl. May be specified more than once. +.TP +\fB\-o, \-O, \-\-output=\fI<PATH>\fR...\fR +Use the provided output path instead of getting it from the URL. If multiple +URLs are provided, all files will have the same name with a number appended to +the end (curl >= 7.83.0). If this option is provided multiple times, only the +last value is considered. +.TP +\fB\-\-dry\-run\fR +Don't actually execute curl, just print what would be invoked. +.TP +\fB\-V, \-\-version\fR +Print version information. +.TP +\fB\-h, \-\-help\fR +Print help message. +.SH CURL_OPTIONS +Any option supported by curl can be set here. +This is not used by \fBwcurl\fR; it's instead forwarded to the curl invocation. +.SH URL +Anything which is not a parameter will be considered an URL. +\fBwcurl\fR will percent-encode whitespaces and pass that to curl, which will perform the +parsing of the URL. +.SH EXAMPLES +Download a single file: +.br +\fBwcurl example.com/filename.txt\fR +.PP +Download two files in parallel: +.br +\fBwcurl example.com/filename1.txt example.com/filename2.txt\fR +.PP +Download a file passing the \fI\-\-progress\-bar\fR and \fI\-\-http2\fR flags to curl: +.br +\fBwcurl \-\-curl\-options="\-\-progress\-bar \-\-http2" example.com/filename.txt\fR +.PP +Resume from an interrupted download (if more options are used, this needs to be the last one in the list): +.br +\fBwcurl \-\-curl\-options="\-\-continue-at \-" example.com/filename.txt\fR +.SH AUTHORS +Samuel Henrique <[email protected]> +.br +Sergio durigan junior <[email protected]> +.br +and many contributors, see the AUTHORS file. +.SH REPORTING BUGS +If you experience any problems with \fBwcurl\fR that you do not experience with curl, +submit an issue on Github: +.br +https://github.com/curl/wcurl +.SH COPYRIGHT +\fBwcurl\fR is licensed under the curl license +.SH SEE ALSO +.BR curl (1)
