Hello, I'd like to propose several improvements in gnupload. The patch is attached.
This version of gnupload has been successfully used with alpha.gnu.org, ftp.gnu.org and download.gnu.org.ua. Additional documentation about its usage is available from: http://puszcza.gnu.org.ua/cookbook/?func=detailitem&item_id=136 The list of changes follows: 1. Ability to upload to download.gnu.org.ua (uploads via sftp). 2. Configuration file. At startup, gnupload searches the current working directory for a file named .gnupload. If found, this file is parsed according to the following rules: * Empty lines are removed. * Lines beginning with a # (comments) are removed. * Newline characters are replaced by spaces. * The resulting string is prepended to the actual command line. This allows to spare typing most commonly used options. E.g.: # A sample .gnupload file --to ftp://ftp.gnu.org.ua:tar --user 55D0C732 # End of file 3. Support for removal and creating symlinks. Three command line options are introduced for this purpose. The --rmsymlink option causes removal of the symlinks listed in the command line. The --symlink option creates symbolic links. It takes even number of arguments, e.g.: --symlink a b c d will create two symbolic links: a -> b and c -> d. Finally, the --symlink-re option allows to create symbolic links along with uploading new releases. For example, the following command: gnupload --to ... --symlink-re foo-1.2.tar.gz uploads the file foo-1.2.tar.gz and creates a symbolic link: foo-latest.tar.gz -> foo-1.2.tar.gz. As an optional argument, this option allows to specify a sed expression to transform file names into link names. 4. Any number of operations can be specified in a single invocation, e.g.: gnupload --to alpha.gnu.org:tar \ --delete tar-1.20.90.tar.gz tar-1.20.90.tar.bz2 \ --rmsymlink tar-latest.tar.gz tar-latest.tar.gz2 \ -- tar-1.20.91.tar.gz (double-dash in this case is needed to separate files to upload from --rmsymlink arguments). 5. Debugging features: The --dry-run option causes gnupload to print what it would have done, without actually uploading anything. It also prints the contents of the created directive file(s). The --to command option allows for the argument in form DIR:SUBDIR, where DIR is an absolute directory name (--to /tmp:tar). This instructs gnupload to copy the created files to /tmp. This is useful for debugging the script itself and for debugging the software implementing automated upload procedures. Regards, Sergey
Index: lib/gnupload --- orig/gnupload 2008-12-28 15:44:04.000000000 +0200 +++ lib/gnupload 2009-01-24 15:54:33.000000000 +0200 @@ -1,9 +1,9 @@ #!/bin/sh # Sign files and upload them. -scriptversion=2008-11-12.21 +scriptversion=2009-01-23.23 -# Copyright (C) 2004, 2005, 2006, 2007, 2008 Free Software Foundation +# Copyright (C) 2004, 2005, 2006, 2007, 2008, 2009 Free Software Foundation # # This program is free software; you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by @@ -24,97 +24,192 @@ set -e GPG='gpg --batch --no-tty' to= -delete=false +DRY_RUN= +SYMLINK_FILES= +DELETE_FILES= +DELETE_SYMLINKS= +COLLECT_VAR= +DBG= -usage="Usage: $0 [OPTIONS]... FILES... +usage="Usage: $0 [OPTIONS]... [COMMAND] FILES... [[COMMAND] FILES...] -Sign all FILES, and upload them to (or delete them from) selected -destinations, according to +Sign all FILES, and upload them to selected destinations, according to <http://www.gnu.org/prep/maintain/html_node/Automated-FTP-Uploads.html>. +Commands: + --delete delete FILES from destination + --symlink create symbolic links + --rmsymlink remove symbolic links + -- treat the remaining arguments as files to upload + Options: --help print this help text and exit --to DEST specify one destination for FILES (multiple --to options are allowed) --user NAME sign with key NAME - --delete delete FILES from destination instead of uploading + --symlink-re[=SED-EXPR] use SED-EXPR to create symbolic links + --dry-run do nothing, show everything --version output version information and exit +If --symlink-re without SED-EXPR is given, symlink target name is +created by replacing version information with the word \`-latest', +e.g.: + + foo-1.3.4.tar.gz -> foo-latest.tar.gz + Recognized destinations are: alpha.gnu.org:DIRECTORY savannah.gnu.org:DIRECTORY savannah.nongnu.org:DIRECTORY ftp.gnu.org:DIRECTORY build directive files and upload files by FTP + download.gnu.org.ua:{alpha|ftp}/DIRECTORY + build directive files and upload files by SFTP [u...@]host:DIRECTORY upload files with scp -Deletion only works for ftp.gnu.org and alpha.gnu.org (using the -archive: directive). Otherwise it is a no-op. Deleting a file foo also -deletes foo.sig; do not specify the .sig explicitly. - -Simple single-target single-file examples: - gnupload --to alpha.gnu.org:automake automake-1.8.2b.tar.gz - gnupload --to ftp.gnu.org:automake automake-1.8.3.tar.gz - gnupload --to alpha.gnu.org:automake --delete automake-oops.tar.gz +If the file .gnupload exists in the current working directory, its contents +is prepended to the actual command line options. Use this to keep your +defaults. Comments (#) and empty lines in .gnupload are allowed. + +Examples: +1. Upload automake-1.8.2b.tar.gz and automake-1.8.2b.tar.bz2 to two sites: + gnupload --to sources.redhat.com:~ftp/pub/automake \\ + --to alpha.gnu.org:automake \\ + automake-1.8.2b.tar.gz automake-1.8.2b.tar.bz2 -Multiple-target multiple-file example: +2. Same as above, but also create symbolic links to automake-latest.tar.*: gnupload --to sources.redhat.com:~ftp/pub/automake \\ --to alpha.gnu.org:automake \\ + --symlink-re \\ + automake-1.8.2b.tar.gz automake-1.8.2b.tar.bz2 + +3. Symlink automake-1.8.2b.tar.gz to automake-latest.tar.gz and +automake-1.8.2b.tar.bz2 to automake-latest.tar.bz2 on both sites: + + gnupload --to sources.redhat.com:~ftp/pub/automake \\ + --to alpha.gnu.org:automake \\ + --symlink automake-1.8.2b.tar.gz automake-latest.tar.gz \\ + automake-latest.tar.bz2 automake-latest.tar.bz2 + +4. Delete automake-1.8.2a.tar.gz and .bz2, remove symlink +automake-latest.tar.gz and upload automake-1.8.2b.tar.gz: + + gnupload --to sources.redhat.com:~ftp/pub/automake \\ + --to alpha.gnu.org:automake \\ + --delete automake-1.8.2a.tar.gz automake-1.8.2a.tar.bz2 \\ + --rmsymlink automake-latest.tar.gz \\ + -- \\ automake-1.8.2b.tar.gz automake-1.8.2b.tar.bz2 Report bugs to <[email protected]>. Send patches to <[email protected]>." +# Read local configuration file +if [ -r .gnupload ]; then + echo "$0: Reading configuration file .gnupload" + eval set -- "`sed 's/#.*$//;/^$/d' .gnupload | tr '\n' ' '` $*" +fi + while test -n "$1"; do case $1 in - --delete) - delete=true - shift - ;; - --help) - echo "$usage" - exit $? - ;; - --to) - if test -z "$2"; then - echo "$0: Missing argument for --to" 1>&2 - exit 1 - else - to="$to $2" - shift 2 - fi + -*) COLLECT_VAR= + case $1 in + --help) + echo "$usage" + exit $? + ;; + --to) + if test -z "$2"; then + echo "$0: Missing argument for --to" 1>&2 + exit 1 + else + to="$to $2" + shift 2 + fi + ;; + --user) + if test -z "$2"; then + echo "$0: Missing argument for --user" 1>&2 + exit 1 + else + GPG="$GPG --local-user $2" + shift 2 + fi + ;; + --delete) + COLLECT_VAR=DELETE_FILES + shift + ;; + --rmsymlink) + COLLECT_VAR=DELETE_SYMLINKS + shift + ;; + --symlink-re=*) + SYMLINK_EXPR=${1##--symlink=} + shift + ;; + --symlink-re) + SYMLINK_EXPR='s|-[0-9][0-9\.]*\(-[0-9][0-9]*\)\?\.|-latest.|' + shift + ;; + --symlink) + COLLECT_VAR=SYMLINK_FILES + shift + ;; + --dry-run|-n) + DRY_RUN=1 + shift + ;; + --version) + echo "gnupload $scriptversion" + exit $? + ;; + --) + shift + break + ;; + -*) + echo "$0: Unknown option \`$1', try \`$0 --help'" 1>&2 + exit 1 + ;; + esac ;; - --user) - if test -z "$2"; then - echo "$0: Missing argument for --user" 1>&2 - exit 1 + *) if test -z "$COLLECT_VAR"; then + break else - GPG="$GPG --local-user $2" - shift 2 + eval $COLLECT_VAR=\"\$$COLLECT_VAR $1\" + shift fi ;; - --version) - echo "gnupload $scriptversion" - exit $? - ;; - -*) - echo "$0: Unknown option \`$1', try \`$0 --help'" 1>&2 - exit 1 - ;; - *) - break - ;; - esac + esac done -if test $# = 0; then - echo "$0: No file to upload or delete" 1>&2 +dprint() { + echo "Running $*..." +} + +if test -n "$DRY_RUN"; then + DBG=dprint +fi + +if test -z "$to"; then + echo "$0: Missing destination sites" >&2 exit 1 -else - : fi -if $delete; then :; else +if test -n "$SYMLINK_FILES"; then + if test -n "`echo "$SYMLINK_FILES" | sed 's/[^ ]//g;s/ //g'`"; then + echo "$0: Odd number of symlink arguments" >&2 + exit 1 + fi +fi + +if test $# = 0; then + if test -z "${SYMLINK_FILES}${DELETE_FILES}${DELETE_SYMLINKS}"; then + echo "$0: No file to upload" 1>&2 + exit 1 + fi +else # Make sure all files exist. We don't want to ask # for the passphrase if the script will fail. for file @@ -122,8 +217,15 @@ if $delete; then :; else if test ! -f $file; then echo "$0: Cannot find \`$file'" 1>&2 exit 1 - else - : + elif test -n "$SYMLINK_EXPR"; then + linkname=`echo $file | sed "$SYMLINK_EXPR"` + if test -z "$linkname"; then + echo "$0: symlink expression produces empty results" >&2 + exit 1 + elif test $linkname = $file; then + echo "$0: symlink expression does not alter file name" >&2 + exit 1 + fi fi done fi @@ -143,8 +245,7 @@ read -r passphrase stty echo echo -# Nothing to sign if deleting. -if $delete; then :; else +if test $# -ne 0; then for file do echo "Signing $file..." @@ -153,59 +254,143 @@ if $delete; then :; else done fi +# mkdirective DESTDIR BASE FILE STMT +# Arguments: See upload, below +mkdirective() { + stmt="$4" + if test -n "$3"; then + stmt=" +filename: $3$stmt" + fi + + cat >${2}.directive<<EOF +version: 1.1 +directory: $1 +comment: gnupload v. $scriptversion$stmt +EOF + if test -n "$DRY_RUN"; then + echo "File ${2}.directive:" + cat ${2}.directive + echo "File ${2}.directive:" | sed 's/./-/g' + fi +} + +mksymlink() { + while test $# -ne 0 + do + echo "symlink: $1 $2" + shift + shift + done +} + +# upload DEST DESTDIR BASE FILE STMT FILES +# Arguments: +# DEST Destination site; +# DESTDIR Destination directory; +# BASE Base name for the directive file; +# FILE Name of the file to distribute (may be empty); +# STMT Additional statements for the directive file; +# FILES List of files to upload. +upload() { + dest=$1 + destdir=$2 + base=$3 + file=$4 + stmt=$5 + files=$6 + + rm -f $base.directive $base.directive.asc + case $dest in + alpha.gnu.org:*) + mkdirective "$destdir" "$base" "$file" "$stmt" + echo "$passphrase" | $GPG --passphrase-fd 0 --clearsign $base.directive + $DBG ncftpput ftp-upload.gnu.org /incoming/alpha $files $base.directive.asc + ;; + ftp.gnu.org:*) + mkdirective "$destdir" "$base" "$file" "$stmt" + echo "$passphrase" | $GPG --passphrase-fd 0 --clearsign $base.directive + $DBG ncftpput ftp-upload.gnu.org /incoming/ftp $files $base.directive.asc + ;; + savannah.gnu.org:*) + if test -z "$files"; then + echo "$0: warning: standalone directives not applicable for $dest" >&2 + fi + $DBG ncftpput savannah.gnu.org /incoming/savannah/$destdir $files + ;; + savannah.nongnu.org:*) + if test -z "$files"; then + echo "$0: warning: standalone directives not applicable for $dest" >&2 + fi + $DBG ncftpput savannah.nongnu.org /incoming/savannah/$destdir $files + ;; + download.gnu.org.ua:alpha/*|download.gnu.org.ua:ftp/*) + mkdirective "${destdir#*/}" "$base" "$file" "$stmt" + echo "$passphrase" | $GPG --passphrase-fd 0 --clearsign $base.directive + for f in $files $base.directive.asc + do + echo put $f + done | $DBG sftp -b - puszcza.gnu.org.ua:/incoming/${destdir%%/*} + ;; + /*) + mkdirective "$destdir" "$base" "$file" "$stmt" + echo "$passphrase" | $GPG --passphrase-fd 0 --clearsign $base.directive + $DBG cp $files $base.directive.asc ${dest%%:*} + ;; + *) + if test -z "$files"; then + echo "$0: warning: standalone directives not applicable for $dest" >&2 + fi + $DBG scp $files $dest + ;; + esac + rm -f $base.directive $base.directive.asc +} + +##### +# Process any standalone directives +stmt= +if test -n "$SYMLINK_FILES"; then + stmt="$stmt +`mksymlink $SYMLINK_FILES`" +fi + +for file in $DELETE_FILES +do + stmt="$stmt +archive: $file" +done + +for file in $DELETE_SYMLINKS +do + stmt="$stmt +rmsymlink: $file" +done + +if test -n "$stmt"; then + for dest in $to + do + destdir=`echo $dest | sed 's/[^:]*://'` + upload "$dest" "$destdir" "`hostname`-$$" "" "$stmt" + done +fi + +# Process actual uploads for dest in $to do for file do - # Prepare arguments. - if $delete; then - echo "Removing $file from $dest..." - files= # nothing to upload if deleting - directive="archive: $file" - else - echo "Uploading $file to $dest..." - files="$file $file.sig" - directive="filename: "`basename -- "$file"` - fi + echo "Uploading $file to $dest..." + stmt= + files="$file $file.sig" destdir=`echo $dest | sed 's/[^:]*://'` - - case $dest in - alpha.gnu.org:*) - rm -f $file.directive $file.directive.asc - cat >$file.directive<<EOF -version: 1.1 -directory: $destdir -$directive -EOF - echo "$passphrase" | $GPG --passphrase-fd 0 --clearsign $file.directive - ncftpput ftp-upload.gnu.org /incoming/alpha $files $file.directive.asc - rm -f $file.directive $file.directive.asc - ;; - ftp.gnu.org:*) - rm -f $file.directive $file.directive.asc - cat >$file.directive<<EOF -version: 1.1 -directory: $destdir -$directive -EOF - echo "$passphrase" | $GPG --passphrase-fd 0 --clearsign $file.directive - ncftpput ftp-upload.gnu.org /incoming/ftp $files $file.directive.asc - rm -f $file.directive $file.directive.asc - ;; - savannah.gnu.org:*) - # We only know how to implement delete for {ftp,alpha}.gnu.org. - $delete \ - || ncftpput savannah.gnu.org /incoming/savannah/$destdir $files - ;; - savannah.nongnu.org:*) - $delete \ - || ncftpput savannah.nongnu.org /incoming/savannah/$destdir $files - ;; - *) - $delete \ - || scp $files $dest - ;; - esac + if test -n "$SYMLINK_EXPR"; then + linkname=`echo $file | sed "$SYMLINK_EXPR"` + stmt="$stmt +symlink: $file $linkname +symlink: $file.sig $linkname.sig" + fi + upload "$dest" "$destdir" "$file" "$file" "$stmt" "$files" done done
