gnupload: Support -h
The first time I ran gnupload, I used -h and got an error. ;-) This patch adds -h as an alias for --help. Cheers, Ben 2018-11-28 Ben Elliston * build-aux/gnupload: Support -h. diff --git a/build-aux/gnupload b/build-aux/gnupload index 37d7b6806..a2856fe6b 100755 --- a/build-aux/gnupload +++ b/build-aux/gnupload @@ -148,7 +148,7 @@ while test -n "$1"; do -*) collect_var= case $1 in ---help) +-h | --help) echo "$usage" exit $? ;; signature.asc Description: PGP signature
gnupload: fix Shellcheck warnings
This patch silences some warnings from Shellcheck, mostly about using
POSIX $(..) command substitutions instead of old backtick
substitutions.
Cheers, Ben
2018-11-28 Ben Elliston
* build-aux/gnupload: Fix some Shellcheck warnings.
diff --git a/build-aux/gnupload b/build-aux/gnupload
index 37d7b6806..a2856fe6b 100755
--- a/build-aux/gnupload
+++ b/build-aux/gnupload
@@ -29,10 +29,10 @@ GPG=gpg
# "gpg-agent is not available in this session" error
# when gpg-agent is version 2 but gpg is still version 1.
# FIXME-2020: remove, once all major distros ship gpg version 2 as /usr/bin/gpg
-gpg_agent_version=`(gpg-agent --version) 2>/dev/null | sed -e '2,$d' -e
's/^[^0-9]*//'`
+gpg_agent_version=$( (gpg-agent --version) 2>/dev/null | sed -e '2,$d' -e
's/^[^0-9]*//')
case "$gpg_agent_version" in
2.*)
-gpg_version=`(gpg --version) 2>/dev/null | sed -e '2,$d' -e 's/^[^0-9]*//'`
+gpg_version=$( (gpg --version) 2>/dev/null | sed -e '2,$d' -e
's/^[^0-9]*//')
case "$gpg_version" in
1.*)
if (type gpg2) >/dev/null 2>/dev/null; then
@@ -138,7 +138,7 @@ Send patches and bug reports to ."
# Read local configuration file
if test -r "$conffile"; then
echo "$0: Reading configuration file $conffile"
- conf=`sed 's/#.*$//;/^$/d' "$conffile" | tr "\015$nl" ' '`
+ conf=$(sed 's/#.*$//;/^$/d' "$conffile" | tr "\\015$nl" ' ')
eval set x "$conf \"\$@\""
shift
fi
@@ -186,7 +186,7 @@ while test -n "$1"; do
collect_var=delete_symlinks
;;
--symlink-regex=*)
- symlink_expr=`expr "$1" : '[^=]*=\(.*\)'`
+ symlink_expr=$(expr "$1" : '[^=]*=\(.*\)')
;;
--symlink-regex)
symlink_expr='s|-[0-9][0-9\.]*\(-[0-9][0-9]*\)\{0,1\}\.|-latest.|'
@@ -237,7 +237,7 @@ if test -z "$to"; then
fi
if test -n "$symlink_files"; then
- x=`echo "$symlink_files" | sed 's/[^ ]//g;s/ //g'`
+ x=$(echo "$symlink_files" | sed 's/[^ ]//g;s/ //g')
if test -n "$x"; then
echo "$0: Odd number of symlink arguments" >&2
exit 1
@@ -258,7 +258,7 @@ else
echo "$0: Cannot find '$file'" 1>&2
exit 1
elif test -n "$symlink_expr"; then
- linkname=`echo $file | sed "$symlink_expr"`
+ linkname=$(echo $file | sed "$symlink_expr")
if test -z "$linkname"; then
echo "$0: symlink expression produces empty results" >&2
exit 1
@@ -282,7 +282,7 @@ GNUPGHOME=${GNUPGHOME:-$HOME/.gnupg}
# Remember this script runs with 'set -e', so if echo is not built-in
# it will exit now.
if $dry_run || grep -q "^use-agent" $GNUPGHOME/gpg.conf; then :; else
- PATH=/empty echo -n "Enter GPG passphrase: "
+ PATH=/empty printf "Enter GPG passphrase: "
stty -echo
read -r passphrase
stty echo
@@ -374,8 +374,8 @@ upload ()
$dbg ncftpput savannah.nongnu.org /incoming/savannah/$destdir $files
;;
download.gnu.org.ua:alpha/*|download.gnu.org.ua:ftp/*)
- destdir_p1=`echo "$destdir" | sed 's,^[^/]*/,,'`
- destdir_topdir=`echo "$destdir" | sed 's,/.*,,'`
+ destdir_p1=$(echo "$destdir" | sed 's,^[^/]*/,,')
+ destdir_topdir=$(echo "$destdir" | sed 's,/.*,,')
mkdirective "$destdir_p1" "$base" "$file" "$stmt"
echo "$passphrase" | $dbg $GPG $passphrase_fd_0 --clearsign
$base.directive
for f in $files $base.directive.asc
@@ -384,7 +384,7 @@ upload ()
done | $dbg sftp -b - puszcza.gnu.org.ua:/incoming/$destdir_topdir
;;
/*)
- dest_host=`echo "$dest" | sed 's,:.*,,'`
+ dest_host=$(echo "$dest" | sed 's,:.*,,')
mkdirective "$destdir" "$base" "$file" "$stmt"
echo "$passphrase" | $dbg $GPG $passphrase_fd_0 --clearsign
$base.directive
$dbg cp $files $base.directive.asc $dest_host
@@ -404,7 +404,7 @@ upload ()
stmt=
if test -n "$symlink_files"; then
stmt="$stmt
-`mksymlink $symlink_files`"
+$(mksymlink $symlink_files)"
fi
for file in $delete_files
@@ -422,8 +422,8 @@ done
if test -n "$stmt"; then
for dest in $to
do
-destdir=`echo $dest | sed 's/[^:]*://'`
-upload "$dest" "$destdir" "`hostname`-$$" "" "$stmt"
+destdir=$(echo $dest | sed 's/[^:]*://')
+upload "$dest" "$destdir" "$(hostname)-$$" "" "$stmt"
done
fi
@@ -441,9 +441,9 @@ $replace"
fi
#
files="$file $file.sig"
-destdir=`echo $dest | sed 's/[^:]*://'`
+destdir=$(echo $dest | sed 's/[^:]*://')
if test -n "$symlink_expr"; then
- linkname=`echo $file | sed "$symlink_expr"`
+ linkname=$(echo $file | sed "$symlink_expr")
stmt="$stmt
symlink: $file $linkname
symlink: $file.sig $linkname.sig"
signature.asc
Description: PGP signature
Re: A new module: bitset
> Le 28 nov. 2018 à 06:03, Paul Eggert a écrit : > > Akim Demaille wrote: >> What do you propose? > > A simple, general solution is to use only malloc/calloc/realloc, and to have > bitset functions fail when they exhaust memory. Then I guess the current state does what you want. If you don't want the two occurrences of xmalloc in bitset, define OBSTACK_CHUNK_ALLOC to what you want.
Re: A new module: bitset
Akim Demaille wrote: What do you propose? A simple, general solution is to use only malloc/calloc/realloc, and to have bitset functions fail when they exhaust memory.
Re: A new module: bitset
> Le 27 nov. 2018 à 21:31, Paul Eggert a écrit : > > On 11/26/18 8:15 PM, Akim Demaille wrote: >> The current implementation uses xmalloc, but there are also calls to calloc >> and realloc. > > How can it be right to combine xmalloc with calloc/realloc? I agree. > Shouldn't the implementation use xmalloc, xcalloc, and xrealloc for > consistency? Also, what about applications that don't want to use the x* > variants? What do you propose?
Re: A new module: bitset
On 11/26/18 8:15 PM, Akim Demaille wrote: The current implementation uses xmalloc, but there are also calls to calloc and realloc. How can it be right to combine xmalloc with calloc/realloc? Shouldn't the implementation use xmalloc, xcalloc, and xrealloc for consistency? Also, what about applications that don't want to use the x* variants?
Re: [PATCH 1/2] maint.mk: Split long argument lists
Roman Bolshakov wrote:
> if test -n "$$files"; then
> \
> if test -n "$$prohibit"; then \
> - grep $$with_grep_options $(_ignore_case) -nE "$$prohibit" $$files \
> + echo "$$files" | xargs -n $(VC_ARG_MAX)
> \
> + grep $$with_grep_options $(_ignore_case) -nE "$$prohibit" \
> | grep -vE "$${exclude:-^$$}"
> \
> && { msg="$$halt" $(_sc_say_and_exit) } || :;
> \
It is incorrect to transform
grep OPTIONS FILES
to
echo FILES | xargs -n N grep OPTIONS
because when the last chunk of FILES consists of just 1 file, 'grep'
produces different output. Instead, you need to transform it to
echo FILES | xargs -n N grep OPTIONS /dev/null
See:
$ cd gnulib/modules
$ grep xalloc *-tests
acl-tests:xalloc
copy-file-tests:xalloc
c-xvasprintf-tests:xalloc
obstack-printf-tests:xalloc
regex-quote-tests:xalloc
userspec-tests:xalloc
xalloc-die-tests:tests/test-xalloc-die.c
xalloc-die-tests:tests/test-xalloc-die.sh
xalloc-die-tests:TESTS += test-xalloc-die.sh
xalloc-die-tests:check_PROGRAMS += test-xalloc-die
xalloc-die-tests:test_xalloc_die_LDADD = $(LDADD) @LIBINTL@
$ echo *-tests | xargs -n 1 grep xalloc
xalloc
xalloc
xalloc
xalloc
xalloc
xalloc
tests/test-xalloc-die.c
tests/test-xalloc-die.sh
TESTS += test-xalloc-die.sh
check_PROGRAMS += test-xalloc-die
test_xalloc_die_LDADD = $(LDADD) @LIBINTL@
$ echo *-tests | xargs -n 1 grep xalloc /dev/null
acl-tests:xalloc
copy-file-tests:xalloc
c-xvasprintf-tests:xalloc
obstack-printf-tests:xalloc
regex-quote-tests:xalloc
userspec-tests:xalloc
xalloc-die-tests:tests/test-xalloc-die.c
xalloc-die-tests:tests/test-xalloc-die.sh
xalloc-die-tests:TESTS += test-xalloc-die.sh
xalloc-die-tests:check_PROGRAMS += test-xalloc-die
xalloc-die-tests:test_xalloc_die_LDADD = $(LDADD) @LIBINTL@
Bruno
Re: [PATCH 2/2] maint.mk: Replace grep with $(GREP)
Eric Blake wrote: > > In other words, can you guarantee that $(GREP) will never expand to empty? > > If we change gl_INIT() to AC_REQUIRE([AC_PROG_GREP]), then that should > be sufficient to ensure $(GREP) is set by the time Makefile is parsed, > which in turn will propagate to maint.mk. Yes. And (question to Roman): what the proper place to change, so that gl_INIT() contains AC_REQUIRE([AC_PROG_GREP]) ? Hint: The answer is contained in this part of the Gnulib documentation: https://www.gnu.org/software/gnulib/manual/html_node/Writing-modules.html Bruno
Re: [PATCH 2/2] maint.mk: Replace grep with $(GREP)
On 11/27/18 12:47 PM, Bruno Haible wrote: Yes. And (question to Roman): what the proper place to change, so that gl_INIT() contains AC_REQUIRE([AC_PROG_GREP]) ? Hint: The answer is contained in this part of the Gnulib documentation: https://www.gnu.org/software/gnulib/manual/html_node/Writing-modules.html Another hint - modules/maintainer-makefile already ensures that $(SED) is non-empty :) -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
Re: [PATCH 2/2] maint.mk: Replace grep with $(GREP)
On 11/27/18 12:21 PM, Bruno Haible wrote: Roman Bolshakov wrote: Autoconf can discover an alias for GNU grep and set it in GREP but it takes no effect for maint.mk Does Automake always invoke this Autoconf macro which sets GREP? In other words, can you guarantee that $(GREP) will never expand to empty? If we change gl_INIT() to AC_REQUIRE([AC_PROG_GREP]), then that should be sufficient to ensure $(GREP) is set by the time Makefile is parsed, which in turn will propagate to maint.mk. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org
Re: [PATCH 2/2] maint.mk: Replace grep with $(GREP)
Roman Bolshakov wrote: > Autoconf can discover an alias for GNU > grep and set it in GREP but it takes no effect for maint.mk Does Automake always invoke this Autoconf macro which sets GREP? In other words, can you guarantee that $(GREP) will never expand to empty? Bruno
Re: [PATCH 1/2] maint.mk: Split long argument lists
Hi, > The workaround is to split argument list into chunks that operating > system can process. "getconf ARG_MAX" is used to determine size of the > chunk. Two questions on this: 1) People say that 'getconf ARG_MAX' returns the appromixate number of bytes in a command line. [1] But you use it with 'xargs -n', which gives a limit on the number of arguments. Shouldn't the patch use 'xargs -s' instead? 2) The really available values are slightly smaller. On Linux: $ getconf ARG_MAX 2097152 $ LC_ALL=C xargs --show-limits Your environment variables take up 4744 bytes POSIX upper limit on argument length (this system): 2090360 POSIX smallest allowable upper limit on argument length (all systems): 4096 Maximum length of command we could actually use: 2085616 Size of command buffer we are actually using: 131072 On FreeBSD/x86_64: $ getconf ARG_MAX 262144 $ LC_ALL=C xargs --show-limits Your environment variables take up 353 bytes POSIX upper limit on argument length (this system): 259743 POSIX smallest allowable upper limit on argument length (all systems): 4096 Maximum length of command we could actually use: 259390 Size of command buffer we are actually using: 131072 On macOS: $ getconf ARG_MAX 262144 $ LC_ALL=C xargs --show-limits Your environment variables take up 1262 bytes POSIX upper limit on argument length (this system): 258834 POSIX smallest allowable upper limit on argument length (all systems): 4096 Maximum length of command we could actually use: 257572 Size of command buffer we are actually using: 131072 How about being conservative and dividing the limit by 2, to avoid this margin error? Could it be that your patch works only because xargs uses a command buffer of length 131072, regardless of the value you pass to '-n'? Bruno [1] https://www.cyberciti.biz/faq/linux-unix-arg_max-maximum-length-of-arguments/
[PATCH 1/2] maint.mk: Split long argument lists
$(VC_LIST_EXCEPT) is usually expanded into arguments for a command.
When a project contains too many, some operating systems can't pass all
the arguments because they hit the limit of arguments. FreeBSD and macOS
are known to have the limit of 256k of arguments.
More on the issue:
http://lists.gnu.org/archive/html/bug-gnulib/2015-08/msg00019.html
https://www.redhat.com/archives/libvir-list/2015-August/msg00758.html
The workaround is to split argument list into chunks that operating
system can process. "getconf ARG_MAX" is used to determine size of the
chunk.
In-Reply-To: 55d5f55e.60...@redhat.com
Signed-off-by: Roman Bolshakov
---
top/maint.mk | 55 ++--
1 file changed, 36 insertions(+), 19 deletions(-)
diff --git a/top/maint.mk b/top/maint.mk
index 4889ebacc..c4f21f947 100644
--- a/top/maint.mk
+++ b/top/maint.mk
@@ -54,6 +54,8 @@ GIT = git
VC = $(GIT)
VC_LIST = $(srcdir)/$(_build-aux)/vc-list-files -C $(srcdir)
+# Most operating systems have a limit of arguments
+VC_ARG_MAX = $(shell getconf ARG_MAX)
# You can override this variable in cfg.mk if your gnulib submodule lives
# in a different location.
@@ -303,18 +305,22 @@ define _sc_search_regexp
\
: Filter by content;
\
test -n "$$files" && test -n "$$containing" \
- && { files=$$(grep -l "$$containing" $$files); } || :;\
+ && { files=$$(echo "$$files" \
+ | xargs -n $(VC_ARG_MAX) grep -l "$$containing"); } || :; \
test -n "$$files" && test -n "$$non_containing" \
- && { files=$$(grep -vl "$$non_containing" $$files); } || :; \
+ && { files=$$(echo "$$files" \
+ | xargs -n $(VC_ARG_MAX) grep -vl "$$non_containing"); } || :; \
\
: Check for the construct; \
if test -n "$$files"; then \
if test -n "$$prohibit"; then \
- grep $$with_grep_options $(_ignore_case) -nE "$$prohibit" $$files \
+ echo "$$files" | xargs -n $(VC_ARG_MAX) \
+ grep $$with_grep_options $(_ignore_case) -nE "$$prohibit" \
| grep -vE "$${exclude:-^$$}" \
&& { msg="$$halt" $(_sc_say_and_exit) } || :; \
else \
- grep $$with_grep_options $(_ignore_case) -LE "$$require" $$files \
+ echo "$$files" | xargs -n $(VC_ARG_MAX) \
+ grep $$with_grep_options $(_ignore_case) -LE "$$require" \
| grep .\
&& { msg="$$halt" $(_sc_say_and_exit) } || :; \
fi
\
@@ -323,9 +329,10 @@ define _sc_search_regexp
endef
sc_avoid_if_before_free:
- @$(srcdir)/$(_build-aux)/useless-if-before-free \
- $(useless_free_options) \
- $$($(VC_LIST_EXCEPT) | grep -v useless-if-before-free) && \
+ @$(VC_LIST_EXCEPT) | grep -v useless-if-before-free \
+| xargs -n $(VC_ARG_MAX) \
+ $(srcdir)/$(_build-aux)/useless-if-before-free\
+ $(useless_free_options)&& \
{ echo '$(ME): found useless "if" before "free" above' 1>&2; \
exit 1; } || :
@@ -399,14 +406,16 @@ sc_error_exit_success:
# "FATAL:" should be fully upper-cased in error messages
# "WARNING:" should be fully upper-cased, or fully lower-cased
sc_error_message_warn_fatal:
- @grep -nEA2 '[^rp]error *\(' $$($(VC_LIST_EXCEPT)) \
+ @$(VC_LIST_EXCEPT) | xargs -n $(VC_ARG_MAX) \
+ grep -nEA2 '[^rp]error *\(' \
| grep -E '"Warning|"Fatal|"fatal' && \
{ echo '$(ME): use FATAL, WARNING or warning' 1>&2; \
exit 1; } || :
# Error messages should not start with a capital letter
sc_error_message_uppercase:
- @grep -nEA2 '[^rp]error *\(' $$($(VC_LIST_EXCEPT)) \
+ @$(VC_LIST_EXCEPT) | xargs -n $(VC_ARG_MAX) \
+ grep -nEA2 '[^rp]error *\(' \
| grep -E '"[A-Z]' \
| grep -vE '"FATAL|"WARNING|"Java|"C#|PRIuMAX' && \
{ echo '$(ME): found capitalized error message' 1>&2;
[PATCH 2/2] maint.mk: Replace grep with $(GREP)
A project that uses maint.mk can specify regular expressions that are
not supported in system grep. Autoconf can discover an alias for GNU
grep and set it in GREP but it takes no effect for maint.mk
The patch provides an ability to use GNU grep if it was discovered by
autoconf and by calling GNU grep we don't get the messages in syntax-check:
prohibit_diagnostic_without_format
grep: empty (sub)expression
grep: empty (sub)expression
grep: empty (sub)expression
grep: empty (sub)expression
grep: empty (sub)expression
grep: empty (sub)expression
Signed-off-by: Roman Bolshakov
---
top/maint.mk | 114 +--
1 file changed, 57 insertions(+), 57 deletions(-)
diff --git a/top/maint.mk b/top/maint.mk
index c4f21f947..fc29631f5 100644
--- a/top/maint.mk
+++ b/top/maint.mk
@@ -46,7 +46,7 @@ member-check =
\
# Do not save the original name or timestamp in the .tar.gz file.
# Use --rsyncable if available.
gzip_rsyncable := \
- $(shell gzip --help 2>/dev/null|grep rsyncable >/dev/null \
+ $(shell gzip --help 2>/dev/null|$(GREP) rsyncable >/dev/null \
&& printf %s --rsyncable)
GZIP_ENV = '--no-name --best $(gzip_rsyncable)'
@@ -85,9 +85,9 @@ _sc_excl = \
$(or $(exclude_file_name_regexp--$@),^$$)
VC_LIST_EXCEPT = \
$(VC_LIST) | $(SED) 's|^$(_dot_escaped_srcdir)/||' \
- | if test -f $(srcdir)/.x-$@; then grep -vEf $(srcdir)/.x-$@; \
- else grep -Ev -e "$${VC_LIST_EXCEPT_DEFAULT-ChangeLog}"; fi \
- | grep -Ev -e '($(VC_LIST_ALWAYS_EXCLUDE_REGEX)|$(_sc_excl))' \
+ | if test -f $(srcdir)/.x-$@; then $(GREP) -vEf $(srcdir)/.x-$@; \
+ else $(GREP) -Ev -e "$${VC_LIST_EXCEPT_DEFAULT-ChangeLog}"; fi \
+ | $(GREP) -Ev -e '($(VC_LIST_ALWAYS_EXCLUDE_REGEX)|$(_sc_excl))' \
$(_prepend_srcdir_prefix)
ifeq ($(origin prev_version_file), undefined)
@@ -294,34 +294,34 @@ define _sc_search_regexp
\
: Filter by file name; \
if test -n "$$in_files"; then \
- files=$$(find $(srcdir) | grep -E "$$in_files"\
- | grep -Ev '$(_sc_excl)'); \
+ files=$$(find $(srcdir) | $(GREP) -E "$$in_files" \
+ | $(GREP) -Ev '$(_sc_excl)');\
else
\
files=$$($(VC_LIST_EXCEPT)); \
if test -n "$$in_vc_files"; then \
- files=$$(echo "$$files" | grep -E "$$in_vc_files"); \
+ files=$$(echo "$$files" | $(GREP) -E "$$in_vc_files"); \
fi; \
fi; \
\
: Filter by content;
\
test -n "$$files" && test -n "$$containing" \
&& { files=$$(echo "$$files" \
- | xargs -n $(VC_ARG_MAX) grep -l "$$containing"); } || :; \
+ | xargs -n $(VC_ARG_MAX) $(GREP) -l "$$containing"); } || :;\
test -n "$$files" && test -n "$$non_containing" \
&& { files=$$(echo "$$files" \
- | xargs -n $(VC_ARG_MAX) grep -vl "$$non_containing"); } || :; \
+ | xargs -n $(VC_ARG_MAX) $(GREP) -vl "$$non_containing"); } || :;\
\
: Check for the construct; \
if test -n "$$files"; then \
if test -n "$$prohibit"; then \
echo "$$files" | xargs -n $(VC_ARG_MAX) \
- grep $$with_grep_options $(_ignore_case) -nE "$$prohibit" \
- | grep -vE "$${exclude:-^$$}" \
+ $(GREP) $$with_grep_options $(_ignore_case) -nE "$$prohibit" \
+ | $(GREP) -vE "$${exclude:-^$$}" \
&& { msg="$$halt" $(_sc_say_and_exit) } || :; \
else \
echo "$$files" | xargs -n $(VC_ARG_MAX) \
- grep $$with_grep_options $(_ignore_case) -LE "$$require" \
- | grep .\
+ $(GREP) $$with_grep_options $(_ignore_case) -LE "$$require" \
+ | $(GREP) . \
&& { msg="$$halt"
[PATCH 0/2] Fix syntax-check on macOS/FreeBSD
Hello, There was an issue with syntax-check on FreeBSD reported a few years ago: https://www.redhat.com/archives/libvir-list/2015-August/msg00758.html http://lists.gnu.org/archive/html/bug-gnulib/2015-08/msg00019.html The patch series attempts to resolve the issue on gnulib side. With related changes on libvirt side I can run make syntax-check on macOS. I wasn't entirely sure on indentation in the long commands. I did my best, sorry if something is wrong. -- Best regards, Roman Roman Bolshakov (2): maint.mk: Split long argument lists maint.mk: Replace grep with $(GREP) top/maint.mk | 147 --- 1 file changed, 82 insertions(+), 65 deletions(-) -- 2.19.1
