Re: --create-testdir eats CPU and does nothing

[Tim Rühsen]

On Samstag, 5. August 2017 18:34:58 CEST Bruno Haible wrote:
> Tim Rühsen wrote:
> > > $ ./gnulib-tool --create-testdir --dir=../testdir1
> > > gnulib-tool: warning: module euidaccess depends on a module with an
> > > incompatible license: group-member
> > 
> > Correction, it came back after ~20 minutes.
> 
> You have a fast machine.
> 
> > Is there any chance to parallelize the work of ./gnulib-tool ? The machine
> > was just 6-7% idle.
> 
> Not parallelize. But rewrite in a decent programming language. IMO the way
> to go would be to merge Dmitry Selyutin's rewrite
>   https://github.com/ghostmansd/gnulib-python
> into gnulib.

But I guess there is a good reason for not doing it !? It dates 2012.

check-copyright seems to be a good candidate for parallelizing. I tested that 
with the 'parallel' tool (with some modifications of course).

A speedup only when certain tools / shell are/is available would be fine, IMO.

But that's just a random idea, I am absolutely not enough into gnulib nor 
shell programming.

Regards, Tim


signature.asc
Description: This is a digitally signed message part.

Re: Problem with valgrind-tests: relies on bash not causing error

[Reuben Thomas]

On 5 August 2017 at 19:46, Paul Eggert  wrote:

> Thanks, I installed the attached somewhat-more-fancier patch; does it work
> for you?
>

​That works fine; thanks very much.

-- 
https://rrt.sc3d.org 

Re: Problem with valgrind-tests: relies on bash not causing error

[Paul Eggert]

Thanks, I installed the attached somewhat-more-fancier patch; does it work for 
you?
>From 59c181fa400ead71e9a28e3db9abc107f8fc9ea3 Mon Sep 17 00:00:00 2001
From: Paul Eggert 
Date: Sat, 5 Aug 2017 11:45:10 -0700
Subject: [PATCH] valgrind-tests: use ls, and cache

* m4/valgrind-tests.m4: Test ls, not bash.
Problem reported by Reuben Thomas.
Also, cache the result so that it can be overridden.
---
 ChangeLog|  7 +++
 m4/valgrind-tests.m4 | 30 +++---
 2 files changed, 22 insertions(+), 15 deletions(-)

diff --git a/ChangeLog b/ChangeLog
index caab63e..74e9831 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,10 @@
+2017-08-05  Paul Eggert  
+
+	valgrind-tests: use ls, and cache
+	* m4/valgrind-tests.m4: Test ls, not bash.
+	Problem reported by Reuben Thomas.
+	Also, cache the result so that it can be overridden.
+
 2017-08-04  Paul Eggert  
 
 	manywarnings: port to 64-bit GCC builds of Emacs
diff --git a/m4/valgrind-tests.m4 b/m4/valgrind-tests.m4
index 00189d8..7e4bf60 100644
--- a/m4/valgrind-tests.m4
+++ b/m4/valgrind-tests.m4
@@ -1,4 +1,4 @@
-# valgrind-tests.m4 serial 3
+# valgrind-tests.m4 serial 4
 dnl Copyright (C) 2008-2017 Free Software Foundation, Inc.
 dnl This file is free software; the Free Software Foundation
 dnl gives unlimited permission to copy and/or distribute it,
@@ -11,27 +11,27 @@ dnl From Simon Josefsson
 # Check if valgrind is available, and set VALGRIND to it if available.
 AC_DEFUN([gl_VALGRIND_TESTS],
 [
-  AC_ARG_ENABLE(valgrind-tests,
+  AC_ARG_ENABLE([valgrind-tests],
 AS_HELP_STRING([--disable-valgrind-tests],
[don't try to run self tests under valgrind]),
 [opt_valgrind_tests=$enableval], [opt_valgrind_tests=yes])
 
   # Run self-tests under valgrind?
   if test "$opt_valgrind_tests" = "yes" && test "$cross_compiling" = no; then
-AC_CHECK_PROGS(VALGRIND, valgrind)
-  fi
+AC_CHECK_PROGS([VALGRIND], [valgrind])
 
-  OPTS="-q --error-exitcode=1 --leak-check=full"
+if test "$VALGRIND"; then
+  AC_CACHE_CHECK([for valgrind options for tests],
+[gl_cv_opt_valgrind_tests],
+[gl_cv_opt_valgrind_tests="-q --error-exitcode=1 --leak-check=full"
+ $VALGRIND $gl_valgrind_opts ls > /dev/null 2>&1 ||
+   gl_cv_opt_valgrind_tests=no])
 
-  if test -n "$VALGRIND" \
- && $VALGRIND $OPTS $SHELL -c 'exit 0' > /dev/null 2>&1; then
-opt_valgrind_tests=yes
-VALGRIND="$VALGRIND $OPTS"
-  else
-opt_valgrind_tests=no
-VALGRIND=
+  if test "$gl_cv_opt_valgrind_tests" != no; then
+VALGRIND="$VALGRIND $gl_cv_opt_valgrind_tests"
+  else
+VALGRIND=
+  fi
+fi
   fi
-
-  AC_MSG_CHECKING([whether self tests are run under valgrind])
-  AC_MSG_RESULT($opt_valgrind_tests)
 ])
-- 
2.7.4

Re: manywarnings for C++

[Paul Eggert]

Thanks, as I try to avoid C++ I hope Bruno has the time to review that one.

Re: Testsuite fails in gltests/havelib

[Bruno Haible]

Tim Rühsen wrote:
> Any idea what's wrong ?
> /bin/bash: /bin/bash: cannot execute binary file
> Makefile:2822: recipe for target 'rpath-1b.log' failed

Since no one touched the 'havelib' module recently, you can ignore it.

Maybe something in your setup (LD_LIBRARY_PATH or such things)...

Bruno

Re: --create-testdir eats CPU and does nothing

[Bruno Haible]

Tim Rühsen wrote:
> > $ ./gnulib-tool --create-testdir --dir=../testdir1
> > gnulib-tool: warning: module euidaccess depends on a module with an
> > incompatible license: group-member
> 
> Correction, it came back after ~20 minutes.

You have a fast machine.

> Is there any chance to parallelize the work of ./gnulib-tool ? The machine 
> was 
> just 6-7% idle.

Not parallelize. But rewrite in a decent programming language. IMO the way to go
would be to merge Dmitry Selyutin's rewrite
  https://github.com/ghostmansd/gnulib-python
into gnulib.

Bruno

Testsuite fails in gltests/havelib

[Tim Rühsen]

Any idea what's wrong ?

$ make clean && make check
...
Testsuite summary for dummy 0

# TOTAL: 1150
# PASS:  1068
# SKIP:  82
# XFAIL: 0
# FAIL:  0
# XPASS: 0
# ERROR: 0

make[5]: Leaving directory '/home/tim/src/testdir1/gltests'
make[4]: Leaving directory '/home/tim/src/testdir1/gltests'
make[3]: Leaving directory '/home/tim/src/testdir1/gltests'
Making check in havelib
make[3]: Entering directory '/home/tim/src/testdir1/gltests/havelib'
make  check-TESTS
make[4]: Entering directory '/home/tim/src/testdir1/gltests/havelib'
make[5]: Entering directory '/home/tim/src/testdir1/gltests/havelib'
/bin/bash: /bin/bash: cannot execute binary file
Makefile:2822: recipe for target 'rpath-1b.log' failed
make[5]: *** [rpath-1b.log] Error 126
make[5]: *** Waiting for unfinished jobs
/bin/bash: /bin/bash: cannot execute binary file
/bin/bash: /bin/bash: cannot execute binary file
Makefile:2815: recipe for target 'rpath-1a.log' failed
make[5]: *** [rpath-1a.log] Error 126
Makefile:2836: recipe for target 'rpath-2aab.log' failed
make[5]: *** [rpath-2aab.log] Error 126
/bin/bash: /bin/bash: cannot execute binary file
/bin/bash: /bin/bash: cannot execute binary file
/bin/bash: /bin/bash: cannot execute binary file
/bin/bash: /bin/bash: cannot execute binary file
/bin/bash: /bin/bash: cannot execute binary file
Makefile:2857: recipe for target 'rpath-2abb.log' failed
/bin/bash: /bin/bash: cannot execute binary file
make[5]: *** [rpath-2abb.log] Error 126
Makefile:2864: recipe for target 'rpath-2abd.log' failed
make[5]: *** [rpath-2abd.log] Error 126
Makefile:2829: recipe for target 'rpath-2aaa.log' failed
make[5]: *** [rpath-2aaa.log] Error 126
Makefile:2885: recipe for target 'rpath-2bad.log' failed
make[5]: *** [rpath-2bad.log] Error 126
Makefile:2843: recipe for target 'rpath-2aad.log' failed


signature.asc
Description: This is a digitally signed message part.

Re: --create-testdir eats CPU and does nothing

[Tim Rühsen]

On Samstag, 5. August 2017 08:46:22 CEST Paul Eggert wrote:
> Tim Rühsen wrote:
> > Is there any chance to parallelize the work of ./gnulib-tool ? The machine
> > was just 6-7% idle.
> 
> There was a GNU project to improve gnulib-tool's performance a while ago. I
> don't know what came of it. That, plus 'configure', is a lot of why
> single-core performance still matters when building GNU software.

There was a thread in gnu-prog-discuss with some ideas how to improve the 
speed of configure runs. But I guess, nobody works on it consequently.
I put some work at it: 
https://gitlab.com/gnuwget/wget2/wikis/Developer-hints:-Increasing-speed-of-GNU-toolchain

Regards, Tim


signature.asc
Description: This is a digitally signed message part.

Re: Problem with valgrind-tests: relies on bash not causing error

[Reuben Thomas]

On 12 April 2017 at 13:36, Reuben Thomas  wrote:

> The test for whether to use valgrind runs:
>
> /bin/bash -c 'exit 0'
>
> This looks pretty harmless; unfortunately, bash itself causes problems:
>
> $ valgrind -q --error-exitcode=1 --leak-check=full /bin/bash -c 'exit 0'
> ==32197== Invalid free() / delete / delete[] / realloc()
> ==32197==at 0x4C2ED5B: free (in /usr/lib/valgrind/vgpreload_
> memcheck-amd64-linux.so)
> ==32197==by 0x45E1D0: ??? (in /bin/bash)
> ==32197==by 0x45E37F: run_unwind_frame (in /bin/bash)
> ==32197==by 0x47B664: parse_and_execute (in /bin/bash)
> ==32197==by 0x4209D6: ??? (in /bin/bash)
> ==32197==by 0x41F893: main (in /bin/bash)
> ==32197==  Address 0x423b828 is in the brk data segment 0x4228000-0x423bfff
> ==32197==
>
> Here I was using the provided Ubuntu 16.04 build of bash:
>
> $ bash --version
> GNU bash, version 4.3.46(1)-release (x86_64-pc-linux-gnu)
> Copyright (C) 2013 Free Software Foundation, Inc.
>
> Maybe use some other known-available utility that doesn't play weird
> tricks with memory? /bin/ls works for me!
>

​Attached, a patch to fix this. (I believe this change is still needed.)

-- 
https://rrt.sc3d.org 


valgrind-tests.m4
Description: application/m4

Re: manywarnings for C++

[Reuben Thomas]

On 7 March 2017 at 12:20, Reuben Thomas  wrote:

> On 22 February 2017 at 13:55, Reuben Thomas  wrote:
>
>>
>> ​I attach the latest state of play, which consists simply of updated
>> versions of manywarnings.m4 and warnings.m4, in place of the new files I
>> had before. (To test them I have simply placed them in a gnulib patch
>> directory which is applied by bootstrap; I love how simple it is!)
>>
>
> ​Ping!
>

​Ping!! Attached, the latest version of my patches (in fact they've not
changed since last time, but there is an extra file I forgot to include
before), and now I attach patches to existing files as such rather than as
complete files.

For reference, these versions do not address this request from Bruno:

Please also fix the AC_REQUIRE([gl_UNKNOWN_WARNINGS_ARE_ERRORS]) invocation in
> gl_WARN_ADD and gl_CXX_WARN_ADD. Since this macro expands to different code
> after AC_LANG_PUSH([C++]) than after AC_LANG_PUSH([C]), it is wrong to
> just AC_REQUIRE it. Needs to be a bit more intelligent.


> Look at how _AC_LANG_ABBREV and _AC_LANG_PREFIX can be used. Maybe
> libtool.m4, which also plays around with AC_LANG, gives you some hint
> about this.


​I previously replied:​

I didn't fully understand this bit. I see that _AC_LANG_ABBREV/_AC_LANG_PREFIX
> are used to inject the name of the language into the names of variables. I
> see that gl_UNKNOWN_WARNINGS_ARE_ERRORS changes depending on the language
> because gl_COMPILER_OPTION_IF changes. (By the way, can we now assume
> autoconf 2.64 (of July 2009) or newer and hence quote gl_Warn, as per the
> comment, there?)
> Since gl_UNKNOWN_WARNINGS_ARE_ERRORS is therefore language-dependent, that
> makes me think it needs to be called, not AC_REQUIREd, by
> gl_{CXX_,}_WARN_ADD (because the language might change each time), but if
> it were that simple, you'd've said so. So I'm not sure what else is needed…


-- 
https://rrt.sc3d.org 
--- gnulib/m4/manywarnings.m4   2017-08-03 18:10:49.139888596 +0100
+++ gl/m4/manywarnings.m4   2017-04-04 22:27:00.204131193 +0100
@@ -293,3 +274,248 @@
 
   $1=$gl_manywarn_set
 ])
+
+# gl_MANYWARN_ALL_GXX(VARIABLE)
+# -
+# Add all documented G++ warning parameters to variable VARIABLE.
+# Note that you need to test them using gl_WARN_ADD if you want to
+# make sure your gcc understands it.
+AC_DEFUN([gl_MANYWARN_ALL_GXX],
+[
+  AC_LANG_PUSH([C++])
+
+  dnl First, check for some issues that only occur when combining multiple
+  dnl gcc warning categories.
+  AC_REQUIRE([AC_PROG_CXX])
+  if test -n "$GXX"; then
+
+dnl Check if -W -Werror -Wno-missing-field-initializers is supported
+dnl with the current $CXX $CXXFLAGS $CPPFLAGS.
+AC_MSG_CHECKING([whether -Wno-missing-field-initializers is supported])
+AC_CACHE_VAL([gl_cv_cxx_nomfi_supported], [
+  gl_save_CXXFLAGS="$CXXFLAGS"
+  CXXFLAGS="$CXXFLAGS -W -Werror -Wno-missing-field-initializers"
+  AC_COMPILE_IFELSE(
+[AC_LANG_PROGRAM([[]], [[]])],
+[gl_cv_cxx_nomfi_supported=yes],
+[gl_cv_cxx_nomfi_supported=no])
+  CXXFLAGS="$gl_save_CXXFLAGS"])
+AC_MSG_RESULT([$gl_cv_cxx_nomfi_supported])
+
+if test "$gl_cv_cxx_nomfi_supported" = yes; then
+  dnl Now check whether -Wno-missing-field-initializers is needed
+  dnl for the { 0, } construct.
+  AC_MSG_CHECKING([whether -Wno-missing-field-initializers is needed])
+  AC_CACHE_VAL([gl_cv_cxx_nomfi_needed], [
+gl_save_CXXFLAGS="$CXXFLAGS"
+CXXFLAGS="$CXXFLAGS -W -Werror"
+AC_COMPILE_IFELSE(
+  [AC_LANG_PROGRAM(
+ [[int f (void)
+   {
+ typedef struct { int a; int b; } s_t;
+ s_t s1 = { 0, };
+ return s1.b;
+   }
+ ]],
+ [[]])],
+  [gl_cv_cxx_nomfi_needed=no],
+  [gl_cv_cxx_nomfi_needed=yes])
+CXXFLAGS="$gl_save_CXXFLAGS"
+  ])
+  AC_MSG_RESULT([$gl_cv_cxx_nomfi_needed])
+fi
+
+dnl Next, check if -Werror -Wuninitialized is useful with the
+dnl user's choice of $CXXFLAGS; some versions of gcc warn that it
+dnl has no effect if -O is not also used
+AC_MSG_CHECKING([whether -Wuninitialized is supported])
+AC_CACHE_VAL([gl_cv_cxx_uninitialized_supported], [
+  gl_save_CXXFLAGS="$CXXFLAGS"
+  CXXFLAGS="$CXXFLAGS -Werror -Wuninitialized"
+  AC_COMPILE_IFELSE(
+[AC_LANG_PROGRAM([[]], [[]])],
+[gl_cv_cxx_uninitialized_supported=yes],
+[gl_cv_cxx_uninitialized_supported=no])
+  CXXFLAGS="$gl_save_CXXFLAGS"])
+AC_MSG_RESULT([$gl_cv_cxx_uninitialized_supported])
+
+  fi
+
+  # List all gcc warning categories.
+  # To compare this list to your installed GCC's, run this Bash command:
+  #
+  # comm -3 \
+  #  <(sed -n 's/^  *\(-[^ ]*\) .*/\1/p' manywarnings.m4 | sort) \
+  #  <(gcc --help=warnings | sed -n 's/^  \(-[^ ]*\) .*/\1/p' | sort |
+  #  grep 

Re: --create-testdir eats CPU and does nothing

[Dmitry Selyutin]

Hi everyone!

I'm amazed that someone still remembers this project. Looking at its
sources, I see that there is room for improvement, but still the overall
project structure may still be integrated into gnulib. I would be glad to
continue working on the project in my free time, though my Python skills
are a bit rusty. As for parallelization, I think it may be a good idea to
use Python 3 since it now has some form of Go-like coroutines IIRC.

Do you think it's a good idea to revive the old project? It seems that it
may still be useful.

--create-testdir eats CPU and does nothing

[Tim Rühsen]

Hi,

with a fresh git clone, this command doesn't come back:

$ ./gnulib-tool --create-testdir --dir=../testdir1
gnulib-tool: warning: module euidaccess depends on a module with an 
incompatible license: group-member

Looks like it eats CPU of one core.

./testdir1 has been created but still empty after 15 minutes.

What did I miss ?

Regards, Tim


signature.asc
Description: This is a digitally signed message part.

Re: --create-testdir eats CPU and does nothing

[Tim Rühsen]

On Samstag, 5. August 2017 14:15:32 CEST Tim Rühsen wrote:
> Hi,
> 
> with a fresh git clone, this command doesn't come back:
> 
> $ ./gnulib-tool --create-testdir --dir=../testdir1
> gnulib-tool: warning: module euidaccess depends on a module with an
> incompatible license: group-member
> 
> Looks like it eats CPU of one core.
> 
> ./testdir1 has been created but still empty after 15 minutes.
> 
> What did I miss ?

Correction, it came back after ~20 minutes.

This is a 8 Core, 16 Threads, 16GB, m.2 SSD (3Gb/s) machine.
Is there any chance to parallelize the work of ./gnulib-tool ? The machine was 
just 6-7% idle.

Regards, Tim


signature.asc
Description: This is a digitally signed message part.

Re: --create-testdir eats CPU and does nothing

[Paul Eggert]

Tim Rühsen wrote:

Is there any chance to parallelize the work of ./gnulib-tool ? The machine was
just 6-7% idle.


There was a GNU project to improve gnulib-tool's performance a while ago. I 
don't know what came of it. That, plus 'configure', is a lot of why single-core 
performance still matters when building GNU software.