bug#65697: numproc, add flag to get number of available cores

[Pádraig Brady]

On 02/09/2023 10:38, Simon Heimberg wrote:

Hello

Depending on what a process is mainly limited by, the ideal number of
processes to run concurrently varies.
A frequent use case is to run one process on each _physical_ core
(which is available) than on each logical processing unit.
I propose to introduce a flag --core which returns the number of
physical cores available to the process (or of all cores with --all).


That option should probably be called --physical.
Moving your use cases for this from below...

> About better using one process for each core, depending on ...:
> https://github.com/giampaolo/psutil/issues/1968
> https://stackoverflow.com/questions/27404532/optimal-number-of-processes
> 
https://stackoverflow.com/questions/36958661/openmp-dont-use-hyperthreading-cores-half-num-threads-w-hyperthreading


I made a draft in python using getaffinity(), assuming the threads of a
core are listed after each other:
https://github.com/giampaolo/psutil/issues/1968#issuecomment-1688687251

Looking at the code in gnulib/lib/nproc.c this could get hard to
implement for all cases. (Many cases. No list for physical cores exists
yet.)

Maybe it is better to extend the doc and explain what the number really
means (with and without --all). Or add a link to more details about
processors and cores and ... . Extended doc could improve:
* As #42044 complains that nproc does not include the cores which go
online automatically. (The current help text _could_ match this.)


Right https://bugs.gnu.org/42044 is an interesting case we should probably 
support


* As I complain that nproc can not report (available) _physical_ cores.
* I just found out (partially) why --all shows more cpus than are in my
system. (It returns the _configured_ ones. And those get set ...:
https://unix.stackexchange.com/questions/609555/how-does-linux-detect-number-of-possible-cpus
) But the help writes "installed processors", not configured ones. (On
my system, processors which are configured only are not listed by ls -l
/sys/devices/system/cpu/cpu*)


Right there does seem to be an issue here.
On my 4 core laptop, there are 2 physical cores, and 2 hyperthread cores per 
physical core.

nproc shows 4 which is correct
nproc --all shows 8 which is surprising

An `strace nproc --all` shows that this 8 value is read from 
/sys/devices/system/cpu/possible
Now nproc uses getconf, so `getconf _NPROCESSORS_CONF` is reading this proc 
value.

Digging into the proc values, on my system we have:

  $ grep - /sys/devices/system/cpu/* 2>/dev/null
  /sys/devices/system/cpu/offline:4-7
  /sys/devices/system/cpu/online:0-3
  /sys/devices/system/cpu/possible:0-7
  /sys/devices/system/cpu/present:0-3

So the "possible" values above represent statically allocated kernel structures.
I.e. the kernel could support 4 more hotplug CPUs.
But that's not the normal hardware setup so it is surprising.

This change in glibc happened recently in glibc-2.36:
https://sourceware.org/git/?p=glibc.git;a=commit;h=97a912f7
So I presume you're also on a recent glibc system.

The associated glibc discussion is at:
https://sourceware.org/bugzilla/show_bug.cgi?id=28991

Now it's debatable as to whether glibc should be using 
/sys/devices/system/cpu/present
or /sys/devices/system/cpu/possible for _NPROCESSORS_CONF.
I notice also that uclibc and musl use /proc enumeration and affinity mask 
respectively,
which would return 4 on my system, rather than 8.

If we don't change any logic, then at least we should tweak the docs like:
diff --git a/doc/coreutils.texi b/doc/coreutils.texi
index 373a407ed..bd7a25219 100644
--- a/doc/coreutils.texi
+++ b/doc/coreutils.texi
@@ -17097,8 +17097,9 @@ The program accepts the following options.  Also see 
@ref{Common options}.

 @item --all
 @opindex --all
-Print the number of installed processors on the system, which may
-be greater than the number online or available to the current process.
+Print the number of potentially installed processors on the system, which may
+be greater than the number currently present or online in the system,
+or available to the current process.
 The @env{OMP_NUM_THREADS} or @env{OMP_THREAD_LIMIT} environment variables
 are not honored in this case.


There are various combinations of present, online, masked, physical CPUs
so I'm not sure if it's feasible to add other selection options to nproc.

cheers,
Pádraig

bug#65599: mv and cp give a pointless warning when moving/copying a directory

[Paul Eggert]

On 2023-09-02 04:18, Bruno Haible wrote:

chmod fails with EACCES, hence per POSIX

it's buggy.


Thanks, I reported the bug with chmod, chown, etc. to the linux-cifs 
mailing list, here:


https://lore.kernel.org/linux-cifs/fe8ab586-c697-583b-650d-3adac64df...@cs.ucla.edu/T/#u

I did some testing with fsetxattr and found that it has a similar issue 
even with ext4; for example, setxattr("/", "a", "b", 1, 0) fails with 
EACCES. This is not a POSIX violation, since POSIX lacks setxattr. 
However it's bad form. And it means coreutils needs the attached 
workaround not just for CIFS, but also for ext4 in some (rare) cases.


I installed the attached workaround on coreutils and am closing the bug 
report. Thanks again for mentioning it.From 9cd52bd9993163c2ef8b3d62b757c573fb5320df Mon Sep 17 00:00:00 2001
From: Paul Eggert 
Date: Sat, 2 Sep 2023 13:27:45 -0700
Subject: [PATCH] cp,mv,install: fix chmod on Linux CIFS

This bug occurs only when temporarily setting the mode to the
intersection of old and new modes when changing ownership.
* src/copy.c (owner_failure_ok): Treat EACCES like EPERM.
---
 NEWS   | 6 +++---
 src/copy.c | 3 ++-
 2 files changed, 5 insertions(+), 4 deletions(-)

diff --git a/NEWS b/NEWS
index 6801832f7..2adea1e11 100644
--- a/NEWS
+++ b/NEWS
@@ -4,9 +4,9 @@ GNU coreutils NEWS-*- outline -*-
 
 ** Bug fixes
 
-  cp, mv, install no longer issue spurious "failed to preserve ownership"
-  diagnostics when copying to GNU/Linux CIFS filesystems.  They do
-  this by working around a Linux CIFS bug.
+  cp, mv, and install no longer issue spurious diagnostics like "failed
+  to preserve ownership" when copying to GNU/Linux CIFS file systems.
+  They do this by working around some Linux CIFS bugs.
 
   numfmt options like --suffix no longer have an arbitrary 127-byte limit.
   [bug introduced with numfmt in coreutils-8.21]
diff --git a/src/copy.c b/src/copy.c
index 0b3de04f3..9f1f3e85a 100644
--- a/src/copy.c
+++ b/src/copy.c
@@ -3468,7 +3468,8 @@ chown_failure_ok (struct cp_options const *x)
 static bool
 owner_failure_ok (struct cp_options const *x)
 {
-  return ((errno == EPERM || errno == EINVAL) && !x->owner_privileges);
+  return ((errno == EPERM || errno == EINVAL || errno == EACCES)
+  && !x->owner_privileges);
 }
 
 /* Return the user's umask, caching the result.
-- 
2.39.2

bug#65659: RFC: changing printf(1) behavior on %b

[Steffen Nurpmeso]

Stephane Chazelas wrote in
 <20230902084912.vdfedsgbnat2w...@chazelas.org>:
 |2023-09-01 23:28:50 +0200, Steffen Nurpmeso via austin-group-l at The \
 |Open Group:
 ...
 |>|FWIW, a "printf %b" github shell code search returns ~ 29k
 |>|entries
 |>|(https://github.com/search?q=printf+%25b+language%3AShell=code=Sh\
 |>|ell)
 ...
 |> Actually this returns a huge amount of false positives where
 |> printf(1) and %b are not on the same line, let alone the same
 ...
 |Apparently, we can also search with regexps and searching for
 |printf.*%b
 |(https://github.com/search?q=%2Fprintf.*%25b%2F+language%3AShell=code)
 |It's probably a lot more accurate. It returns ~ 19k.
 ...
 |> Furthermore it shows a huge amount of false use cases like
 ...
 |Yes, I also see a lot of echo -e stuff that should have been
 |echo -E stuff (or echo alone in those (many) implementations
 |that don't expand by default or use the more reliable printf
 |with %s (not %b)).
 |
 |> It seems people think you need this to get colours mostly, which
 ...
 |Incidentally, ANSI terminal colour escape sequences are somewhat
 |connecting those two %b's as they are RGB (well BGR) in binary
 |(white is 7 = 0b111, red 0b001, green 0b010, blue 0b100), with:
 |
 |R=0 G=1 B=1
 |printf '%bcyan%b\n' "\033[3$(( 2#$B$G$R ))m" '\033[m'
 |
 |(with Korn-like shells, also $(( 0b$B$G$R )) in zsh though zsh
 |has builtin colour output support including RGB-based).

..and, off-topic, but in my opinion that is also false usage, one
should use tput(1) instead, and then simply printf(1) (or echo(1)
(or cat(1))) the output, something like, fwiw :),

  color_init() {
  [ -n "${NO_COLOUR}" ] && return
  # We do not want color for "make test > .LOG"!
  if [ -t 1 ] && command -v tput >/dev/null 2>&1; then
  { sgr0=$(tput sgr0); } 2>/dev/null
  [ $? -eq 0 ] || return
  { saf1=$(tput setaf 1); } 2>/dev/null
  [ $? -eq 0 ] || return
  { saf2=$(tput setaf 2); } 2>/dev/null
  [ $? -eq 0 ] || return
  { saf3=$(tput setaf 3); } 2>/dev/null
  [ $? -eq 0 ] || return
  { saf5=$(tput setaf 5); } 2>/dev/null
  [ $? -eq 0 ] || return
  { b=$(tput bold); } 2>/dev/null
  [ $? -eq 0 ] || return

  COLOR_ERR_ON=${saf1}${b} COLOR_ERR_OFF=${sgr0}
  COLOR_DBGERR_ON=${saf5} COLOR_DBGERR_OFF=${sgr0}
  COLOR_WARN_ON=${saf3}${b} COLOR_WARN_OFF=${sgr0}
  COLOR_OK_ON=${saf2} COLOR_OK_OFF=${sgr0}
  unset saf1 saf2 saf3 b
  fi
  }

  ...

  printf '%s%s%s' "${COLOR_WARN_ON}" "$SOME_MSG" "${COLOR_WARN_OFF}"

Of course this is also only ANSI via sgr0 (:-|

 |Speaking of stackexchange, on the June data dump of
 |unix.stackexchange.com:
 |
 |stackexchange/unix.stackexchange.com$ xml2 < Posts.xml | grep -c 'printf\
 |.*%b'
 |494
 |
 |(FWIW)
 |
 |Compared with %d (though that will have entries for printf(3) as well):
 |
 |stackexchange/unix.stackexchange.com$ xml2 < Posts.xml | grep -c 'printf\
 |.*%d'
 |3444

I am totally stunned by the ratio.  I myself have never used %b
(like this, aka for printf).

 --End of <20230902084912.vdfedsgbnat2w...@chazelas.org>

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

bug#65599: mv and cp give a pointless warning when moving/copying a directory

[Bruno Haible]

Paul Eggert wrote:
> Yes please. If chmod also does not work coreutils might need more 
> workarounds.
> 
> Just to be clear; please try it on a file where chmod should fail 
> because you don't own the file and you're not root. chmod should fail 
> with EPERM in this case; it it fails with EACCES it's buggy. Thanks.

chmod fails with EACCES, hence per POSIX

it's buggy.

See: On the server I have 2 files and 2 directories:

# ls -ld dir1 dir1/file1 dir2 dir2/file2
drwxrwxr-x2 brunoeveryone  4096 Sep  2 12:58 dir1/
-rw-rw-r--1 brunoeveryone 4 Sep  2 00:52 dir1/file1
drwxrwxrwx2 adminadminist  4096 Sep  2 00:54 dir2/
-rw-rw-r--1 adminadminist 4 Sep  2 00:54 dir2/file2

The user that exports the mount point is 'bruno'. But since dir2
and dir2/file2 are not owned by 'bruno', an error will (and should)
occur when these two are manipulated through the CIFS client.

chown fails; that's the problem you already committed a workaround for:

$ ./fchownat-test /media/nas/bruno/dir1/file1
lchown: Permission denied
fchownat: Permission denied
chown: Permission denied
fchown: Permission denied
$ ./fchownat-test /media/nas/bruno/dir1
lchown: Permission denied
fchownat: Permission denied
chown: Permission denied
fchown: Permission denied
$ ./fchownat-test /media/nas/bruno/dir2/file2
lchown: Permission denied
fchownat: Permission denied
chown: Permission denied
fchown: Permission denied
$ ./fchownat-test /media/nas/bruno/dir2
lchown: Permission denied
fchownat: Permission denied
chown: Permission denied
fchown: Permission denied

chmod also fails with EACCES instead of EPERM:

$ ./fchmodat-test /media/nas/bruno/dir1/file1
$ ./fchmodat-test /media/nas/bruno/dir1
$ ./fchmodat-test /media/nas/bruno/dir2/file2
lchmod: Permission denied
fchmodat: Permission denied
chmod: Permission denied
fchmod: Permission denied
$ ./fchmodat-test /media/nas/bruno/dir2
lchmod: Permission denied
fchmodat: Permission denied
chmod: Permission denied
fchmod: Permission denied

The same thing with strace:

$ strace ./fchmodat-test /media/nas/bruno/dir2/file2
...
newfstatat(AT_FDCWD, "/media/nas/bruno/dir2/file2", {st_mode=S_IFREG|0664, 
st_size=4, ...}, AT_SYMLINK_NOFOLLOW) = 0
openat(AT_FDCWD, "/media/nas/bruno/dir2/file2", 
O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH) = 3
newfstatat(3, "", {st_mode=S_IFREG|0664, st_size=4, ...}, AT_EMPTY_PATH) = 0
chmod("/proc/self/fd/3", 0664)  = -1 EACCES (Permission denied)
close(3)= 0
dup(2)  = 3
fcntl(3, F_GETFL)   = 0x402 (flags O_RDWR|O_APPEND)
getrandom("\x68\x8f\xbd\x08\xf8\x4b\xa9\xa5", 8, GRND_NONBLOCK) = 8
brk(NULL)   = 0x5563c5d01000
brk(0x5563c5d22000) = 0x5563c5d22000
newfstatat(3, "", {st_mode=S_IFCHR|0600, st_rdev=makedev(0x88, 0x5), ...}, 
AT_EMPTY_PATH) = 0
write(3, "lchmod: Permission denied\n", 26lchmod: Permission denied
) = 26
close(3)= 0
openat(AT_FDCWD, "/media/nas/bruno/dir2/file2", 
O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH) = 3
newfstatat(3, "", {st_mode=S_IFREG|0664, st_size=4, ...}, AT_EMPTY_PATH) = 0
chmod("/proc/self/fd/3", 0664)  = -1 EACCES (Permission denied)
close(3)= 0
dup(2)  = 3
fcntl(3, F_GETFL)   = 0x402 (flags O_RDWR|O_APPEND)
newfstatat(3, "", {st_mode=S_IFCHR|0600, st_rdev=makedev(0x88, 0x5), ...}, 
AT_EMPTY_PATH) = 0
write(3, "fchmodat: Permission denied\n", 28fchmodat: Permission denied
) = 28
close(3)= 0
chmod("/media/nas/bruno/dir2/file2", 0664) = -1 EACCES (Permission denied)
dup(2)  = 3
fcntl(3, F_GETFL)   = 0x402 (flags O_RDWR|O_APPEND)
newfstatat(3, "", {st_mode=S_IFCHR|0600, st_rdev=makedev(0x88, 0x5), ...}, 
AT_EMPTY_PATH) = 0
write(3, "chmod: Permission denied\n", 25chmod: Permission denied
) = 25
close(3)= 0
openat(AT_FDCWD, "/media/nas/bruno/dir2/file2", O_RDONLY|O_NOFOLLOW) = 3
fchmod(3, 0664) = -1 EACCES (Permission denied)
dup(2)  = 4
fcntl(4, F_GETFL)   = 0x402 (flags O_RDWR|O_APPEND)
newfstatat(4, "", {st_mode=S_IFCHR|0600, st_rdev=makedev(0x88, 0x5), ...}, 
AT_EMPTY_PATH) = 0
write(4, "fchmod: Permission denied\n", 26fchmod: Permission denied
) = 26
close(4)= 0
exit_group(1)   = ?
+++ exited with 1 +++

$ strace ./fchmodat-test /media/nas/bruno/dir2
...
newfstatat(AT_FDCWD, "/media/nas/bruno/dir2", {st_mode=S_IFDIR|0777, st_size=0, 
...}, AT_SYMLINK_NOFOLLOW) = 0
openat(AT_FDCWD, "/media/nas/bruno/dir2", O_RDONLY|O_NOFOLLOW|O_CLOEXEC|O_PATH) 
= 3
newfstatat(3, "", {st_mode=S_IFDIR|0777, st_size=0, ...}, AT_EMPTY_PATH) = 0

bug#65697: numproc, add flag to get numer of available cores

[Simon Heimberg]

Hello

Depending on what a process is mainly limited by, the ideal number of
processes to run concurrently varies.
A frequent use case is to run one process on each _physical_ core
(which is available) than on each logical processing unit.
I propose to introduce a flag --core which returns the number of
physical cores available to the process (or of all cores with --all).

I made a draft in python using getaffinity(), assuming the threads of a
core are listed after each other:
https://github.com/giampaolo/psutil/issues/1968#issuecomment-1688687251

Looking at the code in gnulib/lib/nproc.c this could get hard to
implement for all cases. (Many cases. No list for physical cores exists
yet.)

Maybe it is better to extend the doc and explain what the number really
means (with and without --all). Or add a link to more details about
processors and cores and ... . Extended doc could improve:
* As #42044 complains that nproc does not include the cores which go
online automatically. (The current help text _could_ match this.)
* As I complain that nproc can not report (available) _physical_ cores.
* I just found out (partially) why --all shows more cpus than are in my
system. (It returns the _configured_ ones. And those get set ...:
https://unix.stackexchange.com/questions/609555/how-does-linux-detect-number-of-possible-cpus
) But the help writes "installed processors", not configured ones. (On
my system, processors which are configured only are not listed by ls -l
/sys/devices/system/cpu/cpu*)


About better using one process for each core, depending on ...:
https://github.com/giampaolo/psutil/issues/1968
https://stackoverflow.com/questions/27404532/optimal-number-of-processes
https://stackoverflow.com/questions/36958661/openmp-dont-use-hyperthreading-cores-half-num-threads-w-hyperthreading

bug#65659: RFC: changing printf(1) behavior on %b

[Phi Debian]

On Fri, Sep 1, 2023 at 8:10 PM Stephane Chazelas 
wrote:

> 2023-09-01 07:54:02 -0500, Eric Blake via austin-group-l at The Open Group:
>
>
> FWIW, a "printf %b" github shell code search returns ~ 29k
> entries
> (
> https://github.com/search?q=printf+%25b+language%3AShell=code=Shell
> )
>
>
Ha super, at least some numbers :-), I didn't knew we could make this kind
of request... thanx for that.

bug#65659: RFC: changing printf(1) behavior on %b

[Steffen Nurpmeso]

Stephane Chazelas via austin-group-l at The Open Group wrote in
 <20230901181024.pwx4plwclz7ij...@chazelas.org>:
 |2023-09-01 07:54:02 -0500, Eric Blake via austin-group-l at The Open Group:
 ...
 |> How many scripts in the wild actually use %b, though?  And if there
 |> are such scripts, anything we can do to make it easy to do a drop-in
 |> replacement that still preserves the old behavior (such as changing %b
 |> to %#s) is going to be easier to audit than the only other
 |> currently-portable alternative of actually analyzing the string to see
 |> if it uses any octal or \c escapes that have to be re-written to
 |> portably function as a printf format argument.
 |[...]
 |
 |FWIW, a "printf %b" github shell code search returns ~ 29k
 |entries
 |(https://github.com/search?q=printf+%25b+language%3AShell=code=Sh\
 |ell)
 |
 |That likely returns only a small subset of the code that uses
 |printf with %b inside the format and probably a few false
 |positives, but that gives many examples of how printf %b is used
 |in practice.

Actually this returns a huge amount of false positives where
printf(1) and %b are not on the same line, let alone the same
command, if you just scroll down a bit it starts like neovim match

 pr_title="${pr_title// /,}" # Replace spaces with commas.
 pr_title="$(printf 'vim-patch:%s' "${pr_title#,}")"

(bash only btw).
Furthermore it shows a huge amount of false use cases like

 printf >&2 "%b\n" "The following warnings and non-fatal errors were 
encountered during the installation process:"

This is only the first result page.
It seems people think you need this to get colours mostly, which
then, it has to be said, is also practically mislead.  (To the
best of *my* knowledge that is.)

Ah it is a copy world, and for one Stephane at stackoverflow
there are 99 that fool and mislead you, or do not know for sure
themselves, but also copy and paste!

--steffen
|
|Der Kragenbaer,The moon bear,
|der holt sich munter   he cheerfully and one by one
|einen nach dem anderen runter  wa.ks himself off
|(By Robert Gernhardt)

bug#65659: RFC: changing printf(1) behavior on %b

[Stephane Chazelas]

2023-09-01 23:28:50 +0200, Steffen Nurpmeso via austin-group-l at The Open 
Group:
[...]
>  |FWIW, a "printf %b" github shell code search returns ~ 29k
>  |entries
>  |(https://github.com/search?q=printf+%25b+language%3AShell=code=Sh\
>  |ell)
>  |
>  |That likely returns only a small subset of the code that uses
>  |printf with %b inside the format and probably a few false
>  |positives, but that gives many examples of how printf %b is used
>  |in practice.
> 
> Actually this returns a huge amount of false positives where
> printf(1) and %b are not on the same line, let alone the same
> command, if you just scroll down a bit it starts like neovim match
[...]

You're right, I only looked at the first few results and saw
that already gave interesting ones.

Apparently, we can also search with regexps and searching for
printf.*%b
(https://github.com/search?q=%2Fprintf.*%25b%2F+language%3AShell=code)
It's probably a lot more accurate. It returns ~ 19k.

(still FWIW, that's still just a sample of random code on the
internet)

[...]
> Furthermore it shows a huge amount of false use cases like
> 
>  printf >&2 "%b\n" "The following warnings and non-fatal errors were 
> encountered during the installation process:"
[...]

Yes, I also see a lot of echo -e stuff that should have been
echo -E stuff (or echo alone in those (many) implementations
that don't expand by default or use the more reliable printf
with %s (not %b)).

> It seems people think you need this to get colours mostly, which
> then, it has to be said, is also practically mislead.  (To the
> best of *my* knowledge that is.)
[...]

Incidentally, ANSI terminal colour escape sequences are somewhat
connecting those two %b's as they are RGB (well BGR) in binary
(white is 7 = 0b111, red 0b001, green 0b010, blue 0b100), with:

R=0 G=1 B=1
printf '%bcyan%b\n' "\033[3$(( 2#$B$G$R ))m" '\033[m'

(with Korn-like shells, also $(( 0b$B$G$R )) in zsh though zsh
has builtin colour output support including RGB-based).

Speaking of stackexchange, on the June data dump of
unix.stackexchange.com:

stackexchange/unix.stackexchange.com$ xml2 < Posts.xml | grep -c 'printf.*%b'
494

(FWIW)

Compared with %d (though that will have entries for printf(3) as well):

stackexchange/unix.stackexchange.com$ xml2 < Posts.xml | grep -c 'printf.*%d'
3444

-- 
Stephane