Bug#1072096: PATCH: PR: src/stat.c to add magic for bcachefs filesystem

[Pádraig Brady]

On 28/05/2024 12:50, Paul Hedderly wrote:

Package: coreutils
Version: 9.4-3+b1
Severity: important
Tags: patch upstream d-i

Dear Maintainer,

*** Reporter, please consider answering these questions, where
appropriate ***

* What led up to the situation?

stat does not currently  recognise bcachefs
This also means that grub-probe fails to recognise bcachefs

* What was the outcome of this action?

$ sudo stat -f -c %T /
UNKNOWN (0xca451a4e)

* What outcome did you expect instead?

$ sudo ~/apt-src/coreutils-9.4/src/stat -f -c %T /
bcachefs

*** End of the template - remove these template lines ***

Please consider pulling the patch submitted to upstream since this may
take
some time to get accepted (there are many patches waiting on upstream
currently.)

It is a very minor patch that just adds one more magic to the
recognition code.

https://github.com/coreutils/coreutils/commit/6da16c99ff0c6ec35943dd2db5f51c25efb8b508


This is now incorporated in this upstream commit:
https://github.com/coreutils/coreutils/commit/3ce31e6f1

cheers,
Pádraig

Bug#1068891: coreutils: is join -t '' just comm -12?

[Pádraig Brady]

On 12/04/2024 23:59, наб wrote:

Package: coreutils
Version: 9.1-1
Version: 9.4-3
Severity: normal

Dear Maintainer,

POSIX.1-202x/D3:
−t char
Use character char as a separator, for both input and 
output. Every appearance of
char in a line shall be significant. When this option 
is specified, the collating
sequence shall be the same as sort without the −b 
option.
so obviously allowing -t '' is an extension.

Manual:
-t CHAR
   use CHAR as input and output field separator

Important: FILE1 and FILE2 must be sorted on the  join  fields.  E.g.,
use  "sort  -k  1b,1"  if 'join' has no options, or use "join -t ''" if
'sort' has no options.  Note, comparisons honor the rules specified by
'LC_COLLATE'.   If  the  input  is  not sorted and some lines cannot be
joined, a warning message will be given.

So given
$ cat f1
row1f1  1
urow1   f1  2
$ cat f2
row1f2  1
urow2   f2  2
which are stable against both sort and sort -k 1b,1
$ join f?
row1 f1 1 f2 1
$ join f? -t '  '
row1f1  1   f2  1
is all as expected.

But
$ join f? -t ''
returns empty. What would empty -t mean, anyway?
The empty string can either be found at every position
(clearly not the case here, otherwise this'd be joined on r and u)
or at no positions, so
$ cat g1
row1
urow1
$ cat g2
row1
urow2
$ join g? -t ''
row1
which is, well
$ comm g? -12
row1

Somehow I don't feel like this is a good recommendation?


Well sort with no options operates on the whole line.
So the corresponding join -t '' operates on the whole line.

cheers,
Pádraig

Bug#1068889: coreutils: join -t refuses single-character delimiters as "multi-character tab"s

[Pádraig Brady]

On 12/04/2024 23:42, наб wrote:

Package: coreutils
Version: 9.1-1
Version: 9.4-1
Severity: normal

Dear Maintainer,

Yes good:
$ cat f1
row1f1  1
urow1   f1  2
$ cat f2
row1f2  1
urow2   f2  2
$ join f? -t '  '
row1f1  1   f2  1

Not good:
$ cat f1
row1ąf1ą1
urow1ąf1ą2
$ cat f2
row1ąf2ą1
urow2ąf2ą2
$ join f? -t 'ą'
join: multi-character tab ‘ą’

Compare manual:
-t CHAR use CHAR as input and output field separator

Compare POSIX.1-202x/D3, XCU, join, OPTIONS:
−t char
Use character char as a separator, for both input and 
output. Every appearance of
char in a line shall be significant. When this option 
is specified, the collating
sequence shall be the same as sort without the −b 
option.


Please try coreutils 9.5 which has improved multi-byte char support in join

Bug#1068892: coreutils: join -t accepts \0 to mean NUL; doesn't document it

[Pádraig Brady]

On 13/04/2024 00:10, наб wrote:

Package: coreutils
Version: 9.1-1
Version: 9.4-3
Severity: normal

Dear Maintainer,

When trying to decypher -t behaviour for #1068891,
the second empty-string semantic I tried was **(argv + optind),
but that wasn't it. But what /is/ it is -t '\0'; i.e. given
$ cat f1
row1f1  1
urow1   f1  2
$ cat f2
row1f2  1
urow2   f2  2
$ join <(tr '\t' '\0' < f1) <(tr '\t' '\0' < f2) -t '\0' | cat -A
row1^@f1^@1^@f2^@1$
which is great, but the manual doesn't mention this at all.


This is mentioned in the full documentation referenced from the manual

Bug#1068864: coreutils: join -a3 errors "invalid field number" even when no field number given

[Pádraig Brady]

On 12/04/2024 14:12, наб wrote:

Package: coreutils
Version: 9.1-1
Version: 9.4-3
Severity: normal

Dear Maintainer,

$ join -a3 /dev/null /dev/null
join: invalid field number: ‘3’

Not sure where field 3 came from here.


Indeed, that looks to be a copy n paste issue in:
https://github.com/coreutils/coreutils/commit/6f63d53e1

I'll s/field/file/ upstream.

thanks,
Pádraig

Bug#628815: coreutils: pinky makes crazy DNS queries

[Pádraig Brady]

On 19/03/2024 10:54, deb...@perkelt.hu wrote:

I looked into this and found that
pinky tries to canonicalize the information in the "Where" column
automaticaly, and it has no option to disable this behaviour.
see line 285 pinky.c:
if (*ut_host)
/* See if we can canonicalize it.  */
host = canon_host (ut_host);
this results in many unnecessary dns queries.
For example on a debian system with acpi-support, /etc/acpi/lid.sh will
make many requests to find the host $WAYLAND_DISPLAY every time the lid
is opened.

"who" has the --lookup (attempt to canonicalize hostnames via DNS)
option, and doesn't do it by default.
Maybe it would be more lightweight to forget about canonicalization
altogether.


pinky is slow here on Fedora 39 too.
I agree that it should avoid this by default.
pinky is supposed to be a lightweight finger after all.

I'll apply the attached to upstream coreutils soon.

thanks,
PádraigFrom 6d0367c00e0998bf5b5a25358c1f1854555cd03e Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?P=C3=A1draig=20Brady?= 
Date: Tue, 19 Mar 2024 13:19:16 +
Subject: [PATCH] pinky: disable location canonicalization by default

Behave like who(1) in requiring --lookup to enable this
often slow feature.  pinky(1) is supposed to be lightweight after all.

* doc/coreutils.texi (who invocation): Adjust the description to no
longer reference dialup, and be more general about the still significant
delays.
(pinky invocation): Reference the same --lookup description.
* src/pinky.c (main): Accept --lookup to enable DNS lookups.
* NEWS: Mention the change in behavior.
Fixes https://bugs.debian.org/628815
---
 NEWS   |  3 +++
 doc/coreutils.texi | 10 +++---
 src/pinky.c| 19 ++-
 3 files changed, 28 insertions(+), 4 deletions(-)

diff --git a/NEWS b/NEWS
index 3a7e2aa57..0f4503cbb 100644
--- a/NEWS
+++ b/NEWS
@@ -65,6 +65,9 @@ GNU coreutils NEWS-*- outline -*-
   numfmt will accept lowercase 'k' to indicate Kilo or Kibi units on input,
   and uses lowercase 'k' when outputting such units in '--to=si' mode.
 
+  pinky no longer tries to canonicalize the user's login location by default,
+  rather requiring the new --lookup option to enable this often slow feature.
+
   wc no longer ignores encoding errors when counting words.
   Instead, it treats them as non white space.
 
diff --git a/doc/coreutils.texi b/doc/coreutils.texi
index 92c9ceefb..41b644ecf 100644
--- a/doc/coreutils.texi
+++ b/doc/coreutils.texi
@@ -16238,11 +16238,13 @@ Print a line of column headings.
 List only the entries that correspond to processes via which the
 system is waiting for a user to login.  The user name is always @samp{LOGIN}.
 
+@macro lookupOption
 @item --lookup
 @opindex --lookup
-Attempt to canonicalize hostnames found in utmp through a DNS lookup.  This
-is not the default because it can cause significant delays on systems with
-automatic dial-up internet access.
+Attempt to canonicalize hostnames found in utmp through a DNS lookup.
+This is not the default because of potential delays.
+@end macro
+@lookupOption
 
 @item -m
 @opindex -m
@@ -16370,6 +16372,8 @@ format.
 Omit the user's full name, remote host, and idle time when printing in
 short format.
 
+@lookupOption
+
 @end table
 
 @exitstatus
diff --git a/src/pinky.c b/src/pinky.c
index 0843efd55..77c1c2c01 100644
--- a/src/pinky.c
+++ b/src/pinky.c
@@ -61,6 +61,9 @@ static bool include_home_and_shell = true;
 /* if true, use the "short" output format. */
 static bool do_short_format = true;
 
+/* If true, attempt to canonicalize hostnames via a DNS lookup. */
+static bool do_lookup;
+
 /* if true, display the ut_host field. */
 #if HAVE_STRUCT_XTMP_UT_HOST
 static bool include_where = true;
@@ -71,8 +74,15 @@ static bool include_where = true;
 static char const *time_format;
 static int time_format_width;
 
+/* for long options with no corresponding short option, use enum */
+enum
+{
+  LOOKUP_OPTION = CHAR_MAX + 1
+};
+
 static struct option const longopts[] =
 {
+  {"lookup", no_argument, nullptr, LOOKUP_OPTION},
   {GETOPT_HELP_OPTION_DECL},
   {GETOPT_VERSION_OPTION_DECL},
   {nullptr, 0, nullptr, 0}
@@ -279,7 +289,7 @@ print_entry (struct gl_utmp const *utmp_ent)
   if (display)
 *display++ = '\0';
 
-  if (*ut_host)
+  if (*ut_host && do_lookup)
 /* See if we can canonicalize it.  */
 host = canon_host (ut_host);
   if ( ! host)
@@ -499,6 +509,9 @@ usage (int status)
   -i  omit the user's full name and remote host in short format\n\
   -q  omit the user's full name, remote host and idle time\n\
   in short format\n\
+"), stdout);
+  fputs (_("\
+  --lookupattempt to canonicalize hostnames via DNS\n\
 "), stdout);
   fputs (HELP_OPTION_DESCRIPTION, stdout);
   fputs (VERSION_OPTION_DESCRIPTION, stdout);
@@ -574,6 +587,10 @@ main (int argc, char **argv)
   

Bug#1058752: bug#62572: Bug#1058752: bug#62572: cp --no-clobber behavior has changed

[Pádraig Brady]

On 01/02/2024 00:36, Paul Eggert wrote:

On 1/31/24 06:06, Pádraig Brady wrote:

To my mind the most protective option takes precedence.


That's not how POSIX works with mv -i and mv -f. The last flag wins. I
assume this is so that people can have aliases or shell scripts that
make -i the default, but you can override by specifying -f on the
command line. E.g., in mymv:

 #!/bin/sh
 mv -i "$@"

then "mymv -f a b" works as expected.

Wouldn't a similar argument apply to cp's --update options?

Or perhaps we should play it safe, and reject any combination of
--update etc. options that are incompatible. We can always change our
mind later and say that later options override earlier ones, or do
something else that's less conservative.


OK I err'd on the side of changing as little as possible wrt precedence.
-n still has precedence over any -u,--update option.
That's simplest to understand (while not changing existing precedence),
and shouldn't cause any practical issues.

I plan to push the 2 attached patches for this tomorrow.

thanks,
PádraigFrom 51cf6f3ff272467bc9cde75c48d0250446be9b9c Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?P=C3=A1draig=20Brady?= 
Date: Sat, 24 Feb 2024 19:51:56 +
Subject: [PATCH 1/2] cp,mv: reinstate that -n exits with success if files
 skipped

* src/cp.c (main): Adjust so that -n will exit success if skipped files.
* src/mv.c (main): Likewise.
* doc/coreutils.texi (cp invocation): Adjust the description of -n.
* src/system.h (emit_update_parameters_note): Adjust --update=none
comparison.
* tests/cp/cp-i.sh: Adjust -n exit status checks.
* tests/mv/mv-n.sh: Likewise.
* NEWS: Mention the change in behavior.
Fixes https://bugs.gnu.org/62572
---
 NEWS   |  4 
 doc/coreutils.texi | 17 +
 src/cp.c   | 14 --
 src/mv.c   | 10 ++
 src/system.h   |  4 ++--
 tests/cp/cp-i.sh   | 11 +--
 tests/mv/mv-n.sh   | 11 +--
 7 files changed, 39 insertions(+), 32 deletions(-)

diff --git a/NEWS b/NEWS
index 36b7fd1fe..a52b4cf66 100644
--- a/NEWS
+++ b/NEWS
@@ -43,6 +43,10 @@ GNU coreutils NEWS-*- outline -*-
   basenc --base16 -d now supports lower case hexadecimal characters.
   Previously an error was given for lower case hex digits.
 
+  cp --no-clobber, and mv -n no longer exit with failure status if
+  existing files are encountered in the destination.  Instead they revert
+  to the behavior from before v9.2, silently skipping existing files.
+
   ls --dired now implies long format output without hyperlinks enabled,
   and will take precedence over previously specified formats or hyperlink mode.
 
diff --git a/doc/coreutils.texi b/doc/coreutils.texi
index c42126955..911e15b46 100644
--- a/doc/coreutils.texi
+++ b/doc/coreutils.texi
@@ -9059,12 +9059,13 @@ a regular file in the destination tree.
 @itemx --no-clobber
 @opindex -n
 @opindex --no-clobber
-Do not overwrite an existing file; silently fail instead.
-This option overrides a previous
-@option{-i} option.  This option is mutually exclusive with @option{-b} or
-@option{--backup} option.
-See also the @option{--update=none} option which will
-skip existing files but not fail.
+Do not overwrite an existing file; silently skip instead.
+This option overrides a previous @option{-i} option.
+This option is mutually exclusive with @option{-b} or @option{--backup} option.
+This option is deprecated due to having a different exit status from
+other platforms.  See also the @option{--update} option which will
+give more control over how to deal with existing files in the destination,
+and over the exit status in particular.
 
 @item -P
 @itemx --no-dereference
@@ -9335,8 +9336,8 @@ This is the default operation when an @option{--update} option is not specified,
 and results in all existing files in the destination being replaced.
 
 @item none
-This is similar to the @option{--no-clobber} option, in that no files in the
-destination are replaced, but also skipping a file does not induce a failure.
+This is like the deprecated @option{--no-clobber} option, where no files in the
+destination are replaced, and also skipping a file does not induce a failure.
 
 @item older
 This is the default operation when @option{--update} is specified, and results
diff --git a/src/cp.c b/src/cp.c
index 0355ed97f..36ae4fb66 100644
--- a/src/cp.c
+++ b/src/cp.c
@@ -195,8 +195,8 @@ Copy SOURCE to DEST, or multiple SOURCE(s) to DIRECTORY.\n\
   -L, --dereferencealways follow symbolic links in SOURCE\n\
 "), stdout);
   fputs (_("\
-  -n, --no-clobber ensure no existing files overwritten, and fail\n\
- silently instead. See also --update\n\
+  -n, --no-clobber silently skip existing files.\n\
+ See also --update\n\
 "), stdout);
   fputs (_("\
   -P, --no-dereference never follow symb

Bug#1063837: coreutils: shred: documentation bug: 3>- should be 3>&-

[Pádraig Brady]

On 13/02/2024 12:21, Greg Wooledge wrote:

Package: coreutils
Version: 9.1-1
Severity: minor
Tags: upstream

The info page for shred includes this example code:

  i=$(mktemp)
  exec 3<>"$i"
  rm -- "$i"
  echo "Hello, world" >&3
  shred - >&3
  exec 3>-

The last line is incorrect.  It should be "exec 3>&-" which closes file
descriptor 3.  As written, the example code creates or truncates a
file literally named "-" instead.

This same error appears on
https://www.gnu.org/software/coreutils/manual/html_node/shred-invocation.html
(as of today) so I believe it's an upstream bug.


Fixed upstream.
https://github.com/coreutils/coreutils/commit/d39c52372

Thank you.
Pádraig.

Bug#1058752: bug#62572: Bug#1058752: bug#62572: cp --no-clobber behavior has changed

[Pádraig Brady]

On 30/01/2024 18:31, Paul Eggert wrote:

On 2024-01-30 03:18, Pádraig Brady wrote:

So we now have the proposed change as:

    - revert -n to old silent success behavior
    - document -n as deprecated
    - Leave --update=none as is (will be synonymous with -n)
    - Provide --update=none-fail to diagnose and exit failure


Thanks, that's a better proposal, but I still see several opportunities
for confusion.

If I understand things correctly, cp --update=none is not synonymous
with the proposed (i.e., old-behavior) cp -n, because -n overrides
previous -i options but --update=none does not. Also, -n overrides
either previous or following --update=UPDATE options, but --update=none
overrides only previous --update=UPDATE options. (For what it's worth,
FreeBSD -n overrides


Good point.
Well -n is a protection mechanism really, so should override.
Since --update now incorporates a protection mode, it should also override I 
think.


Some of this complication seems to be for consistency with how mv
behaves with -f, -i, -n, and --update, and similarly with how rm behaves
with -f, -i, -I, and --interactive. To be honest I don't quite
understand the reason for all this complexity, which suggests it should
be documented somewhere (the manual?) if it isn't already.


To my mind the most protective option takes precedence.
So from least to most that would be -f -n --update=none --update=none-fail -i
So the main consideration there is that -n should not override 
--update=none-fail


This raises more questions:

* If we deprecate cp -n, what about mv -n? FreeBSD mv -n behaves like
Coreutils mv -n: it silently does nothing and exits successfully. So
there's no compatibility argument for changing mv -n's behavior.
However, whatever --update option we add to cp (to output a diagnostic
and exit with failure) should surely also be added to mv, to aid
consistency.


Yes I agree.


* Should cp --update=none be changed so that it really behaves like the
old cp -n, in that it overrides other options in ways that differ from
how the other --update=UPDATE options behave? I'm leaning toward "no" as
this adds complexity that I don't see the use for.

* If we don't change cp --update=none's overriding behavior, is it still
OK to tell users to substitute --update=none for -n even though the two
options are not exactly equivalent? I'm leaning towards "yes" but would
like other opinions.


Yes I think we should still give that advice to deprecate -n,
if we ensure that -n does not override --update=none=fail.

thanks,
Pádraig

Bug#1058752: bug#62572: Bug#1058752: bug#62572: cp --no-clobber behavior has changed

[Pádraig Brady]

On 29/01/2024 21:44, Paul Eggert wrote:

On 1/29/24 08:11, Pádraig Brady wrote:


Right, that's why I'm still leaning towards my proposal in the last mail.


Well, I won't insist on doing nothing; however, the proposal needs
ironing out and now's a good time to do it before installing changes.



    - revert to previous exit success -n behavior
    - document -n as deprecated
    - provide --update=noclobber to give exit failure functionality


So --update=noclobber would differ in meaning from the deprecated-in-9.5
--no-clobber, but would agree in meaning with 9.4 --no-clobber? That
sounds pretty confusing for future users. (And a nit: why should one
spelling have a hyphen but the other doesn't?)


That's a fair point; just avoid "no-clobber" entirely.
We could choose a new name so. How about --update=none-fail


      - BTW, it probably makes sense to print a diagnostic for each
skipped file here
    as it's exceptional behavior, for which we're exiting with
failure for.


Coreutils 9.4 cp -n already does that, no? So I'm not sure what's being
proposed here.

$ touch a b
$ cp -n a b; echo $?
cp: not replacing 'b'
1


Sorry I got confused between your suggestion of added diagnostics,
and FreeBSD's silent behavior here. Looks like we just keep the
existing behavior of diagnosing 'not replacing' iff exiting with failure.


    - the existing --update=none provides the exit success functionality


It seems to me that this proposal conflates two questions:

* What rules should cp use to decide whether to update a destination?

* When cp decides not to update a destination, what should it do? Exit
with nonzero status? Output a diagnostic? Both? Neither?

Aren't these independent axes? If so, shouldn't they have independent
options? For example, since we have --update=older, shouldn't there be a
way to say "I want to copy A to B only if B is older than A, and I want
the exit status to be zero only if A was copied to B"?


Well they're not entirely independent.
It's more appropriate to output a diagnostic if exiting failure,
and POSIX also advises along the same lines.
We also have --verbose to control diagnostics somewhat
for the non failure case.

So we now have the proposed change as:

  - revert -n to old silent success behavior
  - document -n as deprecated
  - Leave --update=none as is (will be synonymous with -n)
  - Provide --update=none-fail to diagnose and exit failure

thanks,
Pádraig.

Bug#1058752: bug#62572: Bug#1058752: bug#62572: cp --no-clobber behavior has changed

[Pádraig Brady]

On 29/01/2024 14:01, Michael Stone wrote:

On Sun, Jan 28, 2024 at 11:14:14PM -0800, Paul Eggert wrote:

I'm not sure reverting would be best. It would introduce more
confusion, and would make coreutils incompatible with FreeBSD again.


Reverting makes more sense than the current situation. I do not
understand why you seem to value FreeBSD compatibility more than
compatibility with the vast majority of installed coreutils/linux
systems.


Yes, it's not a good place to be. Surely current coreutils is better
than what Debian is doing.


You've introduced a silent incompatibility and I'm trying to find some
way to make that clear. If upstream would provide a better solution I
would certainly use it. I have despaired of there being such since your
attitude thus far seems to be entirely dismissive of compatibility
concerns.


That's a bit unfair.  The current upstream -n behavior is with a view
to being _more_ compat across all systems.
Now I agree this may not be worth it in this case,
but it is a laudable goal.


Another possibility is to add a warning that is emitted only at the
end of 'cp'. The warning would occur only if the exit code differs
because of this cp -n business.


You'd only emit a notification of a change in behavior if some
(potentially uncommon/rarely encountered) situation arises which would
actually trigger breakage? So people can't prepare ahead of time and
change their script to handle the necessary change in logic, they can
only maybe figure out why something broke at 2am when the uncommon event
occurred?


Yes I agree with this point, mostly.
Outputting a diagnostic would help users diagnose what's going on,
and help users to fix forward and avoid their problematic -n usage.
But yes, the crux of the issue with fixing issues as they occur,
is it depends on the state of the destination and so can become an issue
at any time.  Now I previously did an audit with github and debian code search
and noticed very few problematic uses of cp -n, but that does miss
the mountain of private code.


At the end of the day, -n is basically a useless option with unknowable
semantics which should be avoided by everyone. In the past it was an
option which wasn't portable between coreutils/linux and freebsd systems,
and I guess you've "fixed" that (by making it an option everyone should
avoid entirely), but let's be honest about how common that concern was.


Right, that's why I'm still leaning towards my proposal in the last mail.
  - revert to previous exit success -n behavior
  - document -n as deprecated
  - provide --update=noclobber to give exit failure functionality
- BTW, it probably makes sense to print a diagnostic for each skipped file 
here
  as it's exceptional behavior, for which we're exiting with failure for.
  - the existing --update=none provides the exit success functionality

With the above in place for the next coreutils release,
then debian could remove its noisy patch.

thanks,
Pádraig

Bug#1061612: coreutils: cp -n deprecation warning gives questionable advice

[Pádraig Brady]

On 27/01/2024 16:02, Michael Stone wrote:

On Sat, Jan 27, 2024 at 02:00:14PM +0100, Sven Joachim wrote:

Package: coreutils
Version: 9.4-3+b1

,
| $ cp -n /bin/true tmp
| cp: warning: behavior of -n is non-portable and may change in future; use 
--update=none instead
`

The advice to use the --update=none option is highly questionable,
because this option is even less portable than -n.  It is not available
in coreutils older than 9.3 or in other cp implementations.


There is no alternative that I can see. I didn't create this situation,
it was created upstream. You can continue to use -n and ignore the
warning, but in future if debian stops patching -n to behave the way it
always has in order to match upstream, stuff will break. If debian keeps
patching -n, then then anything you write in debian will be depending on
behavior that differs in other distributions and will break everywhere
else (except older versions of those distributions). It's a mess.

This warning isn't for debian developers of existing packages, because
debian is maintaining compatibility (at least for now); you'll see a
warning message but the actual behavior hasn't changed and won't change
in debian without some coordination with affected packages. But for
developers with *new* upstream code that uses -n, which behavior does
the code expect? There are now two answers and *the only solution is to
not use -n*; it's not possible to simply file bugs with packages and fix
it once, because this is an ongoing incompatibility. I understand that
the messages are somewhat obnoxious, but my attempt to address the
situation upstream instead failed.


That is a very aggressive deprecation.
IMHO it would have been better for debian to have -n behave
like it did previously and (silently) skip files and not set an error exit 
status.
If it was a mess, this is a mess squared.
I guess this forces our hand a bit.
I'll address upstream...

Bug#1059616: "wc -l" gives "illegal instruction"

[Pádraig Brady]

On 29/12/2023 09:52, gates.ocarina...@icloud.com wrote:


Package: coreutils
Version: 9.1-1

When I run command wc -l, it gives “illegal instruction”.
I reinstalled the coreutils package:

$ wc -l

Illegal instruction


$ ls -l | wc -l

Illegal instruction


$ ls -l | wc -lw

       4      29


$ wc test.txt

10  8 18 test.txt


$ cat test.txt | wc -mwl

      10       8      18


$ wc -l test.txt

Illegal instruction


$ cat test.txt | wc -l

Illegal instruction


These upstream patches should address the issue:

https://github.com/coreutils/coreutils/commit/91a74d361.patch
https://github.com/coreutils/coreutils/commit/7814596fa.patch

Bug#1058752: bug#62572: cp --no-clobber behavior has changed

[Pádraig Brady]

On 16/12/2023 21:46, Bernhard Voelker wrote:

On 12/15/23 21:13, Michael Stone wrote:

On Fri, Dec 15, 2023 at 11:21:06AM -0800, Paul Eggert wrote:

Stlll, Pádraig gave a reasonable summary of why the change was made,


To clarify my summary a little, there I said that -n now _immediately_ fails.
I should have said _silently_ fails.  I.e. the complete copy operation
proceeds as before, and only the exit status is at issue here.


despite its incompatibility with previous behavior. (One thing I'd add
is that the FreeBSD behavior is inherently less race-prone.)


Whether the implementation is race-prone or not is an internal thing.
I think we're currently discussing more on a user-perspective level.

IIUC then the question is whether `cp -n` should continue to behave like
the (new) `cp --update=none` which returns EXIT_SUCCESS.

Regardless what other implementations do, when reading the -n description
from a user's point of view:

-n, --no-clobber do not overwrite an existing file (overrides a
   -u or previous -i option). See also --update

then I'd expect the tool to just skip existing files like `rsync 
--ignore-existing`
does.  In that regard I would be surprised if skipping files would result in an 
error.
Well, I would understand if there'd be a '--no-clobber=fail' option.


Agreed we should improve the docs a bit for this option.
I'll apply this at least:

diff --git a/doc/coreutils.texi b/doc/coreutils.texi
index 1f8b356d1..bf0f424d3 100644
--- a/doc/coreutils.texi
+++ b/doc/coreutils.texi
@@ -9057,6 +9057,8 @@ Do not overwrite an existing file; silently fail instead.
 This option overrides a previous
 @option{-i} option.  This option is mutually exclusive with @option{-b} or
 @option{--backup} option.
+See also the @option{--update=none} option which will
+skip existing files but not fail.

 @item -P
 @itemx --no-dereference
diff --git a/src/cp.c b/src/cp.c
index 04a5cbee3..3ccc4c4e6 100644
--- a/src/cp.c
+++ b/src/cp.c
@@ -192,8 +192,8 @@ Copy SOURCE to DEST, or multiple SOURCE(s) to DIRECTORY.\n\
   -L, --dereferencealways follow symbolic links in SOURCE\n\
 "), stdout);
   fputs (_("\
-  -n, --no-clobber do not overwrite an existing file (overrides 
a\n\
- -u or previous -i option). See also 
--update\n\
+  -n, --no-clobber ensure no existing files overwritten, and 
fail\n\
+ silently instead. See also --update\n\
 "), stdout);
   fputs (_("\
   -P, --no-dereference never follow symbolic links in SOURCE\n\



As Kamil added the option in 2009, I'd assume that the same patch was already
active in RHEL versions for quite some longer time.
Now changing the exit code feels kind of rough.


Well RHEL 6 came out a bit after (2010), and had the --no-clobber change,
while RHEL 5 before that did not.

Taking about distros, it's worth noting that the change is Fedora 39
which has been released for a month now.
We'll keep a close eye on issues, but haven't heard much as
of yet at least.


Therefore, from a pure user's perspective and regarding many years of 
precedence,
I am 80:20 for reverting the exit code change.


Thanks for your thoughts,
appreciated as always.

cheers,
Pádraig

Bug#1058752: bug#62572: cp --no-clobber behavior has changed

[Pádraig Brady]

On 15/12/2023 15:56, Michael Stone wrote:

I tend to think this was a serious mistake: it breaks the behavior of
existing scripts with no deprecation period. A stated advantage is
better compatibility with freebsd, but I don't understand why that is
more desirable than compatibility with all deployed gnu/linux systems? I
also don't think it's sufficient to try to lawyer out by saying that the
current behavior was undocumented: the previous documentation said that
-n would "silently do nothing" and that the return code would be zero on
success. Logically, unless cp fails to "do nothing", it should exit with
a zero code.

Such a drastic change in behavior demands a new flag, not a radical
repurposing of a widely used existing flag.

I was hoping to see more action on this bug, but that hasn't happened.
I'm not sure I see a way forward for debian other than reverting to the
old behavior. I am reluctant to do so as that will likely lead to
divergent behavior between distributions, but breaking scripts without a
compelling reason is also not good. I would encourage coreutils to
reconsider the change and finding a non-breaking way forward.


Yes it's a fair point.
It's an awkward case, and worth discussing.

To summarise:

  coreutils >= 7.1 had -n skip existing in dest (2009)
  coreutils >= 9.2 has -n immediately fail if existing in dest
  coreutils >= 9.3 has --update=none to skip existing in dest

  FreeBSD >= 4.7/macos has -n immediately fail if existing in dest

  bash has noclobber as a file protection mechanism,
  and fails immediately upon trying to overwrite a file.
  This is more consistent with the new coreutils behavior.

I see a reasonable amount of cp -n usage across github:
https://github.com/search?q=/cp+.*+-n+.*/+path:*.sh=code

Now it's not clear which behavior these github usages expect,
and the original docs didn't make it clear which behavior to expect.
A quick scan of the github usages also seem mainly to expect
a protection rather than an update use case, so failing
immediately would be the most appropriate action there too.
Also the original coreutils bug report here expected the new behaviour.

So we probably all agree that failing immediately is the
most appropriate / consistent -n behavior,
but GNU had diverged from that so there are about 10 years
of scripts that may expect the silent skip behavior.

Two options I see are:

- Leave as is and fix -n usages that expected the skip behavior
- Deprecate -n entirely and prompt to use --update={fail,none}

Advantages of leaving as is:
We get consistency of "noclobber" behavior across systems / shells.
We fix cases where previously scripts could have proceeded with
stale old files in place.

Disadvantages of leaving as is:
Users expecting the skip behavior, have to change to --update=none.

There is no potential for data loss etc. so it just comes
down to how disruptive it is, or how often -n was used
with the "skip behavior" assumption.

We've not had much push back as of yet,
and my current thinking is it's not that disruptive a change.
So I'd be 55:45 if favor of keeping things as is.

thanks,
Pádraig.

Bug#1054559: numfmt: --to lakh ?

[Pádraig Brady]

On 25/10/2023 22:57, Trent W. Buck wrote:

Package: coreutils
Version: 9.1-1
Severity: wishlist
File: /usr/bin/numfmt
Tags: upstream

In India it is common to write large numbers using combinations of
"lakh" / "L" (10⁷) and "crore" / "cr" (10⁵).

 https://en.wikipedia.org/wiki/Lakh
 https://en.wikipedia.org/wiki/Crore

For example,

 According to […] ₹15.3 lakh crore of the ₹15.41 lakh crore in demonetised 
bank notes, […]

 — https://en.wikipedia.org/wiki/2016_Indian_banknote_demonetisation

When I try to convert these to SI units in my head, I always mess up.

Could numfmt handle this for me?
Something like this (my maths might be wrong!):

 $ numfmt --from=auto --to=लाख 15.3T
 15.3 lakh crore

 $ numfmt --from=auto --to=लाख 1.2G
 120 crore

 $ numfmt --from=auto --to=लाख 1.2M
 12 lakh

 $ numfmt --from=लाख --to=si 12 lakh
 1.2M

 $ numfmt --from=लाख --to=si 15.3 lakh crore
 15.3T


Noted in upstream todo.
This is something that would be nice to have in numfmt.
The designator might be better as "indic" rather than "लाख".
The space between number and suffixes would be hard to support.
So I;m thinking something like this would be more achievable:

  $ numfmt --from=auto --to=indic 15.3T
  15.3Lcr

thanks,
Pádraig

Bug#1054128: coreutils: cp -p --parent not working

[Pádraig Brady]

On 17/10/2023 15:54, Georg Borgström wrote:

Package: coreutils
Version: 9.1-1
Severity: normal

Dear Maintainer,

This doesn't work in bookworm but works in bullseye:

$ mkdir /tmp/newdir
$ cp -p --parent /etc/passwd /tmp/newdir
cp: ‘etc’: No such file or directory

/tmp/newdir/etc gets created but with wrong permission and also the error 
message above.

$ ls -ald /tmp/newdir/etc
drwx-- 2 gb gb 60 Oct 17 14:54 /tmp/newdir/etc

Should be:
drwxr-xr-x 2 gb gb 60 Oct 17 11:09 /tmp/newdir/etc

Could be the same as:
https://bugs.launchpad.net/ubuntu/+source/coreutils/+bug/2017414


I think the upstream fix (released in coreutils 9.4) is:
https://github.com/coreutils/coreutils/commit/c6b1fe434

Pádraig

Bug#1042546: tac: /tmp/tac...: read error: Illegal seek

[Pádraig Brady]

On 30/07/2023 20:30, наб wrote:

On Sun, Jul 30, 2023 at 12:14:10PM +0100, Pádraig Brady wrote:

On 30/07/2023 05:09, наб wrote:

$ truncate -s 5G 5G
$ cat 5G | /bin/tac | wc -c
/bin/tac: /tmp/tacOOOwnf: read error: Illegal seek
0

Couldn't repo with 64 bit tac 9.1

Tested on amd64.


or 32 bit tac latest

I just get the expected:

$ cat 5G | git/coreutils/src/tac | wc -c
git/coreutils/src/tac: /tmp/cutmpZRrn27: write error: No space left on device

Well, what if you let /tmp grow enough to allocate the 5G
(or put it on a compressed filesystem that'll discard the zeroes)?
I ran this on a big system so my /tmp has 47G avail,
but I can reproduce your lack of reproduction by making it too small:

# mount -t tmpfs -o size=1g tmpfs dir
$ cat 5G | TMPDIR=dir /bin/tac | wc -c
/bin/tac: dir/tacbIeNII: write error: No space left on device
0


Indeed.
I'm addressing this upstream with the attached.

cheers,
Pádraig
From ef60e3727ee4e4e5c3232a2f44a4f8ef9681aaba Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?P=C3=A1draig=20Brady?= 
Date: Mon, 31 Jul 2023 12:41:26 +0100
Subject: [PATCH] tac: handle short reads on input

This can be reproduced by getting the read() above 2G,
which induces a short read, thus triggering the erroneous failure.

  $ truncate -s 5G 5G

  $ cat 5G | TMPDIR=$PWD tac | wc -c
  tac: /tmp/tacFt7txA: read error: Illegal seek
  0

With the fix in place we now get:

  $ cat 5G | TMPDIR=$PWD src/tac | wc -c
  5368709120

* src/tac.c (tac_seekable): Use full_read() to handle short reads.
* NEWS: Mention the bug fix.
Reported at https://bugs.debian.org/1042546
---
 NEWS  | 4 
 src/tac.c | 3 ++-
 2 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/NEWS b/NEWS
index 92e591ee2..011058299 100644
--- a/NEWS
+++ b/NEWS
@@ -46,6 +46,10 @@ GNU coreutils NEWS-*- outline -*-
   'pr --length=1 --double-space' no longer enters an infinite loop.
   [This bug was present in "the beginning".]
 
+  tac now handles short reads on its input.  Previously it may have exited
+  erroneously, especially with large input files with no separators.
+  [This bug was present in "the beginning".]
+
   'wc -l' and 'cksum' no longer crash with an "Illegal instruction" error
   on x86 Linux kernels that disable XSAVE YMM.  This was seen on Xen VMs.
   [bug introduced in coreutils-9.0]
diff --git a/src/tac.c b/src/tac.c
index 285f99a74..4c3655895 100644
--- a/src/tac.c
+++ b/src/tac.c
@@ -44,6 +44,7 @@ tac -r -s '.\|
 #include 
 
 #include "filenamecat.h"
+#include "full-read.h"
 #include "safe-read.h"
 #include "temp-stream.h"
 #include "xbinary-io.h"
@@ -336,7 +337,7 @@ tac_seekable (int input_fd, char const *file, off_t file_pos)
   else
 match_start = past_end;
 
-  if (safe_read (input_fd, G_buffer, read_size) != read_size)
+  if (full_read (input_fd, G_buffer, read_size) != read_size)
 {
   error (0, errno, _("%s: read error"), quotef (file));
   return false;
-- 
2.41.0

Bug#1042546: tac: /tmp/tac...: read error: Illegal seek

[Pádraig Brady]

On 30/07/2023 05:09, наб wrote:

Package: coreutils
Version: 9.1-1
Severity: normal

Dear Maintainer,

$ truncate -s 5G 5G
$ cat 5G | /bin/tac | wc -c
/bin/tac: /tmp/tacOOOwnf: read error: Illegal seek
0

наб

-- System Information:
Debian Release: 12.1
   APT prefers stable-updates
   APT policy: (500, 'stable-updates'), (500, 'stable-security'), (500, 
'stable-debug'), (500, 'stable')
Architecture: amd64 (x86_64)
Foreign Architectures: i386


Couldn't repo with 64 bit tac 9.1
or 32 bit tac latest

I just get the expected:

$ cat 5G | git/coreutils/src/tac | wc -c
git/coreutils/src/tac: /tmp/cutmpZRrn27: write error: No space left on device

cheers,
Pádraig

Bug#1039488: coreutils: tail: -f fifo broken when splicing to the fifo (uses inotify)

[Pádraig Brady]

On 26/06/2023 16:03, наб wrote:

Package: coreutils
Version: 9.1-1
Severity: normal

Dear Maintainer,

This came out of a linux-fsdevel@ discussion:
   
https://lore.kernel.org/linux-fsdevel/jbyihkyk5dtaohdwjyivambb2gffyjs3dodpofafnkkunxq7bu@jngkdxx65pux/t/#u

According to Amir, at least, it is "correct" that inotify-watching fifos
is unsupported, and tail(1) speaks of using inotify. Let's see:
-- >8 --
$ echo zupa > fifo
$ echo zupa > fifo
$ echo zupa > fifo
$ echo cupa | strace cat > fifo
...
splice(0, NULL, 1, NULL, 134217728, SPLICE_F_MOVE|SPLICE_F_MORE) = 5
splice(0, NULL, 1, NULL, 134217728, SPLICE_F_MOVE|SPLICE_F_MORE) = 0
...
$ strace < /bin/cat cat > fifo
splice(0, NULL, 1, NULL, 134217728, SPLICE_F_MOVE|SPLICE_F_MORE) = 44016
splice(0, NULL, 1, NULL, 134217728, SPLICE_F_MOVE|SPLICE_F_MORE) = 0
(a couple times until)
$ strace < /bin/cat cat > fifo
splice(0, NULL, 1, NULL, 134217728, SPLICE_F_MOVE|SPLICE_F_MORE) = 32768
splice(0, NULL, 1, NULL, 134217728, SPLICE_F_MOVE|SPLICE_F_MORE   <- 
sleeps here
^C
$ echo qupa > fifo<- 
sleeps in open
-- >8 --
yields
-- >8 --
$ tail -f fifo
zupa
zupa
zupa
-- >8 --
and at this point fifo is full but inotify hasn't returned an event and
thus service has been denied.

According to Amir, at least, this is expected behaviour, and while he
"doesn't think" it is, the only correct way of polling a pipe under Linux
I see is to fall back to classic UNIX sleep()/read().


tail explicitly uses inotify only with regular files and fifos.
We could easily restrict that to just regular files.
You can test that with the undocumented `---disable-inotify` option
(note the three dashes).

cheers,
Pádraig

Bug#1038898: coreutils: tail: -fc +{bigger-than-file} file says "file truncated" and dumps the whole file

[Pádraig Brady]

On 22/06/2023 21:14, наб wrote:

Package: coreutils
Version: 9.1-1
Severity: normal

Dear Maintainer,

-- >8 --
$ ls -l sel.c
-rw-rw-r-- 1 nabijaczleweli users 193 Jun 24  2021 sel.c
$ tail -c +200 -f sel.c
tail: sel.c: file truncated
#include 
#include 
int main() {
printf("%d\n", is_selinux_enabled());
char * con = 0x12345;
printf("%d\n", getcon());
printf("%s\n", con);
freecon(con);
}
(now it sleeps and processes -f normally)
-- >8 --

This is, obviously, wrong. Not least because sel.c wasn't truncated,
but also because the #include is byte 1, not 200.


tail is using a heuristic here and determines
the file was truncated in this case and thus shows
the whole contents of the "new" file.

The particular race significant here was discussed at:
https://lists.gnu.org/r/bug-textutils/2003-05/msg7.html

cheers,
Pádraig

Bug#1037264: cksum crashes intermittently with "Illegal instruction" on some Xen DomU

[Pádraig Brady]

On 13/06/2023 09:38, Axel Beckert wrote:

Hi, especially to Pádraig,




Pádraig Brady wrote:

cksum since v9.0 checks at runtime whether pclmul is supported.
It seems that check is not working appropriately on a Xen DomU.
The routine in question is pclmul_supported() at:
https://github.com/coreutils/coreutils/blob/b841f111/src/cksum.c#L160-L191

That either suggests xen is incorrectly setting PCLMUL and AVX bits,
or perhaps these two bits are not sufficient.
Hmm I wonder do we also need to explicitly check for SSSE3 support?

I.e. I wonder does cksum built with the following help?

[…]

diff --git a/src/cksum.c b/src/cksum.c
index 85afab0ac..98733dadf 100644
--- a/src/cksum.c
+++ b/src/cksum.c
@@ -172,7 +172,7 @@ pclmul_supported (void)
return false;
  }

-  if (! (ecx & bit_PCLMUL) || ! (ecx & bit_AVX))
+  if (! (ecx & bit_PCLMUL) || ! (ecx & bit_AVX) || ! (ecx & bit_SSSE3))
  {
if (cksum_debug)
  error (0, 0, "%s", _("pclmul support not detected"));


No, the patch unfortunately didn't help:


It's great you can test changes at least.
Thanks for trying the above.

At this stage it would be good to get the output from `cpuid -1`
so that hopefully we can get something there to key on
that indicates the cpu doesn't support the instructions.

thanks,
Pádraig

Bug#1037275: dd: regression - posix expression syntax no longer supported

[Pádraig Brady]

On 11/06/2023 18:29, Jim Meyering wrote:

Thanks. The patch looks fine.
My only suggestion would be a stylistic one, to change the ">"
comparison to be the equivalent "<" one, i.e., change this:

+  && *suffix == 'B' && (suffix > str && suffix[-1] != 'B'))

to this (also dropping the unnecessary parentheses):

+  && *suffix == 'B' && str < suffix && suffix[-1] != 'B')

Why? Because of the increasing-to-right number-line argument, where
smaller things are visually on the left of larger ones.

Currently, in src/, the uses of space-delimited < and <= outnumber
uses of > and >= by four to one, 1678 to 428.


Pushed with that adjustment.

thanks
Pádraig

Bug#1037275: dd: regression - posix expression syntax no longer supported

[Pádraig Brady]

On 10/06/2023 02:45, Marc Lehmann wrote:

Package: coreutils
Version: 9.1-1
Severity: normal

Dear Maintainer,

I have a script that was used for some decades on multiple
unices. Beginning with bookworm, it stopped working because dd no longer
understands POSIX expression syntax for bs=:

$ dd if=... bs=1024x1024x32
dd: invalid number: ‘1024x1024x32’

This should be valid syntax according to POSIX, and was understood on
older versions of Debian GNU/Linux:

Two or more positive decimal numbers (with or without k or b) separated by 
x, specifying the product of the indicated values


Yes this was a regression in coreutils 9.1
The patch attached is the proposed upstream fix.

thanks,
PádraigFrom 09186005a77bd24caedc7fa4fc3eb8acf44a4b50 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?P=C3=A1draig=20Brady?= 
Date: Sat, 10 Jun 2023 14:18:00 +0100
Subject: [PATCH] dd: fix parsing of numbers with more than two multipliers

* src/dd.c (parse_integer): Use recursion to support more than two
multipliers.  Also protect suffix[-1] access to ensure we don't
inspect before the passed string.
* tests/dd/bytes.sh: Add test cases.
* NEWS: Mention the bug fix.
Fixes https://bugs.debian.org/1037275
---
 NEWS  |  4 
 src/dd.c  |  8 
 tests/dd/bytes.sh | 12 
 3 files changed, 20 insertions(+), 4 deletions(-)

diff --git a/NEWS b/NEWS
index e47701962..7df9ff5b0 100644
--- a/NEWS
+++ b/NEWS
@@ -11,6 +11,10 @@ GNU coreutils NEWS-*- outline -*-
   cksum again diagnoses read errors in its default CRC32 mode.
   [bug introduced in coreutils-9.0]
 
+  dd again supports more than two multipliers for numbers.
+  Previously numbers of the form '1024x1024x32' gave "invalid number" errors.
+  [bug introduced in coreutils-9.1]
+
   factor, numfmt, and tsort now diagnose read errors on the input.
   [This bug was present in "the beginning".]
 
diff --git a/src/dd.c b/src/dd.c
index 665fc831c..c826d09b8 100644
--- a/src/dd.c
+++ b/src/dd.c
@@ -1428,7 +1428,7 @@ parse_integer (char const *str, strtol_error *invalid)
   intmax_t result;
 
   if ((e & ~LONGINT_OVERFLOW) == LONGINT_INVALID_SUFFIX_CHAR
-  && suffix[-1] != 'B' && *suffix == 'B')
+  && *suffix == 'B' && (suffix > str && suffix[-1] != 'B'))
 {
   suffix++;
   if (!*suffix)
@@ -1436,10 +1436,10 @@ parse_integer (char const *str, strtol_error *invalid)
 }
 
   if ((e & ~LONGINT_OVERFLOW) == LONGINT_INVALID_SUFFIX_CHAR
-  && *suffix == 'x' && ! (suffix[-1] == 'B' && strchr (suffix + 1, 'B')))
+  && *suffix == 'x')
 {
-  uintmax_t o;
-  strtol_error f = xstrtoumax (suffix + 1, , 10, , suffixes);
+  strtol_error f = LONGINT_OK;
+  intmax_t o = parse_integer (suffix + 1, );
   if ((f & ~LONGINT_OVERFLOW) != LONGINT_OK)
 {
   e = f;
diff --git a/tests/dd/bytes.sh b/tests/dd/bytes.sh
index 01753d6a0..21368ead6 100755
--- a/tests/dd/bytes.sh
+++ b/tests/dd/bytes.sh
@@ -60,4 +60,16 @@ for operands in "oseek=8B" "seek=8 oflag=seek_bytes"; do
   compare expected2 out2 || fail=1
 done
 
+# Check recursive integer parsing
+for oseek in '1x2x4 oflag=seek_bytes' '1Bx2x4' '1Bx8' '2Bx4B' '2x4B'; do
+  # seek bytes
+  echo abcdefghijklm |
+dd oseek=$oseek bs=5 > out 2> /dev/null || fail=1
+  compare expected out || fail=1
+done
+
+# Negative checks for integer parsing
+for count in B B1 Bx1 KBB BB KBb KBx x1 1xx1; do
+  returns_ 1 dd count=$count /dev/null || fail=1
+done
 Exit $fail
-- 
2.40.1

Bug#1037264: cksum crashes intermittently with "Illegal instruction" on some Xen DomU

[Pádraig Brady]

On 09/06/2023 18:40, Axel Beckert wrote:

Package: coreutils
Version: 9.1-1
Severity: important
X-Debbugs-Cc: a...@debian.org
Control: affects -1 aptitude-robot

On a Xen DomU running Debian 12, cksum intermittently crashes as
follows:

# while :; do dd if=/dev/urandom count=1 2> /dev/null | cksum ; done
1758277878 512
2101634611 512
Illegal instruction



So to summarise

* Debian 12 in Xen DomU exihibits this behaviour.

* Debian 11 in Xen DomU on same Dom0 does not exihibit this behaviour.

* The Xen Dom0 (Debian 11 though) itself does not exihibit this
   behaviour.

* A Debian 12 installation on bare metal with the same CPU ("AMD EPYC
   7313P 16-Core Processor") as the Dom0 does not exhibit this
   behaviour.



Hence some more details about the system:

* cksum --debug says: "cksum: using pclmul hardware support"

* amd64-microcode on the Dom0 is at 3.20191218.1


cksum since v9.0 checks at runtime whether pclmul is supported.
It seems that check is not working appropriately on a Xen DomU.
The routine in question is pclmul_supported() at:
https://github.com/coreutils/coreutils/blob/b841f111/src/cksum.c#L160-L191

That either suggests xen is incorrectly setting PCLMUL and AVX bits,
or perhaps these two bits are not sufficient.
Hmm I wonder do we also need to explicitly check for SSSE3 support?

I.e. I wonder does cksum built with the following help?

BTW it would be worth checking if ssse3 is mentioned in /proc/cpuinfo also.
If it was NOT, then there would be more of a chance of that change helping.

diff --git a/src/cksum.c b/src/cksum.c
index 85afab0ac..98733dadf 100644
--- a/src/cksum.c
+++ b/src/cksum.c
@@ -172,7 +172,7 @@ pclmul_supported (void)
   return false;
 }

-  if (! (ecx & bit_PCLMUL) || ! (ecx & bit_AVX))
+  if (! (ecx & bit_PCLMUL) || ! (ecx & bit_AVX) || ! (ecx & bit_SSSE3))
 {
   if (cksum_debug)
 error (0, 0, "%s", _("pclmul support not detected"));

Bug#1037217: coreutils: od: -S considers printable, not graphic, characters

[Pádraig Brady]

On 08/06/2023 00:27, наб wrote:

Package: coreutils
Version: 8.32-4+b1
Version: 9.1-1
Severity: normal

Dear Maintainer,

isgraph(3):
isgraph()
   checks for any printable character except space.
od(1):
-S BYTES, --strings[=BYTES]
   output strings of at least BYTES graphic chars; 3 is implied 
when BYTES is not specified
$ printf 'a b\0' | od -S3
000 a b

?


This isn't specified by POSIX,
so let's match the docs to the operation.
Done upstream in the attached.

cheers,
Pádraig
From 66ea09b0fecb4fa1e4de78e3738bdbb1442b3f31 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?P=C3=A1draig=20Brady?= 
Date: Thu, 8 Jun 2023 10:58:10 +0100
Subject: [PATCH] doc: od --strings: clarify operation

* doc/coreutils.texi (od invocation): Remove mention of ASCII,
as all printable characters in unibyte locales are output.
* src/od.c (usage): Clarify that only NUL terminated strings
are displayed, and that it's printable chars, not only graphic chars
that are output. I.e., spaces are output also if part of the string.
Reported at https://bugs.ddebian.org/1037217
---
 doc/coreutils.texi | 2 +-
 src/od.c   | 5 ++---
 2 files changed, 3 insertions(+), 4 deletions(-)

diff --git a/doc/coreutils.texi b/doc/coreutils.texi
index 6a693e283..e9d7b8eb4 100644
--- a/doc/coreutils.texi
+++ b/doc/coreutils.texi
@@ -2058,7 +2058,7 @@ Output at most @var{bytes} bytes of the input.  Prefixes and suffixes on
 @opindex --strings
 @cindex string constants, outputting
 Instead of the normal output, output only @dfn{string constants}: at
-least @var{bytes} consecutive ASCII graphic characters,
+least @var{bytes} consecutive printable characters,
 followed by a zero byte (ASCII NUL).
 Prefixes and suffixes on @var{bytes} are interpreted as for the
 @option{-j} option.
diff --git a/src/od.c b/src/od.c
index 10a28e21f..f68407008 100644
--- a/src/od.c
+++ b/src/od.c
@@ -356,9 +356,8 @@ suffixes may be . for octal and b for multiply by 512.\n\
 "), stdout);
   fputs (_("\
   -N, --read-bytes=BYTES  limit dump to BYTES input bytes\n\
-  -S BYTES, --strings[=BYTES]  output strings of at least BYTES graphic chars;\
-\n\
-3 is implied when BYTES is not specified\n\
+  -S BYTES, --strings[=BYTES]  show only NUL terminated strings\n\
+of at least BYTES (3) printable characters\n\
   -t, --format=TYPE   select output format or formats\n\
   -v, --output-duplicates do not use * to mark line suppression\n\
   -w[BYTES], --width[=BYTES]  output BYTES bytes per output line;\n\
-- 
2.40.1

Bug#1036827: coreutils: split: --additional-suffix= doesn't always find the /, but also shouldn't look for it?

[Pádraig Brady]

On 27/05/2023 13:54, наб wrote:

Package: coreutils
Version: 8.32-4+b1
Version: 9.1-1
Severity: normal

Dear Maintainer,
   $ echo | split --additional-suffix=asd/
   split: xaaasd/: Is a directory
   $ echo | split --additional-suffix=asd/asd/
   split: invalid suffix ‘asd/asd/’, contains directory separator
   Try 'split --help' for more information.
   $ echo | split --additional-suffix=asd/asd
   split: invalid suffix ‘asd/asd’, contains directory separator
   Try 'split --help' for more information.
   $ echo | split --additional-suffix=/
   split: invalid suffix ‘/’, contains directory separator
   Try 'split --help' for more information.

Also, I don't really see a reason to reject slashes in there anyway?
Forbidding
   $ printf abcd > a
   $ printf ABCD > A
   $ mkdir 0 1 2 3
   $ split -b1 --additional-suffix=/a -da1 a ''
   $ split -b1 --additional-suffix=/A -da1 A ''
   $ find
   .
   ./0
   ./0/A
   ./0/a
   ./3
   ./3/A
   ./3/a
   ./2
   ./2/A
   ./2/a
   ./1
   ./1/A
   ./1/a
   ./A
   ./a
   $ grep . ?/*
   0/a:a
   0/A:A
   1/a:b
   1/A:B
   2/a:c
   2/A:C
   3/a:d
   3/A:D
seems relatively pointless.



Yes we should fix the inconsistency here at least.
The "slash detection" logic was copied from mktemp,
so we'll need to look at that too.

$ mktemp --tmpdir=. --suffix=a/
mktemp: failed to create file via template ‘./tmp.XXa/’: Is a directory
$ mktemp --suffix=a/
mktemp: failed to create file via template ‘/tmp/tmp.XXa/’: Is a 
directory

Though mktemp can create dirs, so perhaps the trailing / is allowed for that?
Though docs say a slash isn't allowed in suffix, so it's probably an oversight.

Note templates for mktemp can contain dirs,
so it's only the combo of dirs with /tmp
or sub dirs in --suffix that are disallowed,
which makes sense for mktemp.

For split it may well be simplest to just remove the restriction.

cheers,
Pádraig

Bug#1036651: coreutils: split: -n with (some?) devices fails with EOVERFLOW, accepts some chardevs?

[Pádraig Brady]

On 23/05/2023 20:44, наб wrote:

Package: coreutils
Version: 9.1-1
Version: 8.32-4+b1
Severity: normal

Dear Maintainer,

This happens regardless of the blockdev size:
   $ split -n20 /dev/loop2
   split: /dev/loop2: cannot determine file size: Value too large for defined 
data type
and with
   $ split -n3 /dev/full
   split: /dev/full: cannot determine file size: Value too large for defined 
data type
the normal message is
   $ cat | split -n20
   split: -: cannot determine file size
i.e. w/o strerror.
Nothing's EOVERFLOW-worthy here, one'd think.

However:
   $ split -n20
   split: -: cannot determine file size
   $ split -n20 /dev/pts/0
   split: /dev/pts/0: cannot determine file size
   $ split -n20 /dev/full
   split: /dev/full: cannot determine file size: Value too large for defined 
data type
   $ split -n20 /dev/zero
   split: /dev/zero: cannot determine file size: Value too large for defined 
data type
   $ split -n20 /dev/rfkill
   split: /dev/rfkill: cannot determine file size
so normal unseekable files get no strerror,
/dev/full and /dev/zero are seekable and somehow yield EOVERFLOWs as well.

Oddly:
   $ split -n20 /dev/autofs
   split: /dev/autofs: cannot determine file size: Invalid argument
but /dev/autofs is seekable, and only EINVALs on read()s.

Also oddly:
   $ split -n20 /dev/null
just works.
Is it hard-coded somehow? This isn't noted in the manual.


This has all changed in coreutils 9.2 with:
https://github.com/coreutils/coreutils/commit/aa266f1b3

That causes data to be read, rather than depending on lseek.

cheers,
Pádraig

Bug#1035596: coreutils: pr: --number-lines= yields "extra characters or invalid number in the argument: ‘SHELL=/bin/bash’: ERANGE"

[Pádraig Brady]

On 06/05/2023 01:19, наб wrote:

Package: coreutils
Version: 8.32-4+b1
Version: 9.1-1
Severity: normal

Dear Maintainer,

:v


Fixed upstream with the attached.

cheers,
Pádraig
From fae65623a92a3f150fd35cfef58dcb3d8533f94c Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?P=C3=A1draig=20Brady?= 
Date: Sat, 6 May 2023 11:22:22 +0100
Subject: [PATCH] pr: fix parsing of empty arguments
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Before:
  $ pr --expand-tabs=
  pr: '-e' extra characters or invalid number in the argument:
   ‘SHELL=/bin/bash’: Value too large for defined data type

After:
  $ pr --expand-tabs=
  pr: '-e': Invalid argument: ‘’

* src/pr.c (getoptarg): Ensure we don't parse beyond the
end of an empty argument, thus outputting arbitrary stack
info in subsequent error messages.

Addresses https://bugs.debian.org/1035596
---
 src/pr.c | 6 ++
 1 file changed, 6 insertions(+)

diff --git a/src/pr.c b/src/pr.c
index 14a368b6c..1c32e8c81 100644
--- a/src/pr.c
+++ b/src/pr.c
@@ -1168,6 +1168,12 @@ getoptnum (char const *n_str, int min, int *num, char const *err)
 static void
 getoptarg (char *arg, char switch_char, char *character, int *number)
 {
+  if (!*arg)
+{
+  error (0, 0, _("'-%c': Invalid argument: %s"), switch_char, quote (arg));
+  usage (EXIT_FAILURE);
+}
+
   if (!ISDIGIT (*arg))
 *character = *arg++;
   if (*arg)
-- 
2.26.2

Bug#1035531: coreutils: fix regressions in copy dir-relative operations

[Pádraig Brady]

Package: coreutils
Version: 9.1-1
Severity: important

There are regressions wrt how some directory-relative operations
performed by cp,mv,install as detailed in the following upstream reports:

 From https://bugs.gnu.org/55910
 From https://bugs.gnu.org/63245

These should be addressed by the attached patch,

Note this has already been applied to Fedora at:
https://src.fedoraproject.org/rpms/coreutils/raw/f38/f/coreutils-9.1-copy-relative-dir.patchdiff -Naur coreutils-9.1.orig/src/copy.c coreutils-9.1/src/copy.c
--- coreutils-9.1.orig/src/copy.c	2022-04-15 13:53:28.0 +
+++ coreutils-9.1/src/copy.c	2023-05-04 21:09:46.290253081 +
@@ -1954,6 +1954,7 @@
   bool restore_dst_mode = false;
   char *earlier_file = NULL;
   char *dst_backup = NULL;
+  char const *drelname = *dst_relname ? dst_relname : ".";
   bool delayed_ok;
   bool copied_as_regular = false;
   bool dest_is_symlink = false;
@@ -1971,7 +1972,7 @@
   if (x->move_mode)
 {
   if (rename_errno < 0)
-rename_errno = (renameatu (AT_FDCWD, src_name, dst_dirfd, dst_relname,
+rename_errno = (renameatu (AT_FDCWD, src_name, dst_dirfd, drelname,
RENAME_NOREPLACE)
 ? errno : 0);
   nonexistent_dst = *rename_succeeded = new_dst = rename_errno == 0;
@@ -1983,7 +1984,7 @@
 {
   char const *name = rename_errno == 0 ? dst_name : src_name;
   int dirfd = rename_errno == 0 ? dst_dirfd : AT_FDCWD;
-  char const *relname = rename_errno == 0 ? dst_relname : src_name;
+  char const *relname = rename_errno == 0 ? drelname : src_name;
   int fstatat_flags
 = x->dereference == DEREF_NEVER ? AT_SYMLINK_NOFOLLOW : 0;
   if (follow_fstatat (dirfd, relname, _sb, fstatat_flags) != 0)
@@ -2051,8 +2052,7 @@
   int fstatat_flags = use_lstat ? AT_SYMLINK_NOFOLLOW : 0;
   if (!use_lstat && nonexistent_dst < 0)
 new_dst = true;
-  else if (follow_fstatat (dst_dirfd, dst_relname, _sb,
-   fstatat_flags)
+  else if (follow_fstatat (dst_dirfd, drelname, _sb, fstatat_flags)
== 0)
 {
   have_dst_lstat = use_lstat;
@@ -2077,7 +2077,7 @@
   bool return_now = false;
 
   if (x->interactive != I_ALWAYS_NO
-  && ! same_file_ok (src_name, _sb, dst_dirfd, dst_relname,
+  && ! same_file_ok (src_name, _sb, dst_dirfd, drelname,
  _sb, x, _now))
 {
   error (0, 0, _("%s and %s are the same file"),
@@ -2140,7 +2140,7 @@
  cp and mv treat -i and -f differently.  */
   if (x->move_mode)
 {
-  if (abandon_move (x, dst_name, dst_dirfd, dst_relname, _sb))
+  if (abandon_move (x, dst_name, dst_dirfd, drelname, _sb))
 {
   /* Pretend the rename succeeded, so the caller (mv)
  doesn't end up removing the source file.  */
@@ -2321,14 +2321,11 @@
  Otherwise, use AT_SYMLINK_NOFOLLOW, in case dst_name is a symlink.  */
   if (have_dst_lstat)
 dst_lstat_sb = _sb;
+  else if (fstatat (dst_dirfd, drelname, _buf, AT_SYMLINK_NOFOLLOW)
+   == 0)
+dst_lstat_sb = _buf;
   else
-{
-  if (fstatat (dst_dirfd, dst_relname, _buf,
-   AT_SYMLINK_NOFOLLOW) == 0)
-dst_lstat_sb = _buf;
-  else
-lstat_ok = false;
-}
+lstat_ok = false;
 
   /* Never copy through a symlink we've just created.  */
   if (lstat_ok
@@ -2475,8 +2472,7 @@
   if (x->move_mode)
 {
   if (rename_errno == EEXIST)
-rename_errno = ((renameat (AT_FDCWD, src_name, dst_dirfd, dst_relname)
- == 0)
+rename_errno = (renameat (AT_FDCWD, src_name, dst_dirfd, drelname) == 0
 ? 0 : errno);
 
   if (rename_errno == 0)
@@ -2576,7 +2572,7 @@
  or not, and this is enforced above.  Therefore we check the src_mode
  and operate on dst_name here as a tighter constraint and also because
  src_mode is readily available here.  */
-  if ((unlinkat (dst_dirfd, dst_relname,
+  if ((unlinkat (dst_dirfd, drelname,
  S_ISDIR (src_mode) ? AT_REMOVEDIR : 0)
!= 0)
   && errno != ENOENT)
@@ -2646,7 +2642,7 @@
  to ask mkdir to copy all the CHMOD_MODE_BITS, letting mkdir
  decide what to do with S_ISUID | S_ISGID | S_ISVTX.  */
   mode_t mode = dst_mode_bits & ~omitted_permissions;
-  if (mkdirat (dst_dirfd, dst_relname, mode) != 0)
+  if (mkdirat (dst_dirfd, drelname, mode) != 0)
 {
   error (0, errno, _("cannot create directory %s"),
  quoteaf (dst_name));
@@ -2657,8 +2653,7 @@
  for writing the directory's contents. Check if these
 

Bug#1035530: coreutils: fix regressions in copy backup operations

[Pádraig Brady]

Package: coreutils
Version: 9.1-1
Severity: important

There are regressions wrt how backups are performed by cp,mv,install
as detailed in the following upstream reports:

From https://bugs.gnu.org/55029
From https://bugs.gnu.org/62607

These should be addressed by the attached patch,

Note this has already been applied to Fedora at:
https://src.fedoraproject.org/rpms/coreutils/raw/f38/f/gnulib-simple-backup-fix.patchdiff -Naur coreutils-9.1.orig/lib/backupfile.c coreutils-9.1/lib/backupfile.c
--- coreutils-9.1.orig/lib/backupfile.c	2022-04-08 11:22:26.0 +
+++ coreutils-9.1/lib/backupfile.c	2023-05-04 17:07:20.784911071 +
@@ -332,7 +332,7 @@
 return s;
 
   DIR *dirp = NULL;
-  int sdir = AT_FDCWD;
+  int sdir = -1;
   idx_t base_max = 0;
   while (true)
 {
@@ -371,10 +371,10 @@
   if (! rename)
 break;
 
-  int olddirfd = sdir < 0 ? dir_fd : sdir;
+  dir_fd = sdir < 0 ? dir_fd : sdir;
   idx_t offset = sdir < 0 ? 0 : base_offset;
   unsigned flags = backup_type == simple_backups ? 0 : RENAME_NOREPLACE;
-  if (renameatu (olddirfd, file + offset, sdir, s + offset, flags) == 0)
+  if (renameatu (dir_fd, file + offset, dir_fd, s + offset, flags) == 0)
 break;
   int e = errno;
   if (! (e == EEXIST && extended))
diff -Naur coreutils-9.1.orig/tests/cp/backup-dir.sh coreutils-9.1/tests/cp/backup-dir.sh
--- coreutils-9.1.orig/tests/cp/backup-dir.sh	2022-04-08 11:22:18.0 +
+++ coreutils-9.1/tests/cp/backup-dir.sh	2023-05-04 17:07:24.851960384 +
@@ -1,5 +1,5 @@
 #!/bin/sh
-# Ensure that cp -b doesn't back up directories.
+# Ensure that cp -b handles directories appropriately
 
 # Copyright (C) 2006-2022 Free Software Foundation, Inc.
 
@@ -29,4 +29,10 @@
 test -d y/x || fail=1
 test -d y/x~ && fail=1
 
+# Bug 62607.
+# This would fail to backup using rename, and thus fail to replace the file
+mkdir -p src/foo dst/foo || framework_failure_
+touch src/foo/bar dst/foo/bar || framework_failure_
+cp --recursive --backup src/* dst || fail=1
+
 Exit $fail
diff -Naur coreutils-9.1.orig/tests/mv/backup-dir.sh coreutils-9.1/tests/mv/backup-dir.sh
--- coreutils-9.1.orig/tests/mv/backup-dir.sh	2022-04-08 11:22:18.0 +
+++ coreutils-9.1/tests/mv/backup-dir.sh	2023-05-04 17:03:29.593098230 +
@@ -36,4 +36,10 @@
 mv -T --backup=numbered C E/ || fail=1
 mv -T --backup=numbered D E/ || fail=1
 
+# Bug#55029
+mkdir F && echo 1 >1 && echo 2 >2 && cp 1 F/X && cp 2 X || framework_failure_
+mv --backup=simple X F/ || fail=1
+compare 1 F/X~ || fail=1
+compare 2 F/X || fail=1
+
 Exit $fail

Bug#1034808: coreutils: pr: -d makes -l $odd enter a busy loop

[Pádraig Brady]

On 24/04/2023 23:05, наб wrote:

Package: coreutils
Version: 8.32-4+b1
Version: 9.1-1
Severity: normal

Dear Maintainer,

Be hold:

-- >8 --
$ time timeout 1 pr -dl2 < /dev/null

real0m0.004s
user0m0.004s
sys 0m0.000s
$ time timeout 1 pr -dl1 < /dev/null

real0m1.003s
user0m1.001s
sys 0m0.002s
-- >8 --

Best,
наб


Nice one.
Fixing this upstream with the attached.

cheers,
Pádraig
From 3fb0cc80fa5e1cede9ec05303a70c26d0d23ca1d Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?P=C3=A1draig=20Brady?= 
Date: Tue, 25 Apr 2023 14:07:03 +0100
Subject: [PATCH] pr: fix infinite loop when double spacing

* src/pr.c (init_parameters): Ensure we avoid a 0 lines_per_body
which was possible when adjusting for double spacing.
That caused print_page() to always return true,
causing an infinite loop.
* tests/pr/pr-tests.pl: Add a test case.
* NEWS: Mention the fix.
Fixes https://bugs.debian.org/1034808
---
 NEWS | 3 +++
 src/pr.c | 2 +-
 tests/pr/pr-tests.pl | 3 +++
 3 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/NEWS b/NEWS
index f6f79ae53..3d34a1b3c 100644
--- a/NEWS
+++ b/NEWS
@@ -14,6 +14,9 @@ GNU coreutils NEWS-*- outline -*-
   factor, numfmt, and tsort now diagnose read errors on the input.
   [This bug was present in "the beginning".]
 
+  'pr --length=1 --double-space' no longer enters an infinite loop.
+  [This bug was present in "the beginning".]
+
 ** Changes in behavior
 
   'cp -v' and 'mv -v' will no longer output a message for each file skipped
diff --git a/src/pr.c b/src/pr.c
index 2c5cdceb1..14a368b6c 100644
--- a/src/pr.c
+++ b/src/pr.c
@@ -1209,7 +1209,7 @@ init_parameters (int number_of_files)
 lines_per_body = lines_per_page;
 
   if (double_space)
-lines_per_body = lines_per_body / 2;
+lines_per_body = MAX (1, lines_per_body / 2);
 
   /* If input is stdin, cannot print parallel files.  BSD dumps core
  on this. */
diff --git a/tests/pr/pr-tests.pl b/tests/pr/pr-tests.pl
index 265e6e108..eafc13d81 100755
--- a/tests/pr/pr-tests.pl
+++ b/tests/pr/pr-tests.pl
@@ -415,6 +415,9 @@ my @tv = (
 ['padding2', '-t -n,64', "1\n", (" "x 63)."1,1\n", 0],
 # Ensure we handle buffer truncation correctly
 ['padding3', '-t -N100 -n,1', "1\n", "0,1\n", 0],
+
+# This entered an infinite loop before coreutils-9.4
+['page-length1', '-dl1', "", "", 0],
 );
 
 # Convert the above old-style test vectors to the newer
-- 
2.26.2

Bug#1034429: coreutils: install: -s runs ["strip", $f] instead of ["strip", "--", $f]; should use strip from $STRIP

[Pádraig Brady]

On 15/04/2023 03:13, наб wrote:

Package: coreutils
Version: 8.32-4+b1
Version: 9.1-1
Severity: normal

Dear Maintainer,

-- >8 --
$ install -vs /bin/install -- q
removed 'q/install'
'/bin/install' -> 'q/install'
$ install -vs /bin/install -- -v
'/bin/install' -> '-v'
Usage: strip  in-file(s)


Proposed upstream patch:
https://lists.gnu.org/archive/html/coreutils/2023-04/msg00037.html

cheers,
Pádraig

Bug#1034429: coreutils: install: -s runs ["strip", $f] instead of ["strip", "--", $f]; should use strip from $STRIP

[Pádraig Brady]

On 15/04/2023 15:47, наб wrote:

On Sat, Apr 15, 2023 at 02:04:18PM +0100, Pádraig Brady wrote:

But perhaps the strip program doesn't handle the convention
that -- indicates end of option processing?

Inasmuch as "perhaps strip doesn't use getopt(3)" /and/
 "perhaps strip doesn't conform to the XCU"
(https://pubs.opengroup.org/onlinepubs/9699919799/utilities/strip.html;
  note no changes since Issue 2, which means this has been a requirement
  realistically since 1989
  (it's in my copy of XSI Issue 3, saying it's equivalent to Issue 2,
   but it's only Volume 1, and it's unclear to me if the XBD USG
   was fully developed; Issue 2 hasn't been archived),
  and definitely since SUSv1, whose Guideline 10 and Utility Description
  Defaults, OPTIONS, Default Behaviour are both as present-day)
can both hold at the same time, I guess?
You can probably tell I'm quite sceptical any such strip exists.


I was thinking of the more general case where the strip program
may be a shell wrapper or something that doesn't handle -- appropriately.
The main point is that we should fix this issue in any case.


It would also be nice for $STRIP to provide a default if -S isn't set;
the usefulness of install -s is doubtful if
install -s binary $DESTDIR/bin
only works if binary happens to match the host arch.

Yes it's a fair point that calling strip without options is quite restrictive.
Though I suppose --strip-program can point to a script that calls strip 
appropriately.

My target usecase is for third-party programs running install -s,
which, without modification, cannot be made to work if strip is binutils
strip and the binary is non-native.


Well one could adjust the $PATH so an appropriate strip wrapper is found.

cheers,
Pádraig

Bug#1034602: coreutils: nl: resets line numbers on each new section instead of new page, POSIX and SysV violation

[Pádraig Brady]

On 19/04/2023 13:07, наб wrote:

Package: coreutils
Version: 9.1-1
Severity: normal

Dear Maintainer,

POSIX very clearly says (Issue 8 Draft 2.1, same as Issue 7):
100982  The nl utility views the text it reads in terms of logical pages. Line 
numbering shall be reset at
100983  the start of each logical page. A logical page consists of a header, a 
body, and a footer section.
100984  Empty sections are valid. Different line numbering options are 
independently available for
100985  header, body, and footer (for example, no numbering of header and 
footer lines while
100986  numbering blank lines only in the body).

100987 The starts of logical page sections shall be signaled by input lines 
containing nothing but the
100988 following delimiter characters:
100989   Line Start of
100990   \:\:\:   Header
100991   \:\: Body
100992   \:   Footer
100993 Unless otherwise specified, nl shall assume the text being read is in a 
single logical page body.

101021  −p  Specify that numbering should not be restarted at logical page 
delimiters.


Indeed, both illumos-gate nl and SysVr4 nl behave like this:
-- >8 --
$ printf '%s\n' a '\:\:\:' b '\:\:' c '\:' d '\:\:\:' e '\:\:\:' f | ./a.out 
-ht -ft
  1  a

  1  b

  2  c

  3  d

  1  e

  1  f
$ printf '%s\n' a '\:\:\:' b '\:\:' c '\:' d '\:\:\:' e '\:\:\:' f | ./a.out 
-ht -ft -p
  1  a

  2  b

  3  c

  4  d

  5  e

  6  f
-- >8 --

But coreutils nl resets the line number at the start of each section?
-- >8 --
$ printf '%s\n' a '\:\:\:' b '\:\:' c '\:' d '\:\:\:' e '\:\:\:' f | nl -ht -ft
  1  a

  1  b

  1  c

  1  d

  1  e

  1  f
$ printf '%s\n' a '\:\:\:' b '\:\:' c '\:' d '\:\:\:' e '\:\:\:' f | nl -ht -ft 
-p
  1  a

  2  b

  3  c

  4  d

  5  e

  6  f
-- >8 --


This was changed 7 years ago with:
https://lists.gnu.org/archive/html/coreutils/2016-05/msg0.html

Now re-reading POSIX I agree with your interpretation.
I.e. reading the -p description the current coreutils behavior seems correct,
while also adding the context from the start of the POSIX doc,
your interpretation seems correct.

Note FreeBSD follows the current coreutils behavior by resetting the count at 
each section,
so it is a bit ambiguous as to the correct operation.

cheers,
Pádraig

Bug#1034429: coreutils: install: -s runs ["strip", $f] instead of ["strip", "--", $f]; should use strip from $STRIP

[Pádraig Brady]

On 15/04/2023 03:13, наб wrote:

Package: coreutils
Version: 8.32-4+b1
Version: 9.1-1
Severity: normal

Dear Maintainer,

-- >8 --
$ install -vs /bin/install -- q
removed 'q/install'
'/bin/install' -> 'q/install'
$ install -vs /bin/install -- -v
'/bin/install' -> '-v'
Usage: strip  in-file(s)



install: strip process terminated abnormally
-- >8 --

Probably don't do that.


Fair point.
But perhaps the strip program doesn't handle the convention
that -- indicates end of option processing?
Perhaps a more portable workaround would be to prepend "./"
if the name starts with "-".


It would also be nice for $STRIP to provide a default if -S isn't set;
the usefulness of install -s is doubtful if
   install -s binary $DESTDIR/bin
only works if binary happens to match the host arch.


Yes it's a fair point that calling strip without options is quite restrictive.
Though I suppose --strip-program can point to a script that calls strip 
appropriately.
There are some security issues with env vars that impact what is executed.
I do see that FreeBSD install(1) does support $STRIPBIN, though they
don't support --strip-program, so functionality is equivalent from that point 
of view.

cheers,
Pádraig

Bug#1034416: coreutils: stat.1: are there any shells which actually provide stat as a built-in?

[Pádraig Brady]

On 14/04/2023 18:25, наб wrote:

Package: coreutils
Version: 9.1-1
Severity: minor

Dear Maintainer,

Quoth stat(1):
-- >8 --
NOTE: your shell may have its own version of stat, which usually 
supersedes the version described here.  Please
refer to your shell's documentation for details about the options it 
supports.
-- >8 --

An apt list | grep sh/unstable filtered to "is a shell" gives me
-- >8 --
bash/unstable,now 5.2.15-2+b1 x32 [installed]
csh/unstable 20110502-7 x32
dash/unstable,now 0.5.12-2 x32 [installed]
fish/unstable 3.6.0-3 x32
mksh/unstable 59c-27 x32
posh/unstable 0.14.1 x32
rush/unstable 2.3-1 x32
sash/unstable 3.8-5+b9 x32
tcsh/unstable 6.24.07-1 x32
yash/unstable 2.52-2 x32
zsh/unstable 5.9-4 x32
-- >8 --

and
$ for sh in bash dash fish mksh posh sash tcsh yash zsh; do echo $sh; $sh -c 
'type stat'; done
bash
stat is /bin/stat
dash
stat is /bin/stat
fish
stat is /bin/stat
mksh
stat is a tracked alias for /bin/stat
yash
stat: an external command at /bin/stat
zsh
stat is /bin/stat

csh and tcsh
Don't have type, strace says "stat ." execs stat.

posh
No type, no stat builtin.

sash
No type; has a lot of stuff built in, not stat tho.

rush
idk; source doesn't have "stat" anywhere so probably no


I thought this was either ancient bash or ksh tech,
but it's not in 1.14.4-2 or as far back as the illumos-gate ksh goes.

Probably axe it?


Maybe.
The warning was added in 2005 without mentioning specific shells:
https://github.com/coreutils/coreutils/commit/d209b0f75

I see zsh has a stat module:
https://zsh.sourceforge.io/Doc/Release/Zsh-Modules.html#The-zsh_002fstat-Module

I also see a bash stat loadable:
http://git.savannah.gnu.org/gitweb/?p=bash.git;a=blob;f=examples/loadables/stat.c;hb=HEAD

Though if we're considering loadables,
then that would cover most commands.

It might be best to remove this warning indeed.

cheers,
Pádraig

Bug#1034421: coreutils: install(1): says -v only notes directory creation, but also writes regular file creation

[Pádraig Brady]

On 14/04/2023 20:23, наб wrote:

Package: coreutils
Version: 8.32-4+b1
Version: 9.1-1
Severity: minor

Dear Maintainer,

Quoth install(1):
-- >8 --
-v, --verbose
 print the name of each directory as it is created
-- >8 --

Compare:
-- >8 --
$ install -v $(command -v install) .
'/bin/install' -> './install'


Yes I already fixed that upstream actually as part of:
https://github.com/coreutils/coreutils/commit/d899f9e33

  $ src/ginstall --help | grep -- --verbose
  -v, --verbose   print the name of each created file or directory

cheers,
Pádraig

Bug#1027100: coreutils: wc: total overflows unchecked

[Pádraig Brady]

On 28/12/2022 13:38, Pádraig Brady wrote:

On 27/12/2022 19:31, наб wrote:

Package: coreutils
Version: 8.32-4+b1
Severity: normal

Dear Maintainer,

-- >8 --
$ truncate -s 2E a
$ wc -c a a a a a a a | uniq -c
 7  2305843009213693952 a
 1 16140901064495857664 total
$ wc -c a a a a a a a a | uniq -c
 8 2305843009213693952 a
 1   0 total
$ wc -c a a a a a a a a a | uniq -c
 9 2305843009213693952 a
 1 2305843009213693952 total
$ wc -c a a a a a a a a a a | uniq -c
10 2305843009213693952 a
 1 4611686018427387904 total
-- >8 --

Which is obviously wrong. One of the sensible solutions would be to just
saturate the totals


Proposed solution for upstream 9.3 is at:
https://lists.gnu.org/archive/html/coreutils/2023-03/msg00091.html

Bug#1027413: coreutils: shuf: -i forbids 2^64-1/2^32-1, max value is ...-2?

[Pádraig Brady]

On 31/12/2022 02:02, наб wrote:

Package: coreutils
Version: 9.1-1
Severity: normal

Dear Maintainer,

We all know that for a 4-byte number the maximum value is 2^32-1,
and for an 8-byte one ‒ 2^64-1.

Even if we accept that for some bizzarre reason shuf
wants to be limited to 4-byte numbers,
then it should at least actually allow all of them.

Why, then:
-- >8 --
$ bc -q
2^32
4294967296
$ shuf -n1 -i 0-4294967296
shuf: invalid input range: ‘4294967296’: Value too large for defined data type
$ shuf -n1 -i 0-4294967295
shuf: invalid input range: ‘0-4294967295’
$ shuf -n1 -i 0-4294967294
2310362399
-- >8 --
and
-- >8 --
$ bc
2^64
18446744073709551616
$ shuf -n1 -i 0-18446744073709551616
shuf: invalid input range: ‘18446744073709551616’: Value too large for defined 
data type
$ shuf -n1 -i 0-18446744073709551615
shuf: invalid input range: ‘0-18446744073709551615’
$ shuf -n1 -i 0-18446744073709551614
16168582199677063639
-- >8 --

I mean, to hell with the highlighted bit of the range being different,
or there being no error string, apparently, right,
but why can't you just generate an integer?
Why is the operational range apparently one smaller than [0, 0x]?

This is disproportionately worse on ILP32 platforms given #1027412;
if this were just one off the end of 2^64, eh, who cares.


We shrink the range by one to use as an internal flag.
Since the range is already limited, this isn't seen as a practical issue.
I.e. the same more detailed response in bug #1027412 applies here also.

cheers,
Pádraig

Bug#1027412: coreutils: shuf: -i EOVERFLOW when speccing either side of the range >2^32 (ILP32 only: limited to size_t?!?)

[Pádraig Brady]

On 31/12/2022 01:51, наб wrote:

Package: coreutils
Version: 9.1-1
Severity: normal

Dear Maintainer,

On x32 and i386 I observe:
-- >8 --
$ shuf -i 9223372036854775800-9223372036854775808
shuf: invalid input range: ‘9223372036854775800’: Value too large for defined 
data type
-- >8 --

But cannot reproduce this on amd64:
-- >8 --
$ shuf -i 9223372036854775800-9223372036854775808
9223372036854775803
9223372036854775805
9223372036854775800
9223372036854775808
9223372036854775802
9223372036854775806
9223372036854775807
9223372036854775801
9223372036854775804
-- >8 --

My brother in christ, just define the data type to be bigger.

The manual says nothing of this.

The info manual does mention the range is as per "unsigned integers".
Since this is limited for all platforms, there isn't much advantage
in supporting the 64 bit range on 32 bit systems, given the current 
implementation.
Requiring large ranges would be very unusual. Note we also don't support 
negative
numbers which is another possibly more problematic limitation.
Assuming a reasonable amount of numbers are required, one can
easily / generally shift output to the desired range without limits
as described at https://bugs.gnu.org/61267

cheers,
Pádraig

Bug#1029103: coreutils: csplit: -z misdocumented

[Pádraig Brady]

Fair point.
I'll clarify like:

commit 70dc90c579adc972644c9047543ff1dc611b89cc (HEAD -> master)
Author: Pádraig Brady 
Date:   Tue Jan 17 21:31:31 2023 +

doc: csplit: more accurate --elide-empty-files help

* src/csplit.c (usage): Use "suppress" rather than "remove"
when describing -z so it's more apparent that the effect
is a particular numbered file is not created, rather than
being removed later.  I.e., don't suggest -z may induce
gaps in file numbering.

diff --git a/src/csplit.c b/src/csplit.c
index da4b36a7c..9a31697b6 100644
--- a/src/csplit.c
+++ b/src/csplit.c
@@ -1471,7 +1471,7 @@ Read standard input if FILE is -\n\
   fputs (_("\
   -n, --digits=DIGITSuse specified number of digits instead of 2\n\
   -s, --quiet, --silent  do not print counts of output file sizes\n\
-  -z, --elide-empty-filesremove empty output files\n\
+  -z, --elide-empty-filessuppress empty output files\n\
 "), stdout);
   fputs (HELP_OPTION_DESCRIPTION, stdout);
   fputs (VERSION_OPTION_DESCRIPTION, stdout);

Bug#1027442: coreutils: stty: soft-wrapping broken, wraps to cols+1

[Pádraig Brady]

On 31/12/2022 15:12, наб wrote:

Package: coreutils
Version: 8.30-3
Severity: normal

Dear Maintainer,

On all three versions, at my usual teletype size, I get:



$ COLUMNS=172 stty -a | wc -L
173


good catch!

This off by one bug was introduced in v5.3 (2005)

I'll apply the following upstream:

index b4c2cbecd..f3c7915e1 100644
--- a/src/stty.c
+++ b/src/stty.c
@@ -519,7 +519,7 @@ wrapf (char const *message,...)

   if (0 < current_col)
 {
-  if (max_col - current_col < buflen)
+  if (max_col - current_col <= buflen)
 {
   putchar ('\n');
   current_col = 0;

thanks,
Pádraig

Bug#1027101: coreutils: wc: -c no longer optimises to st_size for regular files

[Pádraig Brady]

On 28/12/2022 13:49, наб wrote:

Control: retitle -1 coreutils: wc: -c st_size optimisation broken for files 
larger than max(size_t)

Hi!

On Wed, Dec 28, 2022 at 01:31:12PM +, Pádraig Brady wrote:

On 27/12/2022 19:38, наб wrote:

To repro this, run truncate -s 2E a, then wc -c a;
this completes instantly on bullseye and spins in a read(2) loop on sid.

I think I see the issue but only on 32 bit size_t.
Ah this is x32, as per your report.


The patch doesn't apply to the Debian source unfortunately, but
I can confirm this behaves as you'd expect for a [iu]64->u32 truncation:
-- >8 --
$ truncate -s $(echo 2^32-1 | bc) a
$ time wc -c a
4294967295 a

real0m0.002s
user0m0.002s
sys 0m0.000s
$ truncate -s $(echo 2^32 | bc) a
$ time wc -c a
4294967296 a

real0m0.718s
user0m0.092s
sys 0m0.626s
-- >8 --

And I can repro this exact behaviour on i386.


Proposed upstream patch attached.

thanks,
Pádraig
From be259a269bee563e4670c5936efb9031835525ef Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?P=C3=A1draig=20Brady?= 
Date: Wed, 28 Dec 2022 14:04:19 +
Subject: [PATCH] wc: fix regression determining file size

* src/wc.c (wc): Use off_t rather than size_t
when calculating where to seek to, so that
we don't seek to a too low offset on systems
where size_t < off_t, which would result in
many read() calls to determine the file size.
* tests/misc/wc-proc.sh: Add a test case
sufficient for 32 bit systems at least.
Reported at https://bugs.debian.org/1027101
---
 src/wc.c  | 2 +-
 tests/misc/wc-proc.sh | 7 +++
 2 files changed, 8 insertions(+), 1 deletion(-)

diff --git a/src/wc.c b/src/wc.c
index bc52a8c0e..df9770396 100644
--- a/src/wc.c
+++ b/src/wc.c
@@ -431,7 +431,7 @@ wc (int fd, char const *file_x, struct fstatus *fstatus, off_t current_pos)
   if (! fstatus->failed && usable_st_size (>st)
   && 0 <= fstatus->st.st_size)
 {
-  size_t end_pos = fstatus->st.st_size;
+  off_t end_pos = fstatus->st.st_size;
   if (current_pos < 0)
 current_pos = lseek (fd, 0, SEEK_CUR);
 
diff --git a/tests/misc/wc-proc.sh b/tests/misc/wc-proc.sh
index 581890ddd..030872a91 100755
--- a/tests/misc/wc-proc.sh
+++ b/tests/misc/wc-proc.sh
@@ -42,4 +42,11 @@ cat <<\EOF > exp
 EOF
 compare exp out || fail=1
 
+# Ensure we don't read too much when reading,
+# as was the case on 32 bit systems
+# from coreutils-8.24 to coreutils-9.1
+if timeout 10 truncate -s1T do_read; then
+  timeout 10 wc -c do_read || fail=1
+fi
+
 Exit $fail
-- 
2.26.2

Bug#1027100: coreutils: wc: total overflows unchecked

[Pádraig Brady]

On 27/12/2022 19:31, наб wrote:

Package: coreutils
Version: 8.32-4+b1
Severity: normal

Dear Maintainer,

-- >8 --
$ truncate -s 2E a
$ wc -c a a a a a a a | uniq -c
7  2305843009213693952 a
1 16140901064495857664 total
$ wc -c a a a a a a a a | uniq -c
8 2305843009213693952 a
1   0 total
$ wc -c a a a a a a a a a | uniq -c
9 2305843009213693952 a
1 2305843009213693952 total
$ wc -c a a a a a a a a a a | uniq -c
   10 2305843009213693952 a
1 4611686018427387904 total
-- >8 --

Which is obviously wrong. One of the sensible solutions would be to just
saturate the totals, which would yield:
-- >8 --
$ wc -c a a a a a a a | uniq -c
   7  2305843009213693952 a
   1 16140901064495857664 total
$ wc -c a a a a a a a a | uniq -c
   8 2305843009213693952 a
   1 18446744073709551615 total
$ wc -c a a a a a a a a a | uniq -c
   9 2305843009213693952 a
   1 18446744073709551615 total
$ wc -c a a a a a a a a a a | uniq -c
  10 2305843009213693952 a
   1 18446744073709551615 total
-- >8 --

Which is just about the best solution here,
since this is hardly a common occurrence.


It's a good point.
We might just output ERR or NAN or something
in place of a total value in the overflow case.
We should also do the same in df --total

cheers,
Pádraig

Bug#1027101: coreutils: wc: -c no longer optimises to st_size for regular files

[Pádraig Brady]

On 27/12/2022 19:38, наб wrote:

Package: coreutils
Version: 9.1-1
Severity: normal

Dear Maintainer,

This is a regression against 8.32-4+b1 (bullseye).

To repro this, run truncate -s 2E a, then wc -c a;
this completes instantly on bullseye and spins in a read(2) loop on sid.

Found as part of #1027100.


I think I see the issue but only on 32 bit size_t.
Ah this is x32, as per your report.
Can you change the end_pos type to off_t like:


diff --git a/src/wc.c b/src/wc.c

index bc52a8c0e..df9770396 100644
--- a/src/wc.c
+++ b/src/wc.c
@@ -431,7 +431,7 @@ wc (int fd, char const *file_x, struct fstatus *fstatus, 
off_t current_pos)
   if (! fstatus->failed && usable_st_size (>st)
   && 0 <= fstatus->st.st_size)
 {
-  size_t end_pos = fstatus->st.st_size;
+  off_t end_pos = fstatus->st.st_size;
   if (current_pos < 0)
 current_pos = lseek (fd, 0, SEEK_CUR);

cheers,
Pádraig

Bug#1022857: [PATCH] printf: with \U syntax, support all valid unicode points

[Pádraig Brady]

On 27/10/2022 21:46, Bernhard Voelker wrote:

On 10/27/22 16:41, Pádraig Brady wrote:

Previously this was restricted to the C standard subset
which restricted most values <= 0x9F, as that simplifies the C lexer.
However printf(1) doesn't need this restriction.
Note also the bash builtin printf already supports all values <= 0x9F.


Nice one.

BTW: do we also want to support the new modifier that bash has since version 
5.2?

r. The `printf' builtin has a new format specifier: %Q. This acts like %q 
but
  applies any specified precision to the original unquoted argument, then
  quotes and outputs the result.


That does sound useful.
However we yet don't support qualifiers on %q
which would need to come first.

  $ printf '%.4q\n' 'sp ace'  # builtin
  sp\

  $ env printf '%.4q\n' 'sp ace'
  printf: %.4q: invalid conversion specification

cheers,
Pádraig

Bug#1022857: [PATCH] printf: with \U syntax, support all valid unicode points

[Pádraig Brady]

Previously this was restricted to the C standard subset
which restricted most values <= 0x9F, as that simplifies the C lexer.
However printf(1) doesn't need this restriction.
Note also the bash builtin printf already supports all values <= 0x9F.

* src/printf.c (main): Relax the restriction on points <= 0x9F.
* doc/coreutils.texi (printf invocation): Adjust description.
* tests/misc/printf-cov.pl: Adjust accordingly.  Add new cases.
* NEWS: Mention the change in behavior.
Reported at https://bugs.debian.org/1022857
---
 NEWS |  4 
 doc/coreutils.texi   |  5 ++---
 src/printf.c | 11 +++
 tests/misc/printf-cov.pl | 11 ---
 4 files changed, 17 insertions(+), 14 deletions(-)

diff --git a/NEWS b/NEWS
index 31b04fe9f..b6b5201e7 100644
--- a/NEWS
+++ b/NEWS
@@ -41,6 +41,10 @@ GNU coreutils NEWS-*- 
outline -*-
   reverting to the behavior in coreutils-9.0 and earlier.
   This behavior is now documented.
 
+  printf unicode \u, \U syntax, now supports all valid
+  unicode code points.  Previously is was restricted to the C
+  universal character subset, which restricted most points <= 0x9F.
+
   runcon now exits with status 125 for internal errors.  Previously upon
   internal errors it would exit with status 1, which was less distinguishable
   from errors from the invoked command.
diff --git a/doc/coreutils.texi b/doc/coreutils.texi
index 8ce23898b..764a7c897 100644
--- a/doc/coreutils.texi
+++ b/doc/coreutils.texi
@@ -13216,9 +13216,8 @@ characters, specified as
 four hexadecimal digits @var{}, and @samp{\U} for 32-bit Unicode
 characters, specified as eight hexadecimal digits @var{}.
 @command{printf} outputs the Unicode characters
-according to the @env{LC_CTYPE} locale.  Unicode characters in the ranges
-U+@dots{}U+009F, U+D800@dots{}U+DFFF cannot be specified by this syntax,
-except for U+0024 ($), U+0040 (@@), and U+0060 (@`).
+according to the @env{LC_CTYPE} locale.  Unicode characters in the range
+U+D800@dots{}U+DFFF cannot be specified by this syntax.
 
 The processing of @samp{\u} and @samp{\U} requires a full-featured
 @code{iconv} facility.  It is activated on systems with glibc 2.2 (or newer),
diff --git a/src/printf.c b/src/printf.c
index 68c388341..5d420cef2 100644
--- a/src/printf.c
+++ b/src/printf.c
@@ -298,14 +298,9 @@ print_esc (char const *escstart, bool octal_0)
   uni_value = uni_value * 16 + hextobin (*p);
 }
 
-  /* A universal character name shall not specify a character short
- identifier in the range  through 0020, 007F through
- 009F, or D800 through DFFF inclusive. A universal
- character name shall not designate a character in the required
- character set.  */
-  if ((uni_value <= 0x9f
-   && uni_value != 0x24 && uni_value != 0x40 && uni_value != 0x60)
-  || (uni_value >= 0xd800 && uni_value <= 0xdfff))
+  /* Error for invalid code points D800 through DFFF inclusive.
+ Note print_unicode_char() would print the literal \u.. in this case. 
*/
+  if (uni_value >= 0xd800 && uni_value <= 0xdfff)
 die (EXIT_FAILURE, 0, _("invalid universal character name \\%c%0*x"),
  esc_char, (esc_char == 'u' ? 4 : 8), uni_value);
 
diff --git a/tests/misc/printf-cov.pl b/tests/misc/printf-cov.pl
index ab6bd159b..59c85cb66 100755
--- a/tests/misc/printf-cov.pl
+++ b/tests/misc/printf-cov.pl
@@ -66,9 +66,14 @@ my @Tests =
   ['esc', q('\xaa\0377'),  {OUT=>"\xaa\0377"}],
   ['esc-bad-hex', q('\x'), {EXIT=>1},
 {ERR=>"$prog: missing hexadecimal number in escape\n"}],
-  # ['u4', q('\u09ac'), {OUT=>"\xe0a6ac"}],
-  ['u-invalid', q('\u'), {EXIT=>1},
-{ERR=>"$prog: invalid universal character name \\u\n"}],
+  ['u-bad-hex', q('\u00'), {EXIT=>1},
+{ERR=>"$prog: missing hexadecimal number in escape\n"}],
+  ['U-bad-hex', q('\U'), {EXIT=>1},
+{ERR=>"$prog: missing hexadecimal number in escape\n"}],
+  ['u4', q('\u0030'), {OUT=>"0"}],
+  ['U8', q('\U0030'), {OUT=>"0"}],
+  ['u-invalid', q('\ud800'), {EXIT=>1},
+{ERR=>"$prog: invalid universal character name \\ud800\n"}],
   ['u-missing', q('\u'), {EXIT=>1},
 {ERR=>"$prog: missing hexadecimal number in escape\n"}],
   ['d-invalid', '%d no-num', {OUT=>'0'}, {EXIT=>1},
-- 
2.26.2

Bug#1022857: coreutils: printf refuses ~all unicode values smaller than 0x7F for \u and \U?

[Pádraig Brady]

On 27/10/2022 01:23, наб wrote:

Package: coreutils
Version: 8.32-4+b1
Severity: normal

Dear Maintainer,

On both of the above versions, on both LC_ALL=en_GB.UTF-8 and
LC_ALL=C.UTF-8, I get:
-- >8 --
$ for i in $(seq 0 255); do /bin/printf '\U'$(printf '%08X' $i)'\n'; done  2>&1 
| paste - - - -
/bin/printf: invalid universal character name \U/bin/printf: 
invalid universal character name \U0001


Right. This is an explicit restriction in printf(1)
which comes from the C standard.  That's so that
C compilers that don't support universal characters
don't have to consider them in other contexts.

Now printf(1) is not C, so we should lift this restriction I think.

I'll do that upstream now.

thanks,
Pádraig

Bug#1022803: base64: URL-safe alphabet

[Pádraig Brady]

On 26/10/2022 11:22, Jakub Wilk wrote:

Package: coreutils
Version: 9.1-1
Severity: wishlist

RFC 4648 §5  defines an
alternative URL-safe alphabet (with "-" and "_" instead of "+" and "/").
Please add an option to base64(1) to use this alphabet.


Already supported with `basenc --base64url`

cheers,
Pádraig

Bug#1019468: coreutils: stty: changing ispeed/ospeed says "unable to perform all requested operations" but the change sticks

[Pádraig Brady]

On 10/09/2022 03:03, наб wrote:

Package: coreutils
Version: 8.32-4+b1
Severity: normal

Dear Maintainer,

What's happening here? strace confirms that the TCSETSW and TCGETS
values are identical in every case:
```
$ stty 134
$ stty 100

$ stty 134
$ stty ispeed 100
stty: 'standard input': unable to perform all requested operations
$ stty ispeed 100

$ stty 134 ispeed 100
stty: 'standard input': unable to perform all requested operations
$ stty 134 ispeed 100
stty: 'standard input': unable to perform all requested operations
```


Fixed upstream with:

https://git.sv.gnu.org/gitweb/?p=coreutils.git;a=commitdiff;h=760998a78
Which avoids warnings from:
  $ stty ispeed 100

https://git.sv.gnu.org/gitweb/?p=coreutils.git;a=commitdiff;h=39f71795d
Which gives explicit warning for:
  $ stty 134 ispeed 100
  stty: asymmetric input (9600), output (134) speeds not supported

cheers,
Pádraig

Bug#1018803: coreutils: stty: drain/-drain doesn't interact with -g/-a/default output?

[Pádraig Brady]

On 31/08/2022 02:31, наб wrote:

Package: coreutils
Version: 8.32-4+b1
Severity: normal

Dear Maintainer,

It appears drain/-drain is special, in that, unlike all operands,
it does /not/ prevent default output and it does /not/ exclude -a/-g:
-- >8 --
$ stty -drain
speed 38400 baud; line = 0;
-brkint -imaxbel
$ stty -drain -g
500:5:bf:8a3b:3:1c:7f:15:4:0:1:0:11:13:1a:0:12:f:17:16:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0:0
$ stty -drain -a
speed 38400 baud; rows 54; columns 172; line = 0;
intr = ^C; quit = ^\; erase = ^?; kill = ^U; eof = ^D; eol = ; eol2 = ; 
swtch = ; start = ^Q; stop = ^S; susp = ^Z; rprnt = ^R; werase = ^W; lnext = ^V;
discard = ^O; min = 1; time = 0;
-parenb -parodd -cmspar cs8 -hupcl -cstopb cread -clocal -crtscts
-ignbrk -brkint -ignpar -parmrk -inpck -istrip -inlcr -igncr icrnl ixon -ixoff 
-iuclc -ixany -imaxbel -iutf8
opost -olcuc -ocrnl onlcr -onocr -onlret -ofill -ofdel nl0 cr0 tab0 bs0 vt0 ff0
isig icanon iexten echo echoe echok -echonl -noflsh -xcase -tostop -echoprt 
echoctl echoke -flusho -extproc
-- >8 --

This is undocumented, and, indeed, directly against the usage string:
   Usage: stty [-F DEVICE | --file=DEVICE] [SETTING]...
 or:  stty [-F DEVICE | --file=DEVICE] [-a|--all]
 or:  stty [-F DEVICE | --file=DEVICE] [-g|--save]

I assume this is an oversight.


Specifying [-]drain in isolation is an edge case,
but yes drain is treated specially to act like an option.

We originally considered a -I option to control this,
but changed to [-]drain for symmetry reasons,
to allow users to enable / disable explicitly.

I'll clarify this upstream with:

diff --git a/doc/coreutils.texi b/doc/coreutils.texi
index de819b6dc..2af761654 100644
--- a/doc/coreutils.texi
+++ b/doc/coreutils.texi
@@ -15518,6 +15518,8 @@ Tell the kernel that the terminal has @var{n} columns.  
Non-POSIX.
 @cindex nonblocking @command{stty} setting
 Apply settings after first waiting for pending output to be transmitted.
 This is enabled by default for GNU @command{stty}.
+Note this is treated as an option rather than a line setting,
+and will follow the option processing rules described in the summary above.
 It is useful to disable this option
 in cases where the system may be in a state where serial transmission
 is not possible.

cheers,
Pádraig.

Bug#1018790: coreutils: stty: "ispeed anything" is valid and doesn't seem to do anything

[Pádraig Brady]

On 30/08/2022 21:40, наб wrote:

Package: coreutils
Version: 8.32-4+b1
Severity: normal

Dear Maintainer,

stty ispeed and ospeed don't seem to check that the speed is valid:
-- >8 --
$ LD_PRELOAD=$PWD/dumpispeed.so stty ispeed 420
cfsetispeed(0xPOINTER, 4294967295)=-1

$ LD_PRELOAD=$PWD/dumpispeed.so stty ispeed 57600
cfsetispeed(0xPOINTER, 4097)=0
-- >8 --

Compare bare speeds:
-- >8 --
$ stty 420
stty: invalid argument 420
Try 'stty --help' for more information.

$ stty 57600
-- >8 --


Good catch.
We'll fix this upstream with the attached.

thanks,
Pádraig
From 544cead8456cf788b47ed13cf17b55bc996b5791 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?P=C3=A1draig=20Brady?= 
Date: Wed, 31 Aug 2022 00:17:21 +0100
Subject: [PATCH] stty: valid ispeed and ospeed arguments

* src/stty.c (apply_settings): Validate [io]speed arguments
against the internal accepted set.
(set_speed): Check the cfset[io]speed() return value so
that we validate against the system supported set.
* tests/misc/stty-invalid.sh: Add a test case.
* NEWS: Mention the bug fix.
Reported in https://bugs.debian.org/1018790
---
 NEWS   |  5 +
 src/stty.c | 29 +
 tests/misc/stty-invalid.sh |  3 +++
 3 files changed, 33 insertions(+), 4 deletions(-)

diff --git a/NEWS b/NEWS
index ab1a2ef91..db5150824 100644
--- a/NEWS
+++ b/NEWS
@@ -26,6 +26,11 @@ GNU coreutils NEWS-*- outline -*-
   long been documented to be platform-dependent.
   [bug introduced 1999-05-02 and only partly fixed in coreutils-8.14]
 
+  stty ispeed and ospeed options no longer accept and silently ignore
+  invalid speed arguments.  Now they're validated against both the
+  general accepted set, and the system supported set of valid speeds.
+  [This bug was present in "the beginning".]
+
 ** Changes in behavior
 
   'cp --reflink=always A B' no longer leaves behind a newly created
diff --git a/src/stty.c b/src/stty.c
index 3b6a592a9..3d515223e 100644
--- a/src/stty.c
+++ b/src/stty.c
@@ -1159,6 +1159,11 @@ apply_settings (bool checking, char const *device_name,
 {
   check_argument (arg);
   ++k;
+  if (string_to_baud (settings[k]) == (speed_t) -1)
+{
+  error (0, 0, _("invalid ispeed %s"), quote (settings[k]));
+  usage (EXIT_FAILURE);
+}
   if (checking)
 continue;
   set_speed (input_speed, settings[k], mode);
@@ -1169,6 +1174,11 @@ apply_settings (bool checking, char const *device_name,
 {
   check_argument (arg);
   ++k;
+  if (string_to_baud (settings[k]) == (speed_t) -1)
+{
+  error (0, 0, _("invalid ospeed %s"), quote (settings[k]));
+  usage (EXIT_FAILURE);
+}
   if (checking)
 continue;
   set_speed (output_speed, settings[k], mode);
@@ -1696,13 +1706,24 @@ set_control_char (struct control_info const *info, char const *arg,
 static void
 set_speed (enum speed_setting type, char const *arg, struct termios *mode)
 {
-  speed_t baud;
+  /* Note cfset[io]speed(), do not check with the device,
+ and only check whether the system logic supports the specified speed.
+ Therefore we don't report the device name in any errors.  */
+
+  speed_t baud = string_to_baud (arg);
+
+  assert (baud != (speed_t) -1);
 
-  baud = string_to_baud (arg);
   if (type == input_speed || type == both_speeds)
-cfsetispeed (mode, baud);
+{
+  if (cfsetispeed (mode, baud))
+die (EXIT_FAILURE, 0, "unsupported ispeed %s", quotef (arg));
+}
   if (type == output_speed || type == both_speeds)
-cfsetospeed (mode, baud);
+{
+  if (cfsetospeed (mode, baud))
+die (EXIT_FAILURE, 0, "unsupported ospeed %s", quotef (arg));
+}
 }
 
 #ifdef TIOCGWINSZ
diff --git a/tests/misc/stty-invalid.sh b/tests/misc/stty-invalid.sh
index 58e51311d..af49b8d89 100755
--- a/tests/misc/stty-invalid.sh
+++ b/tests/misc/stty-invalid.sh
@@ -50,6 +50,9 @@ if tty -s 

Bug#1014008: coreutils: comm --output-delimiter= (empty) separates with a single NUL, not the empty string, vice versa for --total

[Pádraig Brady]

On 01/07/2022 11:29, Pádraig Brady wrote:

On 28/06/2022 21:06, наб wrote:

retitle -1 1014008 coreutils: comm --output-delimiter= (empty) separates with a 
single NUL, not the empty string, vice versa for --total
thanks

The only thing suitable for posting is "lmao". As the title says:
$ comm   --total  --output-delimiter= <(echo a; echo b) <(echo a) | cat -A
^@^@a$
b$
101total$


Support for the NUL delimiter was explicitly added with:
https://github.com/coreutils/coreutils/commit/0e46753d7

You're correct that the support / docs are inconsistent,
especially with the subsequently added --total option.

I'll fix the docs and --total option upstream.


Addressed upstream at:
https://git.sv.gnu.org/gitweb/?p=coreutils.git;a=commitdiff;h=708ae170c

Bug#1017110: coreutils: printf 'X takes X to be a byte, not a character (and missing in manual)?

[Pádraig Brady]

On 14/08/2022 01:41, наб wrote:

Package: coreutils
Version: 8.32-4+b1
Severity: normal

Dear Maintainer,

Consider:
-- >8 --
$ /bin/printf %d\\n \'ą
/bin/printf: warning: �: character(s) following character constant have been 
ignored
196
$ /bin/printf %d\\n \'я
/bin/printf: warning: �: character(s) following character constant have been 
ignored
209
-- >8 --

Compare bash:
-- >8 --
$ printf %d\\n \'ą
261
$ printf %d\\n \'я
1103
-- >8 --

Compare POSIX (Issue 7):

If the leading character is a single-quote or double-quote, the value
shall be the numeric value in the underlying codeset of the character
following the single-quote or double-quote.


Compare printf.1:
[no data (??)]


Fixed in version 9.1 with:
https://github.com/coreutils/coreutils/commit/6367cfe5f

$ /bin/printf --version | head -n1
printf (GNU coreutils) 9.1

$ /bin/printf %d\\n \'я
1103

cheers,
Pádraig

Bug#705566: date does not read the timezone from the environment variable TZ, and there is no other way to view from the CLI times in other timezones.

[Pádraig Brady]

On 12/08/2022 11:18, Uwe Kleine-König wrote:

On Sun, May 05, 2013 at 05:43:38PM -0600, Bob Proulx wrote:

severity 705566 normal
thanks

Laurence Maddox wrote:

I get this output:
Current default time zone: 'America/Chicago'
Local time is now:  Tue Apr 16 14:45:29 CDT 2013.
Universal Time is now:  Tue Apr 16 19:45:29 UTC 2013.


Looks okay.  What output did you expect?


I need to view on the CLI the time in various timezones.
...
I have searched for the "right" way to complete my task, and found a link here:
http://www.linuxquestions.org/questions/programming-9/display-different-timezones-in-command-line-927660/

That link recommends that I complete my task using a command like this one:
TZ=UTC date && TZ=CDT date && TZ=IST date


The concept is reasonable but you must pick a correct timezone for
what you want to accomplish.  UTC is okay.  But CDT and IST are
meaningless to libc.  The GNU libc has no concept of invalid
timezones.  If a timezone doesn't match anything else then the default
is that it is a name for UTC.  So basically you have asked for UTC
three times in a row.


I stumbled about a similar thing just now and would consider this
behaviour at least inconsistent:

uwe@taurus:/usr/share/zoneinfo$ TZ=Europe/London date
Fri Aug 12 11:13:34 AM BST 2022
uwe@taurus:/usr/share/zoneinfo$ TZ=BST date
Fri Aug 12 10:13:38 AM BST 2022

The first one is the right one, but there is no way to determine the
latter to be wrong. Apart from the offset the output is identical and if
you're not aware that TZ=BST is wrong you have no chance to notice that.

If it at least said "Fri Aug 12 11:13:38 AM UTC 2022" or (better yet)
dies with an error code that would be highly convenient.

And given that the timezone specifier in the output is "BST" (in both
cases!) noticing that TZ=BST is wrong isn't as trivial as it should be.

Note that using -R doesn't help here either because then I have to
notice that "+" is wrong, but to find the offset from me to BST was
the actual task to solve here ...


In my experience using POSIX timezone specifications
leads to ambiguity and they're not really usable.
Instead it's best to use the longer form location based zones where possible.
I've expanded on this here:
https://www.pixelbeat.org/docs/linux_timezones/

thanks,
Pádraig

Bug#1016049: coreutils: env --{default,ignore,block}-signal with empty argument is valid and does nothing?

[Pádraig Brady]

On 26/07/2022 05:46, наб wrote:

Package: coreutils
Version: 8.32-4+b1
Severity: minor

Dear Maintainer,

Quoth upstream NEWS:
-- >8 --
* Noteworthy changes in release 8.31 (2019-03-10) [stable]

** New features

   env now supports '--default-signal[=SIG]', '--ignore-signal[=SIG]', and
   '--block-signal[=SIG], to setup signal handling before executing a program.

   env now supports '--list-signal-handling' to indicate non-default
   signal handling before executing a program.
-- >8 --

Quoth env(1):
-- >8 --
--block-signal[=SIG]
   block delivery of SIG signal(s) to COMMAND

--default-signal[=SIG]
   reset handling of SIG signal(s) to the default

--ignore-signal[=SIG]
   set handling of SIG signals(s) to do nothing

SIG may be a signal name like 'PIPE', or a signal number like '13'.  
Without SIG, all known signals are included.  Multiple signals can be 
comma-separated.
-- >8 --

However:
-- >8 --
$ env -v --block-signal=1 ls /dev/null
signal HUP (1) mask set to BLOCK
executing: ls
arg[0]= ‘ls’
arg[1]= ‘/dev/null’
/dev/null
$ env -v --block-signal ls /dev/null 2>&1 | paste - - - -
signal HUP (1) mask set to BLOCKsignal INT (2) mask set to BLOCK
signal QUIT (3) mask set to BLOCK   signal ILL (4) mask set to BLOCK
signal TRAP (5) mask set to BLOCK   signal ABRT (6) mask set to BLOCK   
signal BUS (7) mask set to BLOCKsignal FPE (8) mask set to BLOCK
signal KILL (9) mask set to BLOCK   signal USR1 (10) mask set to BLOCK  
signal SEGV (11) mask set to BLOCK  signal USR2 (12) mask set to BLOCK
signal PIPE (13) mask set to BLOCK  signal ALRM (14) mask set to BLOCK  
signal TERM (15) mask set to BLOCK  signal STKFLT (16) mask set to BLOCK
signal CHLD (17) mask set to BLOCK  signal CONT (18) mask set to BLOCK  
signal STOP (19) mask set to BLOCK  signal TSTP (20) mask set to BLOCK
signal TTIN (21) mask set to BLOCK  signal TTOU (22) mask set to BLOCK  
signal URG (23) mask set to BLOCK   signal XCPU (24) mask set to BLOCK
signal XFSZ (25) mask set to BLOCK  signal VTALRM (26) mask set to BLOCK
signal PROF (27) mask set to BLOCK  signal WINCH (28) mask set to BLOCK
signal POLL (29) mask set to BLOCK  signal PWR (30) mask set to BLOCK   
signal SYS (31) mask set to BLOCK   signal RTMIN (34) mask set to BLOCK
signal RTMIN+1 (35) mask set to BLOCK   signal RTMIN+2 (36) mask set to BLOCK   
signal RTMIN+3 (37) mask set to BLOCK   signal RTMIN+4 (38) mask set to BLOCK
signal RTMIN+5 (39) mask set to BLOCK   signal RTMIN+6 (40) mask set to BLOCK   
signal RTMIN+7 (41) mask set to BLOCK   signal RTMIN+8 (42) mask set to BLOCK
signal RTMIN+9 (43) mask set to BLOCK   signal RTMIN+10 (44) mask set to BLOCK  
signal RTMIN+11 (45) mask set to BLOCK  signal RTMIN+12 (46) mask set to BLOCK
signal RTMIN+13 (47) mask set to BLOCK  signal RTMIN+14 (48) mask set to BLOCK  
signal RTMIN+15 (49) mask set to BLOCK  signal RTMAX-14 (50) mask set to BLOCK
signal RTMAX-13 (51) mask set to BLOCK  signal RTMAX-12 (52) mask set to BLOCK  
signal RTMAX-11 (53) mask set to BLOCK  signal RTMAX-10 (54) mask set to BLOCK
signal RTMAX-9 (55) mask set to BLOCK   signal RTMAX-8 (56) mask set to BLOCK   
signal RTMAX-7 (57) mask set to BLOCK   signal RTMAX-6 (58) mask set to BLOCK
signal RTMAX-5 (59) mask set to BLOCK   signal RTMAX-4 (60) mask set to BLOCK   
signal RTMAX-3 (61) mask set to BLOCK   signal RTMAX-2 (62) mask set to BLOCK
signal RTMAX-1 (63) mask set to BLOCK   signal RTMAX (64) mask set to BLOCK 
executing: ls  arg[0]= ‘ls’
arg[1]= ‘/dev/null’  /dev/null
$ env -v --block-signal= ls /dev/null
executing: ls
arg[0]= ‘ls’
arg[1]= ‘/dev/null’
/dev/null
-- >8 --

Precedent dictates that an empty argument should be refused (right?),
practicality suggests that it's likely a mistake
(since, again, it does nothing).
and the manual says it's forbidden
(clumsily using SIG and prose to describe signal[,signal]...).

Best,
наб

-- System Information:
Debian Release: 11.4
   APT prefers stable-updates
   APT policy: (500, 'stable-updates'), (500, 'stable-security'), (500, 
'stable-debug'), (500, 'stable')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 5.10.0-16-amd64 (SMP w/24 CPU threads)
Kernel taint flags: TAINT_PROPRIETARY_MODULE, TAINT_FIRMWARE_WORKAROUND, 
TAINT_OOT_MODULE, TAINT_UNSIGNED_MODULE
Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8), 
LANGUAGE=en_GB:en
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages coreutils depends on:
ii  libacl1  2.2.53-10
ii  libattr1 1:2.4.48-6
ii  libc62.31-13+deb11u3
ii  libgmp10 2:6.2.1+dfsg-1+deb11u1
ii  libselinux1  3.1-3

coreutils recommends no packages.

coreutils suggests no packages.

-- no debconf information



Not a bug I think.

"Empty" and "No" arguments 

Bug#1013924: coreutils: runcon -c getfscon()s program verbatim but execve()s it; trojan moment?

[Pádraig Brady]

On 27/06/2022 19:13, Pádraig Brady wrote:

On 27/06/2022 16:04, наб wrote:

Package: coreutils
Version: 8.32-4+b1
Severity: normal

Dear Maintainer,

The strace for runcon -c true true (after a > true) contains
getxattr("true", "security.selinux", "unconfined_u:object_r:user_tmp_t", 
255) = 36
execve("/usr/local/sbin/true", ["true", "true"]) = -1 ENOENT
execve("/usr/local/bin/true", ["true", "true"]) = -1 ENOENT
execve("/sbin/true", ["true", "true"]) = -1 ENOENT
execve("/bin/true", ["true", "true"]) = 0

This corresponds to getfscon("true"), execvp("true", ["true", NULL]).
(of course, this also errors if ./true doesn't exist).

So, uh: is this intentional? It certainly feels wrong? All invocations
take a PATH executable except this one which takes a PATH executable
that must *also* be a valid file? And also invites a trivial trojan
because the precomputed transition is to the file in the cwd, but the
program executed lives somewhere in PATH? Should -c just execv()
instead? Am I misunderstanding the usefulness of this?

Best,
наб


This is a fair point.
I.e. the following patch would be more correct operation.
I'll propose this upstream.

Now existing scripts would need to pass absolute paths to `runcon -c`
to work in the first place, so I don't know how much of a security
issue this is in practice.

thanks,
Pádraig

iff --git a/src/runcon.c b/src/runcon.c
index c4227c784..d85411c79 100644
--- a/src/runcon.c
+++ b/src/runcon.c
@@ -255,7 +255,7 @@ main (int argc, char **argv)
 if (cur_context != NULL)
   freecon (cur_context);

-  execvp (argv[optind], argv + optind);
+  (compute_trans ? execv : execvp) (argv[optind], argv + optind);

 int exit_status = errno == ENOENT ? EXIT_ENOENT : EXIT_CANNOT_INVOKE;
 error (0, errno, "%s", quote (argv[optind]));


Pushed the attached upstream.

thanks,
Pádraig
From 96c149941357186abcbd8da914544a7867cab01e Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?P=C3=A1draig=20Brady?= 
Date: Sun, 24 Jul 2022 18:46:10 +0100
Subject: [PATCH] runcon: ensure --compute runs the file it inspects

* src/runcon.c (main): With -c avoid searching the path
to ensure the file specified to --compute is executed.
* tests/misc/runcon-compute.sh: Add a new test.
* tests/local.mk: Reference the new test.
* NEWS: Mention the bug fix.
Reported in https://bugs.debian.org/1013924
---
 NEWS |  4 
 src/runcon.c |  2 +-
 tests/local.mk   |  1 +
 tests/misc/runcon-compute.sh | 28 
 4 files changed, 34 insertions(+), 1 deletion(-)
 create mode 100755 tests/misc/runcon-compute.sh

diff --git a/NEWS b/NEWS
index 816025255..b5b8990f8 100644
--- a/NEWS
+++ b/NEWS
@@ -10,6 +10,10 @@ GNU coreutils NEWS-*- outline -*-
   'mv --backup=simple f d/' no longer mistakenly backs up d/f to f~.
   [bug introduced in coreutils-9.1]
 
+  runcon --compute no longer looks up the specified command in the $PATH
+  so that there is no mismatch between the inspected and executed file.
+  [bug introduced when runcon was introduced in coreutils-6.9.90]
+
   'sort -g' no longer infloops when given multiple NaNs on platforms
   like x86_64 where 'long double' has padding bits in memory.
   Although the fix alters sort -g's NaN ordering, that ordering has
diff --git a/src/runcon.c b/src/runcon.c
index c4227c784..d85411c79 100644
--- a/src/runcon.c
+++ b/src/runcon.c
@@ -255,7 +255,7 @@ main (int argc, char **argv)
   if (cur_context != NULL)
 freecon (cur_context);
 
-  execvp (argv[optind], argv + optind);
+  (compute_trans ? execv : execvp) (argv[optind], argv + optind);
 
   int exit_status = errno == ENOENT ? EXIT_ENOENT : EXIT_CANNOT_INVOKE;
   error (0, errno, "%s", quote (argv[optind]));
diff --git a/tests/local.mk b/tests/local.mk
index 0f7778619..0496c2873 100644
--- a/tests/local.mk
+++ b/tests/local.mk
@@ -351,6 +351,7 @@ all_tests =	\
   tests/misc/readlink-fp-loop.sh		\
   tests/misc/readlink-root.sh			\
   tests/misc/realpath.sh			\
+  tests/misc/runcon-compute.sh			\
   tests/misc/runcon-no-reorder.sh		\
   tests/misc/sha1sum.pl\
   tests/misc/sha1sum-vec.pl			\
diff --git a/tests/misc/runcon-compute.sh b/tests/misc/runcon-compute.sh
new file mode 100755
index 0..1c4e0c060
--- /dev/null
+++ b/tests/misc/runcon-compute.sh
@@ -0,0 +1,28 @@
+#!/bin/sh
+# Ensure that runcon -c uses absolute file names
+
+# Copyright (C) 2022 Free Software Foundation, Inc.
+
+# This program is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+
+# This program is distributed in the hop

Bug#1014008: coreutils: comm --output-delimiter= (empty) separates with a single NUL, not the empty string, vice versa for --total

[Pádraig Brady]

On 28/06/2022 21:06, наб wrote:

retitle -1 1014008 coreutils: comm --output-delimiter= (empty) separates with a 
single NUL, not the empty string, vice versa for --total
thanks

The only thing suitable for posting is "lmao". As the title says:
   $ comm   --total  --output-delimiter= <(echo a; echo b) <(echo a) | cat -A
   ^@^@a$
   b$
   101total$


Support for the NUL delimiter was explicitly added with:
https://github.com/coreutils/coreutils/commit/0e46753d7

You're correct that the support / docs are inconsistent,
especially with the subsequently added --total option.

I'll fix the docs and --total option upstream.

thanks,
Pádraig

Bug#1013924: coreutils: runcon -c getfscon()s program verbatim but execve()s it; trojan moment?

[Pádraig Brady]

On 27/06/2022 16:04, наб wrote:

Package: coreutils
Version: 8.32-4+b1
Severity: normal

Dear Maintainer,

The strace for runcon -c true true (after a > true) contains
   getxattr("true", "security.selinux", "unconfined_u:object_r:user_tmp_t", 
255) = 36
   execve("/usr/local/sbin/true", ["true", "true"]) = -1 ENOENT
   execve("/usr/local/bin/true", ["true", "true"]) = -1 ENOENT
   execve("/sbin/true", ["true", "true"]) = -1 ENOENT
   execve("/bin/true", ["true", "true"]) = 0

This corresponds to getfscon("true"), execvp("true", ["true", NULL]).
(of course, this also errors if ./true doesn't exist).

So, uh: is this intentional? It certainly feels wrong? All invocations
take a PATH executable except this one which takes a PATH executable
that must *also* be a valid file? And also invites a trivial trojan
because the precomputed transition is to the file in the cwd, but the
program executed lives somewhere in PATH? Should -c just execv()
instead? Am I misunderstanding the usefulness of this?

Best,
наб


This is a fair point.
I.e. the following patch would be more correct operation.
I'll propose this upstream.

Now existing scripts would need to pass absolute paths to `runcon -c`
to work in the first place, so I don't know how much of a security
issue this is in practice.

thanks,
Pádraig

iff --git a/src/runcon.c b/src/runcon.c
index c4227c784..d85411c79 100644
--- a/src/runcon.c
+++ b/src/runcon.c
@@ -255,7 +255,7 @@ main (int argc, char **argv)
   if (cur_context != NULL)
 freecon (cur_context);

-  execvp (argv[optind], argv + optind);
+  (compute_trans ? execv : execvp) (argv[optind], argv + optind);

   int exit_status = errno == ENOENT ? EXIT_ENOENT : EXIT_CANNOT_INVOKE;
   error (0, errno, "%s", quote (argv[optind]));

Bug#1012447: coreutils: tr: tr.1 (and tr --help) falsely claims -t is only valid when translating

[Pádraig Brady]

On 07/06/2022 13:41, наб wrote:

Package: coreutils
Version: 8.32-4.1
Severity: minor
File: /usr/bin/tr

Dear Maintainer,

The soup at the end of tr.1 (and tr --help) claims:
   -t may be used only when translating.

This is a very obvious lie:
   $ tr -tds abcd A
   aabbccddAA
   A

Of course, -t only changes anything when translating,
because heirloom tr uses separate tables for tr/-d/-s
and explicitly handles overlong set2 for squeezing,
but it can be specified whenever
(just doesn't do anything unless translating).


Fair point.
Will address this upstream in the attached.

thanks,
Pádraig
From 2616c03a616804e9832e6412474013cd2c6c339e Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?P=C3=A1draig=20Brady?= 
Date: Sun, 19 Jun 2022 13:18:01 +0100
Subject: [PATCH] doc: tr: clarify that -t is ignored unless translating

* src/tr.c (usage): Don't say that -t is disallowed unless translating.
Reported in https://bugs.debian.org/1012447
---
 src/tr.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/tr.c b/src/tr.c
index e2aa33e76..59b0969d2 100644
--- a/src/tr.c
+++ b/src/tr.c
@@ -340,7 +340,7 @@ Interpreted sequences are:\n\
  fputs (_("\
 \n\
 Translation occurs if -d is not given and both STRING1 and STRING2 appear.\n\
--t may be used only when translating.  ARRAY2 is extended to length of\n\
+-t is only significant when translating.  ARRAY2 is extended to length of\n\
 ARRAY1 by repeating its last character as necessary.  Excess characters\n\
 of ARRAY2 are ignored.  Character classes expand in unspecified order;\n\
 while translating, [:lower:] and [:upper:] may be used in pairs to\n\
-- 
2.26.2

Bug#1004476: mv: add reflink option

[Pádraig Brady]

On 28/01/2022 12:07, Matthias Urlichs wrote:

Package: coreutils
Version: 8.32-4+b1
Severity: wishlist
X-Debbugs-Cc: sm...@smurf.noris.de

Moving files between btrfs subvolumes doesn't work because they're
different volumes.

However, it's possible to reflink the files to their new location, then
remove the original.

Rather than forcing the user to do the cp+rm dance that -- in the distant
past -- was required to move files between different hard disks, please add
an option to /bin/mv to use (or try using) reflinks for moving files.

Using reflinks automatigally when a "normal" rename(2) fails may be a
sensible default, but should be turnoffable


mv has used reflinks by default since 2014

Bug#1004476: mv: add reflink option

[Pádraig Brady]

On 28/01/2022 12:07, Matthias Urlichs wrote:

Package: coreutils
Version: 8.32-4+b1
Severity: wishlist
X-Debbugs-Cc: sm...@smurf.noris.de

Moving files between btrfs subvolumes doesn't work because they're
different volumes.

However, it's possible to reflink the files to their new location, then
remove the original.

Rather than forcing the user to do the cp+rm dance that -- in the distant
past -- was required to move files between different hard disks, please add
an option to /bin/mv to use (or try using) reflinks for moving files.

Using reflinks automatigally when a "normal" rename(2) fails may be a
sensible default, but should be turnoffable.


mv should already be doing this since v8.24
https://github.com/coreutils/coreutils/commit/b47231be6

Note we switched from BTRFS_IOC_CLONE to the more general FICLONE
in v8.26, though that should be supported by BTRFS:
https://github.com/coreutils/coreutils/commit/89e1fef97

Could you provide an strace of the failing reflink.

thanks,
Pádraig

Bug#988595: stty: Doesn't work when flow control is on

[Pádraig Brady]

On 16/05/2021 16:00, Matthias Urlichs wrote:

Package: coreutils
Version: 8.32-4+b1
Severity: important
X-Debbugs-Cc: sm...@smurf.noris.de

If terminal output cannot drain, e.g. because of flow control, "stty" does
no longer work because it's using TCSETSW instead of TCSETS.
See ioctl_tty(2).

Fix: use TCSETS.


This is an option in stty, use "-drain" if you want to disable the default 
drain behavior.

cheers,
Pádraig

Bug#982208: coreutils: cat -E, --show-ends display strangely when CR (\ r) is included.

[Pádraig Brady]

On 07/02/2021 13:08, KenichiroMATOHARA wrote:

Package: coreutils
Version: 8.32-4+b1
Severity: minor

Dear Maintainer,

When I create a text file with a newline code of "\r\n" and
`cat` with the `-E, --show-ends` option, "$" is displayed at
the beginning of the line.
When `-A, --show-all` is used, it looks normal.


- Create a file for \r\n

$ echo 'foo

 $ od -c dos.txt

000   f   o   o  \r  \n  \t   b   a   r  \r  \n
013
   bar' | tr '\n' '\r\n' > dos.txt


- `-E, --show-ends` option

$ cat -E dos.txt
$oo
$   bar
$ cat -nE dos.txt
$1  foo
$2  bar


- `-A, --show-all` option

$ cat -A dos.txt
foo ^M$
^Ibar^M$
$ cat -nA dos.txt
  1  foo^M$
  2  ^Ibar^M$


I agree the current behavior is less than useful because:

  * \r\n is common a line end combination
  * catting such a file without options causes it to display normally
  * overwriting the first char with $, loses info

I propose to change the upstream coreutils project
as per the attached, to extend the --show-ends option
to show \r as ^M when the following character is \n.

thanks,
Pádraig
diff --git a/src/cat.c b/src/cat.c
index 68f3c9ac5..c9838cfc5 100644
--- a/src/cat.c
+++ b/src/cat.c
@@ -486,7 +486,15 @@ cat (
   *bpout++ = ch + 64;
 }
   else if (ch != '\n')
-*bpout++ = ch;
+{
+  if (ch == '\r' && (*bpin == '\n') && show_ends)
+{
+  *bpout++ = '^';
+  *bpout++ = ch + 64;
+}
+  else
+*bpout++ = ch;
+}
   else
 {
   newlines = -1;

Bug#982300: split can cause data corruption with --number=K/N construct

[Pádraig Brady]

Package: coreutils
Version: 8.30-3

split --number=K/N can be used when there isn't enough local storage,
and you want to split input to separate output devices in separate runs.

Ever since v8.26 (so including centos 8), there is a nasty
data corruption issue for any file over 128KiB
as part of the input file is skipped incorrectly.

The upstream fix for this is:
https://github.com/coreutils/coreutils/commit/bb21daa1.patch

Bug#969664: coreutils: IF_LINT is defined to nothing, so free is never called

[Pádraig Brady]

On 06/09/2020 20:31, Davide Prina wrote:

Package: coreutils
Version: 8.30-3+b1
Severity: normal

Dear Maintainer,

IF_LINT is defined to nothing, so free is never called.

$ apt source coreutils


This is correct for release builds.
The uncalled free() you're seeing is fine,
as the process is being ended anyway.
Calling the free would just be wasted cycles.
We do call the free() in dev mode to more easily see real leaks.

thanks,
Pádraig

Bug#959497: dircolors from coreutils is missing glob pattern `TERM putty*`

[Pádraig Brady]

On 03/05/2020 02:15, Tobias Wendorff wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Package: coreutils
Version: 8.30-3

`src/dircolors.hin` lists TERM entries, which can be glob patterns,
to match against the TERM environment variable to determine if it
is colorizable.

It also lists `TERM putty`, rather than the expected `TERM putty*`.
Right now, `TERM putty-256color` (from ncurses-term) gets ignored,
although it is capable of displaying colors and widely used by
Putty users.

I suggest to fix the glob pattern in `src/dircolors.hin`, also
in upstream.


Note upstream no longer has putty-256color,
but since 8.25 supports globbing of entries,
and so now uses *color*.

This can be further overridden, like on Fedora at least
where /etc/DIR_COLORS.256color matches *256color*

cheers,
Pádraig

Bug#930965: coreutils: seq 84x slower with --equal-width

[Pádraig Brady]

On 23/06/19 18:05, Barak A. Pearlmutter wrote:
> Package: coreutils
> Version: 8.30-3
> Severity: normal
> 
> Was using seq to write some data to test that a cheapo enormous SD card
> isn't faking its capacity. Thought I'd use seq --equal-width just to
> make calculations easier. But jeepers creepers, what a slowdown!
> 
> $ time seq 0 1000 > /dev/null
> 
> real  0m0.358s
> user  0m0.331s
> sys   0m0.018s
> 
> $ time seq --equal-width 0 1000 > /dev/null
> 
> real0m29.562s
> user0m27.968s
> sys 0m0.100s
> 
> $ echo '27.968 / 0.331' | bc
> 84

Right --equal-width currently goes through stdio formatting,
rather than simple ascii manipulation.

The fastest solution for you is probably:
  yes 1000| tail -n 1000

Bug#915559: bug#34681: mv fails when renaming on external drive in coreutils 8.30-2

[Pádraig Brady]

On 27/02/19 14:20, Derek Dongray wrote:
> When trying to use mv to rename a file on an external drive using coreutils
> 8.30-2 on a debian system (Linux version 4.19.0-3-amd64), the rename fails
> with the error message:
> 
> mv: cannot move 'file1' to a subdirectory of itself 'file2'
> 
> Downgrading to coreutils 8.30-1, the rename executes as expected.
> 
> The following is the result of running a test script. The folder '/backup'
> is an external drive using the ZFS fileystems (zfs-zed 0.7.12-3), but I
> have seen a report on superuser.com (
> https://superuser.com/questions/1409618/renaming-a-file-with-mv-cannot-move-to-a-subdirectory-of-itself)
> that this also happens with NTFS external drives.
> 
> root@capella:~# ./mv-bug
> + apt install -y --allow-downgrades coreutils=8.30-1
> Reading package lists... Done
> Building dependency tree
> Reading state information... Done
> coreutils is already the newest version (8.30-1).
> 0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
> + cd /backup
> + touch t
> + ls s t
> ls: cannot access 's': No such file or directory
> t
> + mv t s
> + ls s t
> ls: cannot access 't': No such file or directory
> s
> + rm s t
> rm: cannot remove 't': No such file or directory
> + cd /root
> + apt install -y coreutils=8.30-2
> Reading package lists... Done
> Building dependency tree
> Reading state information... Done
> The following packages will be upgraded:
>   coreutils
> 1 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
> Need to get 2,707 kB of archives.
> After this operation, 4,096 B disk space will be freed.
> Get:1 http://ftp.uk.debian.org/debian sid/main amd64 coreutils amd64 8.30-2
> [2,707 kB]
> Fetched 2,707 kB in 1s (2,729 kB/s)
> apt-listchanges: Reading changelogs...
> (Reading database ... 226704 files and directories currently installed.)
> Preparing to unpack .../coreutils_8.30-2_amd64.deb ...
> Unpacking coreutils (8.30-2) over (8.30-1) ...
> Setting up coreutils (8.30-2) ...
> Processing triggers for install-info (6.5.0.dfsg.1-4+b1) ...
> Processing triggers for man-db (2.8.5-2) ...
> + cd /backup
> + touch t
> + ls s t
> ls: cannot access 's': No such file or directory
> t
> + mv t s
> mv: cannot move 't' to a subdirectory of itself, 's'
> + ls s t
> ls: cannot access 's': No such file or directory
> t
> + rm s t
> rm: cannot remove 's': No such file or directory
> root@capella:~#

That sounds like unsupported renameat2()
is not falling back to rename()

The only change in debian is:
  coreutils (8.30-2) unstable; urgency=medium
* Use renameat glibc function that can be intercepted by fakechroot
  (Closes: https://bugs.debian.org/915559 )
* Above requires autoreconf turned on again

A quick scan of the patches shows that the name was changed
to renameatu() in gnulib, but copy.c still calls renameat2()
and thus now doesn't have the fallback.

To be clear this seems to only be an issue in the debian patch.

cheers,
Pádraig

Bug#895857: coreutils: sha1sum and other hashing tools are compiled without libcrypto support

[Pádraig Brady]

On 08/02/19 07:12, Matteo Croce wrote:
> Hi,
> 
> latest gnulib have support for sendfile and AF_ALG, the kernel crypto
> framework, if compiled with --with-linux-crypto.
> This allows to have very fast hashing algorythms but without the
> libcrypto dependency,
> 
> coreutils$ ./configure --with-linux-crypto
> [...]
> checking whether linux/if_alg.h has struct sockaddr_alg yes
> [...]
> 
> coreutils$ ldd src/sha1sum /usr/bin/sha1sum
> src/sha1sum:
> linux-vdso.so.1 (0x7fff1d56b000)
> libc.so.6 => /lib64/libc.so.6 (0x7f156b70c000)
> /lib64/ld-linux-x86-64.so.2 (0x7f156b902000)
> /usr/bin/sha1sum:
> linux-vdso.so.1 (0x7ffebef02000)
> libcrypto.so.1.1 => /lib64/libcrypto.so.1.1 (0x7fe977d09000)
> libc.so.6 => /lib64/libc.so.6 (0x7fe977b43000)
> libz.so.1 => /lib64/libz.so.1 (0x7fe977b29000)
> libdl.so.2 => /lib64/libdl.so.2 (0x7fe977b23000)
> libpthread.so.0 => /lib64/libpthread.so.0 (0x7fe977b01000)
> /lib64/ld-linux-x86-64.so.2 (0x7fe97802)
> 
> coreutils$ time /usr/bin/sha1sum zero
> 2a492f15396a6768bcbca016993f4b4c8b0b5307  zero
> 
> real0m1,254s
> user0m1,147s
> sys 0m0,108s
> coreutils$ truncate -s1g zero
> coreutils$ time src/sha1sum zero
> 2a492f15396a6768bcbca016993f4b4c8b0b5307  zero
> 
> real0m1,234s
> user0m0,005s
> sys 0m1,230s
> 
> Think about enabling it.

Depending on which kernel is used this can also be significantly slower.
Enabling this by default should be very carefully considered as
per the reasons detailed at:
https://lists.gnu.org/archive/html/coreutils/2018-06/msg00034.html

BTW openssl 3 is relicensed, so debian should be in a position
to enable use of libcrypto routines then.

cheers,
Pádraig

Bug#911572: coreutils: incorrect and inconsistent quoting in ls output

[Pádraig Brady]

On 21/10/18 18:13, Vincent Lefevre wrote:
> Package: coreutils
> Version: 8.30-1
> Severity: normal
> Tags: upstream
> Forwarded: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=33113
> 
> I get the following with ls (GNU coreutils) 8.30.
> 
> zira% touch a=b a=b\
> zira% ls a=b*
> 'a=b'  'a=b'
> zira% ls -b a=b*
> a=b  a=b
> zira% ls -F a=b*
> 'a=b'  'a=b'
> zira% ls -bF a=b*
> a\=b  a\=b
> 
> AFAIK, the = character is not a shell metacharacter (except with zsh
> but only in the first position), thus does not need to be quoted.
> 
> Moreover, while & is not a metacharacter, it is a special character
> that should be quoted for practical reasons.
> 
> In particular, this inconsistency with -bF makes a\=b unusable by
> copy-paste, as a\=b can't be used directly in a shell command, and
> 'a\=b' is not OK either.
> 
> Note: Such filenames with = and & can be produced by "wget -r".
> 

The reason \= is quoted with -bF is to distinguish socket
names that end in =. An edge case indeed, though the issue
is not particular to '=', but all of the classifier chars.

-b does not support copy and paste at the shell directly.
For that, the default (--quoting-style=shell-escape) is
really the only one directly usable.

Note with `set -k` equals is significant in any arg and so needs quoting.

cheers,
Pádraig

Bug#901752: patch for --reflink=never

[Pádraig Brady]

On 17/06/18 17:38, Adam Borowski wrote:
> Control: tags -1 +patch
> 
> Here's a patch.
> 
> I find upstream's decision to not default to --reflink=auto weird.  Not
> reflinking when it's supported is harmful: the stated reason of "resiliency"
> against disk failures is bogus: common types of failure don't work this way. 
> Neither does "guaranteeing disk space" work: a CoW filesystem has to CoW all
> files you modify, even briefly.  In most cases the old version will be kept
> by a snapshot anyway.  Even if not, if the user uses any method of
> deduplication, the damage is obvious.  Even more (un-)hilarious if you have
> some kind of inline deduplication (for btrfs, bees or that recently posted
> patch set, for zfs that's even the only implemented mode of deduplication)
> where the entire file is read and hashed for no reason whatsoever.
> 
> With my hat of a btrfs mailing list/IRC regular and an occasional
> kernel/btrfs-progs contributor, the only case where non-reflinking might
> possibly be useful are non-CoW files.  But:
>   touch foo
>   chattr +C foo
>   dd if=/dev/zero of=foo bs=1048576 count=10
>   cp --reflink=always foo bar
> cp: failed to clone 'bar' from 'foo': Invalid argument
> So it's already blocked!
> 
> 
> Thus, I doubt there's ever a reason to use a value of --reflink other than
> "auto".  But, if there is, you'd want to be able to temporarily override
> your alias.  Which is what the attached patch does.

Upstream is probably going to change to trying reflink by default
in a new major release of coreutils (v9). I agree with the arguments
you have presented, though it's a change with significant behavior change
and best done in a major version bump.

As for --reflink=never, that will be needed as you say,
and useful now in the edge case of people using --reflink=auto in aliases.
I will apply this for the upcoming 8.30 release.

thanks,
Pádraig.

Bug#895857: coreutils: Additional information about coreutils from sid

[Pádraig Brady]

On 16/04/18 16:52, Claudio Giordano wrote:
> Package: coreutils
> Version: 8.28-1
> Followup-For: Bug #895857
> 
> Dear Maintainer,
> 
> same compile issue is present also on sid release on newer coreutils version:
> 
> $ ldd /usr/bin/sha1sum 
>   linux-vdso.so.1 (0x7ffc39ca8000)
>   libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x7fb2033b1000)
>   /lib64/ld-linux-x86-64.so.2 (0x7fb203977000)
> 
> -- System Information:
> Debian Release: buster/sid
>   APT prefers unstable
>   APT policy: (500, 'unstable'), (1, 'experimental')
> Architecture: amd64 (x86_64)
> Foreign Architectures: i386
> 
> Kernel: Linux 4.15.0-2-amd64 (SMP w/4 CPU cores)
> Locale: LANG=it_IT.UTF-8, LC_CTYPE=it_IT.UTF-8 (charmap=UTF-8), 
> LANGUAGE=it_IT.UTF-8 (charmap=UTF-8)
> Shell: /bin/sh linked to /bin/dash
> Init: systemd (via /run/systemd/system)
> LSM: AppArmor: enabled
> 
> Versions of packages coreutils depends on:
> ii  libacl1  2.2.52-3+b1
> ii  libattr1 1:2.4.47-2+b2
> ii  libc62.27-3
> ii  libselinux1  2.7-2+b2
> 
> coreutils recommends no packages.
> 
> coreutils suggests no packages.
> 
> -- no debconf information

This was a policy decision made by the debian coreutils package maintainers
to not link coreutils GPL code with the openssl licensed system libs.
Note Red Hat / Fedora / Centos did consider this combination fine
and have enabled this for quite a while.

Note openssl licencing has nearly completed a transition to Apache 2
in which case debian would be free to enable this option.

cheers,
Pádraig

Bug#892356: fslint: findutils on testing breaks fslint recursive search

[Pádraig Brady]

On 02/04/18 08:57, jEsuSdA 8) wrote:
> El 10/03/18 a las 14:09, Pádraig Brady escribió:
>> I just tried find from git (not a debian package)
>> and I don't see the issue.
>> Perhaps it's already fixed in find upstream?
>> Can you give more details of the breakage?
> 
> I found the problem:
> 
> fslint does not do recursive duplicate searching when the FOLDER is a 
> SYMBOLIC LINK
> 
> For example:
> 
> /folder/main-folder/lot-of-pics
> 
> Where lot-of-pics has duplicate files.
> 
> 
> ln -s /folder/main-folder /folder/symlink-folder
> 
> 
> If you try to find duplicate pics intro /folder/symlink-folder DOES NOT 
> WORK.
> 
> 
> If you try /folder/main-folder/, fslint founds duplicate files inside 
> lot-of-pics
> 
> If you try /folder/symlink-folder/lot-of-pics, fslint works as expected.
> 
> So, I think, if route ends on symlink, fslint treats this folder as a 
> file. If the route ends on regular folder, it works fine.
> 
> 
> Can you confirm this?
> Thanks a lot!!!

I think this is already fixed in git. See:
https://github.com/pixelb/fslint/commit/9efd7fff

thanks,
Pádraig

Bug#894379: coreutils: timeout (hurd, possibly hppa): fails to detect that a program has exited

[Pádraig Brady]

On 29/03/18 08:31, Eugene V. Lyubimkin wrote:
> Package: coreutils
> Version: 8.28-1
> Severity: normal
> 
> Control: affects -1 cupt
> 
> 
> Hello,
> 
> Thank you for maintaining coreutils. I use timeout(1) to conveniently
> detect hanging problems in the testsuite of cupt. Unfortunately, on hurd
> and hppa it makes the test suite fail. 
> 
> This issue is best illustrated through a test case:
> 
> On hurd-i386 the commands which take less time to execute still make
> 'timeout' wait full time and exit with the error code:
> --8<-
> $ mkdir abc
> $ cd abc
> $ timeout 5s ls
> $ echo $?
> 124
> $ timeout 5s cat
> $ echo $?
> 124
> -->8-
> 
> On, say, amd64 the same scenario works as expected:
> --8<-
> $ mkdir abc
> $ cd abc
> $ timeout 5s ls
> $ echo $?
> 0
> $ timeout 5s cat
> $ echo $?
> 124
> -->8-
> 
> I couldn't check this scenario on hppa (as there are no porterboxes for
> it), but the bug report #882238 suggests a similar issue has been
> encountered on a hppa buildd box.
> 
> On hurd-i386 portexbox, I did verify that the program run under
> timeout(1) does exit, but timeout(1) itself continues to run even
> though it has no child processes anymore.
> 
> 

There was a bug fixed wrt SIGCHLD in coreutils-8.29

Bug#892356: fslint: findutils on testing breaks fslint recursive search

[Pádraig Brady]

On 08/03/18 04:37, jEsuSdA wrote:
> Package: fslint
> Version: 2.44-2
> Severity: important
> 
> Dear Maintainer,
> 
> Upgrading findutils from stable to testing breaks the fslint-gui recursive
> function to search duplicate files.
> 
> I tried to downgrade fslint from 2.44 to 2.42 and the bug persists.
> 
> When downgrading findutils from 4.6.0+git+20170828-2 to 4.6.0+git+20161106-2,
> both fslint version still working again.

I just tried find from git (not a debian package)
and I don't see the issue.
Perhaps it's already fixed in find upstream?
Can you give more details of the breakage?

Note there was a fix to cater for a find change wrt permissions,
though that should have been available in 2.44
https://github.com/pixelb/fslint/commit/d68ee1c

thanks,
Pádraig.

Bug#721358: bug#28574: cross compilnng, man pages

[Pádraig Brady]

On 03/10/17 13:51, Manuel A. Fernandez Montecelo wrote:
> Thanks for taking a look into this, I think that it's a good idea.
> 
> In the case that somehow help2man can still end up being used after
> applying this, I think that Eleanor's patch (or an equivalent) is
> still worth considering, because ultimately when cross-building one
> cannot invoke the binaries in the general case.

Eleanor's patch is a good intermediate solution.
It falls back to dummy-man when cross compiling.

My patch is more invasive and will be effective in the next coreutils release
and falls back to using distributed man pages when cross compiling.

help2man is not invoked in either case.

cheers,
Pádraig

Bug#721358: bug#28574: cross compilnng, man pages

[Pádraig Brady]

On 23/09/17 20:18, Peter Bohning wrote:
> Cross compiling for aarch64 from amd64
>  
> Is there no way to disable the man pages in configure options?  Thanks.
>  
> https://lists.gnu.org/archive/html/coreutils/2014-11/msg0.html

I've seen this issue in various places now (CC'd).

We should probably reinstate distribution of man pages
since these generated pages are so similar on various systems.
Also given the amount of old ChangeLogs removed recently
I don't think the addition of 273K (uncompressed) is prohibitive.

The attached was tested in these modes:
 git repo build
 tarball build with/without help2man
 tarball vpath build with/without help2man
 make distcheck
 make install  (and ensure only enabled program man pages installed)

I've not applied this yet, but hope to soon after a bit more testing.

cheers,
Pádraig
From d05e91c7b6c84b22cab3f30abdf3490b7c83910b Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?P=C3=A1draig=20Brady?= 
Date: Sat, 30 Sep 2017 17:08:02 -0700
Subject: [PATCH] build: reinstate distribution of man pages

man pages change little between systems,
so falling back to distributed pages make sense
when cross compiling or lacking perl.

* man/local.mk: Add all man pages to EXTRA_DIST
so that they're distributed in the generated tarball.
Use the dummy-man page generator if cross compiling.
Set TZ to avoid a distcheck failure where man pages
used a diffent month than those rebuilt (with a .timestamp).
* man/dummy-man: Only fall back to generating a stub
if copying an existing man page fails.
* NEWS: Mention the build-related change.
---
 NEWS  |  5 +
 man/dummy-man | 23 ---
 man/local.mk  | 16 +++-
 3 files changed, 28 insertions(+), 16 deletions(-)

diff --git a/NEWS b/NEWS
index 5d57c72..2878b70 100644
--- a/NEWS
+++ b/NEWS
@@ -11,6 +11,11 @@ GNU coreutils NEWS-*- outline -*-
   to attempt to hide the original length of the file name.
   [bug introduced in coreutils-8.28]
 
+** Build-related
+
+  Default man pages are now distributed which are used if perl is
+  not available on the build system, or when cross compiling.
+
 
 * Noteworthy changes in release 8.28 (2017-09-01) [stable]
 
diff --git a/man/dummy-man b/man/dummy-man
index f05ce2a..e298295 100755
--- a/man/dummy-man
+++ b/man/dummy-man
@@ -1,8 +1,7 @@
 #!/bin/sh
-# Poor man's placeholder for help2man invocation on systems lacking perl;
-# it generates a dummy man page stating that a proper one could not be
-# generated, and redirecting the user back to either the info
-# documentation or the '--help' output.
+# Poor man's placeholder for help2man invocation on systems lacking perl,
+# or when cross compiling.
+# It just copies the distributed man pages.
 
 set -e; set -u
 
@@ -24,12 +23,12 @@ while test $# -gt 0; do
 # Help2man options we recognize and handle.
 --output=*) output=`expr x"$1" : x'--output=\(.*\)'`;;
 --output) shift; output=$1;;
+--include=*) include=`expr x"$1" : x'--include=\(.*\)'`;;
+--include) shift; include=$1;;
 --source=*) source=`expr x"$1" : x'--source=\(.*\)'`;;
 --source) shift; source=$1;;
 # Recognize (as no-op) other help2man options that might be used
 # in the makefile.
---include=*);;
---include) shift;;
 --info-page=*);;
 -*) fatal_ "invalid or unrecognized help2man option '$1'";;
 --) shift; break;;
@@ -41,10 +40,13 @@ done
 test $# -gt 0 || fatal_ "missing argument"
 test $# -le 1 || fatal_ "too many non-option arguments"
 
+dist_man=$(printf '%s\n' "$include" | sed 's/\.x$/.1/')
+test -f "$dist_man" && cp "$dist_man" "$output" && exit || :
+
 baseout=`basename_ "$output"`
 sed 's/^/WARNING: /' >&2 <"$output"  $@-t			\
 	  && rm -rf $$t			\
 	  && chmod a-w $@-t		\
-	  && mv $@-t $@
+	  && rm -f $@ && mv $@-t $@
-- 
2.9.3

Bug#859021: dd: document that bs overrides ibs and obs

[Pádraig Brady]

On 29/03/17 09:04, Chris Davies wrote:
> Package: coreutils
> Version: 8.25-2+b1
> Severity: minor
> 
> Dear Maintainer,
> 
> The man page for dd does not define an interaction between bs and its more
> specific ibs and obs options. POSIX does mandate that bs, if specified,
> shall override ibs and obs. Empirically I have determined that coreutils
> dd implements POSIX behaviour, so I should like to suggest the man page
> is updated to state that bs overrides ibs and/or obs.
> 
> 
> POSIX man page at http://www.unix.com/man-page/posix/1posix/dd/ defines
> the three options bs, ibs, and obs as follows:
> 
> ibs=expr
>   Specify the input block size, in bytes, by expr (default is 512).
> 
> obs=expr
>   Specify the output block size, in bytes, by expr (default is 512).
> 
> bs=expr
>   Set  both input and output block sizes to expr bytes, superseding
>   ibs= and obs=. If no conversion other than sync, noerror, and
>   notrunc is specified, each input  block shall be copied to the
>   output as a single block without aggregating short blocks.
> 
> 
> Extracts from the current coreutils man page read as follows:
> 
> bs=BYTES
>   read and write up to BYTES bytes at a time
> 
> ibs=BYTES
>   read up to BYTES bytes at a time (default: 512)
> 
> obs=BYTES
>   write BYTES bytes at a time (default: 512)
> 
> 
> I would like to suggest that the explanation is amended as follows:
> 
> bs=BYTES
>   read and write up to BYTES bytes at a time (default:
>   512); overrides ibs and obs

Done in your name upstream at:
http://git.sv.gnu.org/gitweb/?p=coreutils.git;a=commitdiff;h=v8.27-15-gc1c558e

Bug#851934: bug#23035: date: regression in timezone printing (+%Z)

[Pádraig Brady]

On 17/03/16 17:38, Paul Eggert wrote:
> On 03/16/2016 08:51 PM, Assaf Gordon wrote:
>> I suspect it has something to do with this commit:
>>commit037e3b9847feb46cf6b58d99ce960d3987faaf52
> 
> You're right, and thanks for that detailed bug report. I installed the 
> attached patch, which fixed the bug for me.

This introduced a bug unfortunately due to the side effects of parse_datetime().
parse_datetime resets the TZ env variable but doesn't call tzset().
Hence using the wrong timezone for output.
Previously localtime() called tzset() implicitly.
Perhaps we should call tzset() in parse_datetime() if needed?
I'm not sure as to whether this undoes the fix for bug 23035?

Anyway this avoids the issue on GNU/Linux.

diff --git a/src/date.c b/src/date.c
index ddc702f..1be410c 100644
--- a/src/date.c
+++ b/src/date.c
@@ -578,6 +578,8 @@ show_date (const char *format, struct timespec when, 
timezone_t tz)
 {
   struct tm tm;

+  tzset ();
+
   if (localtime_rz (tz, _sec, ))
 {
   if (format == rfc_2822_format)

$ TZ="Europe/London" date-before -d 'TZ="Australia/Perth" 2016-08-15 07:00'
Mon Aug 15 07:00:00 AWST 2016

$ TZ="Europe/London" date-after -d 'TZ="Australia/Perth" 2016-08-15 07:00'
Mon Aug 15 00:00:00 BST 2016

p.s. date --debug doesn't have the issue either as it
calls localtime() within parse_datetime()

cheers,
Pádraig

Bug#835904: tail --retry doesn't

[Pádraig Brady]

On 05/09/16 07:07, Harald Dunkel wrote:
> coreutils 8.25-2 seems to have the same problem. Sample:
> 
> 
> % ps -ef | grep tail
> harri 3815  3783  0 Aug30 pts/300:00:00 tail --retry 
> --max-unchanged-stats=5 -f /var/log/messages
> harri11756 11708  0 08:00 pts/400:00:00 grep tail
> 
> % lsof -p 3815
> COMMAND  PID  USER   FD  TYPE DEVICE SIZE/OFFNODE NAME
> tail3815 harri  cwd   DIR8,420480 4456449 /home/harri
> tail3815 harri  rtd   DIR8,3 4096   2 /
> tail3815 harri  txt   REG8,364232   15185 /usr/bin/tail
> tail3815 harri  mem   REG8,3  1697504 1054034 
> /lib/x86_64-linux-gnu/libc-2.23.so
> tail3815 harri  mem   REG8,3   153288 1054030 
> /lib/x86_64-linux-gnu/ld-2.23.so
> tail3815 harri0u  CHR  136,3  0t0   6 /dev/pts/3
> tail3815 harri1u  CHR  136,3  0t0   6 /dev/pts/3
> tail3815 harri2u  CHR  136,3  0t0   6 /dev/pts/3
> tail3815 harri3r  REG8,3   159279  140486 /var/log/messages.1
> tail3815 harri4r  a_inode   0,1104381 inotify
> 
> % ls -alrt /var/log/messages*
> -rw-r- 1 root adm   2057 Aug 15 07:54 /var/log/messages.4.gz
> -rw-r- 1 root adm   1762 Aug 21 08:03 /var/log/messages.3.gz
> -rw-r- 1 root adm  29000 Aug 29 07:53 /var/log/messages.2.gz
> -rw-r- 1 root adm 159279 Sep  4 08:00 /var/log/messages.1
> -rw-r- 1 root adm   2881 Sep  5 07:52 /var/log/messages
> 
> % dpkg -l coreutils
> Desired=Unknown/Install/Remove/Purge/Hold
> | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
> |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
> ||/ NameVersionArchitecture   
> Description
> +++-===-==-==-
> ii  coreutils   8.25-2 amd64  
> GNU core utilities

Oh the file is renamed.
You probably need `tail -F` rather than `tail -f --retry`

Bug#835904: tail --retry doesn't

[Pádraig Brady]

On 29/08/16 08:55, Harald Dunkel wrote:
> Package: coreutils
> Version: 8.23-4
> 
> I am running tail in a dedicated window for some days now:
> 
> % ps -ef | grep tai[l]
> hdunkel   2928  2915  0 Aug15 pts/100:00:00 tail --retry 
> --max-unchanged-stats=5 -f /var/log/messages
> 
> Problem: It's frozen. /var/log/messages is changing all the
> time, but tail doesn't recognize.
> 
> lsof -p shows:
> 
> % lsof -p 2928
> lsof: WARNING: can't stat() tracefs file system /sys/kernel/debug/tracing
>   Output information may be incomplete.
> COMMAND  PIDUSER   FD   TYPE DEVICE SIZE/OFF  NODE NAME
> tail2928 hdunkel  cwdDIR   0,4020480 189573782 /home/hdunkel 
> (nfs-data:/space/home)
> tail2928 hdunkel  rtdDIR  259,0 4096 2 /
> tail2928 hdunkel  txtREG  259,064232657851 /usr/bin/tail
> tail2928 hdunkel  memREG  259,0  1738176   1724334 
> /lib/x86_64-linux-gnu/libc-2.19.so
> tail2928 hdunkel  memREG  259,0   140928   1724331 
> /lib/x86_64-linux-gnu/ld-2.19.so
> tail2928 hdunkel0u   CHR  136,1  0t0 4 /dev/pts/1
> tail2928 hdunkel1u   CHR  136,1  0t0 4 /dev/pts/1
> tail2928 hdunkel2u   CHR  136,1  0t0 4 /dev/pts/1
> tail2928 hdunkel3r   REG  259,0   100176263640 
> /var/log/messages.1 (deleted)
> tail2928 hdunkel4r     0,110  8294 anon_inode
> 
> 
> AFAICT this is not supposed to happen. According to the man
> page "tail --retry --max-unchanged-stats=5" should reopen
> /var/log/messages.

There were tail fixes in this area in v8.24.
Previously renames may have been missed when inotify was used

Bug#833948: i18n: stty --help displays bogus block

[Pádraig Brady]

On 10/08/16 19:01, Andrea Stacchiotti wrote:
> Package: coreutils
> Version: 8.25-2
> Severity: minor
> Tags: l10n
> 
> It follows an invocation of stty --help with the italian locale.
> 
> The starting block of the translation file is printed in the middle of the
> output.
> Somehow gettext decided to put in the translation of "", I guess.
> 
> andreas@trelitri:~/Progetti/scanmem/gui$ LANG=it_IT.utf8 stty --help
> Uso: stty [-F DEVICE] | --file=DEVICE] [IMPOSTAZIONE]...
>  or: stty [-F DEVICE] | --file=DEVICE] [-a|--all]
>  or: stty [-F DEVICE] | --file=DEVICE] [-g|--save]
> Print or change terminal characteristics.
> 
> Mandatory arguments to long options are mandatory for short options too.
>   -a, --all  print all current settings in human-readable form
>   -g, --save print all current settings in a stty-readable form
>   -F, --file=DEVICE  open and use the specified DEVICE instead of stdin
>   --help mostra questo aiuto ed esce
>   --version  stampa le informazioni sulla versione ed esce
> 
> Un - opzionale prima di un'IMPOSTAZIONE indica la negazione. Un * indica le
> impostazioni non POSIX. Il sistema sottostante definisce quali impostazioni
> sono disponibili.
> 
> Special characters:
>  * discard CHAR  CHAR will toggle discarding of output
>eof CHAR  CHAR will send an end of file (terminate the input)
>eol CHAR  CHAR will end the line
>  * eol2 CHAR alternate CHAR for ending the line
>erase CHARCHAR will erase the last character typed
>intr CHAR CHAR will send an interrupt signal
>kill CHAR CHAR will erase the current line
>  * lnext CHARCHAR will enter the next character quoted
>quit CHAR CHAR will send a quit signal
>  * rprnt CHARCHAR will redraw the current line
>start CHARCHAR will restart the output after stopping it
>stop CHAR CHAR will stop the output
>susp CHAR CHAR will send a terminal stop signal
>  * swtch CHARCHAR will switch to a different shell layer
>  * werase CHAR   CHAR will erase the last word typed
> 
> Special settings:
>N set the input and output speeds to N bauds
>  * cols Ntell the kernel that the terminal has N columns
>  * columns N same as cols N
>  * [-]drain  wait for transmission before applying settings (on by 
> default)
>ispeed N  set the input speed to N
>  * line Nuse line discipline N
>min N with -icanon, set N characters minimum for a completed read
>ospeed N  set the output speed to N
>  * rows Ntell the kernel that the terminal has N rows
>  * size  print the number of rows and columns according to the kernel
>speed print the terminal speed
>time Nwith -icanon, set read timeout of N tenths of a second
> 
> Control settings:
>[-]clocal disable modem control signals
>[-]cread  allow input to be received
>  * [-]crtsctsenable RTS/CTS handshaking
>csN   set character size to N bits, N in [5..8]
>[-]cstopb use two stop bits per character (one with '-')
>[-]hupsend a hangup signal when the last process closes the tty
>[-]hupcl  same as [-]hup
>[-]parenb generate parity bit in output and expect parity bit in input
>[-]parodd set odd parity (or even parity with '-')
>  * [-]cmspar use "stick" (mark/space) parity
> 
> Input settings:
>[-]brkint breaks cause an interrupt signal
>[-]icrnl  translate carriage return to newline
>[-]ignbrk ignore break characters
>[-]igncr  ignore carriage return
>[-]ignpar ignore characters with parity errors
>  * [-]imaxbelbeep and do not flush a full input buffer on a character
>[-]inlcr  translate newline to carriage return
>[-]inpck  enable input parity checking
>[-]istrip clear high (8th) bit of input characters
>  * [-]iutf8  assume che i caratteri in ingresso siano codificati UTF-8
>  * [-]iuclc  translate uppercase characters to lowercase
>  * [-]ixany  let any character restart output, not only start character
>[-]ixoff  enable sending of start/stop characters
>[-]ixon   enable XON/XOFF flow control
>[-]parmrk mark parity errors (with a 255-0-character sequence)
>[-]tandem same as [-]ixoff
> 
> Output settings:
>  * bsN   backspace delay style, N in [0..1]
>  * crN   carriage return delay style, N in [0..3]
>  * ffN   form feed delay style, N in [0..1]
>  * nlN   newline delay style, N in [0..1]
>  * [-]ocrnl  translate carriage return to newline
>  * [-]ofdel  use delete characters for fill instead of NUL characters
>  * [-]ofill  use fill (padding) characters instead of timing for delays
>  * [-]olcuc  translate lowercase characters to uppercase
>  * [-]onlcr  translate newline to carriage return-newline
>  * [-]onlret newline performs a carriage return
>  * 

Bug#833932: coreutils: ls -b fails to show Unicode directional characters

[Pádraig Brady]

On 10/08/16 16:15, Peter Ludikovsky wrote:
> 
> 
> Am 10.08.2016 um 16:51 schrieb Pádraig Brady:
>> On 10/08/16 15:21, Peter Ludikovsky wrote:
>>> Package: coreutils
>>> Version: 8.23-4
>>> Severity: normal
>>>
>>> Dear Maintainer,
>>>
>>> This came up due to a posting on debian-user-german [1]. Apparently
>>> certain Unicode characters, at least LEFT-TO-RIGHT EMBEDDING [2] and
>>> RIGHT-TO-LEFT EMBEDDING [3] do not trigger the escape code display for
>>> ls with the -b option.
>>>
>>> An example script is attached, output:
>>>
>>> $ bash unicode_bidir_test.sh 
>>> + touch LTR
>>> + touch RTL
>>> + /bin/ls -l
>>> total 4
>>> -rw-r--r--. 1 peter peter   0 Aug 10 14:00 LTR
>>> -rw-r--r--. 1 peter peter   0 Aug 10 14:00 RTL
>>> -rw-r--r--. 1 peter peter 148 Aug 10 14:00 unicode_bidir_test.sh
>>> + /bin/ls -lb
>>> total 4
>>> -rw-r--r--. 1 peter peter   0 Aug 10 14:00 LTR
>>> -rw-r--r--. 1 peter peter   0 Aug 10 14:00 RTL
>>> -rw-r--r--. 1 peter peter 148 Aug 10 14:00 unicode_bidir_test.sh
>>> + /bin/ls -lb LTR
>>> /bin/ls: cannot access LTR: No such file or directory
>>> + /bin/ls -lb LTR
>>> -rw-r--r--. 1 peter peter 0 Aug 10 14:00 LTR
>>> + /bin/ls -lb RTL
>>> /bin/ls: cannot access RTL: No such file or directory
>>> + /bin/ls -lb RTL
>>> -rw-r--r--. 1 peter peter 0 Aug 10 14:00 RTL
>>>
>>> The expected output would be that those characters be shown, as they are
>>> relevant when accessing a file on the command line.
>>>
>>> [1] https://lists.debian.org/debian-user-german/2016/08/msg00049.html
>>> [2] http://www.fileformat.info/info/unicode/char/202a/index.htm
>>> [3] http://www.fileformat.info/info/unicode/char/202b/index.htm
>>>
>>> -- System Information:
>>> Debian Release: 8.5
>>>   APT prefers stable-updates
>>>   APT policy: (500, 'stable-updates'), (500, 'stable')
>>> Architecture: amd64 (x86_64)
>>>
>>> Kernel: Linux 3.16.0-4-amd64 (SMP w/1 CPU core)
>>> Locale: LANG=C, LC_CTYPE=C (charmap=ANSI_X3.4-1968)
>>> Shell: /bin/sh linked to /bin/dash
>>> Init: systemd (via /run/systemd/system)
>>>
>>> Versions of packages coreutils depends on:
>>> ii  libacl1  2.2.52-2
>>> ii  libattr1 1:2.4.47-2
>>> ii  libc62.19-18+deb8u4
>>> ii  libselinux1  2.3-2
>>>
>>> coreutils recommends no packages.
>>>
>>> coreutils suggests no packages.
>>>
>>> -- no debconf information
>>
>> Is your locale really "C" ?
>> With mine set to "C" I get:
>>
>> $ LANG=C ls -l
>> -rw-rw-r--. 1 padraig padraig 0 Aug 10 15:43 ???LTR
>> -rw-rw-r--. 1 padraig padraig 0 Aug 10 15:43 ???RTL
>>
>> $ LANG=C ls -lb
>> -rw-rw-r--. 1 padraig padraig 0 Aug 10 15:43 \342\200\252LTR
>> -rw-rw-r--. 1 padraig padraig 0 Aug 10 15:43 \342\200\253RTL
>>
>>
>> With the new quoting in version 8.25 you'll get a directly
>> copy and pasteable representation like:
>>
>> $ LANG=C ~/git/coreutils/src/ls -l
>> -rw-rw-r--. 1 padraig padraig 0 Aug 10 15:43 ''$'\342\200\252''LTR'
>> -rw-rw-r--. 1 padraig padraig 0 Aug 10 15:43 ''$'\342\200\253''RTL'
>>
>>
>> I'll need to experiment a bit with non "C" locale handling,
>> and with various terminals, to see how best to handle this case.
>>
>> thanks,
>> Pádraig
>>
> 
> Not really, I haven't set any locale on my servers intentionally. Or
> rather, left it at the "POSIX"(?) default during d-i.
> $ localectl status
>System Locale: n/a
> 
>VC Keymap: n/a
>   X11 Layout: de
>X11 Model: pc105
>  X11 Variant: nodeadkeys
> $ cat /etc/default/locale
> #LANG="C"
> $ env | grep LANG
> $ env | grep LC_
> $
> 
> With both LC_ALL=C and LANG=C it shows at least some indication that
> there are other characters. But why not when no explicit locale has been
> set?

Maybe because it's UTF8 based?
I also noticed that in gnome-terminal you can copy/paste the hidden chars
by also selecting the leading space on the file name (though that's certainly 
not obvious).
xterm gives a visual indication of an extra char, and allows selecting it.
So there is an overlap here with terminal handling of the RTL chars

Bug#833932: coreutils: ls -b fails to show Unicode directional characters

[Pádraig Brady]

On 10/08/16 15:21, Peter Ludikovsky wrote:
> Package: coreutils
> Version: 8.23-4
> Severity: normal
> 
> Dear Maintainer,
> 
> This came up due to a posting on debian-user-german [1]. Apparently
> certain Unicode characters, at least LEFT-TO-RIGHT EMBEDDING [2] and
> RIGHT-TO-LEFT EMBEDDING [3] do not trigger the escape code display for
> ls with the -b option.
> 
> An example script is attached, output:
> 
> $ bash unicode_bidir_test.sh 
> + touch LTR
> + touch RTL
> + /bin/ls -l
> total 4
> -rw-r--r--. 1 peter peter   0 Aug 10 14:00 LTR
> -rw-r--r--. 1 peter peter   0 Aug 10 14:00 RTL
> -rw-r--r--. 1 peter peter 148 Aug 10 14:00 unicode_bidir_test.sh
> + /bin/ls -lb
> total 4
> -rw-r--r--. 1 peter peter   0 Aug 10 14:00 LTR
> -rw-r--r--. 1 peter peter   0 Aug 10 14:00 RTL
> -rw-r--r--. 1 peter peter 148 Aug 10 14:00 unicode_bidir_test.sh
> + /bin/ls -lb LTR
> /bin/ls: cannot access LTR: No such file or directory
> + /bin/ls -lb LTR
> -rw-r--r--. 1 peter peter 0 Aug 10 14:00 LTR
> + /bin/ls -lb RTL
> /bin/ls: cannot access RTL: No such file or directory
> + /bin/ls -lb RTL
> -rw-r--r--. 1 peter peter 0 Aug 10 14:00 RTL
> 
> The expected output would be that those characters be shown, as they are
> relevant when accessing a file on the command line.
> 
> [1] https://lists.debian.org/debian-user-german/2016/08/msg00049.html
> [2] http://www.fileformat.info/info/unicode/char/202a/index.htm
> [3] http://www.fileformat.info/info/unicode/char/202b/index.htm
> 
> -- System Information:
> Debian Release: 8.5
>   APT prefers stable-updates
>   APT policy: (500, 'stable-updates'), (500, 'stable')
> Architecture: amd64 (x86_64)
> 
> Kernel: Linux 3.16.0-4-amd64 (SMP w/1 CPU core)
> Locale: LANG=C, LC_CTYPE=C (charmap=ANSI_X3.4-1968)
> Shell: /bin/sh linked to /bin/dash
> Init: systemd (via /run/systemd/system)
> 
> Versions of packages coreutils depends on:
> ii  libacl1  2.2.52-2
> ii  libattr1 1:2.4.47-2
> ii  libc62.19-18+deb8u4
> ii  libselinux1  2.3-2
> 
> coreutils recommends no packages.
> 
> coreutils suggests no packages.
> 
> -- no debconf information

Is your locale really "C" ?
With mine set to "C" I get:

$ LANG=C ls -l
-rw-rw-r--. 1 padraig padraig 0 Aug 10 15:43 ???LTR
-rw-rw-r--. 1 padraig padraig 0 Aug 10 15:43 ???RTL

$ LANG=C ls -lb
-rw-rw-r--. 1 padraig padraig 0 Aug 10 15:43 \342\200\252LTR
-rw-rw-r--. 1 padraig padraig 0 Aug 10 15:43 \342\200\253RTL


With the new quoting in version 8.25 you'll get a directly
copy and pasteable representation like:

$ LANG=C ~/git/coreutils/src/ls -l
-rw-rw-r--. 1 padraig padraig 0 Aug 10 15:43 ''$'\342\200\252''LTR'
-rw-rw-r--. 1 padraig padraig 0 Aug 10 15:43 ''$'\342\200\253''RTL'


I'll need to experiment a bit with non "C" locale handling,
and with various terminals, to see how best to handle this case.

thanks,
Pádraig

Bug#833710: tail -F /var/log/messages missed the rotation

[Pádraig Brady]

On 08/08/16 08:28, Harald Dunkel wrote:
> Package: coreutils
> Version: 8.23-4
> 
> I am running "tail -f --retry /var/log/messages" in a terminal
> for >10 days. Since inotify is available the --max-unchanged-stats
> option is not set, as recommended in the man page.
> 
> Problem: tail ignored the log file rotation last weekend. lsof -p
> shows that it is watching /var/log/messages.1 now:
> 
> % lsof -p 2905
> COMMAND  PIDUSER   FD   TYPE DEVICE SIZE/OFF  NODE NAME
> tail2905 hdunkel  cwdDIR   0,4020480 189573782 /home/hdunkel
> tail2905 hdunkel  rtdDIR  259,0 4096 2 /
> tail2905 hdunkel  txtREG  259,064232657851 /usr/bin/tail
> tail2905 hdunkel  memREG  259,0  1738176   1724334 
> /lib/x86_64-linux-gnu/libc-2.19.so
> tail2905 hdunkel  memREG  259,0   140928   1724331 
> /lib/x86_64-linux-gnu/ld-2.19.so
> tail2905 hdunkel0u   CHR  136,1  0t0 4 /dev/pts/1
> tail2905 hdunkel1u   CHR  136,1  0t0 4 /dev/pts/1
> tail2905 hdunkel2u   CHR  136,1  0t0 4 /dev/pts/1
> tail2905 hdunkel3r   REG  259,0   141711264950 /var/log/messages.1
> tail2905 hdunkel4r     0,110  8294 anon_inode
> 
> Obviously the inotify support did not work for renamed
> files.

There were lots of tail inotify fixes in 8.24
with -f now working with renamed files being explicitly listed

Bug#832753: coreutils: stdbuf is not usable for multiarch tools (eg. i386 on amd64)

[Pádraig Brady]

On 28/07/16 15:20, Norbert Lange wrote:
> Package: coreutils
> Version: 8.25-2
> Severity: normal
> 
> Dear Maintainer,
> 
> I am trying to use stdbuf tool on a 32bit Binary, this will result
> in a failure:
> 
> ERROR: ld.so: object '/usr/lib/x86_64-linux-gnu/coreutils/libstdbuf.so' from 
> LD_PRELOAD cannot be preloaded (wrong ELF class: ELFCLASS64): ignored.
> 
> To make this work seemlessly, 
> LD_PRELOAD would need to be used with the plain library libstdbuf.so
> and the library would need to be installed in the proper locations
> for 64 and 32bit.
> 
> Likely this would mean using a seperate binary package for the library to
> allow co-installing both?

Notes on this upstream at http://bugs.gnu.org/8960

Stuff I noted previously...

You can't install coreutils 32 bit and 64 bit together,
so to support this the libs would need to be separated out to a separate 
package.
I.E. one would have a coreutils package that depended on
coreutils-lib.i686 and coreutils-lib.x86_64, which would
install to /usr/lib{,64}/coreutils/libstbuf.so respectively.
Then you would have the flexibility to set both LD_PRELOAD_{32,64}

Currently we don't support separate (per arch) libs because of:
http://git.sv.gnu.org/gitweb/?p=coreutils.git;a=commitdiff;h=e23f1795
So we'd have to revert that essentially as automake
is too restrictive here and may support the orig var in newer versions?

We might also have to adjust setting of PKGLIBEXECDIR at build time?

Is this complexity worth it? How often does 64 bit coreutils
need to control buffering for 32 bit programs?
We've not been asked for that flexibility at present.

Bug#816703: regression: "mv from to" now fails when both are the same file

[Pádraig Brady]

On 06/03/16 19:46, Marc Lehmann wrote:

On Sun, Mar 06, 2016 at 02:06:47PM -0700, Bob Proulx  wrote:

The kernel system call rename(2) returns 0 for SUCCESS.  And yet the
operation definitely did not succeed.


It did - this is how rename must behave according to POSIX.


I find the above result surprising!


Right, which isn't asurprising :)


So why did this appear to work previously with previous versions of
coreutils mv command?  Because 'mv' didn't previously rename(2) the
files at all.


It worked because I reported this as a bug long ago, and it was changed.

POSIX allows three behaviours for mv in this case:

1. silently succeed
2. issue a disgnostic
3. unlink the source

So very old coreutils used 1, then switched to 3, and now switched to 2,
essentially to please apple users (the race condition could be fixed by a
rename/unlink combination).


place.  The utilities should not hide race condition behavior.


It could be done without races by first renaming the source to a third
name for example, one that doesn't conflict with a concurrent mv (bceause
it can't know that name).

Nevertheless, mv needs to do more than one system call to detect this issue,
so the change doesn't save on system calls.

The only way to save systems calls would be to silently ignore3 this case
(which is allowed by posix).


the script author still wants the racy behavior then the script author
can perform the same actions explicitly and get identical behavior.)


(which script author?)

If you refer to me, this has nothing to do with scripts, it#s simply me
dpoing mv commands sometimes, and expecting mv to do be useful on my
Debian GNU/Linux box, without thinbking too much about apple users and
their problems because their filesystem isn't posix-compliant.


Which means it was historical behavior on multiple kernels originally
and standardized specifically for the reason that changing it would
make portable programs impossible.


Nothing a rename3 or renameat2 (which already exists) couldn't improve with a
flag. In fact, renameat2 with RENAME_EXCHANGE can already be useful. and
renameat2 with RENAME_NOREPLACE can be useful to save on syscalls to detect
this case.


I hope this additional information helps understand what is happening.


I'm not sure anybody was unclear about it. The bug was about keeping
the existing behaviour as a useful extension to POSIX. Unsurprising
mv behaviour would certainly be a useful extension, but upstream
apparently decided to sacrifice quality for better non-posixs (apple)
compatibility. Tough game, but so be it. I'll just keep a copy of mv for
personal use, problem solved.

I do disagree with your mail's implicationh that this is a technical issue,
though - it isn't. It's a policy issue - what weighs more, surprising
behaviour on non-posix filesystems vs. rational behaviour on posix
filesystems, and the decision went to support non-posix filesystems.



It's not to appease case retentive file systems.
There is a race between multiple mv calls issuing unlink();
The race must be fixed in the kernel.
I proposed a new flag to renameat() on the kernel mailing list,
to get the kernel to handle it. When/If that becomes available,
mv can safely restore this behaviour.

Pádraig

Bug#816703: regression: "mv from to" now fails when both are the same file

[Pádraig Brady]

On 03/03/16 21:24, Marc Lehmann wrote:

Package: coreutils
Version: 8.25-2
Severity: normal

Dear Maintainer,

as an extension, coreutils supported moving one file to another, when both
refer to the same file.

This support has been lost after upgrading from 8.23-4 (stable) to
8.25-2. That is, in 8.25-2:

# touch a; ln -f a b; mv a b
mv: 'a' and 'b' are the same file

and both names still exist. While in 8.24-4, the "obvious" happens, i.e.
only "b" is left after the mv command.

It would be wonderful if this extension could somehow survive, as the
new behaviour is rather counter-intuitive (one expects that "mv a b", if
possible, makes b refer to a, and a is gone afterwards).

Thanks!

-- System Information:
Debian Release: 8.3
   APT prefers stable
   APT policy: (990, 'stable'), (500, 'unstable'), (500, 'testing'), (500, 
'oldstable'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.1.18-040118-generic (SMP w/12 CPU cores)
Locale: LANG=C, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash
Init: systemd (via /run/systemd/system)

Versions of packages coreutils depends on:
ii  libacl1  2.2.52-2
ii  libattr1 1:2.4.47-2
ii  libc62.21-9
ii  libselinux1  2.4-3

coreutils recommends no packages.

coreutils suggests no packages.


This changed in v8.24 with the NEWS:

  mv no longer supports moving a file to a hardlink, instead issuing an error.
  The implementation was susceptible to races in the presence of multiple mv
  instances, which could result in both hardlinks being deleted.  Also on case
  insensitive file systems like HFS, mv would just remove a hardlinked 'file'
  if called like `mv file File`.  The feature was added in coreutils-5.0.1.

Bug#813164: coreutils: ls suddenly quotes output

[Pádraig Brady]

On 12/02/16 01:47, Jaroslav Skarvada wrote:
> Hi,
> 
> please revert this ugly change, it's confusing and against GNU coding 
> standards [1]:
> 
>> Likewise, please don’t make the behavior of a command-line program depend
>> on the type of output device it gets as standard output or standard input

ls already changed output depending on if output is a tty
We really don't want to adhere to that guideline for ls.

Bug#813164: alias ls='ls -N' is not a solution

[Pádraig Brady]

On 03/02/16 06:05, Adam Borowski wrote:
> Also, the proposed workaround, "alias ls='ls -N'" doesn't act reasonably.
> It disables _all_ quoting, including nasty unprintable characters.  When the
> output goes to the terminal, it is meant to be read by a human.  Humans can
> read spaces and apostrophes just fine, they can't read \1 or broken UTF-8.

`ls -N` does revert to the previous behaviour.
I.E. weird chars are replaced with ?

Bug#813164: coreutils: ls suddenly quotes output

[Pádraig Brady]

On 02/02/16 13:19, Jamie Heilman wrote:
> This behavior needs to be reverted.  There are too many assumptions
> being made, the quoting used is shell-specific, and not universally
> supported.  For example, consider a file who's name contains a tab,
> like "ab".
> 
> $ ls
> 'a'$'\t''b'
> 
> OK, so that syntax is supported by bash and zsh, so if you're using
> one of those shells, maybe you know what it means, and you can cut and
> paste that and make use of it, but in csh or dash, it doesn't mean the
> same thing.

$'...' is in the process of being POSIX standardized.

Even when the shell doesn't support it directly yet,
surely \t is better than ?

Bug#813164: coreutils: ls suddenly quotes output

[Pádraig Brady]

On 30/01/16 15:45, Christoph Anton Mitterer wrote:
> Hey.
> 
> I've also just stumbled over this... while the idea is nice in
> principle, I think it's quite dangerous as well... even if behaviour is
> preserved when output goes to a terminal.
> 
> a) The quotation alone doesn't necessarily help with copy,
> depending on where you paste.
> 
> b) When the pasted content is actually further to be processed by e.g.
> old scripts (which expect perhaps one *unquoted* name per line) things
> will fail.
> 
> c) We now have what you see is NOT what you get [when stdout is not a
> terminal]
> People do use ls in all it's variations in countless of scripts, and
> they build up their scripts by first trying it out on the terminal to
> see what they get is what they want to have in the script.
> But now, what they get is different in both.
> And the change isn't necessarily one that get's noticed, but it still
> may lead to all kinds of garbage, wrong files being deleted and so on.
> 
> Consider a user writes a script that shall deleted all files from
> 2012... a stupid solution may be something like:
> rm -f "$( ls -1l | grep  | sed 
> "some pattern that removes everything up the filename" | xargs )"
> On the terminal this may give proper:
> rm -f 'file with spaces mythesis.pdf'
> in the script this would give
> rm -f file with spaces mythesis.pdf

ls output really isn't amenable to further processing.
That's what find(1) is focused on.
Though I see what you're saying.

Let's try and force your example into something concrete.
You're saying that people might assume files are always quoted,
but I'm explicitly adding the --quoting option below to
see if users did notice to do that, whether further
processing of ls is valid anyway.

# Very carefully format ls output for processing
ls -ln --quoting=shell-escape --block-size=1 --time-style='+%m' |
# Get rid of the first few space aligned fields while not
# impacting consecutive spaces in a file name.
sed 's/\([^ ]* \)\{4\}//; s/^ *[^ ]* //' |
# Pick files in January
grep ^01 |
# Get the quoted file names
cut -d' ' -f2-
# Process them
xargs ...


Now this mostly works, right up until xargs,
but that will only strip single quotes.
It will not process $'\n' quoting that ls may also output.

Also given the extreme awkwardness in the filtering above,
users are really going to be and should be using find(1) instead.

So yes you have a valid point,
but it's quite a contrived situation.

cheers,
Pádraig.

Bug#813164: coreutils: ls suddenly quotes output

[Pádraig Brady]

On 29/01/16 16:50, Thorsten Glaser wrote:
> Package: coreutils
> Version: 8.25-1
> Severity: minor
> 
> tglase@tglase-nb:~ $ mkdir -p foo/{a,b\ c}; cd foo; /bin/ls   
>   
> a  'b c'
> 
> ’nuff said… this *should* be:
> 
> (pbuild17294)root@tglase-nb:/# mkdir -p foo/{a,b\ c}; cd foo; /bin/ls
> a  b c

A few points about the change.

- It only happens when outputting to terminals
- It disambiguates the output for users
- Output can be pasted back in the shell for further processing
- Users can get back to the old format by adding -N to their ls alias

cheers,
Pádraig

Bug#810539: coreutils: /usr/bin/stat (stat -c "%a" returns "0" instead of "000")

[Pádraig Brady]

On 09/01/16 18:29, lopiuh wrote:
> Package: coreutils
> Version: 8.23-4
> Severity: normal
> 
> Dear Maintainer,
> 
> stat -c "%a" returns "0" instead of "000" in case of getting octal 
> permissions of a directory with permissions 000 (no permissions). No 
> permissions are set in order to prevent users write into unmounted mountpoints
> 
> # ls /mnt/ -lath
> insgesamt 16K
> drwxr-xr-x 1 root root  14 Jan  9 19:12 .
> d- 1 root root   0 Jan  9 19:12 data
> drwxr-xr-x 1 root root 198 Jan  9 19:00 ..
> 
> # stat -c "%a" /mnt/data/
> 0
> 
> Thanks
> 
> lopiuh
> 
> 
> -- System Information:
> Debian Release: stretch/sid
>   APT prefers testing-updates
>   APT policy: (500, 'testing-updates'), (500, 'testing')
> Architecture: amd64 (x86_64)
> 
> Kernel: Linux 4.3.0-1-amd64 (SMP w/4 CPU cores)
> Locale: LANG=de_DE.UTF-8, LC_CTYPE=de_DE.UTF-8 (charmap=UTF-8)
> Shell: /bin/sh linked to /bin/dash
> Init: systemd (via /run/systemd/system)
> 
> Versions of packages coreutils depends on:
> ii  libacl1  2.2.52-2
> ii  libattr1 1:2.4.47-2
> ii  libc62.21-6
> ii  libselinux1  2.4-3
> 
> coreutils recommends no packages.
> 
> coreutils suggests no packages.


Control over the precise format of the number is given to the user.
So you can get the behavior you desire like:

$ stat -c %#03a data
000

$ find data -maxdepth 0 -printf %#03m\\n
000

We probably should mention those flags in the upstream docs for %a.
I'll do that now.

thanks,
Pádraig

Bug#806321: coreutils: please make the build reproducible (timestamps)

[Pádraig Brady]

On 26/11/15 13:59, Jérémy Bobbio wrote:
> Source: coreutils
> Version: 8.23-4
> Severity: wishlist
> Tags: patch
> User: reproducible-bui...@lists.alioth.debian.org
> Usertags: timestamps
> 
> Hi!
> 
> While working on the “reproducible builds” effort [1], we have noticed
> that coreutils could not be built reproducibly. The build process uses
> help2man to create some of manpages.
> 
> help2man author added support for the SOURCE_DATE_EPOCH environment
> variable [2] in version 1.47.1 which makes it possible to use a
> pre-defined value instead of the time of the build.
> 
> As coreutils source currently contains an old version of help2man, the
> attached patch will copy the version in the help2man Debian package
> before running ./configure on top of setting SOURCE_DATE_EPOCH to the
> date of the latest debian/changelog entry.
> 
> Once applied, coreutils can be built reproducibly in our current
> experimental framework.
> 
>  [1]: https://wiki.debian.org/ReproducibleBuilds
>  [2]: https://wiki.debian.org/ReproducibleBuilds/TimestampsProposal
> 

Note with the soon to be released coreutils 8.25 it will be better to
leave the bundled help2man in place, but continue
to set SOURCE_DATE_EPOCH as you do.

http://git.sv.gnu.org/gitweb/?p=coreutils.git;a=commitdiff;h=c1b3d658

cheers,
Pádraig

Bug#806947: coreutils: improve expr with huge numbers by GMP

[Pádraig Brady]

On 03/12/15 11:09, Hideki Yamane wrote:
> Package: coreutils
> Severity: minor
> Tags: patch
> 
> Hi,
> 
> Here's a tiny B-D patch to build coreutils with GMP(https://gmplib.org/), 
> it can manage huge numbers with expr.
> 
> * Without GMP 
> 
> $ expr `echo 2 ^ 62 | bc` + 0
> 4611686018427387904
> $ expr `echo 2 ^ 63 | bc` + 0
> expr: 9223372036854775808: Numerical result out of range
> 
> * With GMP (libgmp-dev)
> 
> $ expr `echo 2 ^ 62 | bc` + 0
> 4611686018427387904
> $ expr `echo 2 ^ 63 | bc` + 0
> 9223372036854775808
> 
> Good, no "out of range" :)
> 
> Other distro such as RHEL/CentOS also enabled this and ship it 
> for a long time and I cannot see any hurm for it, and we can get 
> fix from them if there would be.
> 
> Could you consider to apply it in next upload, please?

+1
factor(1) also benefits from GMP:

$ grep GMP src/local.mk
# for various GMP functions
src_expr_LDADD += $(LIB_GMP)
src_factor_LDADD += $(LIB_GMP)

Another thing to consider is depending on openssl
and using ./configure --with-openssl which will
get significantly faster sha*sum implementations.

cheers,
Pádraig

Bug#806947: coreutils: improve expr with huge numbers by GMP

[Pádraig Brady]

On 03/12/15 13:21, Michael Stone wrote:
> On Thu, Dec 03, 2015 at 11:43:59AM +0000, Pádraig Brady wrote:
>> Another thing to consider is depending on openssl
>> and using ./configure --with-openssl which will
>> get significantly faster sha*sum implementations.
> 
> In addition to the concern about making openssl a required component, 
> I'm surprised coreutils uses openssl rather than gnutls, and doesn't 
> have an SSL linking exception in the license.

openssl is where dev of optimized versions of routines is concentrated.
This issue is discussed somewhat at:
http://lists.gnu.org/archive/html/bug-gnulib/2013-12/msg00021.html
It would be useful to provide a --with-crypto={openssl,gnutls,...} option.

cheers,
Pádraig

Bug#804338: coreutils: provide a clean way for optional global cp --reflink=auto

[Pádraig Brady]

On 07/11/15 19:22, Christoph Anton Mitterer wrote:
> So do you see it coming that this is actually "solved" (meaning refcopy
> works per default out of the box for e.g. btrfs users) upstream ... or
> is it worth to look into making it configurable at the distro level, as
> I proposed above?

I don't see this as something a distro would want to diverge on,
nor something a distro would change in a minor release.
I.E. we should solve this upstream.

cheers,
Pádraig.