Re: shred bug? [was: Unidentified subject!]

[Michael Stone]

On Sun, Feb 11, 2024 at 08:02:12AM +0100, to...@tuxteam.de wrote:

What Thomas was trying to do is to get a cheap, fast random number
generator. Shred seems to have such.


You're better off with /dev/urandom, it's much easier to understand what 
it's trying to do, vs the rather baroque logic in shred. In fact, 
there's nothing in shred's documenation AFAICT that suggests it should 
be used as a random number generator. For pure speed, playing games with 
openssl enc and /dev/zero will generally win. If speed doesn't matter, 
we're back to /dev/urandom as the simplest and most direct solution.


FWIW, the main use for shred in 2024 is: to be there so someone's old 
script doesn't break. There's basically no good use case for it, and it 
probably shouldn't have gotten into coreutils in the first place. The 
multipass pattern stuff is cargo-cult voodoo--a single overwrite with 
zeros will be as effective as anything else--and on modern 
storage/filesystems there's a good chance your overwrite won't overwrite 
anything anyway. Probably the right answer is a kernel facility 
(userspace can't guarantee anything). If you're really sure that 
overwrites work on your system, `shred -n0 -z` will be the fastest way 
to do that. The docs say don't do that because SSDs might optimize that 
away, but SSDs probably aren't overwriting anything anyway (also 
mentioned in the docs). ¯\_(ツ)_/¯

Re: shred bug? [was: Unidentified subject!]

[David Wright]

On Tue 13 Feb 2024 at 11:21:08 (-0500), Greg Wooledge wrote:
> On Tue, Feb 13, 2024 at 09:35:11AM -0600, David Wright wrote:
> > On Tue 13 Feb 2024 at 07:15:48 (-0500), Greg Wooledge wrote:
> > > On Mon, Feb 12, 2024 at 11:01:47PM -0600, David Wright wrote:
> > > > … but not much. For me, "standard output" is /dev/fd/1, yet it seems
> > > > unlikely that anyone is going to use >&1 in the manner of the example.
> > > 
> > > Standard output means "whatever file descriptor 1 points to".  That
> > > could be a file, a pipe, a terminal (character device), etc.
> > 
> > Why pick on 1?
> 
> It's the definition.  Standard input is FD 0, standard output is FD 1,
> and standard error is FD 2.
> 
> > . It demonstrates the shell syntax element required (&) in order to
> >   avoid truncating the file, rather than shred overwriting it.
> 
> You are confused.  You're making assumptions about shell syntax that
> are simply not true.

You're right. I was looking too hard at the right side of the > and
neglecting the implied left side. It's always worth running these
things past your eyes. Thanks for the clear exposition that followed.

Cheers,
David.

Re: shred bug? [was: Unidentified subject!]

[tomas]

On Tue, Feb 13, 2024 at 01:03:44PM -0800, David Christensen wrote:
> On 2/13/24 09:40, debian-u...@howorth.org.uk wrote:
> > Greg Wooledge  wrote:
> > 
> > > Shred will determine the size of the file, then write data to the
> > > file, rewind, write data again, etc.  On a traditional hard drive,
> > > that will overwrite the original private information.  On modern
> > > devices, it may not.
> > 
> > Thanks for the excellent explanation :)
> > 
> > One nitpick. You say "On a traditional hard drive, that will overwrite
> > the original private information" but that's not quite true. It also
> > needs to be a "traditional" file system! That is, not journalled or COW.
> > 
> > So nowadays I would expect shred not to work unless you got very
> > lucky, or planned carefully.
> 
> 
> Perhaps zerofree(8)?

On a SATA, it won't get at (some) of the spare blocks, since it
doesn't know that they are there.

If your data is so sensitive that you don't want it to escape,
your best bet seems to plan ahead and not let it hit your media
unencrypted.

Use LUKS. And oh, use argon2id as key derivation function [1]
these days.

Cheers
[1] https://mjg59.dreamwidth.org/66429.html
-- 
t


signature.asc
Description: PGP signature

Re: shred bug? [was: Unidentified subject!]

[gene heskett]

On 2/13/24 16:00, David Christensen wrote:

On 2/13/24 11:31, gene heskett wrote:
Next experiment is a pair of 4T Silicon Power SSD's When they & the 
startech usb3 adapters arrive.  I'll get that NAS built for amanda yet.



2.5" SATA SSD's and SATA to USB adapter cables for $187.97 + $10.99 = 
$198.96 each set?


https://www.amazon.com/dp/B0BVLRFFWQ

https://www.amazon.com/dp/B00HJZJI84


Why not external USB drives for $192.99?

https://www.amazon.com/dp/B0C6XVZS4K


For $7 more, you can get the "Pro edition" in black with two cables:

https://www.amazon.com/dp/B0C69QD5NK


You could plug those into the two USB-C 3.1 Gen 2 ports on your Asus 
PRIME Z370-A II motherboard.


Maybe, but these sata types have the the mounting bolts the usb versions 
don't. And fits the drive adapters I already have that put both in one 
drive tray. Not to mention if Taiwan has a similar product, I tend to 
buy it.  So are the 4 2T gigastones I'll fill the next 2 drawers with so 
I should wind up with a 16T backup server's LVM. with a 1/2T Samsung 870 
as a holding disk.  Running a bpi-m5 headless, maybe < 20 watts.  Whats 
not to like?


David


.


Cheers, Gene Heskett, CET.
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis

Re: shred bug? [was: Unidentified subject!]

[David Christensen]

On 2/13/24 09:40, debian-u...@howorth.org.uk wrote:

Greg Wooledge  wrote:


Shred will determine the size of the file, then write data to the
file, rewind, write data again, etc.  On a traditional hard drive,
that will overwrite the original private information.  On modern
devices, it may not.


Thanks for the excellent explanation :)

One nitpick. You say "On a traditional hard drive, that will overwrite
the original private information" but that's not quite true. It also
needs to be a "traditional" file system! That is, not journalled or COW.

So nowadays I would expect shred not to work unless you got very
lucky, or planned carefully.



Perhaps zerofree(8)?


David

Re: shred bug? [was: Unidentified subject!]

[David Christensen]

On 2/13/24 11:31, gene heskett wrote:
Next experiment is a pair of 4T 
Silicon Power SSD's When they & the startech usb3 adapters arrive.  I'll 
get that NAS built for amanda yet.



2.5" SATA SSD's and SATA to USB adapter cables for $187.97 + $10.99 = 
$198.96 each set?


https://www.amazon.com/dp/B0BVLRFFWQ

https://www.amazon.com/dp/B00HJZJI84


Why not external USB drives for $192.99?

https://www.amazon.com/dp/B0C6XVZS4K


For $7 more, you can get the "Pro edition" in black with two cables:

https://www.amazon.com/dp/B0C69QD5NK


You could plug those into the two USB-C 3.1 Gen 2 ports on your Asus 
PRIME Z370-A II motherboard.



David

Re: shred bug? [was: Unidentified subject!]

[gene heskett]

On 2/13/24 14:44, Thomas Schmitt wrote:

Hi,

Gene Heskett wrote:

Next experiment is a pair of 4T Silicon Power SSD's


When f3 has (hopefully) given its OK, the topic of a full write-and-read
test will come up again. I'm looking forward to all the spin-off topics.

I'll have to admit it has been quite educational. Now, can I remember it 
next week? YTBD.>

Have a nice day :)


You too Thomas.
Take care and stay well.


Thomas


Cheers, Gene Heskett, CET.
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis

Re: shred bug? [was: Unidentified subject!]

[Thomas Schmitt]

Hi,

Gene Heskett wrote:
> Next experiment is a pair of 4T Silicon Power SSD's

When f3 has (hopefully) given its OK, the topic of a full write-and-read
test will come up again. I'm looking forward to all the spin-off topics.


Have a nice day :)

Thomas

Re: shred bug? [was: Unidentified subject!]

[Thomas Schmitt]

Hi,

Greg Wooledge wrote:
> Heh.  Don't forget your own attempts to use a shredder as a PRNG stream.

My original idea was to watch a minimal shred run by teeing its work into
a checksummer.

But then topic drift came in. So we got a farm show of random generators
and a discussion about what exactly is a bug in shred's documentation.
Plus the shell programming webinar. And a diagnosis about a rightously
failed attempt to change the partition table type from MBR/DOS to GPT.

And this all because Gene Heskett was adventurous enough to buy a cheap
fake USB disk.


Have a nice day :)

Thomas

Re: shred bug? [was: Unidentified subject!]

[gene heskett]

On 2/13/24 12:56, Thomas Schmitt wrote:

Hi,

Greg Wooledge wrote:

Let me write out the example again, but with the bug fixed, and then
explain what each line does, [... lecture about advanced shell
programming ...]


And this all because Gene Heskett was adventurous enough to buy a cheap
fake USB disk. :))


Guilty as charged, Thomas. My advantage is that it won't affect the 
length of the ladder up my side of the hog.  If I save someone else from 
getting bit by that fraud I'm pleased.  Next experiment is a pair of 4T 
Silicon Power SSD's When they & the startech usb3 adapters arrive.  I'll 
get that NAS built for amanda yet.



Have a nice day :)


You too.


Thomas


Cheers, Gene Heskett, CET.
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis

Re: shred bug? [was: Unidentified subject!]

[Greg Wooledge]

On Tue, Feb 13, 2024 at 06:54:58PM +0100, Thomas Schmitt wrote:
> Greg Wooledge wrote:
> > Let me write out the example again, but with the bug fixed, and then
> > explain what each line does, [... lecture about advanced shell
> > programming ...]
> 
> And this all because Gene Heskett was adventurous enough to buy a cheap
> fake USB disk. :))

Heh.  Don't forget your own attempts to use a shredder as a PRNG stream.

Shell redirections can be complicated, so this topic is going to
come up once in a while.  The example in the shred info page is fairly
unintuitive, so it deserves a bit of explanation.  I imagine most readers
who saw it simply accepted it as written, which is why the bug went
undiscovered for two decades.

Re: shred bug? [was: Unidentified subject!]

[Thomas Schmitt]

Hi,

Greg Wooledge wrote:
> Let me write out the example again, but with the bug fixed, and then
> explain what each line does, [... lecture about advanced shell
> programming ...]

And this all because Gene Heskett was adventurous enough to buy a cheap
fake USB disk. :))


Have a nice day :)

Thomas

Re: shred bug? [was: Unidentified subject!]

[debian-user]

Greg Wooledge  wrote:

> Shred will determine the size of the file, then write data to the
> file, rewind, write data again, etc.  On a traditional hard drive,
> that will overwrite the original private information.  On modern
> devices, it may not.

Thanks for the excellent explanation :)

One nitpick. You say "On a traditional hard drive, that will overwrite
the original private information" but that's not quite true. It also
needs to be a "traditional" file system! That is, not journalled or COW.

So nowadays I would expect shred not to work unless you got very
lucky, or planned carefully.

Re: shred bug? [was: Unidentified subject!]

[Greg Wooledge]

On Tue, Feb 13, 2024 at 09:35:11AM -0600, David Wright wrote:
> On Tue 13 Feb 2024 at 07:15:48 (-0500), Greg Wooledge wrote:
> > On Mon, Feb 12, 2024 at 11:01:47PM -0600, David Wright wrote:
> > > … but not much. For me, "standard output" is /dev/fd/1, yet it seems
> > > unlikely that anyone is going to use >&1 in the manner of the example.
> > 
> > Standard output means "whatever file descriptor 1 points to".  That
> > could be a file, a pipe, a terminal (character device), etc.
> 
> Why pick on 1?

It's the definition.  Standard input is FD 0, standard output is FD 1,
and standard error is FD 2.

> . It demonstrates the shell syntax element required (&) in order to
>   avoid truncating the file, rather than shred overwriting it.

You are confused.  You're making assumptions about shell syntax that
are simply not true.

> > > >A FILE of ‘-’ denotes standard output.  The intended use of this is
> > > >to shred a removed temporary file.  For example:
> > > > 
> > > >   i=$(mktemp)
> > > >   exec 3<>"$i"
> > > >   rm -- "$i"
> > > >   echo "Hello, world" >&3
> > > >   shred - >&3
> > > >   exec 3>-

> Ironic that it truncates a file, and then immediately warns against
> truncating a file instead of shredding it.

No.  This is not what it does (if we fix the bug).

Let me write out the example again, but with the bug fixed, and then
explain what each line does, because apparently there is a LOT of
misunderstanding here.

  i=$(mktemp)
  exec 3<>"$i"
  rm -- "$i"
  echo "Hello, world" >&3
  shred - >&3
  exec 3>&-

The first line runs mktemp(1), which is a GNU coreutils program that
creates a temporary file, and then writes its name to standard output.
The shell syntax grabs that name and stores it in the variable "i".

So, after line 1, we have an (empty) temporary file, which was created
by a child process that has terminated.  We have its name in a variable.

Creation of temporary files works a little bit differently in shell
scripts than it does in regular programs.  In most other languages,
you would call a library function that creates the temporary file
(keeping it open), optionally unlinks it, and returns the open file
descriptor to you for use.  But you can't do that in a shell script
that needs an external program to do the file creation.  So we have this
slightly clumsy approach.

The second line opens this file for reading and writing, and ensures
that file descriptor 3 points to it.  It's important to understand that
while "exec 3>$i" would have truncated the file's contents, "exec 3<>$i"
does not.  Of course, there wasn't any content to truncate, since it was
created empty, but that's not the important part.  The important part
is that this FD is opened for read+write, allowing the temporary file
to be used for storage *and* retrieval.  We aren't doing any retrieval
in this example, but it could be done, with specialized tools.

The third line unlinks the file from the file system.  However, the shell
still has an open file descriptor which points to the file.  Therefore,
the file is still accessible through this FD.  Its inode is not recycled,
and any blocks containing file content are not marked for reuse.

This "unlink before using" technique is traditional on Unix systems.
It allows you to bypass setting up an exit handler to clean up the
temporary file.  Once the open file descriptor is closed, the file
system will mark the inode and any blocks as ready for reuse.  Even if
the script is killed by SIGKILL, that cleanup will still happen.

The fourth line writes some content via the open file descriptor 3.  At
this point, our unlinked file now has data in it.  Presumably this data
is super private, and we don't want anyone to be able to recover it.
When the script exits, the open file descriptor will close, and the file
system will mark the file's blocks as reusable, but it won't *actually*
reuse them until something else comes along and claims them.  But that's
what shred is designed for.

The fifth line calls shred(1), instructing it to destroy the content
that's in the unlinked file.  Since the file is unlinked, it has no name,
and therefore shred can't be *given* a name.  However, we have a file
descriptor that points to it.  So, what we *can* do is point standard
output to the file (that's what >&3 does), and then tell shred to destroy
the file that's pointed to by stdout.

Shred will determine the size of the file, then write data to the file,
rewind, write data again, etc.  On a traditional hard drive, that will
overwrite the original private information.  On modern devices, it may
not.

Finally, the sixth line closes file descriptor 3.  Doing this frees the
file's inode and blocks, allowing them to be reused by a future program.

The key to understanding this example is a firm grasp of the shell's
redirection syntax and semantics.

Let's start with what people normally see:  3>filename

That opens "filename" for writing with truncation, and ensures that 

Re: shred bug? [was: Unidentified subject!]

[David Wright]

On Tue 13 Feb 2024 at 07:15:48 (-0500), Greg Wooledge wrote:
> On Mon, Feb 12, 2024 at 11:01:47PM -0600, David Wright wrote:
> > … but not much. For me, "standard output" is /dev/fd/1, yet it seems
> > unlikely that anyone is going to use >&1 in the manner of the example.
> 
> Standard output means "whatever file descriptor 1 points to".  That
> could be a file, a pipe, a terminal (character device), etc.

Why pick on 1?

> > I might write something like: "The option ‘-’ shreds the file specified
> > by the redirection ‘>’", though there could be a better name for ‘>’.
> 
> You're assuming the program will be used from a shell.  This is *usually*
> going to be true, but nothing prevents you from writing a C program
> which closes stdout, opens a file, ensures that it's using FD 1,
> and then calls "shred -".  The documentation has to support this use
> case as well.

/As well/ — which is why I wrote N in place of 1. The original bug
report (which I hadn't seen until Thomas' post) says:
 "If you redirect output to a file it will work. Shredding a tty doesn't
  make much sense, after all."
  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=155175#10

Now, you can't write "If you redirect output to a file it will work"
in a man page—it needs recasting into something more like what
I wrote above, which contains two key points:

. It points out that '-' is an option, not a filename or a stand-in
  for one, and it doesn't use the word standard, which is totally
  irrelevant in the circumstances.

. It demonstrates the shell syntax element required (&) in order to
  avoid truncating the file, rather than shred overwriting it.

I think that getting the "&" into the man page would be helpful
to anybody who doesn't look at the info page for the example.
It might have shortened the early part of this thread as well.

As for C programmers, neither FD number nor truncation is relevant.
Sure, you can pick 1. But you don't have to document that for shred.
And truncation is an accident that can occur because of shell's
redirect syntax: there's no equivalent in programs.

> > >A FILE of ‘-’ denotes standard output.  The intended use of this is
> > >to shred a removed temporary file.  For example:
> > > 
> > >   i=$(mktemp)
> > >   exec 3<>"$i"
> > >   rm -- "$i"
> > >   echo "Hello, world" >&3
> > >   shred - >&3
> > >   exec 3>-
> > 
> > I can see that the last line truncates the "anonymous" file,
> 
> No, that's not what it does at all.  In fact, that last line is
> written incorrectly.  It should say "exec 3>&-" and what that does
> is close file descriptor 3, which was previously opened on line 2.
> 
> What it actually does *as written* is create/truncate a file whose
> name is "-", close the previously opened FD 3, and make FD 3 point
> to the file named "-".
> 
> unicorn:~$ exec 3>-
> unicorn:~$ ls -ld -- -
> -rw-r--r-- 1 greg greg 0 Feb 13 07:12 -
> unicorn:~$ ls -l /dev/fd/3
> l-wx-- 1 greg greg 64 Feb 13 07:12 /dev/fd/3 -> /home/greg/-
> 
> This is an obvious bug in the info page.  I wonder how many years
> this has gone unnoticed.

Well spotted. That's what an experienced eye brings to a line like
that, whereas I assumed it meant something beyond my experience,
and searched for it.

Ironic that it truncates a file, and then immediately warns against
truncating a file instead of shredding it.

Cheers,
David.

Re: shred bug? [was: Unidentified subject!]

[Thomas Schmitt]

Hi,

"info shred" says:
> > >   i=$(mktemp)
> > >   exec 3<>"$i"
> > >   rm -- "$i"
> > >   echo "Hello, world" >&3
> > >   shred - >&3
> > >   exec 3>-

Greg Wooledge wrote:
> In fact, that last line is
> written incorrectly.  It should say "exec 3>&-" and what that does
> is close file descriptor 3, which was previously opened on line 2.
> [...]
> This is an obvious bug in the info page.  I wonder how many years
> this has gone unnoticed.

  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=155175#36
of 22 Dec 2005 states:

  "I'll assume that this is now adequately explained in the info page
   (below).  If not then please reopen.  // Thomas Hood
   [...]
   exec 3>-"

The bug report is from 02 Aug 2002 and states that the info page contains
the short and broad promise which we can still see in "man shred".

So we can assume a bug age of 18 to 22 years.


Have a nice day :)

Thomas

Re: shred bug? [was: Unidentified subject!]

[tomas]

On Tue, Feb 13, 2024 at 07:36:14AM -0500, Greg Wooledge wrote:
> On Tue, Feb 13, 2024 at 07:15:48AM -0500, Greg Wooledge wrote:
> > This is an obvious bug in the info page.  I wonder how many years
> > this has gone unnoticed.
> 
> I've filed Bug#1063837 for it.  

Well, thanks for doing TRT :-)

Cheers
-- 
t


signature.asc
Description: PGP signature

Re: shred bug? [was: Unidentified subject!]

[Greg Wooledge]

On Tue, Feb 13, 2024 at 07:15:48AM -0500, Greg Wooledge wrote:
> This is an obvious bug in the info page.  I wonder how many years
> this has gone unnoticed.

I've filed Bug#1063837 for it.  

Re: shred bug? [was: Unidentified subject!]

[Greg Wooledge]

On Mon, Feb 12, 2024 at 11:01:47PM -0600, David Wright wrote:
> … but not much. For me, "standard output" is /dev/fd/1, yet it seems
> unlikely that anyone is going to use >&1 in the manner of the example.

Standard output means "whatever file descriptor 1 points to".  That
could be a file, a pipe, a terminal (character device), etc.

> I might write something like: "The option ‘-’ shreds the file specified
> by the redirection ‘>’", though there could be a better name for ‘>’.

You're assuming the program will be used from a shell.  This is *usually*
going to be true, but nothing prevents you from writing a C program
which closes stdout, opens a file, ensures that it's using FD 1,
and then calls "shred -".  The documentation has to support this use
case as well.

> >A FILE of ‘-’ denotes standard output.  The intended use of this is
> >to shred a removed temporary file.  For example:
> > 
> >   i=$(mktemp)
> >   exec 3<>"$i"
> >   rm -- "$i"
> >   echo "Hello, world" >&3
> >   shred - >&3
> >   exec 3>-
> 
> I can see that the last line truncates the "anonymous" file,

No, that's not what it does at all.  In fact, that last line is
written incorrectly.  It should say "exec 3>&-" and what that does
is close file descriptor 3, which was previously opened on line 2.

What it actually does *as written* is create/truncate a file whose
name is "-", close the previously opened FD 3, and make FD 3 point
to the file named "-".

unicorn:~$ exec 3>-
unicorn:~$ ls -ld -- -
-rw-r--r-- 1 greg greg 0 Feb 13 07:12 -
unicorn:~$ ls -l /dev/fd/3
l-wx-- 1 greg greg 64 Feb 13 07:12 /dev/fd/3 -> /home/greg/-

This is an obvious bug in the info page.  I wonder how many years
this has gone unnoticed.

Re: shred bug? [was: Unidentified subject!]

[tomas]

On Mon, Feb 12, 2024 at 10:07:45PM +0100, Thomas Schmitt wrote:
> Hi,
> 
> > https://en.wikipedia.org/wiki/Everything_is_a_file
> > But, there is more than one kind of file.
> 
> "All files are equal.
>  But some files are more equal than others."
> 
> (George Orwell in his dystopic novel "Server Farm".)

:-D

Yesterday I was on the brink of quoting that. Great minds and
all of that...

Reality though is, that if you don't design your file with
magical properties from the get-go, you will keep stumbling
on stuff you want to model and which don't fit the real
life file design you chose.

And yes, Plan 9, as someone else noted in this thread, takes
that to a bigger extreme: in their windowing system, windows
are files, too -- you can remove a window by deleting the
corresponding file.

Cheers
-- 
t


signature.asc
Description: PGP signature

Re: shred bug? [was: Unidentified subject!]

[David Wright]

On Sun 11 Feb 2024 at 09:16:00 (-0600), David Wright wrote:
> On Sun 11 Feb 2024 at 09:54:24 (-0500), Greg Wooledge wrote:
> > On Sun, Feb 11, 2024 at 03:45:21PM +0100, to...@tuxteam.de wrote:
> > > Still there's the discrepancy between doc and behaviour.
> > 
> > There isn't.  The documentation says:
> > 
> > SYNOPSIS
> >shred [OPTION]... FILE...
> > 
> > DESCRIPTION
> >Overwrite  the specified FILE(s) repeatedly, in order to make it 
> > harder
> >for even very expensive hardware probing to recover the data.
> > 
> >If FILE is -, shred standard output.
> > 
> > In every sentence, the word FILE appears.  There's nothing in there
> > which says "you can operate on a non-file".
> > 
> > Once you grasp what the command is *intended* to do (rewind and overwrite
> > a file repeatedly), it makes absolutely perfect sense that it should
> > only operate on rewindable file system objects.
> > 
> > If you want it to write a stream of data instead of performing its normal
> > operation (rewinding and rewriting), that's a new feature.
> > 
> > If you'd prefer the documentation to say explicitly "only regular files
> > and block devices are allowed", that would be an upstream documentation
> > *clarification* request.
> 
> Perhaps info puts it better?

… but not much. For me, "standard output" is /dev/fd/1, yet it seems
unlikely that anyone is going to use >&1 in the manner of the example.

I might write something like: "The option ‘-’ shreds the file specified
by the redirection ‘>’", though there could be a better name for ‘>’.

>A FILE of ‘-’ denotes standard output.  The intended use of this is
>to shred a removed temporary file.  For example:
> 
>   i=$(mktemp)
>   exec 3<>"$i"
>   rm -- "$i"
>   echo "Hello, world" >&3
>   shred - >&3
>   exec 3>-

I can see that the last line truncates the "anonymous" file, but where
is that construction documented¹, and how would one parse the syntax
elements   FD  >  -   to make them mean truncate?

>However, the command ‘shred - >file’ does not shred the contents of
>FILE, since the shell truncates FILE before invoking ‘shred’.  Use
>the command ‘shred file’ or (if using a Bourne-compatible shell) the
>command ‘shred - 1<>file’ instead.

¹ the string ">-" doesn't appear in /usr/share/doc/bash/bashref.pdf,
  ver 5.1, for example.

Cheers,
David.

Re: shred bug? [was: Unidentified subject!]

[Max Nikulin]

On 12/02/2024 05:41, David Christensen wrote:


Apparently, shred(1) has both an info(1) page (?) and a man(1) page. The 
obvious solution is to write one document that is complete and correct, 
and use it everywhere -- e.g. DRY.


https://www.gnu.org/prep/standards/html_node/Man-Pages.html
6.9 Man Pages in "GNU Coding Standards"

In the GNU project, man pages are secondary. It is not necessary or
expected for every GNU program to have a man page, but some of them do.


A standalone man page is not the same as a section in a document 
describing the whole bundle.


Notice that man uses direct formatting, not logical markup. E.g. there 
is no dedicated structure for links to other man pages. They were not 
designed as hypertext documents, they are to be printed on paper. In 
this sense texinfo is a more advanced format.


P.S. I admit that in some cases "man bash" may be more convenient for 
searching than an info browser.

Re: shred bug? [was: Unidentified subject!]

[Thomas Schmitt]

Hi,

> https://en.wikipedia.org/wiki/Everything_is_a_file
> But, there is more than one kind of file.

"All files are equal.
 But some files are more equal than others."

(George Orwell in his dystopic novel "Server Farm".)


Have a nice day :)

Thomas

Re: shred bug? [was: Unidentified subject!]

[David Christensen]

On 2/12/24 08:50, Curt wrote:

On 2024-02-11,   wrote:



On Sun, Feb 11, 2024 at 09:54:24AM -0500, Greg Wooledge wrote:

[...]


If FILE is -, shred standard output.
=20
In every sentence, the word FILE appears.  There's nothing in there
which says "you can operate on a non-file".


Point taken, yes.


I thought everything was a file.



"Everything is a file" is a design feature of the Unix operating system:

https://en.wikipedia.org/wiki/Everything_is_a_file


But, there is more than one kind of file.


And, not every program supports every kind of file.


The manual page for find(1) provides a shopping list of file types it 
supports:


2024-02-12 12:32:13 dpchrist@laalaa ~
$ man find | egrep -A 20 '^   .type c'
   -type c
  File is of type c:

  b  block (buffered) special

  c  character (unbuffered) special

  d  directory

  p  named pipe (FIFO)

  f  regular file

  l  symbolic link; this is never true if  the
 -L option or the -follow option is in ef-
 fect, unless the symbolic link is broken.
 If  you want to search for symbolic links
 when -L is in effect, use -xtype.

  s  socket


As for shred(1), the argument FILE is conventionally a regular file.  We 
are discussing the special case described in the manual page:


   If FILE is -, shred standard output.


David

Re: shred bug? [was: Unidentified subject!]

[Greg Wooledge]

On Mon, Feb 12, 2024 at 04:50:50PM -, Curt wrote:
> On 2024-02-11,   wrote:
> >
> >
> > On Sun, Feb 11, 2024 at 09:54:24AM -0500, Greg Wooledge wrote:
> >
> > [...]
> >
> >>If FILE is -, shred standard output.
> >>=20
> >> In every sentence, the word FILE appears.  There's nothing in there
> >> which says "you can operate on a non-file".
> >
> > Point taken, yes.
> 
> I thought everything was a file.

An anonymous pipe(2) in memory between two processes isn't even close to
a file.  Also, you're confusing Linux and Plan 9.  Linux has a bunch of
things that aren't files, such as network interfaces.

Re: shred bug? [was: Unidentified subject!]

[Curt]

On 2024-02-11,   wrote:
>
>
> On Sun, Feb 11, 2024 at 09:54:24AM -0500, Greg Wooledge wrote:
>
> [...]
>
>>If FILE is -, shred standard output.
>>=20
>> In every sentence, the word FILE appears.  There's nothing in there
>> which says "you can operate on a non-file".
>
> Point taken, yes.

I thought everything was a file.

Re: shred bug? [was: Unidentified subject!]

[David Christensen]

On 2/11/24 06:54, Greg Wooledge wrote:

On Sun, Feb 11, 2024 at 03:45:21PM +0100, to...@tuxteam.de wrote:

On Sun, Feb 11, 2024 at 09:37:31AM -0500, Greg Wooledge wrote:

On Sun, Feb 11, 2024 at 08:02:12AM +0100, to...@tuxteam.de wrote:


[...]


What Thomas was trying to do is to get a cheap, fast random number
generator. Shred seems to have such.


Well... I certainly wouldn't call it a bug.  Maybe a feature request.


Still there's the discrepancy between doc and behaviour.


There isn't.  The documentation says:

SYNOPSIS
shred [OPTION]... FILE...



I interpret the above line to be a prototype for invoking the shred(1) 
program:


* "shred" is the program name

* "[OPTION]..." is one or more option specifiers that may be omitted. 
Each should be described below.


* "FILE..." is one or more argument specifies that should be file system 
paths (strings).




DESCRIPTION
Overwrite  the specified FILE(s) repeatedly, in order to make it harder
for even very expensive hardware probing to recover the data.

If FILE is -, shred standard output.



I interpret the above line at face value -- if the caller provides a 
dash as the argument, shred(1) will operate on standard output.




In every sentence, the word FILE appears.  There's nothing in there
which says "you can operate on a non-file".



Dash is not a file, yet the above sentence says shred(1) can operate on it.



Once you grasp what the command is *intended* to do (rewind and overwrite
a file repeatedly), it makes absolutely perfect sense that it should
only operate on rewindable file system objects.



An expert may infer what you have stated, but I prefer manual pages that 
are explicit.



The GNU project must have found a need for the FILE='-' feature, 
otherwise it would not exist.  The manual page should describe that need 
(e.g. why) and how to properly use shred(1) to solve the need.




If you want it to write a stream of data instead of performing its normal
operation (rewinding and rewriting), that's a new feature.



Humans are (in)famous for doing unexpected things.



If you'd prefer the documentation to say explicitly "only regular files
and block devices are allowed", that would be an upstream documentation
*clarification* request.



Apparently, shred(1) has both an info(1) page (?) and a man(1) page. 
The obvious solution is to write one document that is complete and 
correct, and use it everywhere -- e.g. DRY.



David

Re: shred bug? [was: Unidentified subject!]

[David Wright]

On Sun 11 Feb 2024 at 09:54:24 (-0500), Greg Wooledge wrote:
> On Sun, Feb 11, 2024 at 03:45:21PM +0100, to...@tuxteam.de wrote:
> > On Sun, Feb 11, 2024 at 09:37:31AM -0500, Greg Wooledge wrote:
> > > On Sun, Feb 11, 2024 at 08:02:12AM +0100, to...@tuxteam.de wrote:
> > 
> > [...]
> > 
> > > > What Thomas was trying to do is to get a cheap, fast random number
> > > > generator. Shred seems to have such.
> > > 
> > > Well... I certainly wouldn't call it a bug.  Maybe a feature request.
> > 
> > Still there's the discrepancy between doc and behaviour.
> 
> There isn't.  The documentation says:
> 
> SYNOPSIS
>shred [OPTION]... FILE...
> 
> DESCRIPTION
>Overwrite  the specified FILE(s) repeatedly, in order to make it harder
>for even very expensive hardware probing to recover the data.
> 
>If FILE is -, shred standard output.
> 
> In every sentence, the word FILE appears.  There's nothing in there
> which says "you can operate on a non-file".
> 
> Once you grasp what the command is *intended* to do (rewind and overwrite
> a file repeatedly), it makes absolutely perfect sense that it should
> only operate on rewindable file system objects.
> 
> If you want it to write a stream of data instead of performing its normal
> operation (rewinding and rewriting), that's a new feature.
> 
> If you'd prefer the documentation to say explicitly "only regular files
> and block devices are allowed", that would be an upstream documentation
> *clarification* request.

Perhaps info puts it better?

   A FILE of ‘-’ denotes standard output.  The intended use of this is
   to shred a removed temporary file.  For example:

  i=$(mktemp)
  exec 3<>"$i"
  rm -- "$i"
  echo "Hello, world" >&3
  shred - >&3
  exec 3>-

   However, the command ‘shred - >file’ does not shred the contents of
   FILE, since the shell truncates FILE before invoking ‘shred’.  Use
   the command ‘shred file’ or (if using a Bourne-compatible shell) the
   command ‘shred - 1<>file’ instead.

Cheers,
David.

Re: shred bug? [was: Unidentified subject!]

[Thomas Schmitt]

Hi,

to...@tuxteam.de wrote:
> Still there's the discrepancy between doc and behaviour.

Depends at which documentation you look. Obviously stemming from
  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=155175#36
i read in
  https://www.gnu.org/software/coreutils/manual/html_node/shred-invocation.html

 "A file of ‘-’ denotes standard output. The intended use of this is to
  shred a removed temporary file. For example: [shell wizzardry]"

It works as long as stdout is connected to a data file, or block device,
or directory, or symbolic link, or to a character device that is not a
terminal.
(Maybe it refuses later on some of these types, but not at the location
with the message "invalid file type". I wonder if i can connect stdout
to a symbolic link instead of its target.)

The bug would thus have to be filed against the man page
  https://sources.debian.org/src/coreutils/9.4-3/man/shred.1/
which says only

  "If FILE is \-, shred standard output."

The info empire of coreutils says what above web manual says.
  https://sources.debian.org/src/coreutils/9.4-3/doc/coreutils.texi/#L10705


Have a nice day :)

Thomas

Re: shred bug? [was: Unidentified subject!]

[tomas]

On Sun, Feb 11, 2024 at 09:54:24AM -0500, Greg Wooledge wrote:

[...]

>If FILE is -, shred standard output.
> 
> In every sentence, the word FILE appears.  There's nothing in there
> which says "you can operate on a non-file".

Point taken, yes.

Cheers
-- 
t


signature.asc
Description: PGP signature

Re: shred bug? [was: Unidentified subject!]

[Greg Wooledge]

On Sun, Feb 11, 2024 at 03:45:21PM +0100, to...@tuxteam.de wrote:
> On Sun, Feb 11, 2024 at 09:37:31AM -0500, Greg Wooledge wrote:
> > On Sun, Feb 11, 2024 at 08:02:12AM +0100, to...@tuxteam.de wrote:
> 
> [...]
> 
> > > What Thomas was trying to do is to get a cheap, fast random number
> > > generator. Shred seems to have such.
> > 
> > Well... I certainly wouldn't call it a bug.  Maybe a feature request.
> 
> Still there's the discrepancy between doc and behaviour.

There isn't.  The documentation says:

SYNOPSIS
   shred [OPTION]... FILE...

DESCRIPTION
   Overwrite  the specified FILE(s) repeatedly, in order to make it harder
   for even very expensive hardware probing to recover the data.

   If FILE is -, shred standard output.

In every sentence, the word FILE appears.  There's nothing in there
which says "you can operate on a non-file".

Once you grasp what the command is *intended* to do (rewind and overwrite
a file repeatedly), it makes absolutely perfect sense that it should
only operate on rewindable file system objects.

If you want it to write a stream of data instead of performing its normal
operation (rewinding and rewriting), that's a new feature.

If you'd prefer the documentation to say explicitly "only regular files
and block devices are allowed", that would be an upstream documentation
*clarification* request.

Re: shred bug? [was: Unidentified subject!]

[tomas]

On Sun, Feb 11, 2024 at 09:37:31AM -0500, Greg Wooledge wrote:
> On Sun, Feb 11, 2024 at 08:02:12AM +0100, to...@tuxteam.de wrote:

[...]

> > What Thomas was trying to do is to get a cheap, fast random number
> > generator. Shred seems to have such.
> 
> Well... I certainly wouldn't call it a bug.  Maybe a feature request.

Still there's the discrepancy between doc and behaviour.

Cheers
-- 
t


signature.asc
Description: PGP signature

Re: shred bug? [was: Unidentified subject!]

[Greg Wooledge]

On Sun, Feb 11, 2024 at 08:02:12AM +0100, to...@tuxteam.de wrote:
> On Sat, Feb 10, 2024 at 07:10:54PM -0500, Greg Wooledge wrote:
> > On Sat, Feb 10, 2024 at 04:05:21PM -0800, David Christensen wrote:
> > > 2024-02-10 16:03:50 dpchrist@laalaa ~
> > > $ shred -s 1K - | wc -c
> > > shred: -: invalid file type
> > > 0
> > > 
> > > 
> > > It looks like a shred(1) needs a bug report.
> > 
> > I'm confused what you expected this command to do.  You wanted to
> > "destroy" (by overwriting with random data) a pipe to wc?  What
> > would that even look like?
> 
> What Thomas was trying to do is to get a cheap, fast random number
> generator. Shred seems to have such.

Well... I certainly wouldn't call it a bug.  Maybe a feature request.

Re: shred bug? [was: Unidentified subject!]

[debian-user]

David Christensen  wrote:
> On 2/10/24 16:10, Greg Wooledge wrote:
> > On Sat, Feb 10, 2024 at 04:05:21PM -0800, David Christensen wrote:  
> >> 2024-02-10 16:03:50 dpchrist@laalaa ~
> >> $ shred -s 1K - | wc -c
> >> shred: -: invalid file type
> >> 0
> >>
> >>
> >> It looks like a shred(1) needs a bug report.  
> > 
> > I'm confused what you expected this command to do.  You wanted to
> > "destroy" (by overwriting with random data) a pipe to wc?  What
> > would that even look like?
> > 
> > The basic premise of shred is that it determines the size of the
> > file, then writes data over it, rewinds it, and repeats this a few
> > times. A pipe to a process has no size, and can't be rewound.
> > 
> > Declaring a pipe to be an "invalid file type" for shredding sounds
> > pretty reasonable to me.  
> 
> 
> The documentation is confusing:
> 
> On 2/10/24 16:05, David Christensen wrote:
>  > 2024-02-10 16:03:42 dpchrist@laalaa ~
>  > $ man shred | grep 'If FILE is -'
>  > If FILE is -, shred standard output.  

Maybe it is unstated but mandatory to use -n 1 as well?
And optionally -s N?
I expect reading the code would tell.

First time I've read the man page properly.
Interesting point about COW filesystems such as btrfs :)

Re: shred bug? [was: Unidentified subject!]

[tomas]

On Sat, Feb 10, 2024 at 07:10:54PM -0500, Greg Wooledge wrote:
> On Sat, Feb 10, 2024 at 04:05:21PM -0800, David Christensen wrote:
> > 2024-02-10 16:03:50 dpchrist@laalaa ~
> > $ shred -s 1K - | wc -c
> > shred: -: invalid file type
> > 0
> > 
> > 
> > It looks like a shred(1) needs a bug report.
> 
> I'm confused what you expected this command to do.  You wanted to
> "destroy" (by overwriting with random data) a pipe to wc?  What
> would that even look like?

What Thomas was trying to do is to get a cheap, fast random number
generator. Shred seems to have such.

> The basic premise of shred is that it determines the size of the file,
> then writes data over it, rewinds it, and repeats this a few times.
> A pipe to a process has no size, and can't be rewound.

That's right: stdout is a (potentially) infinite file, so only one
pass (-n 1, as Thomas put in the command line) really makes sense.
Unless you are into transfinite numbers, that is :-)

> Declaring a pipe to be an "invalid file type" for shredding sounds
> pretty reasonable to me.

This is one of those cases of the toolmaker knowing better than the
tool user. One of the things I like UNIX is that this attitude isn't
that widespread (it is slowly spreading, alas). I much prefer those
tool makers who say "surprise me".

Of course, helping people to not shoot themselves in the foot is
also a honourable endeavour. So "you are trying to shred a pipe.
Use option -f for that (see the manual page" would be fully OK
in my book.

I don't like opinionated software. It's messy enough as it is
when people have opinions :-)

Cheers
-- 
t


signature.asc
Description: PGP signature

Re: shred bug? [was: Unidentified subject!]

[David Christensen]

On 2/10/24 16:10, Greg Wooledge wrote:

On Sat, Feb 10, 2024 at 04:05:21PM -0800, David Christensen wrote:

2024-02-10 16:03:50 dpchrist@laalaa ~
$ shred -s 1K - | wc -c
shred: -: invalid file type
0


It looks like a shred(1) needs a bug report.


I'm confused what you expected this command to do.  You wanted to
"destroy" (by overwriting with random data) a pipe to wc?  What
would that even look like?

The basic premise of shred is that it determines the size of the file,
then writes data over it, rewinds it, and repeats this a few times.
A pipe to a process has no size, and can't be rewound.

Declaring a pipe to be an "invalid file type" for shredding sounds
pretty reasonable to me.



The documentation is confusing:

On 2/10/24 16:05, David Christensen wrote:
> 2024-02-10 16:03:42 dpchrist@laalaa ~
> $ man shred | grep 'If FILE is -'
> If FILE is -, shred standard output.


David

Re: shred bug? [was: Unidentified subject!]

[Greg Wooledge]

On Sat, Feb 10, 2024 at 04:05:21PM -0800, David Christensen wrote:
> 2024-02-10 16:03:50 dpchrist@laalaa ~
> $ shred -s 1K - | wc -c
> shred: -: invalid file type
> 0
> 
> 
> It looks like a shred(1) needs a bug report.

I'm confused what you expected this command to do.  You wanted to
"destroy" (by overwriting with random data) a pipe to wc?  What
would that even look like?

The basic premise of shred is that it determines the size of the file,
then writes data over it, rewinds it, and repeats this a few times.
A pipe to a process has no size, and can't be rewound.

Declaring a pipe to be an "invalid file type" for shredding sounds
pretty reasonable to me.

Re: shred bug? [was: Unidentified subject!]

[David Christensen]

On 2/10/24 04:40, to...@tuxteam.de wrote:

On Sat, Feb 10, 2024 at 11:38:21AM +0100, Thomas Schmitt wrote:

[...]


But shred(1) on Debian 11 refuses on "-" contrary to its documentation:
   shred: -: invalid file type
A non-existing file path causes "No such file or directory".


Hmm. This looks like a genuine bug: the man page mentions it.

Also, /dev/stdout as target runs into the very same problem.

Cheers



Testing:

2024-02-10 16:01:54 dpchrist@laalaa ~
$ cat /etc/debian_version ; uname -a
11.8
Linux laalaa 5.10.0-27-amd64 #1 SMP Debian 5.10.205-2 (2023-12-31) 
x86_64 GNU/Linux


2024-02-10 16:02:34 dpchrist@laalaa ~
$ bash --version | head -n 1
GNU bash, version 5.1.4(1)-release (x86_64-pc-linux-gnu)

2024-02-10 16:02:48 dpchrist@laalaa ~
$ shred --version | head -n 1
shred (GNU coreutils) 8.32

2024-02-10 16:03:42 dpchrist@laalaa ~
$ man shred | grep 'If FILE is -'
   If FILE is -, shred standard output.

2024-02-10 16:03:50 dpchrist@laalaa ~
$ shred -s 1K - | wc -c
shred: -: invalid file type
0


It looks like a shred(1) needs a bug report.


David

Re: shred bug? [was: Unidentified subject!]

[tomas]

On Sat, Feb 10, 2024 at 01:40:35PM +0100, to...@tuxteam.de wrote:
> On Sat, Feb 10, 2024 at 11:38:21AM +0100, Thomas Schmitt wrote:
> 
> [...]
> 
> > But shred(1) on Debian 11 refuses on "-" contrary to its documentation:
> >   shred: -: invalid file type
> > A non-existing file path causes "No such file or directory".
> 
> Hmm. This looks like a genuine bug: the man page mentions it.
> 
> Also, /dev/stdout as target runs into the very same problem.

Ah, it seems to be this one, from 2002:

  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=155175

which is archived. The argument seems to be that shred on stdout
doesn't make any sense, because the shell would truncate the
file anyway when you did

  shred - > /this/file/to/be/shredded

... which, of course, undermines shred's purpose. It seems they
hadn't your sneaky use case in mind :-)

Cheers
-- 
t


signature.asc
Description: PGP signature