Unidentified subject!

[jacques.briquet]

Suppression Envoyé depuis mon appareil GalaxySupprimer m. Briquet Jacques qui 
est décédé 

Unidentified subject!

[jacques.briquet]

Supprimer m. Briquet de votre liste Il est décédé Envoyé depuis mon appareil 
Galaxy

Re: shred bug? [was: Unidentified subject!]

[Michael Stone]

On Sun, Feb 11, 2024 at 08:02:12AM +0100, to...@tuxteam.de wrote:

What Thomas was trying to do is to get a cheap, fast random number
generator. Shred seems to have such.


You're better off with /dev/urandom, it's much easier to understand what 
it's trying to do, vs the rather baroque logic in shred. In fact, 
there's nothing in shred's documenation AFAICT that suggests it should 
be used as a random number generator. For pure speed, playing games with 
openssl enc and /dev/zero will generally win. If speed doesn't matter, 
we're back to /dev/urandom as the simplest and most direct solution.


FWIW, the main use for shred in 2024 is: to be there so someone's old 
script doesn't break. There's basically no good use case for it, and it 
probably shouldn't have gotten into coreutils in the first place. The 
multipass pattern stuff is cargo-cult voodoo--a single overwrite with 
zeros will be as effective as anything else--and on modern 
storage/filesystems there's a good chance your overwrite won't overwrite 
anything anyway. Probably the right answer is a kernel facility 
(userspace can't guarantee anything). If you're really sure that 
overwrites work on your system, `shred -n0 -z` will be the fastest way 
to do that. The docs say don't do that because SSDs might optimize that 
away, but SSDs probably aren't overwriting anything anyway (also 
mentioned in the docs). ¯\_(ツ)_/¯

Re: shred bug? [was: Unidentified subject!]

[David Wright]

On Tue 13 Feb 2024 at 11:21:08 (-0500), Greg Wooledge wrote:
> On Tue, Feb 13, 2024 at 09:35:11AM -0600, David Wright wrote:
> > On Tue 13 Feb 2024 at 07:15:48 (-0500), Greg Wooledge wrote:
> > > On Mon, Feb 12, 2024 at 11:01:47PM -0600, David Wright wrote:
> > > > … but not much. For me, "standard output" is /dev/fd/1, yet it seems
> > > > unlikely that anyone is going to use >&1 in the manner of the example.
> > > 
> > > Standard output means "whatever file descriptor 1 points to".  That
> > > could be a file, a pipe, a terminal (character device), etc.
> > 
> > Why pick on 1?
> 
> It's the definition.  Standard input is FD 0, standard output is FD 1,
> and standard error is FD 2.
> 
> > . It demonstrates the shell syntax element required (&) in order to
> >   avoid truncating the file, rather than shred overwriting it.
> 
> You are confused.  You're making assumptions about shell syntax that
> are simply not true.

You're right. I was looking too hard at the right side of the > and
neglecting the implied left side. It's always worth running these
things past your eyes. Thanks for the clear exposition that followed.

Cheers,
David.

Re: shred bug? [was: Unidentified subject!]

[tomas]

On Tue, Feb 13, 2024 at 01:03:44PM -0800, David Christensen wrote:
> On 2/13/24 09:40, debian-u...@howorth.org.uk wrote:
> > Greg Wooledge  wrote:
> > 
> > > Shred will determine the size of the file, then write data to the
> > > file, rewind, write data again, etc.  On a traditional hard drive,
> > > that will overwrite the original private information.  On modern
> > > devices, it may not.
> > 
> > Thanks for the excellent explanation :)
> > 
> > One nitpick. You say "On a traditional hard drive, that will overwrite
> > the original private information" but that's not quite true. It also
> > needs to be a "traditional" file system! That is, not journalled or COW.
> > 
> > So nowadays I would expect shred not to work unless you got very
> > lucky, or planned carefully.
> 
> 
> Perhaps zerofree(8)?

On a SATA, it won't get at (some) of the spare blocks, since it
doesn't know that they are there.

If your data is so sensitive that you don't want it to escape,
your best bet seems to plan ahead and not let it hit your media
unencrypted.

Use LUKS. And oh, use argon2id as key derivation function [1]
these days.

Cheers
[1] https://mjg59.dreamwidth.org/66429.html
-- 
t


signature.asc
Description: PGP signature

Re: shred bug? [was: Unidentified subject!]

[gene heskett]

On 2/13/24 16:00, David Christensen wrote:

On 2/13/24 11:31, gene heskett wrote:
Next experiment is a pair of 4T Silicon Power SSD's When they & the 
startech usb3 adapters arrive.  I'll get that NAS built for amanda yet.



2.5" SATA SSD's and SATA to USB adapter cables for $187.97 + $10.99 = 
$198.96 each set?


https://www.amazon.com/dp/B0BVLRFFWQ

https://www.amazon.com/dp/B00HJZJI84


Why not external USB drives for $192.99?

https://www.amazon.com/dp/B0C6XVZS4K


For $7 more, you can get the "Pro edition" in black with two cables:

https://www.amazon.com/dp/B0C69QD5NK


You could plug those into the two USB-C 3.1 Gen 2 ports on your Asus 
PRIME Z370-A II motherboard.


Maybe, but these sata types have the the mounting bolts the usb versions 
don't. And fits the drive adapters I already have that put both in one 
drive tray. Not to mention if Taiwan has a similar product, I tend to 
buy it.  So are the 4 2T gigastones I'll fill the next 2 drawers with so 
I should wind up with a 16T backup server's LVM. with a 1/2T Samsung 870 
as a holding disk.  Running a bpi-m5 headless, maybe < 20 watts.  Whats 
not to like?


David


.


Cheers, Gene Heskett, CET.
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis

Re: shred bug? [was: Unidentified subject!]

[David Christensen]

On 2/13/24 09:40, debian-u...@howorth.org.uk wrote:

Greg Wooledge  wrote:


Shred will determine the size of the file, then write data to the
file, rewind, write data again, etc.  On a traditional hard drive,
that will overwrite the original private information.  On modern
devices, it may not.


Thanks for the excellent explanation :)

One nitpick. You say "On a traditional hard drive, that will overwrite
the original private information" but that's not quite true. It also
needs to be a "traditional" file system! That is, not journalled or COW.

So nowadays I would expect shred not to work unless you got very
lucky, or planned carefully.



Perhaps zerofree(8)?


David

Re: shred bug? [was: Unidentified subject!]

[David Christensen]

On 2/13/24 11:31, gene heskett wrote:
Next experiment is a pair of 4T 
Silicon Power SSD's When they & the startech usb3 adapters arrive.  I'll 
get that NAS built for amanda yet.



2.5" SATA SSD's and SATA to USB adapter cables for $187.97 + $10.99 = 
$198.96 each set?


https://www.amazon.com/dp/B0BVLRFFWQ

https://www.amazon.com/dp/B00HJZJI84


Why not external USB drives for $192.99?

https://www.amazon.com/dp/B0C6XVZS4K


For $7 more, you can get the "Pro edition" in black with two cables:

https://www.amazon.com/dp/B0C69QD5NK


You could plug those into the two USB-C 3.1 Gen 2 ports on your Asus 
PRIME Z370-A II motherboard.



David

Re: shred bug? [was: Unidentified subject!]

[gene heskett]

On 2/13/24 14:44, Thomas Schmitt wrote:

Hi,

Gene Heskett wrote:

Next experiment is a pair of 4T Silicon Power SSD's


When f3 has (hopefully) given its OK, the topic of a full write-and-read
test will come up again. I'm looking forward to all the spin-off topics.

I'll have to admit it has been quite educational. Now, can I remember it 
next week? YTBD.>

Have a nice day :)


You too Thomas.
Take care and stay well.


Thomas


Cheers, Gene Heskett, CET.
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis

Re: shred bug? [was: Unidentified subject!]

[Thomas Schmitt]

Hi,

Gene Heskett wrote:
> Next experiment is a pair of 4T Silicon Power SSD's

When f3 has (hopefully) given its OK, the topic of a full write-and-read
test will come up again. I'm looking forward to all the spin-off topics.


Have a nice day :)

Thomas

Re: shred bug? [was: Unidentified subject!]

[Thomas Schmitt]

Hi,

Greg Wooledge wrote:
> Heh.  Don't forget your own attempts to use a shredder as a PRNG stream.

My original idea was to watch a minimal shred run by teeing its work into
a checksummer.

But then topic drift came in. So we got a farm show of random generators
and a discussion about what exactly is a bug in shred's documentation.
Plus the shell programming webinar. And a diagnosis about a rightously
failed attempt to change the partition table type from MBR/DOS to GPT.

And this all because Gene Heskett was adventurous enough to buy a cheap
fake USB disk.


Have a nice day :)

Thomas

Re: shred bug? [was: Unidentified subject!]

[gene heskett]

On 2/13/24 12:56, Thomas Schmitt wrote:

Hi,

Greg Wooledge wrote:

Let me write out the example again, but with the bug fixed, and then
explain what each line does, [... lecture about advanced shell
programming ...]


And this all because Gene Heskett was adventurous enough to buy a cheap
fake USB disk. :))


Guilty as charged, Thomas. My advantage is that it won't affect the 
length of the ladder up my side of the hog.  If I save someone else from 
getting bit by that fraud I'm pleased.  Next experiment is a pair of 4T 
Silicon Power SSD's When they & the startech usb3 adapters arrive.  I'll 
get that NAS built for amanda yet.



Have a nice day :)


You too.


Thomas


Cheers, Gene Heskett, CET.
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis

Re: shred bug? [was: Unidentified subject!]

[Greg Wooledge]

On Tue, Feb 13, 2024 at 06:54:58PM +0100, Thomas Schmitt wrote:
> Greg Wooledge wrote:
> > Let me write out the example again, but with the bug fixed, and then
> > explain what each line does, [... lecture about advanced shell
> > programming ...]
> 
> And this all because Gene Heskett was adventurous enough to buy a cheap
> fake USB disk. :))

Heh.  Don't forget your own attempts to use a shredder as a PRNG stream.

Shell redirections can be complicated, so this topic is going to
come up once in a while.  The example in the shred info page is fairly
unintuitive, so it deserves a bit of explanation.  I imagine most readers
who saw it simply accepted it as written, which is why the bug went
undiscovered for two decades.

Re: shred bug? [was: Unidentified subject!]

[Thomas Schmitt]

Hi,

Greg Wooledge wrote:
> Let me write out the example again, but with the bug fixed, and then
> explain what each line does, [... lecture about advanced shell
> programming ...]

And this all because Gene Heskett was adventurous enough to buy a cheap
fake USB disk. :))


Have a nice day :)

Thomas

Re: shred bug? [was: Unidentified subject!]

[debian-user]

Greg Wooledge  wrote:

> Shred will determine the size of the file, then write data to the
> file, rewind, write data again, etc.  On a traditional hard drive,
> that will overwrite the original private information.  On modern
> devices, it may not.

Thanks for the excellent explanation :)

One nitpick. You say "On a traditional hard drive, that will overwrite
the original private information" but that's not quite true. It also
needs to be a "traditional" file system! That is, not journalled or COW.

So nowadays I would expect shred not to work unless you got very
lucky, or planned carefully.

Re: Fast Random Data Generation (Was: Re: Unidentified subject!)

[Linux-Fan]

David Christensen writes:


On 2/12/24 08:30, Linux-Fan wrote:

David Christensen writes:


On 2/11/24 02:26, Linux-Fan wrote:

I wrote a program to automatically generate random bytes in multiple threads:
https://masysma.net/32/big4.xhtml


What algorithm did you implement?


I copied the algorithm from here:
https://www.javamex.com/tutorials/random_numbers/numerical_recipes.shtml


That Java code uses locks, which implies it uses global state and cannot be  
run multi-threaded (?).  (E.g. one process with one JVM.)


Indeed, the example code uses locks which is bad from a performance point of  
view. That is why _my_ implementation works without this fine-grained  
locking and instead ensures that each thread uses its own instance as to  
avoid the lock. IOW: I copied the algorithm but of course adjusted the code  
to my use case.


My version basically runs as follows:

* Create one queue of ByteBuffers
* Create multiple threads
   * Each thread runs their own RNG instance
   * Upon finishing the creation of a buffer, it enqueues
 the resulting ByteBuffer into the queue (this is the only
 part where multiple threads access concurrently)
* The main thread dequeues from the queue and writes the
  buffers to the output file

Is it possible to obtain parallel operation on an SMP machine with multiple  
virtual processors?  (Other than multiple OS processes with one PRNG on one  
JVM each?)


Even the locked random could be instantiated multiple times (each instance  
gets their own lock) and this could still be faster than running just  
one of it. However, since the computation itself is fast, I suppose the  
performance hit from managing the locks could be significant. Multiple OS  
processes would also work, but is pretty uncommon in Java land AFAIR.


I found it during the development of another application where I needed a  
lot of random data for simulation purposes :)


My implementation code is here:
https://github.com/m7a/bo-big/blob/master/latest/Big4.java


See the end of that file to compare with the “Numerical Recipes” RNG linked  
further above to observe the difference wrt. locking :)



If I were to do it again today, I'd probably switch to any of these PRNGS:

* https://burtleburtle.net/bob/rand/smallprng.html
* https://www.pcg-random.org/


Hard core.  I'll let the experts figure it out; and then I will use their  
libraries and programs.


IIRC one of the findings of PCG was that the default RNGs of many  
programming languages and environments are surprisingly bad. I only arrived at  
using a non-default implementation after facing some issues with the Java  
integrated ThreadLocalRandom ”back then” :)


It may indeed be worth pointing out (as Jeffrey Walton already mentioned in  
another subthread) that these RNGs discussed here are _not_ cryptographic  
RNGs. I think for disk testing purposes it is OK to use fast non- 
cryptographic RNGs, but other applications may have higher demands on their  
RNGs.


HTH
Linux-Fan

öö

[...]


pgpLYMPAsjGtj.pgp
Description: PGP signature

Re: shred bug? [was: Unidentified subject!]

[Greg Wooledge]

On Tue, Feb 13, 2024 at 09:35:11AM -0600, David Wright wrote:
> On Tue 13 Feb 2024 at 07:15:48 (-0500), Greg Wooledge wrote:
> > On Mon, Feb 12, 2024 at 11:01:47PM -0600, David Wright wrote:
> > > … but not much. For me, "standard output" is /dev/fd/1, yet it seems
> > > unlikely that anyone is going to use >&1 in the manner of the example.
> > 
> > Standard output means "whatever file descriptor 1 points to".  That
> > could be a file, a pipe, a terminal (character device), etc.
> 
> Why pick on 1?

It's the definition.  Standard input is FD 0, standard output is FD 1,
and standard error is FD 2.

> . It demonstrates the shell syntax element required (&) in order to
>   avoid truncating the file, rather than shred overwriting it.

You are confused.  You're making assumptions about shell syntax that
are simply not true.

> > > >A FILE of ‘-’ denotes standard output.  The intended use of this is
> > > >to shred a removed temporary file.  For example:
> > > > 
> > > >   i=$(mktemp)
> > > >   exec 3<>"$i"
> > > >   rm -- "$i"
> > > >   echo "Hello, world" >&3
> > > >   shred - >&3
> > > >   exec 3>-

> Ironic that it truncates a file, and then immediately warns against
> truncating a file instead of shredding it.

No.  This is not what it does (if we fix the bug).

Let me write out the example again, but with the bug fixed, and then
explain what each line does, because apparently there is a LOT of
misunderstanding here.

  i=$(mktemp)
  exec 3<>"$i"
  rm -- "$i"
  echo "Hello, world" >&3
  shred - >&3
  exec 3>&-

The first line runs mktemp(1), which is a GNU coreutils program that
creates a temporary file, and then writes its name to standard output.
The shell syntax grabs that name and stores it in the variable "i".

So, after line 1, we have an (empty) temporary file, which was created
by a child process that has terminated.  We have its name in a variable.

Creation of temporary files works a little bit differently in shell
scripts than it does in regular programs.  In most other languages,
you would call a library function that creates the temporary file
(keeping it open), optionally unlinks it, and returns the open file
descriptor to you for use.  But you can't do that in a shell script
that needs an external program to do the file creation.  So we have this
slightly clumsy approach.

The second line opens this file for reading and writing, and ensures
that file descriptor 3 points to it.  It's important to understand that
while "exec 3>$i" would have truncated the file's contents, "exec 3<>$i"
does not.  Of course, there wasn't any content to truncate, since it was
created empty, but that's not the important part.  The important part
is that this FD is opened for read+write, allowing the temporary file
to be used for storage *and* retrieval.  We aren't doing any retrieval
in this example, but it could be done, with specialized tools.

The third line unlinks the file from the file system.  However, the shell
still has an open file descriptor which points to the file.  Therefore,
the file is still accessible through this FD.  Its inode is not recycled,
and any blocks containing file content are not marked for reuse.

This "unlink before using" technique is traditional on Unix systems.
It allows you to bypass setting up an exit handler to clean up the
temporary file.  Once the open file descriptor is closed, the file
system will mark the inode and any blocks as ready for reuse.  Even if
the script is killed by SIGKILL, that cleanup will still happen.

The fourth line writes some content via the open file descriptor 3.  At
this point, our unlinked file now has data in it.  Presumably this data
is super private, and we don't want anyone to be able to recover it.
When the script exits, the open file descriptor will close, and the file
system will mark the file's blocks as reusable, but it won't *actually*
reuse them until something else comes along and claims them.  But that's
what shred is designed for.

The fifth line calls shred(1), instructing it to destroy the content
that's in the unlinked file.  Since the file is unlinked, it has no name,
and therefore shred can't be *given* a name.  However, we have a file
descriptor that points to it.  So, what we *can* do is point standard
output to the file (that's what >&3 does), and then tell shred to destroy
the file that's pointed to by stdout.

Shred will determine the size of the file, then write data to the file,
rewind, write data again, etc.  On a traditional hard drive, that will
overwrite the original private information.  On modern devices, it may
not.

Finally, the sixth line closes file descriptor 3.  Doing this frees the
file's inode and blocks, allowing them to be reused by a future program.

The key to understanding this example is a firm grasp of the shell's
redirection syntax and semantics.

Let's start with what people normally see:  3>filename

That opens "filename" for writing with truncation, and ensures that 

Re: shred bug? [was: Unidentified subject!]

[David Wright]

On Tue 13 Feb 2024 at 07:15:48 (-0500), Greg Wooledge wrote:
> On Mon, Feb 12, 2024 at 11:01:47PM -0600, David Wright wrote:
> > … but not much. For me, "standard output" is /dev/fd/1, yet it seems
> > unlikely that anyone is going to use >&1 in the manner of the example.
> 
> Standard output means "whatever file descriptor 1 points to".  That
> could be a file, a pipe, a terminal (character device), etc.

Why pick on 1?

> > I might write something like: "The option ‘-’ shreds the file specified
> > by the redirection ‘>’", though there could be a better name for ‘>’.
> 
> You're assuming the program will be used from a shell.  This is *usually*
> going to be true, but nothing prevents you from writing a C program
> which closes stdout, opens a file, ensures that it's using FD 1,
> and then calls "shred -".  The documentation has to support this use
> case as well.

/As well/ — which is why I wrote N in place of 1. The original bug
report (which I hadn't seen until Thomas' post) says:
 "If you redirect output to a file it will work. Shredding a tty doesn't
  make much sense, after all."
  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=155175#10

Now, you can't write "If you redirect output to a file it will work"
in a man page—it needs recasting into something more like what
I wrote above, which contains two key points:

. It points out that '-' is an option, not a filename or a stand-in
  for one, and it doesn't use the word standard, which is totally
  irrelevant in the circumstances.

. It demonstrates the shell syntax element required (&) in order to
  avoid truncating the file, rather than shred overwriting it.

I think that getting the "&" into the man page would be helpful
to anybody who doesn't look at the info page for the example.
It might have shortened the early part of this thread as well.

As for C programmers, neither FD number nor truncation is relevant.
Sure, you can pick 1. But you don't have to document that for shred.
And truncation is an accident that can occur because of shell's
redirect syntax: there's no equivalent in programs.

> > >A FILE of ‘-’ denotes standard output.  The intended use of this is
> > >to shred a removed temporary file.  For example:
> > > 
> > >   i=$(mktemp)
> > >   exec 3<>"$i"
> > >   rm -- "$i"
> > >   echo "Hello, world" >&3
> > >   shred - >&3
> > >   exec 3>-
> > 
> > I can see that the last line truncates the "anonymous" file,
> 
> No, that's not what it does at all.  In fact, that last line is
> written incorrectly.  It should say "exec 3>&-" and what that does
> is close file descriptor 3, which was previously opened on line 2.
> 
> What it actually does *as written* is create/truncate a file whose
> name is "-", close the previously opened FD 3, and make FD 3 point
> to the file named "-".
> 
> unicorn:~$ exec 3>-
> unicorn:~$ ls -ld -- -
> -rw-r--r-- 1 greg greg 0 Feb 13 07:12 -
> unicorn:~$ ls -l /dev/fd/3
> l-wx-- 1 greg greg 64 Feb 13 07:12 /dev/fd/3 -> /home/greg/-
> 
> This is an obvious bug in the info page.  I wonder how many years
> this has gone unnoticed.

Well spotted. That's what an experienced eye brings to a line like
that, whereas I assumed it meant something beyond my experience,
and searched for it.

Ironic that it truncates a file, and then immediately warns against
truncating a file instead of shredding it.

Cheers,
David.

Re: shred bug? [was: Unidentified subject!]

[Thomas Schmitt]

Hi,

"info shred" says:
> > >   i=$(mktemp)
> > >   exec 3<>"$i"
> > >   rm -- "$i"
> > >   echo "Hello, world" >&3
> > >   shred - >&3
> > >   exec 3>-

Greg Wooledge wrote:
> In fact, that last line is
> written incorrectly.  It should say "exec 3>&-" and what that does
> is close file descriptor 3, which was previously opened on line 2.
> [...]
> This is an obvious bug in the info page.  I wonder how many years
> this has gone unnoticed.

  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=155175#36
of 22 Dec 2005 states:

  "I'll assume that this is now adequately explained in the info page
   (below).  If not then please reopen.  // Thomas Hood
   [...]
   exec 3>-"

The bug report is from 02 Aug 2002 and states that the info page contains
the short and broad promise which we can still see in "man shred".

So we can assume a bug age of 18 to 22 years.


Have a nice day :)

Thomas

Re: shred bug? [was: Unidentified subject!]

[tomas]

On Tue, Feb 13, 2024 at 07:36:14AM -0500, Greg Wooledge wrote:
> On Tue, Feb 13, 2024 at 07:15:48AM -0500, Greg Wooledge wrote:
> > This is an obvious bug in the info page.  I wonder how many years
> > this has gone unnoticed.
> 
> I've filed Bug#1063837 for it.  

Well, thanks for doing TRT :-)

Cheers
-- 
t


signature.asc
Description: PGP signature

Re: shred bug? [was: Unidentified subject!]

[Greg Wooledge]

On Tue, Feb 13, 2024 at 07:15:48AM -0500, Greg Wooledge wrote:
> This is an obvious bug in the info page.  I wonder how many years
> this has gone unnoticed.

I've filed Bug#1063837 for it.  

Re: shred bug? [was: Unidentified subject!]

[Greg Wooledge]

On Mon, Feb 12, 2024 at 11:01:47PM -0600, David Wright wrote:
> … but not much. For me, "standard output" is /dev/fd/1, yet it seems
> unlikely that anyone is going to use >&1 in the manner of the example.

Standard output means "whatever file descriptor 1 points to".  That
could be a file, a pipe, a terminal (character device), etc.

> I might write something like: "The option ‘-’ shreds the file specified
> by the redirection ‘>’", though there could be a better name for ‘>’.

You're assuming the program will be used from a shell.  This is *usually*
going to be true, but nothing prevents you from writing a C program
which closes stdout, opens a file, ensures that it's using FD 1,
and then calls "shred -".  The documentation has to support this use
case as well.

> >A FILE of ‘-’ denotes standard output.  The intended use of this is
> >to shred a removed temporary file.  For example:
> > 
> >   i=$(mktemp)
> >   exec 3<>"$i"
> >   rm -- "$i"
> >   echo "Hello, world" >&3
> >   shred - >&3
> >   exec 3>-
> 
> I can see that the last line truncates the "anonymous" file,

No, that's not what it does at all.  In fact, that last line is
written incorrectly.  It should say "exec 3>&-" and what that does
is close file descriptor 3, which was previously opened on line 2.

What it actually does *as written* is create/truncate a file whose
name is "-", close the previously opened FD 3, and make FD 3 point
to the file named "-".

unicorn:~$ exec 3>-
unicorn:~$ ls -ld -- -
-rw-r--r-- 1 greg greg 0 Feb 13 07:12 -
unicorn:~$ ls -l /dev/fd/3
l-wx-- 1 greg greg 64 Feb 13 07:12 /dev/fd/3 -> /home/greg/-

This is an obvious bug in the info page.  I wonder how many years
this has gone unnoticed.

Re: shred bug? [was: Unidentified subject!]

[tomas]

On Mon, Feb 12, 2024 at 10:07:45PM +0100, Thomas Schmitt wrote:
> Hi,
> 
> > https://en.wikipedia.org/wiki/Everything_is_a_file
> > But, there is more than one kind of file.
> 
> "All files are equal.
>  But some files are more equal than others."
> 
> (George Orwell in his dystopic novel "Server Farm".)

:-D

Yesterday I was on the brink of quoting that. Great minds and
all of that...

Reality though is, that if you don't design your file with
magical properties from the get-go, you will keep stumbling
on stuff you want to model and which don't fit the real
life file design you chose.

And yes, Plan 9, as someone else noted in this thread, takes
that to a bigger extreme: in their windowing system, windows
are files, too -- you can remove a window by deleting the
corresponding file.

Cheers
-- 
t


signature.asc
Description: PGP signature

Re: shred bug? [was: Unidentified subject!]

[David Wright]

On Sun 11 Feb 2024 at 09:16:00 (-0600), David Wright wrote:
> On Sun 11 Feb 2024 at 09:54:24 (-0500), Greg Wooledge wrote:
> > On Sun, Feb 11, 2024 at 03:45:21PM +0100, to...@tuxteam.de wrote:
> > > Still there's the discrepancy between doc and behaviour.
> > 
> > There isn't.  The documentation says:
> > 
> > SYNOPSIS
> >shred [OPTION]... FILE...
> > 
> > DESCRIPTION
> >Overwrite  the specified FILE(s) repeatedly, in order to make it 
> > harder
> >for even very expensive hardware probing to recover the data.
> > 
> >If FILE is -, shred standard output.
> > 
> > In every sentence, the word FILE appears.  There's nothing in there
> > which says "you can operate on a non-file".
> > 
> > Once you grasp what the command is *intended* to do (rewind and overwrite
> > a file repeatedly), it makes absolutely perfect sense that it should
> > only operate on rewindable file system objects.
> > 
> > If you want it to write a stream of data instead of performing its normal
> > operation (rewinding and rewriting), that's a new feature.
> > 
> > If you'd prefer the documentation to say explicitly "only regular files
> > and block devices are allowed", that would be an upstream documentation
> > *clarification* request.
> 
> Perhaps info puts it better?

… but not much. For me, "standard output" is /dev/fd/1, yet it seems
unlikely that anyone is going to use >&1 in the manner of the example.

I might write something like: "The option ‘-’ shreds the file specified
by the redirection ‘>’", though there could be a better name for ‘>’.

>A FILE of ‘-’ denotes standard output.  The intended use of this is
>to shred a removed temporary file.  For example:
> 
>   i=$(mktemp)
>   exec 3<>"$i"
>   rm -- "$i"
>   echo "Hello, world" >&3
>   shred - >&3
>   exec 3>-

I can see that the last line truncates the "anonymous" file, but where
is that construction documented¹, and how would one parse the syntax
elements   FD  >  -   to make them mean truncate?

>However, the command ‘shred - >file’ does not shred the contents of
>FILE, since the shell truncates FILE before invoking ‘shred’.  Use
>the command ‘shred file’ or (if using a Bourne-compatible shell) the
>command ‘shred - 1<>file’ instead.

¹ the string ">-" doesn't appear in /usr/share/doc/bash/bashref.pdf,
  ver 5.1, for example.

Cheers,
David.

Re: shred bug? [was: Unidentified subject!]

[Max Nikulin]

On 12/02/2024 05:41, David Christensen wrote:


Apparently, shred(1) has both an info(1) page (?) and a man(1) page. The 
obvious solution is to write one document that is complete and correct, 
and use it everywhere -- e.g. DRY.


https://www.gnu.org/prep/standards/html_node/Man-Pages.html
6.9 Man Pages in "GNU Coding Standards"

In the GNU project, man pages are secondary. It is not necessary or
expected for every GNU program to have a man page, but some of them do.


A standalone man page is not the same as a section in a document 
describing the whole bundle.


Notice that man uses direct formatting, not logical markup. E.g. there 
is no dedicated structure for links to other man pages. They were not 
designed as hypertext documents, they are to be printed on paper. In 
this sense texinfo is a more advanced format.


P.S. I admit that in some cases "man bash" may be more convenient for 
searching than an info browser.

Re: Fast Random Data Generation (Was: Re: Unidentified subject!)

[David Christensen]

On 2/12/24 08:30, Linux-Fan wrote:

David Christensen writes:


On 2/11/24 02:26, Linux-Fan wrote:
I wrote a program to automatically generate random bytes in multiple 
threads:

https://masysma.net/32/big4.xhtml


What algorithm did you implement?


I copied the algorithm from here:
https://www.javamex.com/tutorials/random_numbers/numerical_recipes.shtml



That Java code uses locks, which implies it uses global state and cannot 
be run multi-threaded (?).  (E.g. one process with one JVM.)



Is it possible to obtain parallel operation on an SMP machine with 
multiple virtual processors?  (Other than multiple OS processes with one 
PRNG on one JVM each?)



I found it during the development of another application where I needed 
a lot of random data for simulation purposes :)


My implementation code is here:
https://github.com/m7a/bo-big/blob/master/latest/Big4.java

If I were to do it again today, I'd probably switch to any of these PRNGS:

* https://burtleburtle.net/bob/rand/smallprng.html
* https://www.pcg-random.org/



Hard core.  I'll let the experts figure it out; and then I will use 
their libraries and programs.



David

Re: Fast Random Data Generation (Was: Re: Unidentified subject!)

[Jeffrey Walton]

On Mon, Feb 12, 2024 at 3:02 PM Linux-Fan  wrote:
>
> David Christensen writes:
>
> > On 2/11/24 02:26, Linux-Fan wrote:
> >> I wrote a program to automatically generate random bytes in multiple 
> >> threads:
> >> https://masysma.net/32/big4.xhtml
> >>
> >> Before knowing about `fio` this way my way to benchmark SSDs :)
> >>
> >> Example:
> >>
> >> | $ big4 -b /dev/null 100 GiB
> >> | Ma_Sys.ma Big 4.0.2, Copyright (c) 2014, 2019, 2020 Ma_Sys.ma.
> >> | For further info send an e-mail to ma_sys...@web.de.
>
> [...]
>
> >> | 99.97% +8426 MiB 7813 MiB/s 102368/102400 MiB
> >> | Wrote 102400 MiB in 13 s @ 7812.023 MiB/s
> >
> >
> > What algorithm did you implement?
>
> I copied the algorithm from here:
> https://www.javamex.com/tutorials/random_numbers/numerical_recipes.shtml
>
> I found it during the development of another application where I needed a
> lot of random data for simulation purposes :)

A PRNG for a simulation has different requirements than a PRNG for
cryptographic purposes. A simulation usually needs numbers fast from a
uniform distribution. Simulations can use predictable numbers. Often a
Linear Congurential Generator (LCG) will do just fine even though they
were broken about 35 years ago. See Also see Joan Boyer's Inferring
Sequences Produced by Pseudo-Random Number Generators,
.

A cryptographic application will have more stringent requirements. A
cryptographic generator may (will?) take longer to generate a number,
the numbers need to come from a uniform distribution, and the numbers
need to be prediction resistant. You can read about the cryptographic
qualities of random numbers in NIST SP800-90 and friends.

> My implementation code is here:
> https://github.com/m7a/bo-big/blob/master/latest/Big4.java
>
> If I were to do it again today, I'd probably switch to any of these PRNGS:
>
>  * https://burtleburtle.net/bob/rand/smallprng.html
>  * https://www.pcg-random.org/
>
> >> Secure Random can be obtained from OpenSSL:
> >>
> >> | $ time for i in `seq 1 100`; do openssl rand -out /dev/null $((1024 * 
> >> 1024
> >> * 1024)); done
> >> |
> >> | real0m49.288s
> >> | user0m44.710s
> >> | sys0m4.579s
> >>
> >> Effectively 2078 MiB/s (quite OK for single-threaded operation). It is not
> >> designed to generate large amounts of random data as the size is limited by
> >> integer range...

Jeff

Re: shred bug? [was: Unidentified subject!]

[Thomas Schmitt]

Hi,

> https://en.wikipedia.org/wiki/Everything_is_a_file
> But, there is more than one kind of file.

"All files are equal.
 But some files are more equal than others."

(George Orwell in his dystopic novel "Server Farm".)


Have a nice day :)

Thomas

Re: shred bug? [was: Unidentified subject!]

[David Christensen]

On 2/12/24 08:50, Curt wrote:

On 2024-02-11,   wrote:



On Sun, Feb 11, 2024 at 09:54:24AM -0500, Greg Wooledge wrote:

[...]


If FILE is -, shred standard output.
=20
In every sentence, the word FILE appears.  There's nothing in there
which says "you can operate on a non-file".


Point taken, yes.


I thought everything was a file.



"Everything is a file" is a design feature of the Unix operating system:

https://en.wikipedia.org/wiki/Everything_is_a_file


But, there is more than one kind of file.


And, not every program supports every kind of file.


The manual page for find(1) provides a shopping list of file types it 
supports:


2024-02-12 12:32:13 dpchrist@laalaa ~
$ man find | egrep -A 20 '^   .type c'
   -type c
  File is of type c:

  b  block (buffered) special

  c  character (unbuffered) special

  d  directory

  p  named pipe (FIFO)

  f  regular file

  l  symbolic link; this is never true if  the
 -L option or the -follow option is in ef-
 fect, unless the symbolic link is broken.
 If  you want to search for symbolic links
 when -L is in effect, use -xtype.

  s  socket


As for shred(1), the argument FILE is conventionally a regular file.  We 
are discussing the special case described in the manual page:


   If FILE is -, shred standard output.


David

Re: shred bug? [was: Unidentified subject!]

[Greg Wooledge]

On Mon, Feb 12, 2024 at 04:50:50PM -, Curt wrote:
> On 2024-02-11,   wrote:
> >
> >
> > On Sun, Feb 11, 2024 at 09:54:24AM -0500, Greg Wooledge wrote:
> >
> > [...]
> >
> >>If FILE is -, shred standard output.
> >>=20
> >> In every sentence, the word FILE appears.  There's nothing in there
> >> which says "you can operate on a non-file".
> >
> > Point taken, yes.
> 
> I thought everything was a file.

An anonymous pipe(2) in memory between two processes isn't even close to
a file.  Also, you're confusing Linux and Plan 9.  Linux has a bunch of
things that aren't files, such as network interfaces.

Re: shred bug? [was: Unidentified subject!]

[Curt]

On 2024-02-11,   wrote:
>
>
> On Sun, Feb 11, 2024 at 09:54:24AM -0500, Greg Wooledge wrote:
>
> [...]
>
>>If FILE is -, shred standard output.
>>=20
>> In every sentence, the word FILE appears.  There's nothing in there
>> which says "you can operate on a non-file".
>
> Point taken, yes.

I thought everything was a file.

Re: Fast Random Data Generation (Was: Re: Unidentified subject!)

[Linux-Fan]

David Christensen writes:


On 2/11/24 02:26, Linux-Fan wrote:

I wrote a program to automatically generate random bytes in multiple threads:
https://masysma.net/32/big4.xhtml

Before knowing about `fio` this way my way to benchmark SSDs :)

Example:

| $ big4 -b /dev/null 100 GiB
| Ma_Sys.ma Big 4.0.2, Copyright (c) 2014, 2019, 2020 Ma_Sys.ma.
| For further info send an e-mail to ma_sys...@web.de.


[...]


| 99.97% +8426 MiB 7813 MiB/s 102368/102400 MiB
| Wrote 102400 MiB in 13 s @ 7812.023 MiB/s



What algorithm did you implement?


I copied the algorithm from here:
https://www.javamex.com/tutorials/random_numbers/numerical_recipes.shtml

I found it during the development of another application where I needed a  
lot of random data for simulation purposes :)


My implementation code is here:
https://github.com/m7a/bo-big/blob/master/latest/Big4.java

If I were to do it again today, I'd probably switch to any of these PRNGS:

* https://burtleburtle.net/bob/rand/smallprng.html
* https://www.pcg-random.org/


Secure Random can be obtained from OpenSSL:

| $ time for i in `seq 1 100`; do openssl rand -out /dev/null $((1024 * 1024  
* 1024)); done

|
| real    0m49.288s
| user    0m44.710s
| sys    0m4.579s

Effectively 2078 MiB/s (quite OK for single-threaded operation). It is not  
designed to generate large amounts of random data as the size is limited by  
integer range...



Thank you for posting the openssl(1) incantation.


You're welcome.

[...]

HTH
Linux-Fan

öö


pgpjvuqb6Fy1L.pgp
Description: PGP signature

Re: Fast Random Data Generation (Was: Re: Unidentified subject!)

[David Christensen]

On 2/11/24 02:26, Linux-Fan wrote:
I wrote a program to automatically generate random bytes in multiple 
threads:

https://masysma.net/32/big4.xhtml

Before knowing about `fio` this way my way to benchmark SSDs :)

Example:

| $ big4 -b /dev/null 100 GiB
| Ma_Sys.ma Big 4.0.2, Copyright (c) 2014, 2019, 2020 Ma_Sys.ma.
| For further info send an e-mail to ma_sys...@web.de.
|| 0.00% +0 MiB 0 MiB/s 0/102400 MiB
| 3.48% +3562 MiB 3255 MiB/s 3562/102400 MiB
| 11.06% +7764 MiB 5407 MiB/s 11329/102400 MiB
| 19.31% +8436 MiB 6387 MiB/s 19768/102400 MiB
| 27.71% +8605 MiB 6928 MiB/s 28378/102400 MiB
| 35.16% +7616 MiB 7062 MiB/s 35999/102400 MiB
| 42.58% +7595 MiB 7150 MiB/s 43598/102400 MiB
| 50.12% +7720 MiB 7230 MiB/s 51321/102400 MiB
| 58.57% +8648 MiB 7405 MiB/s 59975/102400 MiB
| 66.96% +8588 MiB 7535 MiB/s 68569/102400 MiB
| 75.11% +8343 MiB 7615 MiB/s 76916/102400 MiB
| 83.38% +8463 MiB 7691 MiB/s 85383/102400 MiB
| 91.74% +8551 MiB 7762 MiB/s 93937/102400 MiB
| 99.97% +8426 MiB 7813 MiB/s 102368/102400 MiB
|| Wrote 102400 MiB in 13 s @ 7812.023 MiB/s



What algorithm did you implement?



Secure Random can be obtained from OpenSSL:

| $ time for i in `seq 1 100`; do openssl rand -out /dev/null $((1024 * 
1024 * 1024)); done

|
| real    0m49.288s
| user    0m44.710s
| sys    0m4.579s

Effectively 2078 MiB/s (quite OK for single-threaded operation). It is 
not designed to generate large amounts of random data as the size is 
limited by integer range...



Thank you for posting the openssl(1) incantation.


Benchmarking my daily driver laptop without Intel Secure Key:

2024-02-11 21:54:04 dpchrist@laalaa ~
$ lscpu | grep 'Model name'
Model name: Intel(R) Core(TM) i7-2720QM CPU @ 
2.20GHz


2024-02-11 21:54:09 dpchrist@laalaa ~
$ time for i in `seq 1 100`; do openssl rand -out /dev/null $((1024 * 
1024 * 1024)); done


real1m40.149s
user1m25.174s
sys 0m14.952s


So, ~1.072E+9 bytes per second.


Benchmarking a workstation with Intel Secure Key:

2024-02-11 21:54:40 dpchrist@taz ~
$ lscpu | grep 'Model name'
Model name: Intel(R) Xeon(R) E-2174G CPU @ 3.80GHz

2024-02-11 21:54:46 dpchrist@taz ~
$ time for i in `seq 1 100`; do openssl rand -out /dev/null $((1024 * 
1024 * 1024)); done


real1m14.696s
user1m0.338s
sys 0m14.353s


So, ~1.437E+09 bytes per second.


David

Re: shred bug? [was: Unidentified subject!]

[David Christensen]

On 2/11/24 06:54, Greg Wooledge wrote:

On Sun, Feb 11, 2024 at 03:45:21PM +0100, to...@tuxteam.de wrote:

On Sun, Feb 11, 2024 at 09:37:31AM -0500, Greg Wooledge wrote:

On Sun, Feb 11, 2024 at 08:02:12AM +0100, to...@tuxteam.de wrote:


[...]


What Thomas was trying to do is to get a cheap, fast random number
generator. Shred seems to have such.


Well... I certainly wouldn't call it a bug.  Maybe a feature request.


Still there's the discrepancy between doc and behaviour.


There isn't.  The documentation says:

SYNOPSIS
shred [OPTION]... FILE...



I interpret the above line to be a prototype for invoking the shred(1) 
program:


* "shred" is the program name

* "[OPTION]..." is one or more option specifiers that may be omitted. 
Each should be described below.


* "FILE..." is one or more argument specifies that should be file system 
paths (strings).




DESCRIPTION
Overwrite  the specified FILE(s) repeatedly, in order to make it harder
for even very expensive hardware probing to recover the data.

If FILE is -, shred standard output.



I interpret the above line at face value -- if the caller provides a 
dash as the argument, shred(1) will operate on standard output.




In every sentence, the word FILE appears.  There's nothing in there
which says "you can operate on a non-file".



Dash is not a file, yet the above sentence says shred(1) can operate on it.



Once you grasp what the command is *intended* to do (rewind and overwrite
a file repeatedly), it makes absolutely perfect sense that it should
only operate on rewindable file system objects.



An expert may infer what you have stated, but I prefer manual pages that 
are explicit.



The GNU project must have found a need for the FILE='-' feature, 
otherwise it would not exist.  The manual page should describe that need 
(e.g. why) and how to properly use shred(1) to solve the need.




If you want it to write a stream of data instead of performing its normal
operation (rewinding and rewriting), that's a new feature.



Humans are (in)famous for doing unexpected things.



If you'd prefer the documentation to say explicitly "only regular files
and block devices are allowed", that would be an upstream documentation
*clarification* request.



Apparently, shred(1) has both an info(1) page (?) and a man(1) page. 
The obvious solution is to write one document that is complete and 
correct, and use it everywhere -- e.g. DRY.



David

Re: Unidentified subject!

[David Christensen]

On 2/11/24 03:13, Thomas Schmitt wrote:

Hi,

David Christensen wrote:

Concurrency:
threads throughput
8   205+198+180+195+205+184+184+189=1,540 MB/s


There remains the question how to join these streams without losing speed
in order to produce a single checksum. (Or one would have to divide the
target into 8 areas which get checked separately.)



I had similar thoughts.  A FIFO should be able to join the streams. 
But, dividing the device by the number of virtual cores and putting a 
thread on each makes more sense.  Either done right should fill the 
drive I/O capacity.




Does this 8 thread generator cause any problems with the usability of
the rest of the system ? Sluggish program behavior or so ?



CPU Graph shows all eight virtual cores at 100%, so everything else on 
the system would be sluggish (unless you use nice(1)).



Here is a processor with Intel Secure Key and otherwise unloaded:

2024-02-11 11:48:21 dpchrist@taz ~
$ lscpu | grep 'Model name'
Model name: Intel(R) Xeon(R) E-2174G CPU @ 3.80GHz

2024-02-11 11:59:55 dpchrist@taz ~
$ cat /etc/debian_version ; uname -a
11.8
Linux taz 5.10.0-27-amd64 #1 SMP Debian 5.10.205-2 (2023-12-31) x86_64 
GNU/Linux


2024-02-11 12:02:52 dpchrist@taz ~
$ dd if=/dev/urandom of=/dev/null bs=1M count=10K
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB, 10 GiB) copied, 20.0469 s, 536 MB/s

threads throughput
1   536 MB/s
2   512+512 = 1,024 MB/s
3   502+503+503 = 1,508 MB/s
4   492+491+492+492 = 1,967 MB/s
5   492+384+491+385+491 = 2,243 MB/s
6   379+491+492+379+379+379 = 2,499 MB/s
7   352+491+356+388+352+357+388 = 2,684 MB/s
8   355+354+344+348+344+354+353+349 = 2,801 MB/s



I have to correct my previous measurement on the 4 GHz Xeon, which was
made with a debuggable version of the program that produced the stream.
The production binary which is compiled with -O2 can write 2500 MB/s into
a pipe with a pacifier program which counts the data:

   $ time $(scdbackup -where bin)/cd_backup_planer -write_random - 100g 
2s62gss463ar46492bni | $(scdbackup -where bin)/raedchen -step 100m -no_output 
-print_count
   100.0g bytes

   real0m39.884s
   user0m30.629s
   sys 0m41.013s

(One would have to install scdbackup to reproduce this and to see raedchen
  count the bytes while spinning the classic SunOS boot wheel: |/-\|/-\|/-\
http://scdbackup.webframe.org/main_eng.html
http://scdbackup.webframe.org/examples.html
  Oh nostalgy ...
)

Totally useless but yielding nearly 4000 MB/s:

   $ time $(scdbackup -where bin)/cd_backup_planer -write_random - 100g 
2s62gss463ar46492bni >/dev/null

   real0m27.064s
   user0m23.433s
   sys 0m3.646s

The main bottleneck in my proposal would be the checksummer:

   $ time $(scdbackup -where bin)/cd_backup_planer -write_random - 100g 
2s62gss463ar46492bni | md5sum
   5a6ba41c2c18423fa33355005445c183  -

   real2m8.160s
   user2m25.599s
   sys 0m22.663s

That's quite exactly 800 MiB/s ~= 6.7 Gbps.
Still good enough for vanilla USB-3 with a fast SSD, i'd say.



Yes -- more than enough throughput.


Before I knew of fdupes(1) and jdupes(1), I wrote a Perl script to find 
duplicate files.  It uses the Digest module, and supports any algorithm 
supported by that module.  Here are some runs against a local ext4 on 
LUKS (with AES-NI) on Intel SSD 520 Series 60 GB and check summing whole 
files:


2024-02-11 13:32:47 dpchrist@taz ~
$ time finddups --filter w --digest MD4 .thunderbird/ >/dev/null

real0m0.878s
user0m0.741s
sys 0m0.137s

2024-02-11 13:33:14 dpchrist@taz ~
$ time finddups --filter w --digest MD5 .thunderbird/ >/dev/null

real0m1.110s
user0m0.977s
sys 0m0.132s

2024-02-11 13:33:19 dpchrist@taz ~
$ time finddups --filter w --digest SHA-1 .thunderbird/ >/dev/null

real0m1.306s
user0m1.151s
sys 0m0.156s

2024-02-11 13:36:40 dpchrist@taz ~
$ time finddups --filter w --digest SHA-256 .thunderbird/ >/dev/null

real0m2.545s
user0m2.424s
sys 0m0.121s

2024-02-11 13:36:51 dpchrist@taz ~
$ time finddups --filter w --digest SHA-384 .thunderbird/ >/dev/null

real0m1.808s
user0m1.652s
sys 0m0.157s

2024-02-11 13:37:00 dpchrist@taz ~
$ time finddups --filter w --digest SHA-512 .thunderbird/ >/dev/null

real0m1.814s
user0m1.673s
sys 0m0.141s


It is curious that SHA-384 and SHA-512 are faster than SHA-256.  I can 
confirm similar results on:


2024-02-11 13:39:58 dpchrist@laalaa ~
$ lscpu | grep 'Model name'
Model name: Intel(R) Core(TM) i7-2720QM CPU @ 
2.20GHz



David

Re: Unidentified subject!

[David Christensen]

On 2/11/24 00:07, Thomas Schmitt wrote:

In the other thread about the /dev/sdm test:

Gene Heskett wrote:

Creating file 39.h2w ... 1.98% -- 1.90 MB/s -- 257:11:32
[...]
$ sudo f3probe --destructive --time-ops /dev/sdm
Bad news: The device `/dev/sdm' is a counterfeit of type limbo
Device geometry:
  *Usable* size: 59.15 GB (124050944 blocks)
 Announced size: 1.91 TB (409600 blocks)


David Christensen wrote:

Which other thread?  Please provide a URL to archived post.


https://lists.debian.org/msgid-search/e7a0c1a1-f973-4007-a86d-8d91d8d91...@shentel.net
https://lists.debian.org/msgid-search/b566504b-5677-4d84-b5b3-b0de63230...@shentel.net



Thank you.  :-)


David

Re: shred bug? [was: Unidentified subject!]

[David Wright]

On Sun 11 Feb 2024 at 09:54:24 (-0500), Greg Wooledge wrote:
> On Sun, Feb 11, 2024 at 03:45:21PM +0100, to...@tuxteam.de wrote:
> > On Sun, Feb 11, 2024 at 09:37:31AM -0500, Greg Wooledge wrote:
> > > On Sun, Feb 11, 2024 at 08:02:12AM +0100, to...@tuxteam.de wrote:
> > 
> > [...]
> > 
> > > > What Thomas was trying to do is to get a cheap, fast random number
> > > > generator. Shred seems to have such.
> > > 
> > > Well... I certainly wouldn't call it a bug.  Maybe a feature request.
> > 
> > Still there's the discrepancy between doc and behaviour.
> 
> There isn't.  The documentation says:
> 
> SYNOPSIS
>shred [OPTION]... FILE...
> 
> DESCRIPTION
>Overwrite  the specified FILE(s) repeatedly, in order to make it harder
>for even very expensive hardware probing to recover the data.
> 
>If FILE is -, shred standard output.
> 
> In every sentence, the word FILE appears.  There's nothing in there
> which says "you can operate on a non-file".
> 
> Once you grasp what the command is *intended* to do (rewind and overwrite
> a file repeatedly), it makes absolutely perfect sense that it should
> only operate on rewindable file system objects.
> 
> If you want it to write a stream of data instead of performing its normal
> operation (rewinding and rewriting), that's a new feature.
> 
> If you'd prefer the documentation to say explicitly "only regular files
> and block devices are allowed", that would be an upstream documentation
> *clarification* request.

Perhaps info puts it better?

   A FILE of ‘-’ denotes standard output.  The intended use of this is
   to shred a removed temporary file.  For example:

  i=$(mktemp)
  exec 3<>"$i"
  rm -- "$i"
  echo "Hello, world" >&3
  shred - >&3
  exec 3>-

   However, the command ‘shred - >file’ does not shred the contents of
   FILE, since the shell truncates FILE before invoking ‘shred’.  Use
   the command ‘shred file’ or (if using a Bourne-compatible shell) the
   command ‘shred - 1<>file’ instead.

Cheers,
David.

Re: shred bug? [was: Unidentified subject!]

[Thomas Schmitt]

Hi,

to...@tuxteam.de wrote:
> Still there's the discrepancy between doc and behaviour.

Depends at which documentation you look. Obviously stemming from
  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=155175#36
i read in
  https://www.gnu.org/software/coreutils/manual/html_node/shred-invocation.html

 "A file of ‘-’ denotes standard output. The intended use of this is to
  shred a removed temporary file. For example: [shell wizzardry]"

It works as long as stdout is connected to a data file, or block device,
or directory, or symbolic link, or to a character device that is not a
terminal.
(Maybe it refuses later on some of these types, but not at the location
with the message "invalid file type". I wonder if i can connect stdout
to a symbolic link instead of its target.)

The bug would thus have to be filed against the man page
  https://sources.debian.org/src/coreutils/9.4-3/man/shred.1/
which says only

  "If FILE is \-, shred standard output."

The info empire of coreutils says what above web manual says.
  https://sources.debian.org/src/coreutils/9.4-3/doc/coreutils.texi/#L10705


Have a nice day :)

Thomas

Re: Unidentified subject!

[Jeffrey Walton]

On Sun, Feb 11, 2024 at 9:52 AM Thomas Schmitt  wrote:
>
> David Christensen wrote:
> > Concurrency:
> > threads throughput
> > 8   205+198+180+195+205+184+184+189=1,540 MB/s
>
> There remains the question how to join these streams without losing speed
> in order to produce a single checksum. (Or one would have to divide the
> target into 8 areas which get checked separately.)

Hash Tree or Merkle Tree. They are used in blockchains.

> Does this 8 thread generator cause any problems with the usability of
> the rest of the system ? Sluggish program behavior or so ?
>
> The main reason to have my own low-quality implementation of a random
> stream was that /dev/urandom was too slow for 12x speed (= 1.8 MB/s) CD-RW
> media and that higher random quality still put too much load on a
> single-core 600 MHz Pentium system. That was nearly 25 years ago.

Jeff

Re: shred bug? [was: Unidentified subject!]

[tomas]

On Sun, Feb 11, 2024 at 09:54:24AM -0500, Greg Wooledge wrote:

[...]

>If FILE is -, shred standard output.
> 
> In every sentence, the word FILE appears.  There's nothing in there
> which says "you can operate on a non-file".

Point taken, yes.

Cheers
-- 
t


signature.asc
Description: PGP signature

Re: shred bug? [was: Unidentified subject!]

[Greg Wooledge]

On Sun, Feb 11, 2024 at 03:45:21PM +0100, to...@tuxteam.de wrote:
> On Sun, Feb 11, 2024 at 09:37:31AM -0500, Greg Wooledge wrote:
> > On Sun, Feb 11, 2024 at 08:02:12AM +0100, to...@tuxteam.de wrote:
> 
> [...]
> 
> > > What Thomas was trying to do is to get a cheap, fast random number
> > > generator. Shred seems to have such.
> > 
> > Well... I certainly wouldn't call it a bug.  Maybe a feature request.
> 
> Still there's the discrepancy between doc and behaviour.

There isn't.  The documentation says:

SYNOPSIS
   shred [OPTION]... FILE...

DESCRIPTION
   Overwrite  the specified FILE(s) repeatedly, in order to make it harder
   for even very expensive hardware probing to recover the data.

   If FILE is -, shred standard output.

In every sentence, the word FILE appears.  There's nothing in there
which says "you can operate on a non-file".

Once you grasp what the command is *intended* to do (rewind and overwrite
a file repeatedly), it makes absolutely perfect sense that it should
only operate on rewindable file system objects.

If you want it to write a stream of data instead of performing its normal
operation (rewinding and rewriting), that's a new feature.

If you'd prefer the documentation to say explicitly "only regular files
and block devices are allowed", that would be an upstream documentation
*clarification* request.

Re: shred bug? [was: Unidentified subject!]

[tomas]

On Sun, Feb 11, 2024 at 09:37:31AM -0500, Greg Wooledge wrote:
> On Sun, Feb 11, 2024 at 08:02:12AM +0100, to...@tuxteam.de wrote:

[...]

> > What Thomas was trying to do is to get a cheap, fast random number
> > generator. Shred seems to have such.
> 
> Well... I certainly wouldn't call it a bug.  Maybe a feature request.

Still there's the discrepancy between doc and behaviour.

Cheers
-- 
t


signature.asc
Description: PGP signature

Re: shred bug? [was: Unidentified subject!]

[Greg Wooledge]

On Sun, Feb 11, 2024 at 08:02:12AM +0100, to...@tuxteam.de wrote:
> On Sat, Feb 10, 2024 at 07:10:54PM -0500, Greg Wooledge wrote:
> > On Sat, Feb 10, 2024 at 04:05:21PM -0800, David Christensen wrote:
> > > 2024-02-10 16:03:50 dpchrist@laalaa ~
> > > $ shred -s 1K - | wc -c
> > > shred: -: invalid file type
> > > 0
> > > 
> > > 
> > > It looks like a shred(1) needs a bug report.
> > 
> > I'm confused what you expected this command to do.  You wanted to
> > "destroy" (by overwriting with random data) a pipe to wc?  What
> > would that even look like?
> 
> What Thomas was trying to do is to get a cheap, fast random number
> generator. Shred seems to have such.

Well... I certainly wouldn't call it a bug.  Maybe a feature request.

Re: shred bug? [was: Unidentified subject!]

[debian-user]

David Christensen  wrote:
> On 2/10/24 16:10, Greg Wooledge wrote:
> > On Sat, Feb 10, 2024 at 04:05:21PM -0800, David Christensen wrote:  
> >> 2024-02-10 16:03:50 dpchrist@laalaa ~
> >> $ shred -s 1K - | wc -c
> >> shred: -: invalid file type
> >> 0
> >>
> >>
> >> It looks like a shred(1) needs a bug report.  
> > 
> > I'm confused what you expected this command to do.  You wanted to
> > "destroy" (by overwriting with random data) a pipe to wc?  What
> > would that even look like?
> > 
> > The basic premise of shred is that it determines the size of the
> > file, then writes data over it, rewinds it, and repeats this a few
> > times. A pipe to a process has no size, and can't be rewound.
> > 
> > Declaring a pipe to be an "invalid file type" for shredding sounds
> > pretty reasonable to me.  
> 
> 
> The documentation is confusing:
> 
> On 2/10/24 16:05, David Christensen wrote:
>  > 2024-02-10 16:03:42 dpchrist@laalaa ~
>  > $ man shred | grep 'If FILE is -'
>  > If FILE is -, shred standard output.  

Maybe it is unstated but mandatory to use -n 1 as well?
And optionally -s N?
I expect reading the code would tell.

First time I've read the man page properly.
Interesting point about COW filesystems such as btrfs :)

Re: Fast Random Data Generation (Was: Re: Unidentified subject!)

[Gremlin]

On 2/11/24 05:26, Linux-Fan wrote:

David Christensen writes:


On 2/11/24 00:11, Thomas Schmitt wrote:


[...]


Increase block size:

2024-02-11 01:18:51 dpchrist@laalaa ~
$ dd if=/dev/urandom of=/dev/null bs=1M count=1K
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 3.62874 s, 296 MB/s


Here (Intel Xeon W-2295)

| $ dd if=/dev/urandom of=/dev/null bs=1M count=1K
| 1024+0 records in
| 1024+0 records out
| 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.15018 s, 499 MB/s



Raspberry pi 5:

dd if=/dev/urandom of=/dev/null bs=1M count=1K
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 3.0122 s, 356 MB/s

Now lets do it right and use random..
dd if=/dev/random of=/dev/null bs=1M count=1K
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.9859 s, 360 MB/s


Secure Random can be obtained from OpenSSL:

| $ time for i in `seq 1 100`; do openssl rand -out /dev/null $((1024 * 
1024 * 1024)); done

|
| real    0m49.288s
| user    0m44.710s
| sys    0m4.579s


time for i in `seq 1 100`; do openssl rand -out /dev/null $((1024 * 1024 
* 1024)); done


real1m30.884s
user1m24.528s
sys 0m6.116s

Re: Unidentified subject!

[Thomas Schmitt]

Hi,

David Christensen wrote:
> Concurrency:
> threads throughput
> 8   205+198+180+195+205+184+184+189=1,540 MB/s

There remains the question how to join these streams without losing speed
in order to produce a single checksum. (Or one would have to divide the
target into 8 areas which get checked separately.)

Does this 8 thread generator cause any problems with the usability of
the rest of the system ? Sluggish program behavior or so ?

The main reason to have my own low-quality implementation of a random
stream was that /dev/urandom was too slow for 12x speed (= 1.8 MB/s) CD-RW
media and that higher random quality still put too much load on a
single-core 600 MHz Pentium system. That was nearly 25 years ago.


I wrote:
> > Last time i tested /dev/urandom it was much slower on comparable machines
> > and also became slower as the amount grew.

> Did you figure out why the Linux random number subsystem slowed, and at what
> amount?

No. I cannot even remember when and why i had reason to compare it with
my own stream generator. Maybe 5 or 10 years ago. The throughput was
more than 100 times better.


I have to correct my previous measurement on the 4 GHz Xeon, which was
made with a debuggable version of the program that produced the stream.
The production binary which is compiled with -O2 can write 2500 MB/s into
a pipe with a pacifier program which counts the data:

  $ time $(scdbackup -where bin)/cd_backup_planer -write_random - 100g 
2s62gss463ar46492bni | $(scdbackup -where bin)/raedchen -step 100m -no_output 
-print_count
  100.0g bytes

  real0m39.884s
  user0m30.629s
  sys 0m41.013s

(One would have to install scdbackup to reproduce this and to see raedchen
 count the bytes while spinning the classic SunOS boot wheel: |/-\|/-\|/-\
   http://scdbackup.webframe.org/main_eng.html
   http://scdbackup.webframe.org/examples.html
 Oh nostalgy ...
)

Totally useless but yielding nearly 4000 MB/s:

  $ time $(scdbackup -where bin)/cd_backup_planer -write_random - 100g 
2s62gss463ar46492bni >/dev/null

  real0m27.064s
  user0m23.433s
  sys 0m3.646s

The main bottleneck in my proposal would be the checksummer:

  $ time $(scdbackup -where bin)/cd_backup_planer -write_random - 100g 
2s62gss463ar46492bni | md5sum
  5a6ba41c2c18423fa33355005445c183  -

  real2m8.160s
  user2m25.599s
  sys 0m22.663s

That's quite exactly 800 MiB/s ~= 6.7 Gbps.
Still good enough for vanilla USB-3 with a fast SSD, i'd say.


> TIMTOWTDI.  :-)

Looks like another example of a weak random stream. :))


Have a nice day :)

Thomas

Fast Random Data Generation (Was: Re: Unidentified subject!)

[Linux-Fan]

David Christensen writes:


On 2/11/24 00:11, Thomas Schmitt wrote:


[...]


Increase block size:

2024-02-11 01:18:51 dpchrist@laalaa ~
$ dd if=/dev/urandom of=/dev/null bs=1M count=1K
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 3.62874 s, 296 MB/s


Here (Intel Xeon W-2295)

| $ dd if=/dev/urandom of=/dev/null bs=1M count=1K
| 1024+0 records in
| 1024+0 records out
| 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.15018 s, 499 MB/s


Concurrency:

threads throughput
1   296 MB/s
2   285+286=571 MB/s
3   271+264+266=801 MB/s
4   249+250+241+262=1,002 MB/s
5   225+214+210+224+225=1,098 MB/s
6   223+199+199+204+213+205=1,243 MB/s
7   191+209+210+204+213+201+197=1,425 MB/s
8   205+198+180+195+205+184+184+189=1,540 MB/s


I wrote a program to automatically generate random bytes in multiple threads:
https://masysma.net/32/big4.xhtml

Before knowing about `fio` this way my way to benchmark SSDs :)

Example:

| $ big4 -b /dev/null 100 GiB
| Ma_Sys.ma Big 4.0.2, Copyright (c) 2014, 2019, 2020 Ma_Sys.ma.
| For further info send an e-mail to ma_sys...@web.de.
| 
| 0.00% +0 MiB 0 MiB/s 0/102400 MiB

| 3.48% +3562 MiB 3255 MiB/s 3562/102400 MiB
| 11.06% +7764 MiB 5407 MiB/s 11329/102400 MiB
| 19.31% +8436 MiB 6387 MiB/s 19768/102400 MiB
| 27.71% +8605 MiB 6928 MiB/s 28378/102400 MiB
| 35.16% +7616 MiB 7062 MiB/s 35999/102400 MiB
| 42.58% +7595 MiB 7150 MiB/s 43598/102400 MiB
| 50.12% +7720 MiB 7230 MiB/s 51321/102400 MiB
| 58.57% +8648 MiB 7405 MiB/s 59975/102400 MiB
| 66.96% +8588 MiB 7535 MiB/s 68569/102400 MiB
| 75.11% +8343 MiB 7615 MiB/s 76916/102400 MiB
| 83.38% +8463 MiB 7691 MiB/s 85383/102400 MiB
| 91.74% +8551 MiB 7762 MiB/s 93937/102400 MiB
| 99.97% +8426 MiB 7813 MiB/s 102368/102400 MiB
| 
| Wrote 102400 MiB in 13 s @ 7812.023 MiB/s


[...]

Secure Random can be obtained from OpenSSL:

| $ time for i in `seq 1 100`; do openssl rand -out /dev/null $((1024 * 1024 * 
1024)); done
|
| real  0m49.288s
| user  0m44.710s
| sys   0m4.579s

Effectively 2078 MiB/s (quite OK for single-threaded operation). It is not  
designed to generate large amounts of random data as the size is limited by  
integer range...


HTH
Linux-Fan

öö


pgpoAijtbLtka.pgp
Description: PGP signature

Re: Unidentified subject!

[David Christensen]

On 2/11/24 00:11, Thomas Schmitt wrote:

Hi,

David Christensen wrote:

$ time dd if=/dev/urandom bs=8K count=128K | wc -c
[...]
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.30652 s, 249 MB/s


This looks good enough for practical use on spinning rust and slow SSD.



Yes.



Maybe the "wc" pipe slows it down ?
... not much on 4 GHz Xeon with Debian 11:

   $ time dd if=/dev/urandom bs=8K count=128K | wc -c
   ...
   1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.13074 s, 260 MB/s
   $ time dd if=/dev/urandom bs=8K count=128K of=/dev/null
   ...
   1073741824 bytes (1.1 GB, 1.0 GiB) copied, 3.95569 s, 271 MB/s



My CPU has a Max Turbo Frequency of 3.3 GHz.  I would expect a 4 GHz 
processor to be ~21% faster, but apparently not.



Baseline with pipeline, wc(1), and bs=8K due to unknown Bash pipeline 
bottleneck (?):


2024-02-11 01:18:33 dpchrist@laalaa ~
$ dd if=/dev/urandom bs=8K count=128K | wc -c
1073741824
131072+0 records in
131072+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.27283 s, 251 MB/s


Eliminate pipeline and wc(1):

2024-02-11 01:18:44 dpchrist@laalaa ~
$ dd if=/dev/urandom of=/dev/null bs=8K count=128K
131072+0 records in
131072+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 3.75946 s, 286 MB/s


Increase block size:

2024-02-11 01:18:51 dpchrist@laalaa ~
$ dd if=/dev/urandom of=/dev/null bs=1M count=1K
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 3.62874 s, 296 MB/s


Concurrency:

threads throughput
1   296 MB/s
2   285+286=571 MB/s
3   271+264+266=801 MB/s
4   249+250+241+262=1,002 MB/s
5   225+214+210+224+225=1,098 MB/s
6   223+199+199+204+213+205=1,243 MB/s
7   191+209+210+204+213+201+197=1,425 MB/s
8   205+198+180+195+205+184+184+189=1,540 MB/s



Last time i tested /dev/urandom it was much slower on comparable machines
and also became slower as the amount grew.



Did you figure out why the Linux random number subsystem slowed, and at 
what amount?




Therefore i still have my amateur RNG which works with a little bit of
MD5 and a lot of EXOR. It produces about 1100 MiB/s on the 4 GHz machine.
No cryptographical strength, but chaotic enough to avoid any systematic
pattern which could be recognized by a cheater and represented with
some high compression factor.
The original purpose was to avoid any systematic interference with the
encoding of data blocks on optical media.

I am sure there are faster RNGs around with better random quality.



I assume the Linux kernel in Debian 11 is new enough to support RDRAND (?):

https://en.wikipedia.org/wiki/RdRand


But, my processor is too old to have Secure Key.



$ time perl -MMath::Random::ISAAC::XS -e 
'$i=Math::Random::ISAAC::XS->new(12345678); print pack 'L',$i->irand while 1' | 
dd bs=8K count=128K | wc -c
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 82.6523 s, 13.0 MB/s


Now that's merely sufficient for shredding the content of a BD-RE medium
or a slow USB stick.



Okay.



I suggest using /dev/urandom and tee(1) to write the same CSPRN
stream to all of the devices and using cmp(1) to validate.


I'd propose to use a checksummer like md5sum or sha256sum instead of cmp:

  $random_generator | tee $target | $checksummer
  dd if=$target bs=... count=... | $checksummer

This way one can use unreproducible random streams and does not have to
store the whole stream on a reliable device for comparison.



TIMTOWTDI.  :-)


David

Re: Unidentified subject!

[Thomas Schmitt]

Hi,

David Christensen wrote:
> $ time dd if=/dev/urandom bs=8K count=128K | wc -c
> [...]
> 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.30652 s, 249 MB/s

This looks good enough for practical use on spinning rust and slow SSD.

Maybe the "wc" pipe slows it down ?
... not much on 4 GHz Xeon with Debian 11:

  $ time dd if=/dev/urandom bs=8K count=128K | wc -c
  ...
  1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.13074 s, 260 MB/s
  $ time dd if=/dev/urandom bs=8K count=128K of=/dev/null
  ...
  1073741824 bytes (1.1 GB, 1.0 GiB) copied, 3.95569 s, 271 MB/s

Last time i tested /dev/urandom it was much slower on comparable machines
and also became slower as the amount grew.

Therefore i still have my amateur RNG which works with a little bit of
MD5 and a lot of EXOR. It produces about 1100 MiB/s on the 4 GHz machine.
No cryptographical strength, but chaotic enough to avoid any systematic
pattern which could be recognized by a cheater and represented with
some high compression factor.
The original purpose was to avoid any systematic interference with the
encoding of data blocks on optical media.

I am sure there are faster RNGs around with better random quality.


> $ time perl -MMath::Random::ISAAC::XS -e 
> '$i=Math::Random::ISAAC::XS->new(12345678); print pack 'L',$i->irand while 1' 
> | dd bs=8K count=128K | wc -c
> 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 82.6523 s, 13.0 MB/s

Now that's merely sufficient for shredding the content of a BD-RE medium
or a slow USB stick.


> I suggest using /dev/urandom and tee(1) to write the same CSPRN
> stream to all of the devices and using cmp(1) to validate.

I'd propose to use a checksummer like md5sum or sha256sum instead of cmp:

 $random_generator | tee $target | $checksummer
 dd if=$target bs=... count=... | $checksummer

This way one can use unreproducible random streams and does not have to
store the whole stream on a reliable device for comparison.


Have a nice day :)

Thomas

Re: Unidentified subject!

[Thomas Schmitt]

Hi,

i wrote:
> > In the other thread about the /dev/sdm test:
Gene Heskett wrote:
> > > Creating file 39.h2w ... 1.98% -- 1.90 MB/s -- 257:11:32
> > > [...]
> > > $ sudo f3probe --destructive --time-ops /dev/sdm
> > > Bad news: The device `/dev/sdm' is a counterfeit of type limbo
> > > Device geometry:
> > >  *Usable* size: 59.15 GB (124050944 blocks)
> > > Announced size: 1.91 TB (409600 blocks)

David Christensen wrote:
> Which other thread?  Please provide a URL to archived post.

https://lists.debian.org/msgid-search/e7a0c1a1-f973-4007-a86d-8d91d8d91...@shentel.net
https://lists.debian.org/msgid-search/b566504b-5677-4d84-b5b3-b0de63230...@shentel.net


> So, the 2 TB USB SSD's are really scam 64 GB devices?

The manufacturer would probably state that it's no intentional scam but
just poor product quality. (Exploiting Hanlon's Razor.)

It might be intentional that the write speed gets slower and slower as
the end of the usable area is approached. This avoids the need for
confessing the failure to the operating system.
But it might also be an honest attempt of the disk's firmware to find
some blocks which can take and give back the arriving data.


Have a nice day :)

Thomas

Re: shred bug? [was: Unidentified subject!]

[tomas]

On Sat, Feb 10, 2024 at 07:10:54PM -0500, Greg Wooledge wrote:
> On Sat, Feb 10, 2024 at 04:05:21PM -0800, David Christensen wrote:
> > 2024-02-10 16:03:50 dpchrist@laalaa ~
> > $ shred -s 1K - | wc -c
> > shred: -: invalid file type
> > 0
> > 
> > 
> > It looks like a shred(1) needs a bug report.
> 
> I'm confused what you expected this command to do.  You wanted to
> "destroy" (by overwriting with random data) a pipe to wc?  What
> would that even look like?

What Thomas was trying to do is to get a cheap, fast random number
generator. Shred seems to have such.

> The basic premise of shred is that it determines the size of the file,
> then writes data over it, rewinds it, and repeats this a few times.
> A pipe to a process has no size, and can't be rewound.

That's right: stdout is a (potentially) infinite file, so only one
pass (-n 1, as Thomas put in the command line) really makes sense.
Unless you are into transfinite numbers, that is :-)

> Declaring a pipe to be an "invalid file type" for shredding sounds
> pretty reasonable to me.

This is one of those cases of the toolmaker knowing better than the
tool user. One of the things I like UNIX is that this attitude isn't
that widespread (it is slowly spreading, alas). I much prefer those
tool makers who say "surprise me".

Of course, helping people to not shoot themselves in the foot is
also a honourable endeavour. So "you are trying to shred a pipe.
Use option -f for that (see the manual page" would be fully OK
in my book.

I don't like opinionated software. It's messy enough as it is
when people have opinions :-)

Cheers
-- 
t


signature.asc
Description: PGP signature

Re: shred bug? [was: Unidentified subject!]

[David Christensen]

On 2/10/24 16:10, Greg Wooledge wrote:

On Sat, Feb 10, 2024 at 04:05:21PM -0800, David Christensen wrote:

2024-02-10 16:03:50 dpchrist@laalaa ~
$ shred -s 1K - | wc -c
shred: -: invalid file type
0


It looks like a shred(1) needs a bug report.


I'm confused what you expected this command to do.  You wanted to
"destroy" (by overwriting with random data) a pipe to wc?  What
would that even look like?

The basic premise of shred is that it determines the size of the file,
then writes data over it, rewinds it, and repeats this a few times.
A pipe to a process has no size, and can't be rewound.

Declaring a pipe to be an "invalid file type" for shredding sounds
pretty reasonable to me.



The documentation is confusing:

On 2/10/24 16:05, David Christensen wrote:
> 2024-02-10 16:03:42 dpchrist@laalaa ~
> $ man shred | grep 'If FILE is -'
> If FILE is -, shred standard output.


David

Re: shred bug? [was: Unidentified subject!]

[Greg Wooledge]

On Sat, Feb 10, 2024 at 04:05:21PM -0800, David Christensen wrote:
> 2024-02-10 16:03:50 dpchrist@laalaa ~
> $ shred -s 1K - | wc -c
> shred: -: invalid file type
> 0
> 
> 
> It looks like a shred(1) needs a bug report.

I'm confused what you expected this command to do.  You wanted to
"destroy" (by overwriting with random data) a pipe to wc?  What
would that even look like?

The basic premise of shred is that it determines the size of the file,
then writes data over it, rewinds it, and repeats this a few times.
A pipe to a process has no size, and can't be rewound.

Declaring a pipe to be an "invalid file type" for shredding sounds
pretty reasonable to me.

Re: shred bug? [was: Unidentified subject!]

[David Christensen]

On 2/10/24 04:40, to...@tuxteam.de wrote:

On Sat, Feb 10, 2024 at 11:38:21AM +0100, Thomas Schmitt wrote:

[...]


But shred(1) on Debian 11 refuses on "-" contrary to its documentation:
   shred: -: invalid file type
A non-existing file path causes "No such file or directory".


Hmm. This looks like a genuine bug: the man page mentions it.

Also, /dev/stdout as target runs into the very same problem.

Cheers



Testing:

2024-02-10 16:01:54 dpchrist@laalaa ~
$ cat /etc/debian_version ; uname -a
11.8
Linux laalaa 5.10.0-27-amd64 #1 SMP Debian 5.10.205-2 (2023-12-31) 
x86_64 GNU/Linux


2024-02-10 16:02:34 dpchrist@laalaa ~
$ bash --version | head -n 1
GNU bash, version 5.1.4(1)-release (x86_64-pc-linux-gnu)

2024-02-10 16:02:48 dpchrist@laalaa ~
$ shred --version | head -n 1
shred (GNU coreutils) 8.32

2024-02-10 16:03:42 dpchrist@laalaa ~
$ man shred | grep 'If FILE is -'
   If FILE is -, shred standard output.

2024-02-10 16:03:50 dpchrist@laalaa ~
$ shred -s 1K - | wc -c
shred: -: invalid file type
0


It looks like a shred(1) needs a bug report.


David

Re: Unidentified subject!

[David Christensen]

On 2/10/24 10:28, Thomas Schmitt wrote:

In the other thread about the /dev/sdm test:

Creating file 39.h2w ... 1.98% -- 1.90 MB/s -- 257:11:32
but is taking a few bytes now and then.
[...]
$ ls -l
total 40627044
[...]
$ sudo f3probe --destructive --time-ops /dev/sdm
Bad news: The device `/dev/sdm' is a counterfeit of type limbo
Device geometry:
 *Usable* size: 59.15 GB (124050944 blocks)
Announced size: 1.91 TB (409600 blocks)
...
Probe time: 2.07s



Which other thread?  Please provide a URL to archived post.


So, the 2 TB USB SSD's are really scam 64 GB devices?


David

Re: Unidentified subject!

[David Christensen]

On 2/10/24 02:38, Thomas Schmitt wrote:

I have an own weak-random generator, but shred beats it by a factor of 10
when writing to /dev/null.



As a baseline, here is a 2011 Dell Latitude E6520 with Debian generating 
a non-repeatable 1 GiB stream of cryptographically secure pseudo-random 
numbers:


2024-02-10 14:10:27 dpchrist@laalaa ~
$ lscpu | grep 'Model name'
Model name: Intel(R) Core(TM) i7-2720QM CPU @ 
2.20GHz


2024-02-10 14:01:25 dpchrist@laalaa ~
$ cat /etc/debian_version ; uname -a
11.8
Linux laalaa 5.10.0-27-amd64 #1 SMP Debian 5.10.205-2 (2023-12-31) 
x86_64 GNU/Linux


2024-02-10 14:01:47 dpchrist@laalaa ~
$ bash --version | head -n 1
GNU bash, version 5.1.4(1)-release (x86_64-pc-linux-gnu)

2024-02-10 14:13:08 dpchrist@laalaa ~
$ time dd if=/dev/urandom bs=8K count=128K | wc -c
1073741824
131072+0 records in
131072+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.30652 s, 249 MB/s

real0m4.311s
user0m0.107s
sys 0m4.695s


If the OP has a known good storage device of equal or larger size than 
the UUT(s), I suggest using /dev/urandom and tee(1) to write the same 
CSPRN stream to all of the devices and using cmp(1) to validate.



I use Perl.  When I needed a CSPRNG, I searched and found:

https://metacpan.org/pod/Math::Random::ISAAC::XS


Using Perl and Math::Random::ISAAC::XS to generate a repeatable stream 
of cryptographically secure pseudo-random numbers:


2024-02-10 14:09:12 dpchrist@laalaa ~
$ perl -v | head -n 2 | grep .
This is perl 5, version 32, subversion 1 (v5.32.1) built for 
x86_64-linux-gnu-thread-multi


2024-02-10 14:09:53 dpchrist@laalaa ~
$ perl -MMath::Random::ISAAC::XS -e 'print 
$Math::Random::ISAAC::XS::VERSION, $/'

1.004

2024-02-10 14:10:32 dpchrist@laalaa ~
$ time perl -MMath::Random::ISAAC::XS -e 
'$i=Math::Random::ISAAC::XS->new(12345678); print pack 'L',$i->irand 
while 1' | dd bs=8K count=128K | wc -c

1073741824
131072+0 records in
131072+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 82.6523 s, 13.0 MB/s

real1m22.657s
user1m22.892s
sys 0m5.810s


The 'perl ... | dd ...' pipeline appears to be limited to a block size 
of 8,192 bytes.  I do not know if this is due to bash(1), perl(1), or 
dd(1) (?).



The repeatability of the ISAAC stream would substitute for the known 
good storage device, but the throughput is ~19x slower than 
/dev/urandom.  A drive capacity validation tool would want to feature 
concurrent threads and I/O on SMP machines.



David

Re: Unidentified subject!

[gene heskett]

On 2/10/24 13:30, Thomas Schmitt wrote:

Hi,

gene heskett wrote:

my fading eyesight couldn't see
the diffs between () and {} in a 6 point font.  I need a bigger, more
legible font in t-bird.


That's why i propose to copy+paste problematic command lines.

Your mouse can read it, your mail client can send it, and we have
youngsters here of merely 60 years who will be glad to tell our
most senior regular why the lines don't work.

With some luck this creates nostalgic tangents about how the used features
evolved since the early 1980s.

Or even the later 1970's when I made a cosmac super elf RCA 1802 board 
do anything I could dream up. Made the video hdwe, and the interface to 
control a broadcast vcr. S-100 bus adapter was the only thing I bought, 
and had to build that from a kit.  Built it for KRCR-TV in Redding CA, 
it was so useful to the production folks they used it many times a day 
from '79 to mid 90's when it burnt to the ground, That's eons in a tv 
station control room where stuff is often replaced before its fully 
written off tax wise in 5 years. Fun times back then. Now I'm searching 
amazon for a pi-clone hat with a 6 pack of sata-iii sockets on it, and 
coming up MT.  Sniff...


In the other thread about the /dev/sdm test:

Creating file 39.h2w ... 1.98% -- 1.90 MB/s -- 257:11:32
but is taking a few bytes now and then.
[...]
$ ls -l
total 40627044
[...]
$ sudo f3probe --destructive --time-ops /dev/sdm
Bad news: The device `/dev/sdm' is a counterfeit of type limbo
Device geometry:
 *Usable* size: 59.15 GB (124050944 blocks)
Announced size: 1.91 TB (409600 blocks)


That's really barefaced.
How can the seller believe to get away with a problem which will show
already after a few dozen GB were written ?



Probe time: 2.07s


That's indeed a quick diagnosis. Congrats to the developers of f3probe.


Have a nice day :)

Thomas

.


Cheers, Gene Heskett, CET.
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis

Re: Unidentified subject!

[Thomas Schmitt]

Hi,

gene heskett wrote:
> my fading eyesight couldn't see
> the diffs between () and {} in a 6 point font.  I need a bigger, more
> legible font in t-bird.

That's why i propose to copy+paste problematic command lines.

Your mouse can read it, your mail client can send it, and we have
youngsters here of merely 60 years who will be glad to tell our
most senior regular why the lines don't work.

With some luck this creates nostalgic tangents about how the used features
evolved since the early 1980s.


In the other thread about the /dev/sdm test:
> Creating file 39.h2w ... 1.98% -- 1.90 MB/s -- 257:11:32
> but is taking a few bytes now and then.
> [...]
> $ ls -l
> total 40627044
> [...]
> $ sudo f3probe --destructive --time-ops /dev/sdm
> Bad news: The device `/dev/sdm' is a counterfeit of type limbo
> Device geometry:
> *Usable* size: 59.15 GB (124050944 blocks)
>Announced size: 1.91 TB (409600 blocks)

That's really barefaced.
How can the seller believe to get away with a problem which will show
already after a few dozen GB were written ?


> Probe time: 2.07s

That's indeed a quick diagnosis. Congrats to the developers of f3probe.


Have a nice day :)

Thomas

Re: Unidentified subject!

[gene heskett]

On 2/10/24 05:39, Thomas Schmitt wrote:

Hi,

Gene Heskett wrote:

Is bash not actually bash these days? It is not doing for loops for me.


Come on Gene, be no sophie. Copy+paste your failing line here. :))

Alexander M. posted it a few days ago but my fading eyesight couldn't 
see the diffs between () and {} in a 6 point font.  I need a bigger, 
more legible font in t-bird. Or blow it up about 3x. Which doesn't 
stick, its gone with next msg.  G.


Thank Thomas.
[...]> Have a nice day :)


Thomas

.


Cheers, Gene Heskett, CET.
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis

Re: Things I don't touch with a 3.048m barge pole: USB storage(WasRe: Unidentified subject!)

[gene heskett]

On 2/10/24 00:54, David Christensen wrote:

On 2/9/24 04:53, gene heskett wrote:

Interesting report from gdisk however:
GPT fdisk (gdisk) version 1.0.9

Partition table scan:
   MBR: MBR only
   BSD: not present
   APM: not present
   GPT: not present


***
Found invalid GPT and valid MBR; converting MBR to GPT format
in memory. THIS OPERATION IS POTENTIALLY DESTRUCTIVE! Exit by
typing 'q' if you don't want to convert your MBR partitions
to GPT format!
***


Warning! Secondary partition table overlaps the last partition by
33 blocks!
You will need to delete this partition or resize it in another utility.

Command (? for help):
Command (? for help): p
Disk /dev/sdm: 409600 sectors, 1.9 TiB
Model: SSD 3.0
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): 3230045D-589D-4601-8C4D-E9C4684B9657
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 409566
Partitions will be aligned on 64-sector boundaries
Total free space is 30 sectors (15.0 KiB)

Number  Start (sector)    End (sector)  Size   Code  Name
    1  64  409599   1.9 TiB 0700  Microsoft 
basic data


Command (? for help): q

What do we make of that?  Some sort of NTFS?



Do these commands produce any clues or error messages?

# fdisk -l /dev/sdm

# tail /var/log/messages

# dmesg | tail


They suggested an exfat format ootb, which matches most sd cards.
Thanks David, Take care.


David

.


Cheers, Gene Heskett, CET.
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis

Re: Unidentified subject!

[Stefan Monnier]

>> AFAIK the bogus 128TB drives do properly report such ridiculous sizes:
>> the reality only hits when you try to actually store that amount of
>> information on them.
>> [ I'm not sure how it works under the hood, but since SSDs store their
>>data "anywhere" in the flash, they can easily pretend to have any size
>>they want, and allocate the physical flash blocks only on-the-fly as
>>logical blocks are being written.
>>Also, some Flash controllers use compression, so if you store data
>>that compresses well, they can let you store a lot more than if you
>>store already compressed data.  ]
>> IOW, to really check, try to save 2TB of videos (or other already
>> compressed data), and then try and read it back.
>> 
> Sounds like a lawsuit to me. If I can get Alexanders script from a few days
> back to run.  Is bash not actually bash these days? It is not doing for
> loops for me.

As discussed in related threads, there's the `f3` package in Debian
designed specifically for that.
You can try `f3probe /dev/sdX` (or use `f3write` and `f3read` if you
prefer to test at the filesystem level rather than at the block level).


Stefan

Re: shred bug? [was: Unidentified subject!]

[tomas]

On Sat, Feb 10, 2024 at 01:40:35PM +0100, to...@tuxteam.de wrote:
> On Sat, Feb 10, 2024 at 11:38:21AM +0100, Thomas Schmitt wrote:
> 
> [...]
> 
> > But shred(1) on Debian 11 refuses on "-" contrary to its documentation:
> >   shred: -: invalid file type
> > A non-existing file path causes "No such file or directory".
> 
> Hmm. This looks like a genuine bug: the man page mentions it.
> 
> Also, /dev/stdout as target runs into the very same problem.

Ah, it seems to be this one, from 2002:

  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=155175

which is archived. The argument seems to be that shred on stdout
doesn't make any sense, because the shell would truncate the
file anyway when you did

  shred - > /this/file/to/be/shredded

... which, of course, undermines shred's purpose. It seems they
hadn't your sneaky use case in mind :-)

Cheers
-- 
t


signature.asc
Description: PGP signature

shred bug? [was: Unidentified subject!]

[tomas]

On Sat, Feb 10, 2024 at 11:38:21AM +0100, Thomas Schmitt wrote:

[...]

> But shred(1) on Debian 11 refuses on "-" contrary to its documentation:
>   shred: -: invalid file type
> A non-existing file path causes "No such file or directory".

Hmm. This looks like a genuine bug: the man page mentions it.

Also, /dev/stdout as target runs into the very same problem.

Cheers
-- 
t


signature.asc
Description: PGP signature

Re: Unidentified subject!

[Thomas Schmitt]

Hi,

Gene Heskett wrote:
> Is bash not actually bash these days? It is not doing for loops for me.

Come on Gene, be no sophie. Copy+paste your failing line here. :))


IIRC the for-loop in question writes several copies of the same file.
( https://lists.debian.org/debian-user/2024/02/msg00318.html )
Others already pointed out that this invites for firmware scams like
deduplication or silent overwriting of older data.

I would vote for a filesystem killer like
  shred -n 1 -s 204768K -v - | tee /dev/sdm1 | sha256sum
  dd if=/dev/sdm1 bs=32K count=6399 skip=32 | sha256sum

But shred(1) on Debian 11 refuses on "-" contrary to its documentation:
  shred: -: invalid file type
A non-existing file path causes "No such file or directory".


The filesystem killer aspect could be removed by creating large data files
in the readily partitioned and formatted filesystems of the disk.
(Replace "/dev/sdm1" by "/where/mounted/random_test_file" and reduce the
 numbers 204768K and bs=32K count=6399 to what you expect to
 fit into the filesystem. Like 20K and bs=32K count=6250.
 Ask df for storage capacities.)

But there needs to be a fast pseudo-random byte stream (which shred would
provide if i could talk it into writing to stdout) which you can split
for the device and for sha256sum.
I have an own weak-random generator, but shred beats it by a factor of 10
when writing to /dev/null.


Have a nice day :)

Thomas

Re: Unidentified subject!

[gene heskett]

On 2/7/24 23:28, Stefan Monnier wrote:

Well the 2T memory everybody was curious about 3 weeks ago got here early.

 From dmesg after plugging one in:
[629240.916163] usb 1-2: new high-speed USB device number 39 using xhci_hcd
[629241.066221] usb 1-2: New USB device found, idVendor=048d,
idProduct=1234, bcdDevice= 2.00
[629241.066234] usb 1-2: New USB device strings: Mfr=1, Product=2,
SerialNumber=3
[629241.066239] usb 1-2: Product: Disk 3.0
[629241.066242] usb 1-2: Manufacturer: USB
[629241.066246] usb 1-2: SerialNumber: 2697241127107725123
[629241.069485] usb-storage 1-2:1.0: USB Mass Storage device detected
[629241.074187] scsi host37: usb-storage 1-2:1.0
[629242.100738] scsi 37:0:0:0: Direct-Access  SSD 3.02.00
PQ: 0 ANSI: 4
[629242.100959] sd 37:0:0:0: Attached scsi generic sg13 type 0
[629242.101190] sd 37:0:0:0: [sdm] 409600 512-byte logical blocks: (2.10
TB/1.91 TiB)
[629242.101289] sd 37:0:0:0: [sdm] Write Protect is off
[629242.101290] sd 37:0:0:0: [sdm] Mode Sense: 03 00 00 00
[629242.101409] sd 37:0:0:0: [sdm] No Caching mode page found
[629242.101410] sd 37:0:0:0: [sdm] Assuming drive cache: write through
[629242.103927]  sdm: sdm1
[629242.104047] sd 37:0:0:0: [sdm] Attached SCSI disk
gene@coyote:

Looks like a reasonable facsimile of a 2T disk to me.


AFAIK the bogus 128TB drives do properly report such ridiculous sizes:
the reality only hits when you try to actually store that amount of
information on them.

[ I'm not sure how it works under the hood, but since SSDs store their
   data "anywhere" in the flash, they can easily pretend to have any size
   they want, and allocate the physical flash blocks only on-the-fly as
   logical blocks are being written.
   Also, some Flash controllers use compression, so if you store data
   that compresses well, they can let you store a lot more than if you
   store already compressed data.  ]

IOW, to really check, try to save 2TB of videos (or other already
compressed data), and then try and read it back.

Sounds like a lawsuit to me. If I can get Alexanders script from a few 
days back to run.  Is bash not actually bash these days? It is not doing 
for loops for me.


Thanks for the heads up, Stefan.


 Stefan

.


Cheers, Gene Heskett, CET.
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis

Re: Things I don't touch with a 3.048m barge pole: USB storage (WasRe: Unidentified subject!)

[David Christensen]

On 2/9/24 04:53, gene heskett wrote:

Interesting report from gdisk however:
GPT fdisk (gdisk) version 1.0.9

Partition table scan:
   MBR: MBR only
   BSD: not present
   APM: not present
   GPT: not present


***
Found invalid GPT and valid MBR; converting MBR to GPT format
in memory. THIS OPERATION IS POTENTIALLY DESTRUCTIVE! Exit by
typing 'q' if you don't want to convert your MBR partitions
to GPT format!
***


Warning! Secondary partition table overlaps the last partition by
33 blocks!
You will need to delete this partition or resize it in another utility.

Command (? for help):
Command (? for help): p
Disk /dev/sdm: 409600 sectors, 1.9 TiB
Model: SSD 3.0
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): 3230045D-589D-4601-8C4D-E9C4684B9657
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 409566
Partitions will be aligned on 64-sector boundaries
Total free space is 30 sectors (15.0 KiB)

Number  Start (sector)    End (sector)  Size   Code  Name
    1  64  409599   1.9 TiB 0700  Microsoft 
basic data


Command (? for help): q

What do we make of that?  Some sort of NTFS?



Do these commands produce any clues or error messages?

# fdisk -l /dev/sdm

# tail /var/log/messages

# dmesg | tail


David

Re: Things I don't touch with a 3.048m barge pole: USB storage (WasRe: Unidentified subject!)

[gene heskett]

On 2/8/24 15:43, Andy Smith wrote:

Hello,

On Thu, Feb 08, 2024 at 02:20:59PM -0500, Jeffrey Walton wrote:

On Thu, Feb 8, 2024 at 11:57 AM Ralph Aichinger  wrote:

How does a breaking USB disk differ from a breaking SATA disk?


I may be mistaken, but I believe AS is talking about USB thumb drives,
SDcards and the like. I don't think he's talking about external SSD's
and NVME's over USB. But I don't want to put words in his mouth.


I really do mean all forms of USB that come over a USB port.

I wouldn't have much issue with taking a USB drive out of its caddy
to get the SATA drive from inside, except that it would have to be
an amazingly good deal to make it worth voiding the warranty, so I
generally wouldn't bother.

If I need directly attached storage I'd much rather explore options
like SAS and eSATA, or even networked storage, before I would ever
consider USB for a permanent installation.

Thanks,
Andy

OTOH. I have a couple sata-II SSD's, a kingston 256G and an adata 120G 
plugged into the usb-3 ports of what was an rpi3b with usb2, rigged it 
up first in 2016 IIRC, but swapped the rpi3b for an rpi4b in Feb 2020.
I can build linuxcnc from master in around an hour, and a 4.19.120 or so 
kernal for the pi in a little less. And was doing that linuxcnc thing 4 
to 8 times a week for several years but have stopped that with armhf 
since it may be dropped in favor of arm64 which isn't as good for 
latency, but is good enough to run linuxcnc w/o making the machine 
stutter from lack of data.


I have had one failure, the adapter for the 120G adata, wasn't a 
startech, is now for around 5 years. That's beats the performance of 
spinning rust like a white mouthed mule. I'd had  Spinning rust failures 
have totaled around around a dozen in triple that time frame. I built my 
first linux box with a 30G drive in 1998. 26 years ago. I've paid the 
window tax once, buying a lappy in 2002 to run a road map gps thing as I 
did a decade and change worth of consulting since I retired, but that 
lappy got its windows replaced in 2 weeks by mandrake when I found the 
windose xp install could not run the radio in it but mandrake could. 
Long found the out bin, bad ac adapter. but I got most of a decade out 
of it.  Kept me company from the passengers seat for around 20k miles 
though.


If these $23 drives pass the A. M. test, they will get mounted in 
adapters I'll have to design and print, plugged into a 8 port usb3 hub, 
plugged into a usb3 port of an bpi-m5, making a drive cage into a 12TB 
with 6 of these I bought into an amanda backup server I may hide in the 
bookshelves surrounding me. Headless, probably running it all on a 5v5a 
psu. There is a 5v8a psu being rowed across the big pond just in case.


My scope watching the 5 volt line will determine the need.

If they pass the test.  That is YTBD.
Interesting report from gdisk however:
GPT fdisk (gdisk) version 1.0.9

Partition table scan:
  MBR: MBR only
  BSD: not present
  APM: not present
  GPT: not present


***
Found invalid GPT and valid MBR; converting MBR to GPT format
in memory. THIS OPERATION IS POTENTIALLY DESTRUCTIVE! Exit by
typing 'q' if you don't want to convert your MBR partitions
to GPT format!
***


Warning! Secondary partition table overlaps the last partition by
33 blocks!
You will need to delete this partition or resize it in another utility.

Command (? for help):
Command (? for help): p
Disk /dev/sdm: 409600 sectors, 1.9 TiB
Model: SSD 3.0
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): 3230045D-589D-4601-8C4D-E9C4684B9657
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 409566
Partitions will be aligned on 64-sector boundaries
Total free space is 30 sectors (15.0 KiB)

Number  Start (sector)End (sector)  Size   Code  Name
   1  64  409599   1.9 TiB 0700  Microsoft 
basic data


Command (? for help): q

What do we make of that?  Some sort of NTFS?



Cheers, Gene Heskett, CET.
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis

Re: Things I don't touch with a 3.048m barge pole: USB storage (WasRe: Unidentified subject!)

[gene heskett]

On 2/8/24 11:15, Gremlin wrote:

On 2/8/24 10:36, Andy Smith wrote:

Hello,

On Wed, Feb 07, 2024 at 03:30:29PM -0500, gene heskett wrote:

[629241.074187] scsi host37: usb-storage 1-2:1.0


USB storage is for phones and cameras etc, not for serious
computing. Many people will disagree with that statement and say
they use it all the time and it is fine. They will keep saying that
until it isn't fine, and then they'll be in a world of hurt.



LOL,  So my main desktop a raspberry pi 4 is not serious computing? Or 
is it that my name server, web server email server which is a raspberry 
pi 4 not serious computing?


They both boot to USB SSDs and only have USB SSD drives, so they are not 
serious computing?  The desktop RPI has an NVME drive as the boot drive 
connected by you guessed it USB.




I learned not to go there a long time ago and have seen plenty of
reminders along the way from others' misfortunes to not ever go
there again myself.



Well, most of what you attributed to me came from an earlier post.  I'd 
never call a pi4b inadequate.  Its running an 11x54 lathe just like a 
wintel box can.  All the other stuff that makes a desktop computer, web 
browsing, the office suites, web server, you name it, it Just Works. 
Not as fast, but it works as advertised.  And does all that on a 5x5 
psu, with an AOC monitors whose label claim it uses 10 watts. I don't 
even shut them off.





.


Cheers, Gene Heskett, CET.
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis

Re: Things I don't touch with a 3.048m barge pole: USB storage (Was Re: Unidentified subject!)

[Arno Lehmann]

Hi all,

Am 08.02.2024 um 21:38 schrieb Andy Smith:

Hello,

On Thu, Feb 08, 2024 at 05:40:54PM +0100, Ralph Aichinger wrote:

On Thu, 2024-02-08 at 15:36 +, Andy Smith wrote:

I learned not to go there a long time ago and have seen plenty of
reminders along the way from others' misfortunes to not ever go
there again myself.


How does a breaking USB disk differ from a breaking SATA disk?


My own experience is that it's often harder to notice and diagnose -- 
because on top of the actual storage and its "native" interface such as 
SATA or NVMe/PCIe, you have the whole stack of USB things.


And misbehaving USB devices usually result in first working on the USB 
end -- try different port, port directly on mainboard, or a powered hub, 
watch out for native USB 3 or 3.0 Gen 1 -- we can see this on this 
mailing list, too.


Then, USB storage is usually a single device single, if it breaks, it's 
data is lost, whereas SATA/SAS/NVMe can more easily be integrated into 
redundancy providing systems.


On top of all that, my own, admittedly anecdotal, experience is that 
USB/Firewire-to-IDE/SATA adapters and their power supplies are more 
fragile than actual disks. Most of the external hard disks I ever used 
have been replaced because of their enclosures or power supplies failing.


So, I tend to agree with Andy, and I also don't notice any moving 
goalposts in his statements...



In my experience it happens more often and also brings with it
frequent issues of poor performance and other reliability issues
like just dropping off the USB bus. There is almost always a better
way.


For home users / small office environments, that leaves the problem of 
how to do backups -- USB drives are the most appealing storage system 
for such purposes, but also seem to be less reliable than the primary 
storage. What do you do? Throw more of the USB disks onto the problem?


Or is "public" cloud the solution?

Whatever you do, even purely personal storage requirements become a bit 
of a nightmare when you start thinking about how to make sure your 
photos and videos are around when your kids are grown up...



Cheers,

Arno


Thanks,
Andy



--
Arno Lehmann

IT-Service Lehmann
Sandstr. 6, 49080 Osnabrück

Re: Unidentified subject!

[Richmond]

Charles Curley  writes:

> On Thu, 08 Feb 2024 18:02:36 -0500
> Stefan Monnier  wrote:
>
>> > Test it with Validrive.
>> > https://www.grc.com/validrive.htm  
>> 
>> Looks like proprietary software for Windows.
>
> badblocks, available in a Debian repo near you, might be a suitable
> replacement.

I am not sure badblocks would do the same thing.

"The drive appears to be the 1 or 2 terabyte drive you purchased. You
plug it into your computer and everything looks fine. You can even copy
files to the drive; as many as you want. And when you look at the
drive's contents the files are there. But what's insidious is that the
files' contents may have never been stored.  "

So you need to store a lot of data and then verify that it has written
with 'diff'.

Re: Unidentified subject!

[Charles Curley]

On Thu, 08 Feb 2024 18:02:36 -0500
Stefan Monnier  wrote:

> > Test it with Validrive.
> > https://www.grc.com/validrive.htm  
> 
> Looks like proprietary software for Windows.

badblocks, available in a Debian repo near you, might be a suitable
replacement.

-- 
Does anybody read signatures any more?

https://charlescurley.com
https://charlescurley.com/blog/

Re: Unidentified subject!

[Stefan Monnier]

> Test it with Validrive.
> https://www.grc.com/validrive.htm

Looks like proprietary software for Windows.


Stefan

Re: Unidentified subject!

[Richmond]

gene heskett  writes:

> Well the 2T memory everybody was curious about 3 weeks ago got here early.
>
> From dmesg after plugging one in:
> [629240.916163] usb 1-2: new high-speed USB device number 39 using xhci_hcd
> [629241.066221] usb 1-2: New USB device found, idVendor=048d,
> idProduct=1234, bcdDevice= 2.00
> [629241.066234] usb 1-2: New USB device strings: Mfr=1, Product=2,
> SerialNumber=3
> [629241.066239] usb 1-2: Product: Disk 3.0
> [629241.066242] usb 1-2: Manufacturer: USB
> [629241.066246] usb 1-2: SerialNumber: 2697241127107725123
> [629241.069485] usb-storage 1-2:1.0: USB Mass Storage device detected
> [629241.074187] scsi host37: usb-storage 1-2:1.0
> [629242.100738] scsi 37:0:0:0: Direct-Access  SSD 3.0
> 2.00 PQ: 0 ANSI: 4
> [629242.100959] sd 37:0:0:0: Attached scsi generic sg13 type 0
> [629242.101190] sd 37:0:0:0: [sdm] 409600 512-byte logical blocks:
> (2.10 TB/1.91 TiB)
> [629242.101289] sd 37:0:0:0: [sdm] Write Protect is off
> [629242.101290] sd 37:0:0:0: [sdm] Mode Sense: 03 00 00 00
> [629242.101409] sd 37:0:0:0: [sdm] No Caching mode page found
> [629242.101410] sd 37:0:0:0: [sdm] Assuming drive cache: write through
> [629242.103927]  sdm: sdm1
> [629242.104047] sd 37:0:0:0: [sdm] Attached SCSI disk
> gene@coyote:
>
> Looks like a reasonable facsimile of a 2T disk to me.
>
> Cheers, Gene Heskett, CET.

Test it with Validrive.

https://www.grc.com/validrive.htm

Re: Things I don't touch with a 3.048m barge pole: USB storage (Was Re: Unidentified subject!)

[Gremlin]

On 2/8/24 16:28, Andy Smith wrote:

Hello,

On Thu, Feb 08, 2024 at 04:22:49PM -0500, Gremlin wrote:

On Thu, Feb 08, 2024 at 08:43:17PM +, Andy Smith wrote:

I really do mean all forms of USB that come over a USB port.


That line was meant to read

 I really do mean all forms of storage that come over a USB port.

Changing the goal post now are we.


Erm no, it was a simple mistaken repetition of the word "USB" that I
only noticed when I read it back. It would be clearly very difficult
to refuse to use any kind of USB device at all! I have been
consistently talking about storage devices.

You have been very clear that you do not agree though, so let's just
agree to disagree.

Thanks,
Andy



You back pedal really well don't you!  Where did you learn that?

Re: Things I don't touch with a 3.048m barge pole: USB storage (Was Re: Unidentified subject!)

[Andy Smith]

Hello,

On Thu, Feb 08, 2024 at 04:22:49PM -0500, Gremlin wrote:
> On Thu, Feb 08, 2024 at 08:43:17PM +, Andy Smith wrote:
> > I really do mean all forms of USB that come over a USB port.
> 
> That line was meant to read
> 
> I really do mean all forms of storage that come over a USB port.
> 
> Changing the goal post now are we.

Erm no, it was a simple mistaken repetition of the word "USB" that I
only noticed when I read it back. It would be clearly very difficult
to refuse to use any kind of USB device at all! I have been
consistently talking about storage devices.

You have been very clear that you do not agree though, so let's just
agree to disagree.

Thanks,
Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

Re: Things I don't touch with a 3.048m barge pole: USB storage (Was Re: Unidentified subject!)

[Gremlin]

On 2/8/24 16:16, Andy Smith wrote:

On Thu, Feb 08, 2024 at 03:56:19PM -0500, Gremlin wrote:

On 2/8/24 15:43, Andy Smith wrote:

I wouldn't have much issue with taking a USB drive out of its caddy
to get the SATA drive from inside, except that it would have to be
an amazingly good deal to make it worth voiding the warranty, so I
generally wouldn't bother.


Why would it void the warranty?  I put it in the caddy


I mean the USB drives that come as a sealed unit that you can
sometimes find a lot cheaper than the same model SATA drive that is
actually inside them. Some people do enjoy taking those apart to get
the SATA drive out.

Thanks,
Andy



On Thu, Feb 08, 2024 at 08:43:17PM +, Andy Smith wrote:
> I really do mean all forms of USB that come over a USB port.

That line was meant to read

I really do mean all forms of storage that come over a USB port.

Thanks,
Andy


Changing the goal post now are we.

Re: Things I don't touch with a 3.048m barge pole: USB storage (Was Re: Unidentified subject!)

[Andy Smith]

Hello,

On Thu, Feb 08, 2024 at 04:00:01PM -0500, Gremlin wrote:
> I have been using USB attached HDDs and SSDs for 10 years now and
> have never had one unexpectedly go off line.  Your postings
> suggest you don't know what your talking about.

Okay then. Despite this uncharitable comment, I do still wish you
what I consider to be continued good fortune!

Thanks,
Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

Re: Things I don't touch with a 3.048m barge pole: USB storage (Was Re: Unidentified subject!)

[Andy Smith]

On Thu, Feb 08, 2024 at 03:56:19PM -0500, Gremlin wrote:
> On 2/8/24 15:43, Andy Smith wrote:
> > I wouldn't have much issue with taking a USB drive out of its caddy
> > to get the SATA drive from inside, except that it would have to be
> > an amazingly good deal to make it worth voiding the warranty, so I
> > generally wouldn't bother.
> 
> Why would it void the warranty?  I put it in the caddy

I mean the USB drives that come as a sealed unit that you can
sometimes find a lot cheaper than the same model SATA drive that is
actually inside them. Some people do enjoy taking those apart to get
the SATA drive out.

Thanks,
Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

Re: Things I don't touch with a 3.048m barge pole: USB storage (Was Re: Unidentified subject!)

[Gremlin]

On 2/8/24 15:35, Andy Smith wrote:

Hello,

On Fri, Feb 09, 2024 at 12:23:45AM +0700, Max Nikulin wrote:

On 08/02/2024 22:36, Andy Smith wrote:

On Wed, Feb 07, 2024 at 03:30:29PM -0500, gene heskett wrote:

[629241.074187] scsi host37: usb-storage 1-2:1.0


USB storage is for phones and cameras etc, not for serious
computing.


Do you mean that a proper backup drive should use uas (USB Attached Storage)
instead of usb-storage driver?


No, I just mean I advise to never ever use storage that comes to you
via a USB port for anything you care about.

I might consider it okay for temporary shifting of data about, but I
would never use it as part of a permanent setup without fully
expecting one day to find it just not working. But then that is also
how I feel about any storage device. It's just worse when USB is
added to the mix.

I have been using USB attached HDDs and SSDs for 10 years now and have 
never had one unexpectedly go off line.  Your postings suggest you don't 
know what your talking about.

Re: Things I don't touch with a 3.048m barge pole: USB storage (Was Re: Unidentified subject!)

[Gremlin]

On 2/8/24 15:43, Andy Smith wrote:

Hello,

On Thu, Feb 08, 2024 at 02:20:59PM -0500, Jeffrey Walton wrote:

On Thu, Feb 8, 2024 at 11:57 AM Ralph Aichinger  wrote:

How does a breaking USB disk differ from a breaking SATA disk?


I may be mistaken, but I believe AS is talking about USB thumb drives,
SDcards and the like. I don't think he's talking about external SSD's
and NVME's over USB. But I don't want to put words in his mouth.


I really do mean all forms of USB that come over a USB port.

I wouldn't have much issue with taking a USB drive out of its caddy
to get the SATA drive from inside, except that it would have to be
an amazingly good deal to make it worth voiding the warranty, so I
generally wouldn't bother.


Why would it void the warranty?  I put it in the caddy



If I need directly attached storage I'd much rather explore options
like SAS and eSATA, or even networked storage, before I would ever
consider USB for a permanent installation.


You need to start thinking outside the box.

Re: Things I don't touch with a 3.048m barge pole: USB storage (Was Re: Unidentified subject!)

[Andy Smith]

On Thu, Feb 08, 2024 at 08:43:17PM +, Andy Smith wrote:
> I really do mean all forms of USB that come over a USB port.

That line was meant to read

I really do mean all forms of storage that come over a USB port.

Thanks,
Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

Re: Things I don't touch with a 3.048m barge pole: USB storage (Was Re: Unidentified subject!)

[Andy Smith]

Hello,

On Thu, Feb 08, 2024 at 02:20:59PM -0500, Jeffrey Walton wrote:
> On Thu, Feb 8, 2024 at 11:57 AM Ralph Aichinger  wrote:
> > How does a breaking USB disk differ from a breaking SATA disk?
> 
> I may be mistaken, but I believe AS is talking about USB thumb drives,
> SDcards and the like. I don't think he's talking about external SSD's
> and NVME's over USB. But I don't want to put words in his mouth.

I really do mean all forms of USB that come over a USB port.

I wouldn't have much issue with taking a USB drive out of its caddy
to get the SATA drive from inside, except that it would have to be
an amazingly good deal to make it worth voiding the warranty, so I
generally wouldn't bother.

If I need directly attached storage I'd much rather explore options
like SAS and eSATA, or even networked storage, before I would ever
consider USB for a permanent installation.

Thanks,
Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

Re: Things I don't touch with a 3.048m barge pole: USB storage (Was Re: Unidentified subject!)

[Andy Smith]

Hello,

On Thu, Feb 08, 2024 at 05:40:54PM +0100, Ralph Aichinger wrote:
> On Thu, 2024-02-08 at 15:36 +, Andy Smith wrote:
> > I learned not to go there a long time ago and have seen plenty of
> > reminders along the way from others' misfortunes to not ever go
> > there again myself.
> 
> How does a breaking USB disk differ from a breaking SATA disk?

In my experience it happens more often and also brings with it
frequent issues of poor performance and other reliability issues
like just dropping off the USB bus. There is almost always a better
way.

Thanks,
Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

Re: Things I don't touch with a 3.048m barge pole: USB storage (Was Re: Unidentified subject!)

[Andy Smith]

Hello,

On Fri, Feb 09, 2024 at 12:23:45AM +0700, Max Nikulin wrote:
> On 08/02/2024 22:36, Andy Smith wrote:
> > On Wed, Feb 07, 2024 at 03:30:29PM -0500, gene heskett wrote:
> > > [629241.074187] scsi host37: usb-storage 1-2:1.0
> > 
> > USB storage is for phones and cameras etc, not for serious
> > computing.
> 
> Do you mean that a proper backup drive should use uas (USB Attached Storage)
> instead of usb-storage driver?

No, I just mean I advise to never ever use storage that comes to you
via a USB port for anything you care about.

I might consider it okay for temporary shifting of data about, but I
would never use it as part of a permanent setup without fully
expecting one day to find it just not working. But then that is also
how I feel about any storage device. It's just worse when USB is
added to the mix.

Thanks,
Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

Re: Things I don't touch with a 3.048m barge pole: USB storage (Was Re: Unidentified subject!)

[Jeffrey Walton]

On Thu, Feb 8, 2024 at 11:57 AM Ralph Aichinger  wrote:
>
> On Thu, 2024-02-08 at 15:36 +, Andy Smith wrote:
> > USB storage is for phones and cameras etc, not for serious
> > computing. Many people will disagree with that statement and say
> > they use it all the time and it is fine.
>
> I am clearly in the latter camp. This mail is delivered via a Raspberry
> Pi 4 that has a 500G USB SSD. Before the Pi4 I used a Pi3 and a Pi2 (I
> think) with USB disks (first rotating, then SSD). Probably for 5 years
> or so. Never had a problem (unlike with the SD cards I used before, SD
> cards always died on me from to many writes after a few months).
>
> > They will keep saying that
> > until it isn't fine, and then they'll be in a world of hurt.
>
> This is the same with any hard disk or SSD. If you buy the most
> expensive "enterprise" disk, with SAS or whatever, it still can
> break on the next day, taking all your data with you.
>
> Actually with USB disks, sometimes you can remove the USB
> controller, replace it in case of breakage, giving you more
> or less the same reliability as any "normal" disk.
> I've never had USB controllers break, though, so I do not
> care. I just take backups as with any other disk.
>
> > I learned not to go there a long time ago and have seen plenty of
> > reminders along the way from others' misfortunes to not ever go
> > there again myself.
>
> How does a breaking USB disk differ from a breaking SATA disk?

I may be mistaken, but I believe AS is talking about USB thumb drives,
SDcards and the like. I don't think he's talking about external SSD's
and NVME's over USB. But I don't want to put words in his mouth.

My experience with SDcards and thumb drives is along the lines of
AS's. I own a lot of dev boards (dating back to the early 2010's) for
testing, and I could go through a storage device, like an SDcard, in
about 6 months. But I would also add a swap file to the installation
because the dev boards were so resource constrained. You simply can't
run a C++ compiler on a Beagleboard with 256MB of RAM. The swap file,
even with a low swappiness, would eat up SDcards and thumb drives.

Jeff

Re: Things I don't touch with a 3.048m barge pole: USB storage (Was Re: Unidentified subject!)

[Max Nikulin]

On 08/02/2024 22:36, Andy Smith wrote:

On Wed, Feb 07, 2024 at 03:30:29PM -0500, gene heskett wrote:

[629241.074187] scsi host37: usb-storage 1-2:1.0


USB storage is for phones and cameras etc, not for serious
computing.


Do you mean that a proper backup drive should use uas (USB Attached 
Storage) instead of usb-storage driver?

Re: Things I don't touch with a 3.048m barge pole: USB storage (Was Re: Unidentified subject!)

[Ralph Aichinger]

On Thu, 2024-02-08 at 15:36 +, Andy Smith wrote:
> USB storage is for phones and cameras etc, not for serious
> computing. Many people will disagree with that statement and say
> they use it all the time and it is fine. 

I am clearly in the latter camp. This mail is delivered via a Raspberry
Pi 4 that has a 500G USB SSD. Before the Pi4 I used a Pi3 and a Pi2 (I
think) with USB disks (first rotating, then SSD). Probably for 5 years
or so. Never had a problem (unlike with the SD cards I used before, SD
cards always died on me from to many writes after a few months).

> They will keep saying that
> until it isn't fine, and then they'll be in a world of hurt.

This is the same with any hard disk or SSD. If you buy the most
expensive "enterprise" disk, with SAS or whatever, it still can 
break on the next day, taking all your data with you. 

Actually with USB disks, sometimes you can remove the USB 
controller, replace it in case of breakage, giving you more
or less the same reliability as any "normal" disk.
I've never had USB controllers break, though, so I do not
care. I just take backups as with any other disk.

> I learned not to go there a long time ago and have seen plenty of
> reminders along the way from others' misfortunes to not ever go
> there again myself.

How does a breaking USB disk differ from a breaking SATA disk?

/ralph

Re: Things I don't touch with a 3.048m barge pole: USB storage (Was Re: Unidentified subject!)

[Andy Smith]

Hi,

On Thu, Feb 08, 2024 at 11:14:24AM -0500, Gremlin wrote:
> On 2/8/24 10:36, Andy Smith wrote:
> > USB storage is for phones and cameras etc, not for serious
> > computing. Many people will disagree with that statement and say
> > they use it all the time and it is fine. They will keep saying that
> > until it isn't fine, and then they'll be in a world of hurt.
> > 
> 
> LOL,  So my main desktop a raspberry pi 4 is not serious computing? Or is it
> that my name server, web server email server which is a raspberry pi 4 not
> serious computing?

Not in my opinion, no¹, but I don't mind at all if you don't agree
and I also wish you the best of ongoing luck!

Thanks,
Andy

¹ Of course, sometimes you just have a device that only has USB and
  there's no way around it. If I have to go there, I try to make it
  serious by preparing for the storage of those devices to just
  disappear one day and take steps to minimise the downtime lost to
  that.

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

Re: Things I don't touch with a 3.048m barge pole: USB storage (Was Re: Unidentified subject!)

[Gremlin]

On 2/8/24 10:36, Andy Smith wrote:

Hello,

On Wed, Feb 07, 2024 at 03:30:29PM -0500, gene heskett wrote:

[629241.074187] scsi host37: usb-storage 1-2:1.0


USB storage is for phones and cameras etc, not for serious
computing. Many people will disagree with that statement and say
they use it all the time and it is fine. They will keep saying that
until it isn't fine, and then they'll be in a world of hurt.



LOL,  So my main desktop a raspberry pi 4 is not serious computing? Or 
is it that my name server, web server email server which is a raspberry 
pi 4 not serious computing?


They both boot to USB SSDs and only have USB SSD drives, so they are not 
serious computing?  The desktop RPI has an NVME drive as the boot drive 
connected by you guessed it USB.




I learned not to go there a long time ago and have seen plenty of
reminders along the way from others' misfortunes to not ever go
there again myself.

Things I don't touch with a 3.048m barge pole: USB storage (Was Re: Unidentified subject!)

[Andy Smith]

Hello,

On Wed, Feb 07, 2024 at 03:30:29PM -0500, gene heskett wrote:
> [629241.074187] scsi host37: usb-storage 1-2:1.0

USB storage is for phones and cameras etc, not for serious
computing. Many people will disagree with that statement and say
they use it all the time and it is fine. They will keep saying that
until it isn't fine, and then they'll be in a world of hurt.

I learned not to go there a long time ago and have seen plenty of
reminders along the way from others' misfortunes to not ever go
there again myself.

> Looks like a reasonable facsimile of a 2T disk to me.

Good luck.

Thanks,
Andy

-- 
https://bitfolk.com/ -- No-nonsense VPS hosting

Re: Portable External Hard Drive 2TB (was: Unidentified subject!)

[gene heskett]

On 2/7/24 21:55, David Christensen wrote:

On 1/22/24 19:55, gene heskett wrote:
 > 2T ssd's:
 >
 > 




It appears Amazon took down the above web page.  Using Amazon search, I 
see:


https://www.amazon.com/Portable-External-USB3-1-Desktop-Chromebook/dp/B0CSNGML52/


On 2/7/24 12:30, gene heskett wrote:
 > Well the 2T memory everybody was curious about 3 weeks ago got here 
early.

 >
 >  From dmesg after plugging one in:
 > [629240.916163] usb 1-2: new high-speed USB device number 39 using 
xhci_hcd

 > [629241.066221] usb 1-2: New USB device found, idVendor=048d,
 > idProduct=1234, bcdDevice= 2.00
 > [629241.066234] usb 1-2: New USB device strings: Mfr=1, Product=2,
 > SerialNumber=3
 > [629241.066239] usb 1-2: Product: Disk 3.0
 > [629241.066242] usb 1-2: Manufacturer: USB
 > [629241.066246] usb 1-2: SerialNumber: 2697241127107725123
 > [629241.069485] usb-storage 1-2:1.0: USB Mass Storage device detected
 > [629241.074187] scsi host37: usb-storage 1-2:1.0
 > [629242.100738] scsi 37:0:0:0: Direct-Access  SSD 3.0   2.00
 > PQ: 0 ANSI: 4
 > [629242.100959] sd 37:0:0:0: Attached scsi generic sg13 type 0
 > [629242.101190] sd 37:0:0:0: [sdm] 409600 512-byte logical blocks:
 > (2.10 TB/1.91 TiB)
 > [629242.101289] sd 37:0:0:0: [sdm] Write Protect is off
 > [629242.101290] sd 37:0:0:0: [sdm] Mode Sense: 03 00 00 00
 > [629242.101409] sd 37:0:0:0: [sdm] No Caching mode page found
 > [629242.101410] sd 37:0:0:0: [sdm] Assuming drive cache: write through
 > [629242.103927]  sdm: sdm1
 > [629242.104047] sd 37:0:0:0: [sdm] Attached SCSI disk
 > gene@coyote:
 >
 > Looks like a reasonable facsimile of a 2T disk to me.
 >
 > Cheers, Gene Heskett, CET.


I would be curious to see a smartctl(8) full report:

# smartctl -x /dev/sdm


Have you figured out how to verify it actually holds 2 TB?


I think its and m2 device, and smartctl says it does know about them yet
gene@coyote:/etc$ sudo smartctl --all --scan
/dev/sda -d scsi # /dev/sda, SCSI device
/dev/sdd -d scsi # /dev/sdd, SCSI device
/dev/sde -d scsi # /dev/sde, SCSI device
/dev/sdf -d scsi # /dev/sdf, SCSI device
/dev/sdg -d scsi # /dev/sdg, SCSI device
/dev/sdh -d scsi # /dev/sdh, SCSI device
/dev/sdi -d scsi # /dev/sdi, SCSI device
/dev/sdj -d scsi # /dev/sdj, SCSI device
/dev/sdk -d scsi # /dev/sdk, SCSI device
/dev/sdl -d scsi # /dev/sdl, SCSI device
gene@coyote:/etc$
gartped says it look ok at 1.92 T


David

.


Cheers, Gene Heskett, CET.
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis

Re: Portable External Hard Drive 2TB (was: Unidentified subject!)

[gene heskett]

On 2/7/24 21:55, David Christensen wrote:

On 1/22/24 19:55, gene heskett wrote:
 > 2T ssd's:
 >
 > 




It appears Amazon took down the above web page.  Using Amazon search, I 
see:


https://www.amazon.com/Portable-External-USB3-1-Desktop-Chromebook/dp/B0CSNGML52/


On 2/7/24 12:30, gene heskett wrote:
 > Well the 2T memory everybody was curious about 3 weeks ago got here 
early.

 >
 >  From dmesg after plugging one in:
 > [629240.916163] usb 1-2: new high-speed USB device number 39 using 
xhci_hcd

 > [629241.066221] usb 1-2: New USB device found, idVendor=048d,
 > idProduct=1234, bcdDevice= 2.00
 > [629241.066234] usb 1-2: New USB device strings: Mfr=1, Product=2,
 > SerialNumber=3
 > [629241.066239] usb 1-2: Product: Disk 3.0
 > [629241.066242] usb 1-2: Manufacturer: USB
 > [629241.066246] usb 1-2: SerialNumber: 2697241127107725123
 > [629241.069485] usb-storage 1-2:1.0: USB Mass Storage device detected
 > [629241.074187] scsi host37: usb-storage 1-2:1.0
 > [629242.100738] scsi 37:0:0:0: Direct-Access  SSD 3.0   2.00
 > PQ: 0 ANSI: 4
 > [629242.100959] sd 37:0:0:0: Attached scsi generic sg13 type 0
 > [629242.101190] sd 37:0:0:0: [sdm] 409600 512-byte logical blocks:
 > (2.10 TB/1.91 TiB)
 > [629242.101289] sd 37:0:0:0: [sdm] Write Protect is off
 > [629242.101290] sd 37:0:0:0: [sdm] Mode Sense: 03 00 00 00
 > [629242.101409] sd 37:0:0:0: [sdm] No Caching mode page found
 > [629242.101410] sd 37:0:0:0: [sdm] Assuming drive cache: write through
 > [629242.103927]  sdm: sdm1
 > [629242.104047] sd 37:0:0:0: [sdm] Attached SCSI disk
 > gene@coyote:
 >
 > Looks like a reasonable facsimile of a 2T disk to me.
 >
 > Cheers, Gene Heskett, CET.


I would be curious to see a smartctl(8) full report:

# smartctl -x /dev/sdm


Have you figured out how to verify it actually holds 2 TB?

No, not teted that yet. Up to my adam's apple in other things ATM.  Will 
lets the list know when I do.


David

.


Cheers, Gene Heskett, CET.
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis

Re: Unidentified subject!

[Stefan Monnier]

> Well the 2T memory everybody was curious about 3 weeks ago got here early.
>
> From dmesg after plugging one in:
> [629240.916163] usb 1-2: new high-speed USB device number 39 using xhci_hcd
> [629241.066221] usb 1-2: New USB device found, idVendor=048d,
> idProduct=1234, bcdDevice= 2.00
> [629241.066234] usb 1-2: New USB device strings: Mfr=1, Product=2,
> SerialNumber=3
> [629241.066239] usb 1-2: Product: Disk 3.0
> [629241.066242] usb 1-2: Manufacturer: USB
> [629241.066246] usb 1-2: SerialNumber: 2697241127107725123
> [629241.069485] usb-storage 1-2:1.0: USB Mass Storage device detected
> [629241.074187] scsi host37: usb-storage 1-2:1.0
> [629242.100738] scsi 37:0:0:0: Direct-Access  SSD 3.02.00
> PQ: 0 ANSI: 4
> [629242.100959] sd 37:0:0:0: Attached scsi generic sg13 type 0
> [629242.101190] sd 37:0:0:0: [sdm] 409600 512-byte logical blocks: (2.10
> TB/1.91 TiB)
> [629242.101289] sd 37:0:0:0: [sdm] Write Protect is off
> [629242.101290] sd 37:0:0:0: [sdm] Mode Sense: 03 00 00 00
> [629242.101409] sd 37:0:0:0: [sdm] No Caching mode page found
> [629242.101410] sd 37:0:0:0: [sdm] Assuming drive cache: write through
> [629242.103927]  sdm: sdm1
> [629242.104047] sd 37:0:0:0: [sdm] Attached SCSI disk
> gene@coyote:
>
> Looks like a reasonable facsimile of a 2T disk to me.

AFAIK the bogus 128TB drives do properly report such ridiculous sizes:
the reality only hits when you try to actually store that amount of
information on them.

[ I'm not sure how it works under the hood, but since SSDs store their
  data "anywhere" in the flash, they can easily pretend to have any size
  they want, and allocate the physical flash blocks only on-the-fly as
  logical blocks are being written.
  Also, some Flash controllers use compression, so if you store data
  that compresses well, they can let you store a lot more than if you
  store already compressed data.  ]

IOW, to really check, try to save 2TB of videos (or other already
compressed data), and then try and read it back.


Stefan

Portable External Hard Drive 2TB (was: Unidentified subject!)

[David Christensen]

On 1/22/24 19:55, gene heskett wrote:
> 2T ssd's:
>
> 




It appears Amazon took down the above web page.  Using Amazon search, I see:

https://www.amazon.com/Portable-External-USB3-1-Desktop-Chromebook/dp/B0CSNGML52/


On 2/7/24 12:30, gene heskett wrote:
> Well the 2T memory everybody was curious about 3 weeks ago got here 
early.

>
>  From dmesg after plugging one in:
> [629240.916163] usb 1-2: new high-speed USB device number 39 using 
xhci_hcd

> [629241.066221] usb 1-2: New USB device found, idVendor=048d,
> idProduct=1234, bcdDevice= 2.00
> [629241.066234] usb 1-2: New USB device strings: Mfr=1, Product=2,
> SerialNumber=3
> [629241.066239] usb 1-2: Product: Disk 3.0
> [629241.066242] usb 1-2: Manufacturer: USB
> [629241.066246] usb 1-2: SerialNumber: 2697241127107725123
> [629241.069485] usb-storage 1-2:1.0: USB Mass Storage device detected
> [629241.074187] scsi host37: usb-storage 1-2:1.0
> [629242.100738] scsi 37:0:0:0: Direct-Access  SSD 3.0   2.00
> PQ: 0 ANSI: 4
> [629242.100959] sd 37:0:0:0: Attached scsi generic sg13 type 0
> [629242.101190] sd 37:0:0:0: [sdm] 409600 512-byte logical blocks:
> (2.10 TB/1.91 TiB)
> [629242.101289] sd 37:0:0:0: [sdm] Write Protect is off
> [629242.101290] sd 37:0:0:0: [sdm] Mode Sense: 03 00 00 00
> [629242.101409] sd 37:0:0:0: [sdm] No Caching mode page found
> [629242.101410] sd 37:0:0:0: [sdm] Assuming drive cache: write through
> [629242.103927]  sdm: sdm1
> [629242.104047] sd 37:0:0:0: [sdm] Attached SCSI disk
> gene@coyote:
>
> Looks like a reasonable facsimile of a 2T disk to me.
>
> Cheers, Gene Heskett, CET.


I would be curious to see a smartctl(8) full report:

# smartctl -x /dev/sdm


Have you figured out how to verify it actually holds 2 TB?


David

Unidentified subject!

[gene heskett]

Well the 2T memory everybody was curious about 3 weeks ago got here early.

From dmesg after plugging one in:
[629240.916163] usb 1-2: new high-speed USB device number 39 using xhci_hcd
[629241.066221] usb 1-2: New USB device found, idVendor=048d, 
idProduct=1234, bcdDevice= 2.00
[629241.066234] usb 1-2: New USB device strings: Mfr=1, Product=2, 
SerialNumber=3

[629241.066239] usb 1-2: Product: Disk 3.0
[629241.066242] usb 1-2: Manufacturer: USB
[629241.066246] usb 1-2: SerialNumber: 2697241127107725123
[629241.069485] usb-storage 1-2:1.0: USB Mass Storage device detected
[629241.074187] scsi host37: usb-storage 1-2:1.0
[629242.100738] scsi 37:0:0:0: Direct-Access  SSD 3.0 
  2.00 PQ: 0 ANSI: 4

[629242.100959] sd 37:0:0:0: Attached scsi generic sg13 type 0
[629242.101190] sd 37:0:0:0: [sdm] 409600 512-byte logical blocks: 
(2.10 TB/1.91 TiB)

[629242.101289] sd 37:0:0:0: [sdm] Write Protect is off
[629242.101290] sd 37:0:0:0: [sdm] Mode Sense: 03 00 00 00
[629242.101409] sd 37:0:0:0: [sdm] No Caching mode page found
[629242.101410] sd 37:0:0:0: [sdm] Assuming drive cache: write through
[629242.103927]  sdm: sdm1
[629242.104047] sd 37:0:0:0: [sdm] Attached SCSI disk
gene@coyote:

Looks like a reasonable facsimile of a 2T disk to me.

Cheers, Gene Heskett, CET.
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis

Unidentified subject!

[Pocket]

which poc...@columbus.rr.com

--
Hindi madali ang maging ako

Unidentified subject!

[bounce-debian-user=archive=mail-archive.com]

Unidentified subject!

[Stella Ashburne]

The partition table scheme is GPT and UEFI with Secure Boot is enabled. I do 
not use legacy BIOS with master boot record.

Below is the partition layout of my SDD:

536.9MB EFI system partition (ESP)
511.7MB /boot (unencrypted)
100GB encrypted logical volumes (contains 99GB of / partition, 1GB of swap 
area.Debian Buster was installed on this partition)
16MB Microsoft reserved area (automatically created by Microsoft Windows' 
installer)
100GB Microsoft Windows 10

1. Debian Buster's 64bit installer (version 10.10) was used to create the EFI 
System Partition (ESP), the /boot partition and the encrypted logical volumes. 
Installation was successful and I was able to boot into the GRUB menu with a 
blue background. It had an entry named Debian GNU/Linux.

2. Next I installed Microsoft Windows 10 and the installation was successful.

3. I rebooted into Debian and used sudo os-prober to add the Microsoft Windows' 
entry to GRUB followed by sudo update-grub

4. Dual-boot of Debian and Windows was possible

Problem(s) happened after I did the following:

5. With a USB stick containing the latest weekly build of Debian Testing 
(Bullseye), I booted into the Debian's installer screen and deleted the 100GB 
encrypted logical volumes.

6. As a result, 100GB of free space was made available. I configured it to have 
two encrypted logical volumes: 99GB of the / partition, 1GB of swap area.

7. Debian Testing was installed on the 100GB partition. Installation was 
successful.

8. However, I am now unable to boot into the GRUB menu with the blue 
background. Instead all I have is a black screen with the word grub> _ (The 
underscore is actually the position of the cursor)

After reading some stuff on the internet, please tell me if my understanding of 
the following is correct:

Grub's UEFI Stub is located in EFI System Partition (ESP) while its second 
stage modules are in the /boot partition. /boot also contains Grub's config 
file. It would appear that the bootloader in ESP is not updated to match the 
modules in the /boot partition or it could be that /boot/grub/grub.cfg is 
missing.

Below's my attempt at getting back the Grub menu with the blue background:

A. I used Debian Bullseye's weekly installer to boot up my machine and chose 
Rescue mode.

B. After entering the encrypted passphrase, below were some relevant messages 
on the screen:

[beginning of message]
Enter a device you wish to use as your root file system.

Device to use as root file system:

/dev/perfect-vg/root
/dev/perfect-vg/swap
/dev/dm-1
/dev/dm-2
/dev/mapper/sda3_crypt
/dev/sda1
/dev/sda2
etc, etc
Assemble a RAID system
Do not use a root file system
[end of message]

I highlighted /dev/sda1 and pressed Enter.

I was given four choices one of which was to Execute a shell in the installer 
environment. I highlighted it and pressed Enter.

There was a message on the screen stating that /dev/sda1 would be mounted as 
"/target", that the tools of the installer environment would be available for 
use and that I could use chroot to "chroot /target".

There was a small grey box at the bottom of the screen.

I tried the following options:


~# "chroot /target"
/bin/sh: "chroot /target" not found



~# chroot /target
chroot: can't execute "/bin/sh". No such file or directory



~# apt install --reinstall grub-efi
/bin/sh: apt: not found



I am stuck with the above as my technical knowledge of Linux is limited and 
appreciate your help with the above matter.

Unidentified subject!

[peter]

 
 
 
 

Subject: Re: Network connection of a qemu guest.

From: Reco 
Date: Fri, 12 Mar 2021 08:26:51 +0300
> No, it does not work that way.

Got the network connection and added documention here.

https://en.wikibooks.org/wiki/Oberon/ETH_Oberon/QEMUinstall#Network_Connection_on_a_Virtual_Machine

Further tips always welcome.

Thanks,... P.


-- 
cell: +1 236 464 1479Bcc: peter at easthope. ca
VoIP: +1 604 670 0140

Re: Unidentified subject!

[songbird]

Dan Ritter wrote:
...
> You want to ignore the USB ports and focus on the attached
> devices. udev is the mechanism here.
>
> For example, in /etc/udev/rules.d/70-ups  I have:
>
> SUBSYSTEM=="usb", ATTR{idVendor}=="0764", ATTR{idProduct}=="0501", 
> SYMLINK+="ups0", GROUP="nut"
>
> Which means that /dev/ups0 will be assigned to the USB device
> that has an idVendor of 0764 and an idProduct of 0501, and be
> accessible by anyone in the group nut.
>
> So if you have a MIDI keyboard with idVendor 0499 (Yamaha) and 
> idProduct 1617 (PSR-E353 digital keyboard) then you could plug
> those numbers into a rule like the one above and whenever it was
> connected, it would appear at, say, "/dev/midikb0".
>
> SUBSYSTEM=="usb", ATTR{idVendor}=="0499", ATTR{idProduct}=="1617", 
> SYMLINK+="midikb0", GROUP="audio"
>
> lsusb (possibly with a -v) will help you find the currently
> attached devices and their numbers.
>
> Does that help?

  excellent reply thanks!


  songbird  (not the OP or in need of this, but very useful
 information anyways...  :)