subject:"\[no subject\]"

Re: Subject: Glitchy sound in Steam games after hard drive upgrade

2024-04-28 Thread Curt

On 2024-04-22, Charlie Gibbs  wrote:
>
> TL;DR: Copying an existing /home into a fresh Debian installation
> causes audio in Steam games to glitch - but all other sound is OK.

I have only the most vaporous ideas about Steam, but have you tried
backing up and then recreating (if such a thing is possible) your user profile
~/.steam or ~/.config/steam files?

Re: Subject: Glitchy sound in Steam games after hard drive upgrade

2024-04-23 Thread Stefan Monnier

> I doubt the new drive is slower than the old drive:

Overall, agreed.  Tho AFAICT the new drive spins slower (5400rpm vs
7200rpm), so it has a slightly higher rotational latency.  This means
that in *some* cases it can be slower.
Now, I have no idea whether that's the cause of the glitches.


Stefan

Re: Subject: Glitchy sound in Steam games after hard drive upgrade

2024-04-23 Thread David Christensen


On 4/23/24 09:02, debian-u...@howorth.org.uk wrote:

Charlie Gibbs  wrote:

On 2024-04-22 16:50, Jeffrey Walton wrote:


What are the old and new hard drive model numbers and specs?


The old drive is a Western Digital WD5000YS (500GB SATA).
The new drive is a Western Digital Red, WF40EFPX (4TB SATA).


According to my searches, there's no such disk as a WF40EFPX. Are you
sure that's what it is? If by any chance it is a WD40EFRX then that is
certainly slower than your old drive, so may cause some problems as
suggested.



I doubt the new drive is slower than the old drive:

- https://www.harddrivebenchmark.net/hdd.php?hdd=WDC%20WD5000YS

WDC WD5000YS  425

- https://www.harddrivebenchmark.net/hdd.php?hdd=WDC%20WD40EFRX

WDC WD40EFRX1,943


David

Re: Subject: Glitchy sound in Steam games after hard drive upgrade

2024-04-23 Thread David Christensen

On 4/22/24 21:26, Charlie Gibbs wrote:

On 2024-04-22 16:50, Jeffrey Walton wrote:

What are the old and new hard drive model numbers and specs?

The old drive is a Western Digital WD5000YS (500GB SATA).

https://www.newegg.com/western-digital-re2-wd5000ys-500gb/p/N82E16822136032?Item=N82E16822136032

The new drive is a Western Digital Red, WF40EFPX (4TB SATA).

https://www.westerndigital.com/products/internal-drives/wd-red-plus-sata-3-5-hdd?sku=WD40EFPX

Both drives are spinning rust. I'm upgrading for the increased
capacity, i.e. to store more MP3s and videos.

Many thanks to all who have replied. When my schedule permits me to
continue experimenting, I'm going to try copying /etc from the old drive
to the new one. I've already learned how _not_ to do this:

Boot from the new drive
$ su root
# cd /
# mv etc etc.ori
# rsync -av /mnt/backup/etc .

The second line makes the system fall over and makes logins impossible.
It took a boot from the rescue CD to undo the damage, which fortunately
was easy since the deadly step at least succeeded in backing up /etc.

Next time I'll do it while booted from the old drive.

Copying an entire /etc directory from one machine to another requires a
highly controlled environment and lot of engineering. I have always
migrated /etc settings from one OS instance to another OS instance by
hand, one service/ configuration file at a time.

Can you leave the 500 GB HDD operational and use the 4 TB HDD for data?

David

Re: Subject: Glitchy sound in Steam games after hard drive upgrade

2024-04-23 Thread debian-user

Charlie Gibbs  wrote:
> On 2024-04-22 16:50, Jeffrey Walton wrote:
> 
> > What are the old and new hard drive model numbers and specs?  
> 
> The old drive is a Western Digital WD5000YS (500GB SATA).
> The new drive is a Western Digital Red, WF40EFPX (4TB SATA).

According to my searches, there's no such disk as a WF40EFPX. Are you
sure that's what it is? If by any chance it is a WD40EFRX then that is
certainly slower than your old drive, so may cause some problems as
suggested.

Re: Subject: Glitchy sound in Steam games after hard drive upgrade

2024-04-22 Thread Charlie Gibbs


On 2024-04-22 16:50, Jeffrey Walton wrote:


What are the old and new hard drive model numbers and specs?


The old drive is a Western Digital WD5000YS (500GB SATA).
The new drive is a Western Digital Red, WF40EFPX (4TB SATA).

If the old hard drive was spinning rust, it is acceptable to replace it 
with a solid state drive. I did it several times in the past. But 
nowadays a new machine usually (always?) comes with a SSD, so you 
usually don't need to upgrade for performance reasons.


Both drives are spinning rust.  I'm upgrading for the increased 
capacity, i.e. to store more MP3s and videos.


Many thanks to all who have replied.  When my schedule permits me to 
continue experimenting, I'm going to try copying /etc from the old drive 
to the new one.   I've already learned how _not_ to do this:


Boot from the new drive
$ su root
# cd /
# mv etc etc.ori
# rsync -av /mnt/backup/etc .

The second line makes the system fall over and makes logins impossible. 
It took a boot from the rescue CD to undo the damage, which fortunately 
was easy since the deadly step at least succeeded in backing up /etc.


Next time I'll do it while booted from the old drive.

--
/~\  Charlie Gibbs  |  "Some of you may die,
\ /|  but it's a sacrifice
 X   I'm really at ac.dekanfrus |  I'm willing to make."
/ \  if you read it the right way.  |-- Lord Farquaad (Shrek)

Re: Subject: Glitchy sound in Steam games after hard drive upgrade

2024-04-22 Thread Jeffrey Walton

On Mon, Apr 22, 2024 at 5:03 AM Charlie Gibbs  wrote:

> I should probably be posting this to the Steam forums, but
> most of the denizens there are Windows people so I might be
> better off letting you Debian gurus have a go at it first.
>
> TL;DR: Copying an existing /home into a fresh Debian installation
> causes audio in Steam games to glitch - but all other sound is OK.
>
> Full description:
>
> I have a machine in the living room that stores MP3s and videos
> and serves them to other machines on our network as well as playing
> them locally on our TV's big screen.  I also play a few Steam games
> (e.g. Portal) on it.  It's a 2007-vintage machine, but it has 8GB
> of RAM and enough CPU power to do the job, and runs the latest
> version of Bookworm.
>
> Recently I decided to upgrade its storage capacity, and replaced
> its 500GB hard drive (which was pretty large at the time I bought
> it) with a 4TB drive.  I did an install from scratch using a
> network install CD, then copied my /home partition (using rsync)
> from the old drive.  Everything works great with one exception:
> when I fire up Portal the sound gets glitches about once a second.
> This only happens with Steam games; I can play MP3s and videos
> with mpv and the sound is perfect, as it is when watching YouTube
> videos.  If I swap the old drive back in everything is fine.
>
> Obviously my Steam programs and configuration files are in my
> home directory, since the updated system comes up icons and all
> without re-installing Steam, and can find everything it needs to
> run the games.  But perhaps there are a few files somewhere else
> (/usr?) containing information critical to audio for Steam.
>
> Any ideas?
>

What are the old and new hard drive model numbers and specs?

If the old hard drive was spinning rust, it is acceptable to replace it
with a solid state drive. I did it several times in the past. But nowadays
a new machine usually (always?) comes with a SSD, so you usually don't need
to upgrade for performance reasons.

Re: Subject: Glitchy sound in Steam games after hard drive upgrade

2024-04-22 Thread Stefan Monnier

> Recently I decided to upgrade its storage capacity, and replaced
> its 500GB hard drive (which was pretty large at the time I bought
> it) with a 4TB drive.  I did an install from scratch using a
> network install CD, then copied my /home partition (using rsync)
> from the old drive.
[...]
> (Side question: is this an acceptable way to upgrade a hard drive?)

It's acceptable enough that we'll keep talking to you.  

Personally, assuming the 500GB drive was basically full, I suspect I'd
have just done a `dd` copy of the 500GB drive to the new drive, followed
by a quick `gparted` run to resize on-the-fly the partitions (in order
to get access to the extra 3.5GB).

> Everything works great with one exception:
> when I fire up Portal the sound gets glitches about once a second.
> This only happens with Steam games; I can play MP3s and videos
> with mpv and the sound is perfect, as it is when watching YouTube
> videos.  If I swap the old drive back in everything is fine.

I suspect the difference is that the Steam games keep your machine very
busy whereas playing a video isn't nearly as demanding, so the machine
ends up too busy to refill the sound buffer before its empty.

As for why this happens with the new disk and not with the old
disk, ...
AFAICT it can be either due to the new install such as a difference in
the configuration and/or installed software (e.g. one using pulseaudio
and the other pipewire), or due to the new hardware, presumably because
some operations are slower.

Can you boot with both disks connected?  If so, can you try to boot off
of the 500GB and then use the /home from the 4TB drive (and vice versa)?
I think you should be able to do that by booting to "rescue" where
(after entering the root password) you'd do something like

umount /home
mount /dev/the/other/home/partition /home
exit

I'd tend to think that a modern 4TB drive should be no slower than
a 500GB drive, no matter the operation, but maybe the new drive has
a particularly small cache, or maybe it's shingled and the Steam game
makes a fair amount of writes to the disk which ends up affecting the
reads needed to fetch the next chunk of sound?


Stefan

Re: Subject: Glitchy sound in Steam games after hard drive upgrade

2024-04-22 Thread David Christensen


On 4/21/24 22:33, Charlie Gibbs wrote:

I should probably be posting this to the Steam forums, but
most of the denizens there are Windows people so I might be
better off letting you Debian gurus have a go at it first.

TL;DR: Copying an existing /home into a fresh Debian installation
causes audio in Steam games to glitch - but all other sound is OK.

Full description:

I have a machine in the living room that stores MP3s and videos
and serves them to other machines on our network as well as playing
them locally on our TV's big screen.  I also play a few Steam games
(e.g. Portal) on it.  It's a 2007-vintage machine, but it has 8GB
of RAM and enough CPU power to do the job, and runs the latest
version of Bookworm.

Recently I decided to upgrade its storage capacity, and replaced
its 500GB hard drive (which was pretty large at the time I bought
it) with a 4TB drive.  I did an install from scratch using a
network install CD, then copied my /home partition (using rsync)
from the old drive.  Everything works great with one exception:
when I fire up Portal the sound gets glitches about once a second.
This only happens with Steam games; I can play MP3s and videos
with mpv and the sound is perfect, as it is when watching YouTube
videos.  If I swap the old drive back in everything is fine.

Obviously my Steam programs and configuration files are in my
home directory, since the updated system comes up icons and all
without re-installing Steam, and can find everything it needs to
run the games.  But perhaps there are a few files somewhere else
(/usr?) containing information critical to audio for Steam.

Any ideas?

(Side question: is this an acceptable way to upgrade a hard drive?)



Copying a home directory from one OS instance to another OS instance 
sounds risky, especially as I run various OS's.  I have several 
instances of Debian 11, and would not consider them to be identical 
enough to try it.  I only touch the content I create or have learned how 
to manage.



I put my OS on a small SSD and the vast majority of my data on HDD RAID 
in a file server.



As I am the only user on my Debian daily driver, I leave the /home 
directory on the root file system and keep as little as possible in it.



I mount the file server shares under /mnt, and create symlinks in my 
home directory that point into the mounted file system.



I use CVS for project working directories.  To migrate to a new home 
directory, I check in the projects in the old home and check out the 
project in the new home.



I use Firefox and its sync feature.  To migrate to a new home, I start 
Firefox, log in, wait for my settings to sync, and then check all of the 
settings by hand.



I use Thunderbird.  To migrate to a new home, I create a tarball of my 
Thunderbird profile directory on the old machine, expand the tarball on 
the new machine, and configure Thunderbird to use that profile.



I do not attempt to migrate any of the various home directory 
configuration directories; I let the installer and/or package manager 
create them, and let the desktop, apps, etc., manage them.



David

Re: Subject: Glitchy sound in Steam games after hard drive upgrade

2024-04-22 Thread Michel Verdier

On 2024-04-21, Charlie Gibbs wrote:

> Obviously my Steam programs and configuration files are in my
> home directory, since the updated system comes up icons and all
> without re-installing Steam, and can find everything it needs to
> run the games.  But perhaps there are a few files somewhere else
> (/usr?) containing information critical to audio for Steam.

Do you sync /etc ? Configuration is mainly there.

> (Side question: is this an acceptable way to upgrade a hard drive?)

There is other ways but yes if you also sync /etc :)

Subject: Glitchy sound in Steam games after hard drive upgrade

2024-04-21 Thread Charlie Gibbs


I should probably be posting this to the Steam forums, but
most of the denizens there are Windows people so I might be
better off letting you Debian gurus have a go at it first.

TL;DR: Copying an existing /home into a fresh Debian installation
causes audio in Steam games to glitch - but all other sound is OK.

Full description:

I have a machine in the living room that stores MP3s and videos
and serves them to other machines on our network as well as playing
them locally on our TV's big screen.  I also play a few Steam games
(e.g. Portal) on it.  It's a 2007-vintage machine, but it has 8GB
of RAM and enough CPU power to do the job, and runs the latest
version of Bookworm.

Recently I decided to upgrade its storage capacity, and replaced
its 500GB hard drive (which was pretty large at the time I bought
it) with a 4TB drive.  I did an install from scratch using a
network install CD, then copied my /home partition (using rsync)
from the old drive.  Everything works great with one exception:
when I fire up Portal the sound gets glitches about once a second.
This only happens with Steam games; I can play MP3s and videos
with mpv and the sound is perfect, as it is when watching YouTube
videos.  If I swap the old drive back in everything is fine.

Obviously my Steam programs and configuration files are in my
home directory, since the updated system comes up icons and all
without re-installing Steam, and can find everything it needs to
run the games.  But perhaps there are a few files somewhere else
(/usr?) containing information critical to audio for Steam.

Any ideas?

(Side question: is this an acceptable way to upgrade a hard drive?)

--
/~\  Charlie Gibbs  |  Life is perverse.
\ /|  It can be beautiful -
 X   I'm really at ac.dekanfrus |  but it won't.
/ \  if you read it the right way.  |-- Lily Tomlin

[no subject]

2024-03-03 Thread Andre Rodier

Hello,

I was checking the Debian domain, and noticed that it is DNSSEC compliant.

However, when I check "deb.debian.org", the DNS validation fails.

Is there any reason behind this, please ?

Thanks,
André Rodier.

Re: shred bug? [was: Unidentified subject!]

2024-02-16 Thread Michael Stone


On Sun, Feb 11, 2024 at 08:02:12AM +0100, to...@tuxteam.de wrote:

What Thomas was trying to do is to get a cheap, fast random number
generator. Shred seems to have such.


You're better off with /dev/urandom, it's much easier to understand what 
it's trying to do, vs the rather baroque logic in shred. In fact, 
there's nothing in shred's documenation AFAICT that suggests it should 
be used as a random number generator. For pure speed, playing games with 
openssl enc and /dev/zero will generally win. If speed doesn't matter, 
we're back to /dev/urandom as the simplest and most direct solution.


FWIW, the main use for shred in 2024 is: to be there so someone's old 
script doesn't break. There's basically no good use case for it, and it 
probably shouldn't have gotten into coreutils in the first place. The 
multipass pattern stuff is cargo-cult voodoo--a single overwrite with 
zeros will be as effective as anything else--and on modern 
storage/filesystems there's a good chance your overwrite won't overwrite 
anything anyway. Probably the right answer is a kernel facility 
(userspace can't guarantee anything). If you're really sure that 
overwrites work on your system, `shred -n0 -z` will be the fastest way 
to do that. The docs say don't do that because SSDs might optimize that 
away, but SSDs probably aren't overwriting anything anyway (also 
mentioned in the docs). ¯\_(ツ)_/¯

Re: shred bug? [was: Unidentified subject!]

2024-02-14 Thread David Wright

On Tue 13 Feb 2024 at 11:21:08 (-0500), Greg Wooledge wrote:
> On Tue, Feb 13, 2024 at 09:35:11AM -0600, David Wright wrote:
> > On Tue 13 Feb 2024 at 07:15:48 (-0500), Greg Wooledge wrote:
> > > On Mon, Feb 12, 2024 at 11:01:47PM -0600, David Wright wrote:
> > > > … but not much. For me, "standard output" is /dev/fd/1, yet it seems
> > > > unlikely that anyone is going to use >&1 in the manner of the example.
> > > 
> > > Standard output means "whatever file descriptor 1 points to".  That
> > > could be a file, a pipe, a terminal (character device), etc.
> > 
> > Why pick on 1?
> 
> It's the definition.  Standard input is FD 0, standard output is FD 1,
> and standard error is FD 2.
> 
> > . It demonstrates the shell syntax element required (&) in order to
> >   avoid truncating the file, rather than shred overwriting it.
> 
> You are confused.  You're making assumptions about shell syntax that
> are simply not true.

You're right. I was looking too hard at the right side of the > and
neglecting the implied left side. It's always worth running these
things past your eyes. Thanks for the clear exposition that followed.

Cheers,
David.

Re: shred bug? [was: Unidentified subject!]

2024-02-13 Thread tomas

On Tue, Feb 13, 2024 at 01:03:44PM -0800, David Christensen wrote:
> On 2/13/24 09:40, debian-u...@howorth.org.uk wrote:
> > Greg Wooledge  wrote:
> > 
> > > Shred will determine the size of the file, then write data to the
> > > file, rewind, write data again, etc.  On a traditional hard drive,
> > > that will overwrite the original private information.  On modern
> > > devices, it may not.
> > 
> > Thanks for the excellent explanation :)
> > 
> > One nitpick. You say "On a traditional hard drive, that will overwrite
> > the original private information" but that's not quite true. It also
> > needs to be a "traditional" file system! That is, not journalled or COW.
> > 
> > So nowadays I would expect shred not to work unless you got very
> > lucky, or planned carefully.
> 
> 
> Perhaps zerofree(8)?

On a SATA, it won't get at (some) of the spare blocks, since it
doesn't know that they are there.

If your data is so sensitive that you don't want it to escape,
your best bet seems to plan ahead and not let it hit your media
unencrypted.

Use LUKS. And oh, use argon2id as key derivation function [1]
these days.

Cheers
[1] https://mjg59.dreamwidth.org/66429.html
-- 
t


signature.asc
Description: PGP signature

Re: shred bug? [was: Unidentified subject!]

2024-02-13 Thread gene heskett


On 2/13/24 16:00, David Christensen wrote:

On 2/13/24 11:31, gene heskett wrote:
Next experiment is a pair of 4T Silicon Power SSD's When they & the 
startech usb3 adapters arrive.  I'll get that NAS built for amanda yet.



2.5" SATA SSD's and SATA to USB adapter cables for $187.97 + $10.99 = 
$198.96 each set?


https://www.amazon.com/dp/B0BVLRFFWQ

https://www.amazon.com/dp/B00HJZJI84


Why not external USB drives for $192.99?

https://www.amazon.com/dp/B0C6XVZS4K


For $7 more, you can get the "Pro edition" in black with two cables:

https://www.amazon.com/dp/B0C69QD5NK


You could plug those into the two USB-C 3.1 Gen 2 ports on your Asus 
PRIME Z370-A II motherboard.


Maybe, but these sata types have the the mounting bolts the usb versions 
don't. And fits the drive adapters I already have that put both in one 
drive tray. Not to mention if Taiwan has a similar product, I tend to 
buy it.  So are the 4 2T gigastones I'll fill the next 2 drawers with so 
I should wind up with a 16T backup server's LVM. with a 1/2T Samsung 870 
as a holding disk.  Running a bpi-m5 headless, maybe < 20 watts.  Whats 
not to like?


David


.


Cheers, Gene Heskett, CET.
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis

Re: shred bug? [was: Unidentified subject!]

2024-02-13 Thread David Christensen


On 2/13/24 09:40, debian-u...@howorth.org.uk wrote:

Greg Wooledge  wrote:


Shred will determine the size of the file, then write data to the
file, rewind, write data again, etc.  On a traditional hard drive,
that will overwrite the original private information.  On modern
devices, it may not.


Thanks for the excellent explanation :)

One nitpick. You say "On a traditional hard drive, that will overwrite
the original private information" but that's not quite true. It also
needs to be a "traditional" file system! That is, not journalled or COW.

So nowadays I would expect shred not to work unless you got very
lucky, or planned carefully.



Perhaps zerofree(8)?


David

Re: shred bug? [was: Unidentified subject!]

2024-02-13 Thread David Christensen


On 2/13/24 11:31, gene heskett wrote:
Next experiment is a pair of 4T 
Silicon Power SSD's When they & the startech usb3 adapters arrive.  I'll 
get that NAS built for amanda yet.



2.5" SATA SSD's and SATA to USB adapter cables for $187.97 + $10.99 = 
$198.96 each set?


https://www.amazon.com/dp/B0BVLRFFWQ

https://www.amazon.com/dp/B00HJZJI84


Why not external USB drives for $192.99?

https://www.amazon.com/dp/B0C6XVZS4K


For $7 more, you can get the "Pro edition" in black with two cables:

https://www.amazon.com/dp/B0C69QD5NK


You could plug those into the two USB-C 3.1 Gen 2 ports on your Asus 
PRIME Z370-A II motherboard.



David

Re: shred bug? [was: Unidentified subject!]

2024-02-13 Thread gene heskett


On 2/13/24 14:44, Thomas Schmitt wrote:

Hi,

Gene Heskett wrote:

Next experiment is a pair of 4T Silicon Power SSD's


When f3 has (hopefully) given its OK, the topic of a full write-and-read
test will come up again. I'm looking forward to all the spin-off topics.

I'll have to admit it has been quite educational. Now, can I remember it 
next week? YTBD.>

Have a nice day :)


You too Thomas.
Take care and stay well.


Thomas


Cheers, Gene Heskett, CET.
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis

Re: shred bug? [was: Unidentified subject!]

2024-02-13 Thread Thomas Schmitt

Hi,

Gene Heskett wrote:
> Next experiment is a pair of 4T Silicon Power SSD's

When f3 has (hopefully) given its OK, the topic of a full write-and-read
test will come up again. I'm looking forward to all the spin-off topics.


Have a nice day :)

Thomas

Re: shred bug? [was: Unidentified subject!]

2024-02-13 Thread Thomas Schmitt

Hi,

Greg Wooledge wrote:
> Heh.  Don't forget your own attempts to use a shredder as a PRNG stream.

My original idea was to watch a minimal shred run by teeing its work into
a checksummer.

But then topic drift came in. So we got a farm show of random generators
and a discussion about what exactly is a bug in shred's documentation.
Plus the shell programming webinar. And a diagnosis about a rightously
failed attempt to change the partition table type from MBR/DOS to GPT.

And this all because Gene Heskett was adventurous enough to buy a cheap
fake USB disk.


Have a nice day :)

Thomas

Re: shred bug? [was: Unidentified subject!]

2024-02-13 Thread gene heskett


On 2/13/24 12:56, Thomas Schmitt wrote:

Hi,

Greg Wooledge wrote:

Let me write out the example again, but with the bug fixed, and then
explain what each line does, [... lecture about advanced shell
programming ...]


And this all because Gene Heskett was adventurous enough to buy a cheap
fake USB disk. :))


Guilty as charged, Thomas. My advantage is that it won't affect the 
length of the ladder up my side of the hog.  If I save someone else from 
getting bit by that fraud I'm pleased.  Next experiment is a pair of 4T 
Silicon Power SSD's When they & the startech usb3 adapters arrive.  I'll 
get that NAS built for amanda yet.



Have a nice day :)


You too.


Thomas


Cheers, Gene Heskett, CET.
--
"There are four boxes to be used in defense of liberty:
 soap, ballot, jury, and ammo. Please use in that order."
-Ed Howdershelt (Author, 1940)
If we desire respect for the law, we must first make the law respectable.
 - Louis D. Brandeis

Re: shred bug? [was: Unidentified subject!]

2024-02-13 Thread Greg Wooledge

On Tue, Feb 13, 2024 at 06:54:58PM +0100, Thomas Schmitt wrote:
> Greg Wooledge wrote:
> > Let me write out the example again, but with the bug fixed, and then
> > explain what each line does, [... lecture about advanced shell
> > programming ...]
> 
> And this all because Gene Heskett was adventurous enough to buy a cheap
> fake USB disk. :))

Heh.  Don't forget your own attempts to use a shredder as a PRNG stream.

Shell redirections can be complicated, so this topic is going to
come up once in a while.  The example in the shred info page is fairly
unintuitive, so it deserves a bit of explanation.  I imagine most readers
who saw it simply accepted it as written, which is why the bug went
undiscovered for two decades.

Re: shred bug? [was: Unidentified subject!]

2024-02-13 Thread Thomas Schmitt

Hi,

Greg Wooledge wrote:
> Let me write out the example again, but with the bug fixed, and then
> explain what each line does, [... lecture about advanced shell
> programming ...]

And this all because Gene Heskett was adventurous enough to buy a cheap
fake USB disk. :))


Have a nice day :)

Thomas

Re: shred bug? [was: Unidentified subject!]

2024-02-13 Thread debian-user

Greg Wooledge  wrote:

> Shred will determine the size of the file, then write data to the
> file, rewind, write data again, etc.  On a traditional hard drive,
> that will overwrite the original private information.  On modern
> devices, it may not.

Thanks for the excellent explanation :)

One nitpick. You say "On a traditional hard drive, that will overwrite
the original private information" but that's not quite true. It also
needs to be a "traditional" file system! That is, not journalled or COW.

So nowadays I would expect shred not to work unless you got very
lucky, or planned carefully.

Re: Fast Random Data Generation (Was: Re: Unidentified subject!)

2024-02-13 Thread Linux-Fan


David Christensen writes:


On 2/12/24 08:30, Linux-Fan wrote:

David Christensen writes:


On 2/11/24 02:26, Linux-Fan wrote:

I wrote a program to automatically generate random bytes in multiple threads:
https://masysma.net/32/big4.xhtml


What algorithm did you implement?


I copied the algorithm from here:
https://www.javamex.com/tutorials/random_numbers/numerical_recipes.shtml


That Java code uses locks, which implies it uses global state and cannot be  
run multi-threaded (?).  (E.g. one process with one JVM.)


Indeed, the example code uses locks which is bad from a performance point of  
view. That is why _my_ implementation works without this fine-grained  
locking and instead ensures that each thread uses its own instance as to  
avoid the lock. IOW: I copied the algorithm but of course adjusted the code  
to my use case.


My version basically runs as follows:

* Create one queue of ByteBuffers
* Create multiple threads
   * Each thread runs their own RNG instance
   * Upon finishing the creation of a buffer, it enqueues
 the resulting ByteBuffer into the queue (this is the only
 part where multiple threads access concurrently)
* The main thread dequeues from the queue and writes the
  buffers to the output file

Is it possible to obtain parallel operation on an SMP machine with multiple  
virtual processors?  (Other than multiple OS processes with one PRNG on one  
JVM each?)


Even the locked random could be instantiated multiple times (each instance  
gets their own lock) and this could still be faster than running just  
one of it. However, since the computation itself is fast, I suppose the  
performance hit from managing the locks could be significant. Multiple OS  
processes would also work, but is pretty uncommon in Java land AFAIR.


I found it during the development of another application where I needed a  
lot of random data for simulation purposes :)


My implementation code is here:
https://github.com/m7a/bo-big/blob/master/latest/Big4.java


See the end of that file to compare with the “Numerical Recipes” RNG linked  
further above to observe the difference wrt. locking :)



If I were to do it again today, I'd probably switch to any of these PRNGS:

* https://burtleburtle.net/bob/rand/smallprng.html
* https://www.pcg-random.org/


Hard core.  I'll let the experts figure it out; and then I will use their  
libraries and programs.


IIRC one of the findings of PCG was that the default RNGs of many  
programming languages and environments are surprisingly bad. I only arrived at  
using a non-default implementation after facing some issues with the Java  
integrated ThreadLocalRandom ”back then” :)


It may indeed be worth pointing out (as Jeffrey Walton already mentioned in  
another subthread) that these RNGs discussed here are _not_ cryptographic  
RNGs. I think for disk testing purposes it is OK to use fast non- 
cryptographic RNGs, but other applications may have higher demands on their  
RNGs.


HTH
Linux-Fan

öö

[...]


pgpLYMPAsjGtj.pgp
Description: PGP signature

Re: shred bug? [was: Unidentified subject!]

2024-02-13 Thread Greg Wooledge

On Tue, Feb 13, 2024 at 09:35:11AM -0600, David Wright wrote:
> On Tue 13 Feb 2024 at 07:15:48 (-0500), Greg Wooledge wrote:
> > On Mon, Feb 12, 2024 at 11:01:47PM -0600, David Wright wrote:
> > > … but not much. For me, "standard output" is /dev/fd/1, yet it seems
> > > unlikely that anyone is going to use >&1 in the manner of the example.
> > 
> > Standard output means "whatever file descriptor 1 points to".  That
> > could be a file, a pipe, a terminal (character device), etc.
> 
> Why pick on 1?

It's the definition.  Standard input is FD 0, standard output is FD 1,
and standard error is FD 2.

> . It demonstrates the shell syntax element required (&) in order to
>   avoid truncating the file, rather than shred overwriting it.

You are confused.  You're making assumptions about shell syntax that
are simply not true.

> > > >A FILE of ‘-’ denotes standard output.  The intended use of this is
> > > >to shred a removed temporary file.  For example:
> > > > 
> > > >   i=$(mktemp)
> > > >   exec 3<>"$i"
> > > >   rm -- "$i"
> > > >   echo "Hello, world" >&3
> > > >   shred - >&3
> > > >   exec 3>-

> Ironic that it truncates a file, and then immediately warns against
> truncating a file instead of shredding it.

No.  This is not what it does (if we fix the bug).

Let me write out the example again, but with the bug fixed, and then
explain what each line does, because apparently there is a LOT of
misunderstanding here.

  i=$(mktemp)
  exec 3<>"$i"
  rm -- "$i"
  echo "Hello, world" >&3
  shred - >&3
  exec 3>&-

The first line runs mktemp(1), which is a GNU coreutils program that
creates a temporary file, and then writes its name to standard output.
The shell syntax grabs that name and stores it in the variable "i".

So, after line 1, we have an (empty) temporary file, which was created
by a child process that has terminated.  We have its name in a variable.

Creation of temporary files works a little bit differently in shell
scripts than it does in regular programs.  In most other languages,
you would call a library function that creates the temporary file
(keeping it open), optionally unlinks it, and returns the open file
descriptor to you for use.  But you can't do that in a shell script
that needs an external program to do the file creation.  So we have this
slightly clumsy approach.

The second line opens this file for reading and writing, and ensures
that file descriptor 3 points to it.  It's important to understand that
while "exec 3>$i" would have truncated the file's contents, "exec 3<>$i"
does not.  Of course, there wasn't any content to truncate, since it was
created empty, but that's not the important part.  The important part
is that this FD is opened for read+write, allowing the temporary file
to be used for storage *and* retrieval.  We aren't doing any retrieval
in this example, but it could be done, with specialized tools.

The third line unlinks the file from the file system.  However, the shell
still has an open file descriptor which points to the file.  Therefore,
the file is still accessible through this FD.  Its inode is not recycled,
and any blocks containing file content are not marked for reuse.

This "unlink before using" technique is traditional on Unix systems.
It allows you to bypass setting up an exit handler to clean up the
temporary file.  Once the open file descriptor is closed, the file
system will mark the inode and any blocks as ready for reuse.  Even if
the script is killed by SIGKILL, that cleanup will still happen.

The fourth line writes some content via the open file descriptor 3.  At
this point, our unlinked file now has data in it.  Presumably this data
is super private, and we don't want anyone to be able to recover it.
When the script exits, the open file descriptor will close, and the file
system will mark the file's blocks as reusable, but it won't *actually*
reuse them until something else comes along and claims them.  But that's
what shred is designed for.

The fifth line calls shred(1), instructing it to destroy the content
that's in the unlinked file.  Since the file is unlinked, it has no name,
and therefore shred can't be *given* a name.  However, we have a file
descriptor that points to it.  So, what we *can* do is point standard
output to the file (that's what >&3 does), and then tell shred to destroy
the file that's pointed to by stdout.

Shred will determine the size of the file, then write data to the file,
rewind, write data again, etc.  On a traditional hard drive, that will
overwrite the original private information.  On modern devices, it may
not.

Finally, the sixth line closes file descriptor 3.  Doing this frees the
file's inode and blocks, allowing them to be reused by a future program.

The key to understanding this example is a firm grasp of the shell's
redirection syntax and semantics.

Let's start with what people normally see:  3>filename

That opens "filename" for writing with truncation, and ensures that

Re: shred bug? [was: Unidentified subject!]

2024-02-13 Thread David Wright

On Tue 13 Feb 2024 at 07:15:48 (-0500), Greg Wooledge wrote:
> On Mon, Feb 12, 2024 at 11:01:47PM -0600, David Wright wrote:
> > … but not much. For me, "standard output" is /dev/fd/1, yet it seems
> > unlikely that anyone is going to use >&1 in the manner of the example.
> 
> Standard output means "whatever file descriptor 1 points to".  That
> could be a file, a pipe, a terminal (character device), etc.

Why pick on 1?

> > I might write something like: "The option ‘-’ shreds the file specified
> > by the redirection ‘>’", though there could be a better name for ‘>’.
> 
> You're assuming the program will be used from a shell.  This is *usually*
> going to be true, but nothing prevents you from writing a C program
> which closes stdout, opens a file, ensures that it's using FD 1,
> and then calls "shred -".  The documentation has to support this use
> case as well.

/As well/ — which is why I wrote N in place of 1. The original bug
report (which I hadn't seen until Thomas' post) says:
 "If you redirect output to a file it will work. Shredding a tty doesn't
  make much sense, after all."
  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=155175#10

Now, you can't write "If you redirect output to a file it will work"
in a man page—it needs recasting into something more like what
I wrote above, which contains two key points:

. It points out that '-' is an option, not a filename or a stand-in
  for one, and it doesn't use the word standard, which is totally
  irrelevant in the circumstances.

. It demonstrates the shell syntax element required (&) in order to
  avoid truncating the file, rather than shred overwriting it.

I think that getting the "&" into the man page would be helpful
to anybody who doesn't look at the info page for the example.
It might have shortened the early part of this thread as well.

As for C programmers, neither FD number nor truncation is relevant.
Sure, you can pick 1. But you don't have to document that for shred.
And truncation is an accident that can occur because of shell's
redirect syntax: there's no equivalent in programs.

> > >A FILE of ‘-’ denotes standard output.  The intended use of this is
> > >to shred a removed temporary file.  For example:
> > > 
> > >   i=$(mktemp)
> > >   exec 3<>"$i"
> > >   rm -- "$i"
> > >   echo "Hello, world" >&3
> > >   shred - >&3
> > >   exec 3>-
> > 
> > I can see that the last line truncates the "anonymous" file,
> 
> No, that's not what it does at all.  In fact, that last line is
> written incorrectly.  It should say "exec 3>&-" and what that does
> is close file descriptor 3, which was previously opened on line 2.
> 
> What it actually does *as written* is create/truncate a file whose
> name is "-", close the previously opened FD 3, and make FD 3 point
> to the file named "-".
> 
> unicorn:~$ exec 3>-
> unicorn:~$ ls -ld -- -
> -rw-r--r-- 1 greg greg 0 Feb 13 07:12 -
> unicorn:~$ ls -l /dev/fd/3
> l-wx-- 1 greg greg 64 Feb 13 07:12 /dev/fd/3 -> /home/greg/-
> 
> This is an obvious bug in the info page.  I wonder how many years
> this has gone unnoticed.

Well spotted. That's what an experienced eye brings to a line like
that, whereas I assumed it meant something beyond my experience,
and searched for it.

Ironic that it truncates a file, and then immediately warns against
truncating a file instead of shredding it.

Cheers,
David.

Re: shred bug? [was: Unidentified subject!]

2024-02-13 Thread Thomas Schmitt

Hi,

"info shred" says:
> > >   i=$(mktemp)
> > >   exec 3<>"$i"
> > >   rm -- "$i"
> > >   echo "Hello, world" >&3
> > >   shred - >&3
> > >   exec 3>-

Greg Wooledge wrote:
> In fact, that last line is
> written incorrectly.  It should say "exec 3>&-" and what that does
> is close file descriptor 3, which was previously opened on line 2.
> [...]
> This is an obvious bug in the info page.  I wonder how many years
> this has gone unnoticed.

  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=155175#36
of 22 Dec 2005 states:

  "I'll assume that this is now adequately explained in the info page
   (below).  If not then please reopen.  // Thomas Hood
   [...]
   exec 3>-"

The bug report is from 02 Aug 2002 and states that the info page contains
the short and broad promise which we can still see in "man shred".

So we can assume a bug age of 18 to 22 years.

Have a nice day :)

Thomas

Re: shred bug? [was: Unidentified subject!]

2024-02-13 Thread tomas

On Tue, Feb 13, 2024 at 07:36:14AM -0500, Greg Wooledge wrote:
> On Tue, Feb 13, 2024 at 07:15:48AM -0500, Greg Wooledge wrote:
> > This is an obvious bug in the info page.  I wonder how many years
> > this has gone unnoticed.
> 
> I've filed Bug#1063837 for it.  

Well, thanks for doing TRT :-)

Cheers
-- 
t


signature.asc
Description: PGP signature

Re: shred bug? [was: Unidentified subject!]

2024-02-13 Thread Greg Wooledge

On Tue, Feb 13, 2024 at 07:15:48AM -0500, Greg Wooledge wrote:
> This is an obvious bug in the info page.  I wonder how many years
> this has gone unnoticed.

I've filed Bug#1063837 for it.

Re: shred bug? [was: Unidentified subject!]

2024-02-13 Thread Greg Wooledge

On Mon, Feb 12, 2024 at 11:01:47PM -0600, David Wright wrote:
> … but not much. For me, "standard output" is /dev/fd/1, yet it seems
> unlikely that anyone is going to use >&1 in the manner of the example.

Standard output means "whatever file descriptor 1 points to".  That
could be a file, a pipe, a terminal (character device), etc.

> I might write something like: "The option ‘-’ shreds the file specified
> by the redirection ‘>’", though there could be a better name for ‘>’.

You're assuming the program will be used from a shell.  This is *usually*
going to be true, but nothing prevents you from writing a C program
which closes stdout, opens a file, ensures that it's using FD 1,
and then calls "shred -".  The documentation has to support this use
case as well.

> >A FILE of ‘-’ denotes standard output.  The intended use of this is
> >to shred a removed temporary file.  For example:
> > 
> >   i=$(mktemp)
> >   exec 3<>"$i"
> >   rm -- "$i"
> >   echo "Hello, world" >&3
> >   shred - >&3
> >   exec 3>-
> 
> I can see that the last line truncates the "anonymous" file,

No, that's not what it does at all.  In fact, that last line is
written incorrectly.  It should say "exec 3>&-" and what that does
is close file descriptor 3, which was previously opened on line 2.

What it actually does *as written* is create/truncate a file whose
name is "-", close the previously opened FD 3, and make FD 3 point
to the file named "-".

unicorn:~$ exec 3>-
unicorn:~$ ls -ld -- -
-rw-r--r-- 1 greg greg 0 Feb 13 07:12 -
unicorn:~$ ls -l /dev/fd/3
l-wx-- 1 greg greg 64 Feb 13 07:12 /dev/fd/3 -> /home/greg/-

This is an obvious bug in the info page.  I wonder how many years
this has gone unnoticed.

Re: shred bug? [was: Unidentified subject!]

2024-02-12 Thread tomas

On Mon, Feb 12, 2024 at 10:07:45PM +0100, Thomas Schmitt wrote:
> Hi,
> 
> > https://en.wikipedia.org/wiki/Everything_is_a_file
> > But, there is more than one kind of file.
> 
> "All files are equal.
>  But some files are more equal than others."
> 
> (George Orwell in his dystopic novel "Server Farm".)

:-D

Yesterday I was on the brink of quoting that. Great minds and
all of that...

Reality though is, that if you don't design your file with
magical properties from the get-go, you will keep stumbling
on stuff you want to model and which don't fit the real
life file design you chose.

And yes, Plan 9, as someone else noted in this thread, takes
that to a bigger extreme: in their windowing system, windows
are files, too -- you can remove a window by deleting the
corresponding file.

Cheers
-- 
t

signature.asc
Description: PGP signature

Re: shred bug? [was: Unidentified subject!]

2024-02-12 Thread David Wright

On Sun 11 Feb 2024 at 09:16:00 (-0600), David Wright wrote:
> On Sun 11 Feb 2024 at 09:54:24 (-0500), Greg Wooledge wrote:
> > On Sun, Feb 11, 2024 at 03:45:21PM +0100, to...@tuxteam.de wrote:
> > > Still there's the discrepancy between doc and behaviour.
> > 
> > There isn't.  The documentation says:
> > 
> > SYNOPSIS
> >shred [OPTION]... FILE...
> > 
> > DESCRIPTION
> >Overwrite  the specified FILE(s) repeatedly, in order to make it 
> > harder
> >for even very expensive hardware probing to recover the data.
> > 
> >If FILE is -, shred standard output.
> > 
> > In every sentence, the word FILE appears.  There's nothing in there
> > which says "you can operate on a non-file".
> > 
> > Once you grasp what the command is *intended* to do (rewind and overwrite
> > a file repeatedly), it makes absolutely perfect sense that it should
> > only operate on rewindable file system objects.
> > 
> > If you want it to write a stream of data instead of performing its normal
> > operation (rewinding and rewriting), that's a new feature.
> > 
> > If you'd prefer the documentation to say explicitly "only regular files
> > and block devices are allowed", that would be an upstream documentation
> > *clarification* request.
> 
> Perhaps info puts it better?

… but not much. For me, "standard output" is /dev/fd/1, yet it seems
unlikely that anyone is going to use >&1 in the manner of the example.

I might write something like: "The option ‘-’ shreds the file specified
by the redirection ‘>’", though there could be a better name for ‘>’.

>A FILE of ‘-’ denotes standard output.  The intended use of this is
>to shred a removed temporary file.  For example:
> 
>   i=$(mktemp)
>   exec 3<>"$i"
>   rm -- "$i"
>   echo "Hello, world" >&3
>   shred - >&3
>   exec 3>-

I can see that the last line truncates the "anonymous" file, but where
is that construction documented¹, and how would one parse the syntax
elements   FD  >  -   to make them mean truncate?

>However, the command ‘shred - >file’ does not shred the contents of
>FILE, since the shell truncates FILE before invoking ‘shred’.  Use
>the command ‘shred file’ or (if using a Bourne-compatible shell) the
>command ‘shred - 1<>file’ instead.

¹ the string ">-" doesn't appear in /usr/share/doc/bash/bashref.pdf,
  ver 5.1, for example.

Cheers,
David.

Re: shred bug? [was: Unidentified subject!]

2024-02-12 Thread Max Nikulin


On 12/02/2024 05:41, David Christensen wrote:


Apparently, shred(1) has both an info(1) page (?) and a man(1) page. The 
obvious solution is to write one document that is complete and correct, 
and use it everywhere -- e.g. DRY.


https://www.gnu.org/prep/standards/html_node/Man-Pages.html
6.9 Man Pages in "GNU Coding Standards"

In the GNU project, man pages are secondary. It is not necessary or
expected for every GNU program to have a man page, but some of them do.


A standalone man page is not the same as a section in a document 
describing the whole bundle.


Notice that man uses direct formatting, not logical markup. E.g. there 
is no dedicated structure for links to other man pages. They were not 
designed as hypertext documents, they are to be printed on paper. In 
this sense texinfo is a more advanced format.


P.S. I admit that in some cases "man bash" may be more convenient for 
searching than an info browser.

Re: Fast Random Data Generation (Was: Re: Unidentified subject!)

2024-02-12 Thread David Christensen


On 2/12/24 08:30, Linux-Fan wrote:

David Christensen writes:


On 2/11/24 02:26, Linux-Fan wrote:
I wrote a program to automatically generate random bytes in multiple 
threads:

https://masysma.net/32/big4.xhtml


What algorithm did you implement?


I copied the algorithm from here:
https://www.javamex.com/tutorials/random_numbers/numerical_recipes.shtml



That Java code uses locks, which implies it uses global state and cannot 
be run multi-threaded (?).  (E.g. one process with one JVM.)



Is it possible to obtain parallel operation on an SMP machine with 
multiple virtual processors?  (Other than multiple OS processes with one 
PRNG on one JVM each?)



I found it during the development of another application where I needed 
a lot of random data for simulation purposes :)


My implementation code is here:
https://github.com/m7a/bo-big/blob/master/latest/Big4.java

If I were to do it again today, I'd probably switch to any of these PRNGS:

* https://burtleburtle.net/bob/rand/smallprng.html
* https://www.pcg-random.org/



Hard core.  I'll let the experts figure it out; and then I will use 
their libraries and programs.



David

Re: Fast Random Data Generation (Was: Re: Unidentified subject!)

2024-02-12 Thread Jeffrey Walton

On Mon, Feb 12, 2024 at 3:02 PM Linux-Fan  wrote:
>
> David Christensen writes:
>
> > On 2/11/24 02:26, Linux-Fan wrote:
> >> I wrote a program to automatically generate random bytes in multiple 
> >> threads:
> >> https://masysma.net/32/big4.xhtml
> >>
> >> Before knowing about `fio` this way my way to benchmark SSDs :)
> >>
> >> Example:
> >>
> >> | $ big4 -b /dev/null 100 GiB
> >> | Ma_Sys.ma Big 4.0.2, Copyright (c) 2014, 2019, 2020 Ma_Sys.ma.
> >> | For further info send an e-mail to ma_sys...@web.de.
>
> [...]
>
> >> | 99.97% +8426 MiB 7813 MiB/s 102368/102400 MiB
> >> | Wrote 102400 MiB in 13 s @ 7812.023 MiB/s
> >
> >
> > What algorithm did you implement?
>
> I copied the algorithm from here:
> https://www.javamex.com/tutorials/random_numbers/numerical_recipes.shtml
>
> I found it during the development of another application where I needed a
> lot of random data for simulation purposes :)

A PRNG for a simulation has different requirements than a PRNG for
cryptographic purposes. A simulation usually needs numbers fast from a
uniform distribution. Simulations can use predictable numbers. Often a
Linear Congurential Generator (LCG) will do just fine even though they
were broken about 35 years ago. See Also see Joan Boyer's Inferring
Sequences Produced by Pseudo-Random Number Generators,
.

A cryptographic application will have more stringent requirements. A
cryptographic generator may (will?) take longer to generate a number,
the numbers need to come from a uniform distribution, and the numbers
need to be prediction resistant. You can read about the cryptographic
qualities of random numbers in NIST SP800-90 and friends.

> My implementation code is here:
> https://github.com/m7a/bo-big/blob/master/latest/Big4.java
>
> If I were to do it again today, I'd probably switch to any of these PRNGS:
>
>  * https://burtleburtle.net/bob/rand/smallprng.html
>  * https://www.pcg-random.org/
>
> >> Secure Random can be obtained from OpenSSL:
> >>
> >> | $ time for i in `seq 1 100`; do openssl rand -out /dev/null $((1024 * 
> >> 1024
> >> * 1024)); done
> >> |
> >> | real0m49.288s
> >> | user0m44.710s
> >> | sys0m4.579s
> >>
> >> Effectively 2078 MiB/s (quite OK for single-threaded operation). It is not
> >> designed to generate large amounts of random data as the size is limited by
> >> integer range...

Jeff

Re: shred bug? [was: Unidentified subject!]

2024-02-12 Thread Thomas Schmitt

Hi,

> https://en.wikipedia.org/wiki/Everything_is_a_file
> But, there is more than one kind of file.

"All files are equal.
 But some files are more equal than others."

(George Orwell in his dystopic novel "Server Farm".)


Have a nice day :)

Thomas

Re: shred bug? [was: Unidentified subject!]

2024-02-12 Thread David Christensen


On 2/12/24 08:50, Curt wrote:

On 2024-02-11,   wrote:



On Sun, Feb 11, 2024 at 09:54:24AM -0500, Greg Wooledge wrote:

[...]


If FILE is -, shred standard output.
=20
In every sentence, the word FILE appears.  There's nothing in there
which says "you can operate on a non-file".


Point taken, yes.


I thought everything was a file.



"Everything is a file" is a design feature of the Unix operating system:

https://en.wikipedia.org/wiki/Everything_is_a_file


But, there is more than one kind of file.


And, not every program supports every kind of file.


The manual page for find(1) provides a shopping list of file types it 
supports:


2024-02-12 12:32:13 dpchrist@laalaa ~
$ man find | egrep -A 20 '^   .type c'
   -type c
  File is of type c:

  b  block (buffered) special

  c  character (unbuffered) special

  d  directory

  p  named pipe (FIFO)

  f  regular file

  l  symbolic link; this is never true if  the
 -L option or the -follow option is in ef-
 fect, unless the symbolic link is broken.
 If  you want to search for symbolic links
 when -L is in effect, use -xtype.

  s  socket


As for shred(1), the argument FILE is conventionally a regular file.  We 
are discussing the special case described in the manual page:


   If FILE is -, shred standard output.


David

Re: shred bug? [was: Unidentified subject!]

2024-02-12 Thread Greg Wooledge

On Mon, Feb 12, 2024 at 04:50:50PM -, Curt wrote:
> On 2024-02-11,   wrote:
> >
> >
> > On Sun, Feb 11, 2024 at 09:54:24AM -0500, Greg Wooledge wrote:
> >
> > [...]
> >
> >>If FILE is -, shred standard output.
> >>=20
> >> In every sentence, the word FILE appears.  There's nothing in there
> >> which says "you can operate on a non-file".
> >
> > Point taken, yes.
> 
> I thought everything was a file.

An anonymous pipe(2) in memory between two processes isn't even close to
a file.  Also, you're confusing Linux and Plan 9.  Linux has a bunch of
things that aren't files, such as network interfaces.

Re: shred bug? [was: Unidentified subject!]

2024-02-12 Thread Curt

On 2024-02-11,   wrote:
>
>
> On Sun, Feb 11, 2024 at 09:54:24AM -0500, Greg Wooledge wrote:
>
> [...]
>
>>If FILE is -, shred standard output.
>>=20
>> In every sentence, the word FILE appears.  There's nothing in there
>> which says "you can operate on a non-file".
>
> Point taken, yes.

I thought everything was a file.

Re: Fast Random Data Generation (Was: Re: Unidentified subject!)

2024-02-12 Thread Linux-Fan


David Christensen writes:


On 2/11/24 02:26, Linux-Fan wrote:

I wrote a program to automatically generate random bytes in multiple threads:
https://masysma.net/32/big4.xhtml

Before knowing about `fio` this way my way to benchmark SSDs :)

Example:

| $ big4 -b /dev/null 100 GiB
| Ma_Sys.ma Big 4.0.2, Copyright (c) 2014, 2019, 2020 Ma_Sys.ma.
| For further info send an e-mail to ma_sys...@web.de.


[...]


| 99.97% +8426 MiB 7813 MiB/s 102368/102400 MiB
| Wrote 102400 MiB in 13 s @ 7812.023 MiB/s



What algorithm did you implement?


I copied the algorithm from here:
https://www.javamex.com/tutorials/random_numbers/numerical_recipes.shtml

I found it during the development of another application where I needed a  
lot of random data for simulation purposes :)


My implementation code is here:
https://github.com/m7a/bo-big/blob/master/latest/Big4.java

If I were to do it again today, I'd probably switch to any of these PRNGS:

* https://burtleburtle.net/bob/rand/smallprng.html
* https://www.pcg-random.org/


Secure Random can be obtained from OpenSSL:

| $ time for i in `seq 1 100`; do openssl rand -out /dev/null $((1024 * 1024  
* 1024)); done

|
| real    0m49.288s
| user    0m44.710s
| sys    0m4.579s

Effectively 2078 MiB/s (quite OK for single-threaded operation). It is not  
designed to generate large amounts of random data as the size is limited by  
integer range...



Thank you for posting the openssl(1) incantation.


You're welcome.

[...]

HTH
Linux-Fan

öö


pgpjvuqb6Fy1L.pgp
Description: PGP signature

Re: Fast Random Data Generation (Was: Re: Unidentified subject!)

2024-02-11 Thread David Christensen


On 2/11/24 02:26, Linux-Fan wrote:
I wrote a program to automatically generate random bytes in multiple 
threads:

https://masysma.net/32/big4.xhtml

Before knowing about `fio` this way my way to benchmark SSDs :)

Example:

| $ big4 -b /dev/null 100 GiB
| Ma_Sys.ma Big 4.0.2, Copyright (c) 2014, 2019, 2020 Ma_Sys.ma.
| For further info send an e-mail to ma_sys...@web.de.
|| 0.00% +0 MiB 0 MiB/s 0/102400 MiB
| 3.48% +3562 MiB 3255 MiB/s 3562/102400 MiB
| 11.06% +7764 MiB 5407 MiB/s 11329/102400 MiB
| 19.31% +8436 MiB 6387 MiB/s 19768/102400 MiB
| 27.71% +8605 MiB 6928 MiB/s 28378/102400 MiB
| 35.16% +7616 MiB 7062 MiB/s 35999/102400 MiB
| 42.58% +7595 MiB 7150 MiB/s 43598/102400 MiB
| 50.12% +7720 MiB 7230 MiB/s 51321/102400 MiB
| 58.57% +8648 MiB 7405 MiB/s 59975/102400 MiB
| 66.96% +8588 MiB 7535 MiB/s 68569/102400 MiB
| 75.11% +8343 MiB 7615 MiB/s 76916/102400 MiB
| 83.38% +8463 MiB 7691 MiB/s 85383/102400 MiB
| 91.74% +8551 MiB 7762 MiB/s 93937/102400 MiB
| 99.97% +8426 MiB 7813 MiB/s 102368/102400 MiB
|| Wrote 102400 MiB in 13 s @ 7812.023 MiB/s



What algorithm did you implement?



Secure Random can be obtained from OpenSSL:

| $ time for i in `seq 1 100`; do openssl rand -out /dev/null $((1024 * 
1024 * 1024)); done

|
| real    0m49.288s
| user    0m44.710s
| sys    0m4.579s

Effectively 2078 MiB/s (quite OK for single-threaded operation). It is 
not designed to generate large amounts of random data as the size is 
limited by integer range...



Thank you for posting the openssl(1) incantation.


Benchmarking my daily driver laptop without Intel Secure Key:

2024-02-11 21:54:04 dpchrist@laalaa ~
$ lscpu | grep 'Model name'
Model name: Intel(R) Core(TM) i7-2720QM CPU @ 
2.20GHz


2024-02-11 21:54:09 dpchrist@laalaa ~
$ time for i in `seq 1 100`; do openssl rand -out /dev/null $((1024 * 
1024 * 1024)); done


real1m40.149s
user1m25.174s
sys 0m14.952s


So, ~1.072E+9 bytes per second.


Benchmarking a workstation with Intel Secure Key:

2024-02-11 21:54:40 dpchrist@taz ~
$ lscpu | grep 'Model name'
Model name: Intel(R) Xeon(R) E-2174G CPU @ 3.80GHz

2024-02-11 21:54:46 dpchrist@taz ~
$ time for i in `seq 1 100`; do openssl rand -out /dev/null $((1024 * 
1024 * 1024)); done


real1m14.696s
user1m0.338s
sys 0m14.353s


So, ~1.437E+09 bytes per second.


David

Re: shred bug? [was: Unidentified subject!]

2024-02-11 Thread David Christensen


On 2/11/24 06:54, Greg Wooledge wrote:

On Sun, Feb 11, 2024 at 03:45:21PM +0100, to...@tuxteam.de wrote:

On Sun, Feb 11, 2024 at 09:37:31AM -0500, Greg Wooledge wrote:

On Sun, Feb 11, 2024 at 08:02:12AM +0100, to...@tuxteam.de wrote:


[...]


What Thomas was trying to do is to get a cheap, fast random number
generator. Shred seems to have such.


Well... I certainly wouldn't call it a bug.  Maybe a feature request.


Still there's the discrepancy between doc and behaviour.


There isn't.  The documentation says:

SYNOPSIS
shred [OPTION]... FILE...



I interpret the above line to be a prototype for invoking the shred(1) 
program:


* "shred" is the program name

* "[OPTION]..." is one or more option specifiers that may be omitted. 
Each should be described below.


* "FILE..." is one or more argument specifies that should be file system 
paths (strings).




DESCRIPTION
Overwrite  the specified FILE(s) repeatedly, in order to make it harder
for even very expensive hardware probing to recover the data.

If FILE is -, shred standard output.



I interpret the above line at face value -- if the caller provides a 
dash as the argument, shred(1) will operate on standard output.




In every sentence, the word FILE appears.  There's nothing in there
which says "you can operate on a non-file".



Dash is not a file, yet the above sentence says shred(1) can operate on it.



Once you grasp what the command is *intended* to do (rewind and overwrite
a file repeatedly), it makes absolutely perfect sense that it should
only operate on rewindable file system objects.



An expert may infer what you have stated, but I prefer manual pages that 
are explicit.



The GNU project must have found a need for the FILE='-' feature, 
otherwise it would not exist.  The manual page should describe that need 
(e.g. why) and how to properly use shred(1) to solve the need.




If you want it to write a stream of data instead of performing its normal
operation (rewinding and rewriting), that's a new feature.



Humans are (in)famous for doing unexpected things.



If you'd prefer the documentation to say explicitly "only regular files
and block devices are allowed", that would be an upstream documentation
*clarification* request.



Apparently, shred(1) has both an info(1) page (?) and a man(1) page. 
The obvious solution is to write one document that is complete and 
correct, and use it everywhere -- e.g. DRY.



David

1 2 3 4 5 6 7 8 9 10 >

1 - 100 of 5186 matches

Mail list logo