Re: "Repeaters", etc.

[David Christensen]

On 5/30/24 03:14, Roger Price wrote:

On Wed, 29 May 2024, David Christensen wrote:

On 5/29/24 03:36, Roger Price wrote:

On Tue, 28 May 2024, David Christensen wrote:

On 5/28/24 00:28, Roger Price wrote:
I wired my place Cat5. A lot of work, and I regretted it.  I live 
in the hills behind Nice, an area with a lot of lightning.  The 
overhead line to my place took a hit and thanks to the Cat5 
conductivity I lost equipment.


How do you know that the damage your equipment suffered was due to 
the Cat 5e wiring and not due to the electrical power conductors?


Electrical power to my computers comes through 30mA differential 
circuit breakers to Eaton Ellipse 1600 UPS units. I had no such 
protection for the telephone signal, and I saw flashes at the 
telephone junction box.  So I summise that the Cat5 wiring did the 
damage.


Those UPS's should be able to protect telephone and Ethernet, in 
addition to electrical power.  Have you applied the UPS's to the 
former two?


The UPS's stand next to the workstations and well away from the place 
where the telephone line arrives, so I didn't use the UPS's to protect 
the telephone line. My fault.  Later I added a surge protector to the 
copper telephone line.  I am now in the process of migrating from copper 
to fiber so I will need an extra UPS next the fiber terminator.


Roger

===

https://www.eaton.com/sg/en-us/catalog/backup-power-ups-surge-it-power-distribution/eaton-ellipse-pro-ups.html


PS: I once had a lightning strike direct to the house.  Frightening.  
Although every differential circuit breaker in the house tripped, the 
circuit board in the UPS melted.  But even when melting, it protected 
the Dell T7500. No damage to the T7500, no data lost. I took a photo of 
the melt, sent it to Eaton, and they replaced the UPS.



Have you consider applying lightning protection to your house?

1.  Lightning rods, down lines, ground rods, perimeter ground loop, etc..

2. Lightning arresters at the electrical, telephone, CATV, etc. service 
entrance points.



David

Re: "Repeaters", etc.

[David Christensen]

On 5/29/24 03:36, Roger Price wrote:

On Tue, 28 May 2024, David Christensen wrote:

On 5/28/24 00:28, Roger Price wrote:
I wired my place Cat5. A lot of work, and I regretted it.  I live in 
the hills behind Nice, an area with a lot of lightning.  The overhead 
line to my place took a hit and thanks to the Cat5 conductivity I 
lost equipment.


How do you know that the damage your equipment suffered was due to the 
Cat 5e wiring and not due to the electrical power conductors?


Electrical power to my computers comes through 30mA differential circuit 
breakers to Eaton Ellipse 1600 UPS units. I had no such protection for 
the telephone signal, and I saw flashes at the telephone junction box.  
So I summise that the Cat5 wiring did the damage.


Roger



https://www.eaton.com/sg/en-us/catalog/backup-power-ups-surge-it-power-distribution/eaton-ellipse-pro-ups.html

https://standards.globalspec.com/std/104626/iec-61643-1


Those UPS's should be able to protect telephone and Ethernet, in 
addition to electrical power.  Have you applied the UPS's to the former two?



David

Re: "Repeaters", etc.

[David Christensen]

On 5/28/24 17:10, John Hasler wrote:

David writes:

AIUI in the USA for residential 120/240V single-phase three-wire service
drops, electrical utilities either run all three phases along the
distribution line or they run two phases.  Running one phase and a neutral
instead of two phases would reduce the power by the square root of 3


Here in rural Wisconsin the 7200V distribution line leaves the
substation as three phases and a grounded neutral.  This eventually
branches out into three single phase lines consisting of a phase and a
grounded neutral.  The pole pigs are connected phase to neutral.



Interesting.  STFW I found an article and a web site that clarifies the 
above arrangements and more:


https://electrical-engineering-portal.com/primary-distribution-circuits


David

Re: "Repeaters", etc.

[David Christensen]

On 5/28/24 12:47, gene heskett wrote:

On 5/28/24 15:29, Roy J. Tellason, Sr. wrote:

On Tuesday 28 May 2024 01:49:52 pm Paul M Foster wrote:

I've never see a 3 phase in a house. Common in commercial/industrial,
though.
Residential installations (talking in the US here) typically involve 
*one* transformer tapping a single phase out of the three that are up 
there on the pole.  The secondary is center-tapped,  and it's that 
point which is grounded at the service entrance.  Running 3-phase 
power requires *three* transformers up on the pole,  much more in the 
way of expense if you want that for some reason,  and I don't know of 
anybody that does that.  Even those who are into having some 
nontrivial machinery around seem these days to use a VFD to give them 
multiple phases at the machine,  rather than going through the expense 
of having it run in from the pole...


And here you have it from another CET.

Cheers, Gene Heskett, CET.



AIUI in the USA for residential 120/240V single-phase three-wire service 
drops, electrical utilities either run all three phases along the 
distribution line or they run two phases.  Running one phase and a 
neutral instead of two phases would reduce the power by the square root 
of 3.



Running one phase and using the Earth as the return conductor is very 
dangerous and not modern practice.



David

Re: "Repeaters", etc.

[David Christensen]

On 5/28/24 00:28, Roger Price wrote:
I wired my place Cat5. A lot of work, and I regretted it.  I live in the 
hills behind Nice, an area with a lot of lightning.  The overhead line 
to my place took a hit and thanks to the Cat5 conductivity I lost 
equipment.



If your electrical utility uses pole-mounted distribution lines, 
transformers, service drops, etc., and lightning strikes the 
high-voltage conductors, there will be a surge on the customer service 
conductors that places persons and property at risk.  If lightning jumps 
to the customer service conductors, then the risk to persons and 
property can be extreme.



How do you know that the damage your equipment suffered was due to the 
Cat 5e wiring and not due to the electrical power conductors?



David

Re: "Repeaters", etc.

[David Christensen]

On 5/27/24 19:05, Paul M Foster wrote:

I did some more research, and it looks like I must have misstated the
problem.

Let's assume I can't get in the attic and wire the place. Let's 
assume that I've got a wireless router/modem in, say, the garage. 
Let's say I have three rooms with devices I want to connect (one way 
or another) to my router/modem.


It appears there are two solutions. One is wifi extenders, and one
is a mesh network. In both cases, the device sits in the room and 
communicates via wifi to the modem/router. The devices in the room 
connect to the device via ethernet cable.


How does that sound? Any dissenting opinions? Any brand 
recommendations?



On 5/27/24 20:14, Paul M Foster wrote:
Coincidentally, I used to be an electrician too, but we almost never 
ran low voltage except for doorbells.


The house in question appears to have a generous attic, but they've 
blown in two feet of insulation I'd rather not disturb. And that

much insulation makes the headers of walls very hard to find. Also,
I'm not in my 20s anymore, and crawling around in attics is
difficult.

In the house I'm living in now, I did go into the attic years ago 
with cat 5e and wired up the living room.


FWIW, in the house we're buying, I need internet (wired) in the 
living room, bedroom 2 and bedroom 4. Also, it's concrete block 
construction (outer walls).



On 5/27/24 21:50, Paul M Foster wrote:

Well, if I'm gonna run cat 5, I might as well just put a jack in each
room.> No POE needed. The only reason for wifi at all in this case is
so I don't *have*  to run cat 5.



From what I've read, TP-Link gets good reviews.


FYI my previous coax cable suggestion will likely involve MoCA:

https://en.wikipedia.org/wiki/Multimedia_over_Coax_Alliance


Another idea is power-line communication.  I seem to recall reading that 
an RF choke should be installed on the incoming electrical service to 
prevent interference to/from the neighbors, but STFW I do not see any 
mention of that today:


https://en.wikipedia.org/wiki/Powerline_Ethernet


David

Re: "Repeaters", etc.

[David Christensen]

On 5/27/24 14:09, Paul M Foster wrote:

Folks:

At some point this year, I'm moving into a new house, and it is not wired
for internet (WHY aren't new houses wired with Cat5/6/7?). The local
internet provider will likely provide a wireless router, as they all do. My
idea is to put a device which receives wireless signal from the
router/modem, and has an RJ45 jack in it in each room. So each room would
have one of these, and the devices in it would be hooked to that device via
cat 5e. I hope that's clear.

I'd like to shop for such a device, but I don't know what it's called. Can
anyone provide advice, and possibly preferred brand names? I'd appreciate
it.

Paul



Is the house wired for cable television (RG-6 coaxial cable)?  If so, 
and you choose the right Internet provider, you might be able to get a 
"main box" with 1+ type F connectors, 4 @ RJ-45 Gigabit ports,  Wi-Fi 
access point, etc., and "satellite boxes" with 1+ type F connector, 1+ 
RJ-45 Gigabit ports, Wi-Fi access point, etc..  Ask you cable 
television/ Internet provider.



An alternative to running Ethernet cables inside walls is to run the 
cables on the surface -- e.g. staple to wall along floor molding, drill 
and pull through walls as required, paint to match, etc..  Cat 5e is 
smaller diameter and easier to work with than Cat 6a.  I surface wired 
my house with Cat 5e ~20 years ago and have 1000BASE-T (Gigabit 
Ethernet) switches and NIC's.  All of the cable runs are under ~20 
meters, so I should be able to upgrade to 2.5GBASE-T.



David

Re: HDD long-term data storage with ensured integrity

[David Christensen]

On 5/3/24 04:26, Marc SCHAEFER wrote:

On Mon, Apr 08, 2024 at 10:04:01PM +0200, Marc SCHAEFER wrote:

For off-site long-term offline archiving, no, I am not using RAID.


Now, as I had to think a bit about ONLINE integrity, I found this
comparison:

https://github.com/t13a/dm-integrity-benchmarks

Contenders are btrfs, zfs, and notably ext4+dm-integrity+dm-raid

I tend to have a biais favoring UNIX layered solutions against
"all-into-one" solutions, and it seems that performance-wise,
it's also quite good.

I wrote this script to convince myself of auto-correction
of the ext4+dm-integrity+dm-raid layered approach.



Thank you for devising a benchmark and posting some data.  :-)


FreeBSD also offers a layered solution.  From the top down:

* UFS2 file system, which supports snapshots (requires partitions with 
soft updates enabled).


* gpart(8) for partitions (volumes).

* graid(8) for redundancy and self-healing.

* geli(8) providers with continuous integrity checking.


AFAICT the FreeBSD stack is mature and production quality, which I find 
very appealing.  But the feature set is not as sophisticated as ZFS, 
which leaves me wanting.  Notably, I have not found a way to replicate 
UFS snapshots directly -- the best I can dream up is synchronizing a 
snapshot to a backup UFS2 filesystem and then taking a snapshot with the 
same name.



I am coming to the conclusion that the long-term survivability of data 
requires several components -- good live file system, good backups, good 
archives, continuous internal integrity checking with self-healing, 
periodic external integrity checking (e.g. mtree(1)) with some form of 
recovery (e.g. manual), etc.. If I get the other pieces right, I could 
go with OpenZFS for the live and backup systems, and worry less about 
data corruption bugs.



David

Re: realpath quoting

[David Christensen]

On 5/3/24 04:34, jeremy ardley wrote:


On 3/5/24 19:06, Greg Wooledge wrote:

I would suggest that if you need to use a debugger to track down a bug
in your program, you should use filenames that don't require quoting
when you set up your tests.


1970's style static test cases are not relevant here.

In the real world...  I download files generated by another system that 
are constantly changing content and with names I don't control.


My workflow is to download a new file from a remote source and then run 
my processor over it.


As a necessary consequence I need the fully quoted or escaped file name 
of the new file to feed to the processor/debugger.


I can obviously add an extra step to the process to convert the new file 
name to something acceptable before processing. However, my question was 
how to avoid that extra step by getting fully quoted filenames to process.





So, you are copying and pasting file names via some clipboard?  emacs(1) 
might have a way to put a filter into that process, but I am unaware of 
a similar feature using Xfce and Terminal (my platform).



I have tried renaming files in similar situations, but you will want to 
rename them everywhere if you use rsync(1).



What if you downloaded files to a directory with a well-formed name and 
added a feature to your script to process files that appear in that 
directory?



David

Re: realpath quoting

[David Christensen]

On 5/3/24 04:09, Greg Wooledge wrote:

On Thu, May 02, 2024 at 10:18:03PM -0700, David Christensen wrote:

I am unable to find $'string' in the dash(1) man page (?).  As I typically
write "#!/bin/sh" shell scripts, writing such to deal with file names
containing non-printing characters is going to baffle me.


Currently, $' quoting is a bash extension.  It's supposed to appear in
some future edition of POSIX, at which point shells like dash will be
required to adopt it (whenever they get around to it).  For now, though,
you should consider it bash only.



Thank you for the clarification.  :-)


David

Re: realpath quoting

[David Christensen]

On 5/2/24 19:56, Max Nikulin wrote:

On 03/05/2024 09:19, Greg Wooledge wrote:

I still insist that this is a workaround that should *not*  be used
to try to cancel out quoting bugs in one's shell scripts.


There are still specific cases when quoting is necessary, e.g. ssh 
remote command 



+1


(however you have to be sure concerning shell on the 
remote host).



+1


In BASH printf has %q format. GNU coreutils supports it as well, but 
dash does not, so be careful.



My practice is to start with '#!/bin/sh' and migrate to '#!/usr/bin/env 
perl' as complexity increases.




Likely Jeremy's case does not really require this kind of quoting.



We need to see the full range of file names the OP is trying to deal with.


While "ls -l" output is for humans, realpath is often used in scripts. 
Certainly it should nor return quoted output by default. I am in doubts 
if a dedicated option should be added to realpath.



Thank you for helping me realize that my solution fails to print the 
resolved absolute file name.  Here is the updated solution:


2024-05-02 21:57:56 dpchrist@laalaa ~
$ touch 'name with spaces'

2024-05-02 22:23:01 dpchrist@laalaa ~
$ touch 'name with
> newline'

2024-05-02 22:28:36 dpchrist@laalaa ~
$ perl -MString::ShellQuote '-MFile::Spec::Functions qw(rel2abs)' -e 
'print shell_quote(map { rel2abs $_ } @ARGV), "\n"' name*

'/home/dpchrist/name with
newline' '/home/dpchrist/name with spaces'


David

Re: realpath quoting

[David Christensen]

On 5/2/24 19:19, Greg Wooledge wrote:

On Thu, May 02, 2024 at 07:11:46PM -0700, David Christensen wrote:

Perhaps Perl and the module String::ShellQuote ?

2024-05-02 18:50:28 dpchrist@laalaa ~
$ touch "name with spaces"

2024-05-02 18:50:45 dpchrist@laalaa ~
$ touch "name with\nnewline"


You didn't create a name with a newline in it here.  You created a name
with a backslash in it.  If you wanted a newline, you would have to use
the $'...' quoting form (in bash).

 touch $'name with\nnewline'



Thank you for the clarification.


RTFM bash(1):

QUOTING
...
Enclosing  characters  in  double quotes preserves the literal
value of all characters within the quotes, with the exception of
$, `,  \,  and, when history expansion is enabled, !. ...
The backslash retains its special  meaning  only when followed
by one of the following characters: $, `, ", \, or .
...
Words of the form $'string' are treated specially.  The word
expands to string,  with backslash-escaped characters replaced
as specified by the ANSI C standard.


I found another way to obtain a file name containing a newline -- by 
pressing  when typing a double-quoted string literal:


2024-05-02 21:52:23 dpchrist@laalaa ~
$ touch "foo
> bar"

2024-05-02 21:52:29 dpchrist@laalaa ~
$ ls -l foo*
-rw-r--r-- 1 dpchrist dpchrist 0 May  2 21:52 'foo'$'\n''bar'

2024-05-02 21:52:36 dpchrist@laalaa ~
$ perl -MString::ShellQuote -e 'print shell_quote(@ARGV), "\n"' foo*
'foo
bar'


It also seems to work for single-quoted string literals:

2024-05-02 21:57:08 dpchrist@laalaa ~
$ touch 'foo
> bar'

2024-05-02 21:57:14 dpchrist@laalaa ~
$ ls -l foo*
-rw-r--r-- 1 dpchrist dpchrist 0 May  2 21:57 'foo'$'\n''bar'

2024-05-02 21:57:18 dpchrist@laalaa ~
$ perl -MString::ShellQuote -e 'print shell_quote(@ARGV), "\n"' foo*
'foo
bar'


I am unable to find $'string' in the dash(1) man page (?).  As I 
typically write "#!/bin/sh" shell scripts, writing such to deal with 
file names containing non-printing characters is going to baffle me.




I still insist that this is a workaround that should *not* be used
to try to cancel out quoting bugs in one's shell scripts.  Just write
the shell scripts correctly in the first place.



I would if I could.


While I am also unable to write Perl scripts correctly in the first 
place, the quoting rules are easier.



David

Re: realpath quoting

[David Christensen]

On 5/2/24 15:59, jeremy ardley wrote:
I have a need  to get the full path of a file that has spaces in its 
name to use as a program argument


e.g.

jeremy@client:~$ ls -l name\ with\ spaces
-rw-r--r-- 1 jeremy jeremy 0 May  3 06:51 'name with spaces'
jeremy@client:~$ realpath name\ with\ spaces
/home/jeremy/name with spaces


The spaces without quotes cause problems with subsequent processing.

Can realpath or other utility return a quoted pathname?



Perhaps Perl and the module String::ShellQuote ?

2024-05-02 18:50:28 dpchrist@laalaa ~
$ touch "name with spaces"

2024-05-02 18:50:45 dpchrist@laalaa ~
$ touch "name with\nnewline"

2024-05-02 19:06:01 dpchrist@laalaa ~
$ perl -MString::ShellQuote -e 'print shell_quote(@ARGV), "\n"' name*
'name with spaces' 'name with\nnewline'


David

Re: Debian@IBMx3550

[David Christensen]

On 4/23/24 14:35, Greg wrote:

Hi there,

I got refurb IBM x3550 M3 7944 server and I'm a bit lost. Is there any 
Linux/Debian software (some gui would be nice) to monitor fan speed, 
temperatures, voltages, disks.. ?


Thanks in advance for any help
Greg




If you installed the Xfce desktop, add the "Sensor" plug-in/ applet to 
the Panel.  Panel can display various temperatures, fan speeds, etc..



Note that you may to set the SUID bit on /usr/sbin/hddtemp for disk 
drive readings to be available:


# chmod u+s /usr/sbin/hddtemp


David

Re: Subject: Glitchy sound in Steam games after hard drive upgrade

[David Christensen]

On 4/23/24 09:02, debian-u...@howorth.org.uk wrote:

Charlie Gibbs  wrote:

On 2024-04-22 16:50, Jeffrey Walton wrote:


What are the old and new hard drive model numbers and specs?


The old drive is a Western Digital WD5000YS (500GB SATA).
The new drive is a Western Digital Red, WF40EFPX (4TB SATA).


According to my searches, there's no such disk as a WF40EFPX. Are you
sure that's what it is? If by any chance it is a WD40EFRX then that is
certainly slower than your old drive, so may cause some problems as
suggested.



I doubt the new drive is slower than the old drive:

- https://www.harddrivebenchmark.net/hdd.php?hdd=WDC%20WD5000YS

WDC WD5000YS  425

- https://www.harddrivebenchmark.net/hdd.php?hdd=WDC%20WD40EFRX

WDC WD40EFRX1,943


David

Re: Subject: Glitchy sound in Steam games after hard drive upgrade

[David Christensen]

On 4/22/24 21:26, Charlie Gibbs wrote:

On 2024-04-22 16:50, Jeffrey Walton wrote:

What are the old and new hard drive model numbers and specs?


The old drive is a Western Digital WD5000YS (500GB SATA).



https://www.newegg.com/western-digital-re2-wd5000ys-500gb/p/N82E16822136032?Item=N82E16822136032



The new drive is a Western Digital Red, WF40EFPX (4TB SATA).



https://www.westerndigital.com/products/internal-drives/wd-red-plus-sata-3-5-hdd?sku=WD40EFPX


Both drives are spinning rust.  I'm upgrading for the increased 
capacity, i.e. to store more MP3s and videos.


Many thanks to all who have replied.  When my schedule permits me to 
continue experimenting, I'm going to try copying /etc from the old drive 
to the new one.   I've already learned how _not_ to do this:


Boot from the new drive
$ su root
# cd /
# mv etc etc.ori
# rsync -av /mnt/backup/etc .

The second line makes the system fall over and makes logins impossible. 
It took a boot from the rescue CD to undo the damage, which fortunately 
was easy since the deadly step at least succeeded in backing up /etc.


Next time I'll do it while booted from the old drive.



Copying an entire /etc directory from one machine to another requires a 
highly controlled environment and lot of engineering.  I have always 
migrated /etc settings from one OS instance to another OS instance by 
hand, one service/ configuration file at a time.



Can you leave the 500 GB HDD operational and use the 4 TB HDD for data?


David

Re: Strange New Installation Behavior

[David Christensen]

On 4/22/24 06:00, Stephen P. Molnar wrote:
I am running Bookworm and cleaned up a couple of files too many 
resulting in a messed up Xfce Desktop. I decided that this would be a

 good time to reinstall the Bullseye.

I made a backup of my /home/comp directory using Deja-dup.

I downloaded and ran the 512 check sum on a copy of 
Debian-12.5.0-amd64-DVD-1.iso and ran the Graphical Install mode on

the 1.0 TD SSD on my Computer. The installation went smoothly without
any warning or error messages.

I logged in as root to set up the Desktop and, much to my surprise, 
found that my previous Desktop configuration was still there!!???

This was also the case when I logged in user!!!???

I have been using computers in my work since the 1960, the era of the
 Hollerith Card and tape drives and Linux since early days of
Slackware and the Red Hat Mother's Day Edition. Now I am not a
computer expert but a Research Chemist.  I have installed Linux OS's
many times and consider Linux my primary computational platform. I
have never encountered the situation and have no ideas as to what is
going on.

I have been runnind Debian since Etch.

I would appreciate some insight into what might be going on.

Thanks in advance.

Stephen P. Molnar, Ph.D. https://insilicochemistry.net (614)312-7528
(c) Skype: smolnar1



On 4/22/24 09:34, Stephen P. Molnar wrote:

I did not want to revert to Bullseye, but to reinstall to Bookworm.


I suggest that you buy a good 16 GB USB flash drive and install Debian 
12 with Xfce onto it.  Having a working live USB stick is very useful 
for low-level disk drive chores such as examining, backing up, testing, 
repairing, restoring, wiping, etc..  Use it to:


1.  Ensure that you have a good backup of your 1 TB SSD.

2.  Make additional backups or archives of all or part of your 1 TB SSD. 
 Note the mantra: "Data does not exist unless it exists in three places".


3.  Wipe the SSD so that the Debian installer will see a blank disk and 
respond accordingly when you later install Debian onto the SSD.



Regarding copying a home directory from one OS installation to another 
OS installation, please see my comments on another thread:


https://lists.debian.org/debian-user/2024/04/msg00336.html


Once you have logged in to your new account on your fresh install, I 
suggest that you restore your /home/comp backup to a subdirectory and 
manually copy/ move/ edit/ merge files and directories from the restore 
subdirectory into your fresh home directory.  Be very careful not to 
damage or delete anything needed by your fresh desktop or applications.



David

Re: Subject: Glitchy sound in Steam games after hard drive upgrade

[David Christensen]

On 4/21/24 22:33, Charlie Gibbs wrote:

I should probably be posting this to the Steam forums, but
most of the denizens there are Windows people so I might be
better off letting you Debian gurus have a go at it first.

TL;DR: Copying an existing /home into a fresh Debian installation
causes audio in Steam games to glitch - but all other sound is OK.

Full description:

I have a machine in the living room that stores MP3s and videos
and serves them to other machines on our network as well as playing
them locally on our TV's big screen.  I also play a few Steam games
(e.g. Portal) on it.  It's a 2007-vintage machine, but it has 8GB
of RAM and enough CPU power to do the job, and runs the latest
version of Bookworm.

Recently I decided to upgrade its storage capacity, and replaced
its 500GB hard drive (which was pretty large at the time I bought
it) with a 4TB drive.  I did an install from scratch using a
network install CD, then copied my /home partition (using rsync)
from the old drive.  Everything works great with one exception:
when I fire up Portal the sound gets glitches about once a second.
This only happens with Steam games; I can play MP3s and videos
with mpv and the sound is perfect, as it is when watching YouTube
videos.  If I swap the old drive back in everything is fine.

Obviously my Steam programs and configuration files are in my
home directory, since the updated system comes up icons and all
without re-installing Steam, and can find everything it needs to
run the games.  But perhaps there are a few files somewhere else
(/usr?) containing information critical to audio for Steam.

Any ideas?

(Side question: is this an acceptable way to upgrade a hard drive?)



Copying a home directory from one OS instance to another OS instance 
sounds risky, especially as I run various OS's.  I have several 
instances of Debian 11, and would not consider them to be identical 
enough to try it.  I only touch the content I create or have learned how 
to manage.



I put my OS on a small SSD and the vast majority of my data on HDD RAID 
in a file server.



As I am the only user on my Debian daily driver, I leave the /home 
directory on the root file system and keep as little as possible in it.



I mount the file server shares under /mnt, and create symlinks in my 
home directory that point into the mounted file system.



I use CVS for project working directories.  To migrate to a new home 
directory, I check in the projects in the old home and check out the 
project in the new home.



I use Firefox and its sync feature.  To migrate to a new home, I start 
Firefox, log in, wait for my settings to sync, and then check all of the 
settings by hand.



I use Thunderbird.  To migrate to a new home, I create a tarball of my 
Thunderbird profile directory on the old machine, expand the tarball on 
the new machine, and configure Thunderbird to use that profile.



I do not attempt to migrate any of the various home directory 
configuration directories; I let the installer and/or package manager 
create them, and let the desktop, apps, etc., manage them.



David

Re: Debian 11 Xfce panel Network Manager applet has disappeared

[David Christensen]

On 4/14/24 05:29, David Christensen wrote:

debian-user:

I have a Dell Latitude E6520:

2024-04-14 04:28:39 dpchrist@laalaa ~
$ cat /etc/debian_version ; uname -a
11.9
Linux laalaa 5.10.0-28-amd64 #1 SMP Debian 5.10.209-2 (2024-01-31) 
x86_64 GNU/Linux


2024-04-14 04:34:40 dpchrist@laalaa ~
$ dpkg-query -l xfce4 network-manager network-manager-gnome
Desired=Unknown/Install/Remove/Purge/Hold
| 
Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend

|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name  Version  Architecture Description
+++-=---=
ii  network-manager   1.30.6-1+deb11u1 amd64    network 
management framework (daemon and userspace tools)
ii  network-manager-gnome 1.20.0-3 amd64    network 
management framework (GNOME frontend)
ii  xfce4 4.16 all  Meta-package for 
the Xfce Lightweight Desktop Environment



I have used the Xfce panel Network Manager applet for many years. 
Tonight, I noticed that it has disappeared (!).



I compared the problem machine against another with a working Xfce panel 
Network Manager applet, and discovered that the Status Tray Plugin was 
missing.  I may have deleted Status Tray Plugin while cleaning, but did 
not notice the change immediately (?).



So, I added Status Tray Plugin and now Network Manager has returned.


David

Re: Fwd: Debian 11 Xfce panel Network Manager applet has disappeared

[David Christensen]

On 4/19/24 00:16, Florent Rougon wrote:

Another thing: did you look into ~/.xsession-errors?
(Sorry if this was already mentioned and I missed it.)



Please see attached copy of ~/.xsession-errors, taken immediately after 
system restart and login.



"nm-applet" does not appear in .xsession-errors, but there are plenty of 
other warnings and error messages.  Perhaps the failure of nm-applet is 
a symptom of a more fundamental failure (?).




More involved: if you can't find any trace of the applet doing
something, maybe rebuilding the package after adding a few fprintf()
calls would help.



I think it is time for a bug report.


David


.xsession-errors-20240419-121605.gz
Description: application/gzip

Re: Fwd: Debian 11 Xfce panel Network Manager applet has disappeared

[David Christensen]

On 4/18/24 09:46, Gareth Evans wrote:

On Thu 18/04/2024 at 11:05, David Christensen  wrote:


Move aside the ~/.config/xfce4 directory:
...
Restart -- screen with wallpaper alone.
...


Hi David,

Starting from Mate DE only and some old (bookworm) XFCE config files, if I:

$ sudo apt install task-xfce-desktop

then log out and into XFCE, I get an XFCE desktop with wallpaper, desktop 
icons, panel, main menu, launchers, window buttons, notification area.

Then I:

$ mv ~/.config/xfce4 ~/.config/xfce4.old

then logout and into XFCE again, I get wallpaper, desktop icons, panel, main 
menu, window buttons, notification area, but only non-functional launcher 
placeholder icons.

$ diff ~/.config/xfce4 ~/.config/xfce4.old
Common subdirectories: /home/user/.config/xfce4/desktop and 
/home/user/.config/xfce4.old/desktop
Only in /home/user/.config/xfce4.old: panel   }
Only in /home/user/.config/xfce4.old: src }  <-- these are 
subdirectories
Only in /home/user/.config/xfce4.old: terminal}
Common subdirectories: /home/user/.config/xfce4/xfconf and 
/home/user/.config/xfce4.old/xfconf
Common subdirectories: /home/user/.config/xfce4/xfwm4 and 
/home/user/.config/xfce4.old/xfwm4
$

(though there probably wasn't much there to start with)

I'm not sure what would cause this difference in behaviour (though wonder if 
this might suggest more amiss with your XFCE installation) and I will watch 
this thread with interest.

Also I just "rediscovered" that reinstalling task-xfce-desktop doesn't 
reinstall those packages which it brings in in the first place, though I'm sure you knew 
that :)

Sorry I couldn't be of more help.



Thanks for the suggestions.  We have more information now.  :-)


David

Re: Fwd: Debian 11 Xfce panel Network Manager applet has disappeared

[David Christensen]

On 4/18/24 07:28, Max Nikulin wrote:

On 18/04/2024 17:05, David Christensen wrote:

$ mv .config/xfce4-20240418-180045/ .config/xfce4

Restart -- back to Xfce panel with no Network Manager.


Try to create a new system user and log in. Is nm-applet present?



Logging in using another previously working account produces the same 
result -- Xfce Panel displays Notification Plugin (bell icon), but no 
Network Manager icon.



Creating a new account and logging in produces the same result.


David

Re: Fwd: Debian 11 Xfce panel Network Manager applet has disappeared

[David Christensen]

On 4/18/24 05:34, e...@gmx.us wrote:

On 4/18/24 05:27, David Christensen wrote:

On 4/17/24 12:37, Richmond wrote:

What are the permissions on the nm-applet binary? maybe it doesn't have
permission to execute, or the process which starts it doesn't have
permission.



2024-04-18 02:24:20 root@laalaa ~
# ls -l `which nm-applet`
-rwxr-xr-x 1 root root 250784 Feb 27  2021 /usr/bin/nm-applet


I do not know what binary starts nm-applet, but here is a WAG:

2024-04-18 02:27:13 root@laalaa ~
# ll -l `which xfce4-panel`
-rwxr-xr-x 1 root root 363328 2021-02-27 08:29:44 /usr/bin/xfce4-panel


Can PPID tell you?



Yes.  xfce4-session is the parent of nm-applet:

2024-04-18 10:27:32 dpchrist@laalaa ~
$ ps -Ao "%p %P %a" | grep nm-applet
   15301379 nm-applet
   39373865 grep nm-applet

2024-04-18 10:27:44 dpchrist@laalaa ~
$ ps -Ao "%p %P %a" | grep 1379
   13791344 xfce4-session
   14211379 /usr/bin/ssh-agent x-session-manager
   14581379 xfwm4 --display :0.0 --sm-client-id 
2d1bdc0a5-06f2-4835-944e-7e402203bd9f
   14711379 xfsettingsd --display :0.0 --sm-client-id 
29896f657-477c-498a-8156-db9d51f74bcf
   14971379 xfce4-panel --display :0.0 --sm-client-id 
23fd394e3-a4ee-485b-b43c-38bf845d6699
   15011379 xfdesktop --display :0.0 --sm-client-id 
2c913de99-acac-4d70-b6e9-8eda44b6f6da
   15061379 xfce4-power-manager --restart --sm-client-id 
24cebdb39-bdc2-496f-aab0-7b11397b48d3

   15131379 /usr/lib/x86_64-linux-gnu/xfce4/notifyd/xfce4-notifyd
   15151379 light-locker
   15161379 /usr/bin/python3 /usr/share/system-config-printer/applet.py
   15171379 
/usr/lib/policykit-1-gnome/polkit-gnome-authentication-agent-1

   15241379 xiccd
   15301379 nm-applet
   39403865 grep 1379

2024-04-18 10:29:20 dpchrist@laalaa ~
$ ls -l `which xfce4-session`
-rwxr-xr-x 1 root root 260496 Dec 23  2020 /usr/bin/xfce4-session


David

Re: Debian 11 Xfce panel Network Manager applet has disappeared

[David Christensen]

On 4/17/24 12:07, Charles Curley wrote:

On Wed, 17 Apr 2024 11:41:24 -0700
David Christensen  wrote:


My WAG is that nm-applet is failing to start, but I have been unable
to find if and where any error message is reported.


My instance of nm-applet does run, and I see this as part of the boot
process:

root@hawk:~# journalctl -b | grep nm-app
Apr 15 11:27:42 hawk NetworkManager[1354]:   [1713202062.7737] 
agent-manager: agent[108f011a1115d508,:1.131/org.freedesktop.nm-applet/1000]: agent 
registered
root@hawk:~#

I suspect that if nm-applet doesn't start, you won't see any output
from that command.



2024-04-18 03:07:20 root@laalaa ~
# journalctl -b | grep nm-app
Apr 18 03:01:39 laalaa NetworkManager[833]:   [1713434499.3660] 
agent-manager: 
agent[2a08cda7fa35849b,:1.48/org.freedesktop.nm-applet/13250]: agent 
registered



David

Re: Fwd: Debian 11 Xfce panel Network Manager applet has disappeared

[David Christensen]

On 4/17/24 19:41, Gareth Evans wrote:

On Wed 17/04/2024 at 19:41, David Christensen  wrote:

On 4/17/24 03:47, Gareth Evans wrote:

On Wed 17/04/2024 at 09:18, David Christensen  wrote:

On 4/16/24 08:56, Gareth Evans wrote:

On 16 Apr 2024, at 00:18, David Christensen  wrote:

On 4/15/24 09:21, Gareth Evans wrote:

On Sun 14/04/2024 at 13:29, David Christensen wrote:
...
I have used the Xfce panel Network Manager applet for many years.
Tonight, I noticed that it has disappeared (!).
...


There is apparently a long history of nm-applet/XFCE panel-related issues (and 
not many great answers), as evidenced by such as

https://bbs.archlinux.org/viewtopic.php?id=161998

https://forum.xfce.org/viewtopic.php?id=6105

https://superuser.com/questions/900490/networkmanager-icon-on-notification-area-is-not-present



I tried:

1.  Remove Notification applet, restart, add Notification applet, no 
Network Manager, restart -- no Network Manager.


2.  Look at /etc/NetworkManager/NetworkManager.conf

2024-04-18 02:40:49 dpchrist@laalaa ~
$ cat /etc/NetworkManager/NetworkManager.conf
[main]
plugins=ifupdown,keyfile

[ifupdown]
managed=false

Change it to:

2024-04-18 02:43:35 root@laalaa ~
# cat /etc/NetworkManager/NetworkManager.conf
[main]
plugins=ifupdown,keyfile

[ifupdown]
managed=true

Restart -- no Network Manager.

Verify /etc//NetworkManager/NetworkManager.conf:

2024-04-18 02:45:05 dpchrist@laalaa ~
$ cat /etc/NetworkManager/NetworkManager.conf
[main]
plugins=ifupdown,keyfile

[ifupdown]
managed=true



If you get ps output like this directly after a reboot:


$ ps aux | grep nm-applet
dpchrist1518  0.1  0.2 426500 35380 ?Sl   16:06   0:00 nm-applet



2024-04-18 02:45:10 dpchrist@laalaa ~
$ ps aux | grep nm-applet
dpchrist1526  0.1  0.2 426484 35256 ?Sl   02:44   0:00 nm-applet
dpchrist1869  0.0  0.0   3240   712 pts/0S+   02:47   0:00 grep 
nm-applet




then I don't think the issue is with the starting/running of nm-applet itself, 
but rather some issue with the notification plugin, which I'm not sure how to 
begin troubleshooting.  I reluctantly abandoned XFCE partly due to panel 
instability some time ago.

The advice in the final link to rm -rf  ~/.config/xfce* might be a bit extreme, 
but I might try renaming it to force a rebuild on next login, and possibly 
reinstall task-xfce-desktop (or selected packages)



Move aside the ~/.config/xfce4 directory:


2024-04-18 02:50:20 dpchrist@laalaa ~
$ ls -ld .config/xfce4/
drwxr-xr-x 8 dpchrist dpchrist 4096 Apr 18 00:45 .config/xfce4/

2024-04-18 02:50:23 dpchrist@laalaa ~
$ mv .config/xfce4/ .config/xfce4-20240418-180045


Restart -- screen with wallpaper alone.


Right-click on desktop -> Applications -> Settings -> Panel brings up 
the Panel Preferences application.  The Items tab shows my previous 
settings.  Fumbling around, I am unable to get that panel to display 
(?).  If I create a new panel, I am unable to get it to display (?).



Open a Terminal and look if Xfce created a new configuration directory:

2024-04-18 02:57:31 dpchrist@laalaa ~
$ ls -ld .config/xfce4*
drwxr-xr-x 7 dpchrist dpchrist 4096 Apr 18 02:57 .config/xfce4
drwxr-xr-x 8 dpchrist dpchrist 4096 Apr 18 00:45 
.config/xfce4-20240418-180045



Yes, it did.  Compare the old directory to the new directory:

2024-04-18 02:58:11 dpchrist@laalaa ~
$ diff -rq .config/xfce4-20240418-180045/ .config/xfce4
diff: .config/xfce4-20240418-180045/desktop/icons.screen.latest.rc: No 
such file or directory

Only in .config/xfce4-20240418-180045/desktop: icons.screen0-1008x725.rc
Only in .config/xfce4-20240418-180045/desktop: icons.screen0-1424x857.rc
Only in .config/xfce4-20240418-180045/desktop: icons.screen0-1664x1007.rc
Only in .config/xfce4-20240418-180045/desktop: icons.screen0-1904x1037.rc
Only in .config/xfce4-20240418-180045/desktop: icons.screen0-1904x1064.rc
Only in .config/xfce4-20240418-180045/desktop: icons.screen0-2928x1037.rc
Files .config/xfce4-20240418-180045/desktop/icons.screen0-3824x1037.rc 
and .config/xfce4/desktop/icons.screen0-3824x1037.rc differ

Only in .config/xfce4-20240418-180045/: help.rc
Only in .config/xfce4-20240418-180045/panel: launcher-15
Only in .config/xfce4/panel: launcher-17
Only in .config/xfce4/panel: launcher-18
Only in .config/xfce4/panel: launcher-19
Only in .config/xfce4/panel: launcher-20
Only in .config/xfce4-20240418-180045/panel: launcher-24
Only in .config/xfce4-20240418-180045/panel: launcher-28
Only in .config/xfce4-20240418-180045/panel: launcher-7
Only in .config/xfce4-20240418-180045/: src
Files .config/xfce4-20240418-180045/terminal/accels.scm and 
.config/xfce4/terminal/accels.scm differ

Only in .config/xfce4-20240418-180045/terminal: terminalrc
Only in .config/xfce4-20240418-180045/: xfce4-screenshooter
Only in .config/xfce4-20240418-18004

Re: Fwd: Debian 11 Xfce panel Network Manager applet has disappeared

[David Christensen]

On 4/17/24 13:56, e...@gmx.us wrote:

On 4/17/24 15:37, Richmond wrote:

David Christensen writes:

My WAG is that nm-applet is failing to start, but I have been unable to
find if and where any error message is reported.


What are the permissions on the nm-applet binary?


And is its filesystem mounted with noexec?



2024-04-18 02:27:18 root@laalaa ~
# df `which nm-applet`
Filesystem 1M-blocks  Used Available Use% Mounted on
/dev/mapper/sdb3_crypt12084M 8927M 2522M  78% /

2024-04-18 02:28:57 root@laalaa ~
# mount | grep sdb3
/dev/mapper/sdb3_crypt on / type ext4 (rw,relatime,errors=remount-ro)


David

Re: Fwd: Debian 11 Xfce panel Network Manager applet has disappeared

[David Christensen]

On 4/17/24 12:37, Richmond wrote:

David Christensen  writes:

On Sun 14/04/2024 at 13:29, David Christensen wrote:
...
I have used the Xfce panel Network Manager applet for many years.
Tonight, I noticed that it has disappeared (!).
...

What are the permissions on the nm-applet binary? maybe it doesn't have
permission to execute, or the process which starts it doesn't have
permission.



2024-04-18 02:24:20 root@laalaa ~
# ls -l `which nm-applet`
-rwxr-xr-x 1 root root 250784 Feb 27  2021 /usr/bin/nm-applet


I do not know what binary starts nm-applet, but here is a WAG:

2024-04-18 02:27:13 root@laalaa ~
# ll -l `which xfce4-panel`
-rwxr-xr-x 1 root root 363328 2021-02-27 08:29:44 /usr/bin/xfce4-panel


David

Fwd: Debian 11 Xfce panel Network Manager applet has disappeared

[David Christensen]

 Forwarded Message 
Subject: Re: Debian 11 Xfce panel Network Manager applet has disappeared
Date: Wed, 17 Apr 2024 01:18:34 -0700
From: David Christensen 
To: Gareth Evans 

On 4/16/24 08:56, Gareth Evans wrote:

On 16 Apr 2024, at 00:18, David Christensen  wrote:

On 4/15/24 09:21, Gareth Evans wrote:

On Sun 14/04/2024 at 13:29, David Christensen wrote:
...
I have used the Xfce panel Network Manager applet for many years.
Tonight, I noticed that it has disappeared (!).
...

2024-04-15 16:08:23 dpchrist@laalaa ~
$ ps aux | grep nm-applet
dpchrist1518  0.1  0.2 426500 35380 ?Sl   16:06   0:00 nm-applet
dpchrist1940  0.0  0.0   3240   644 pts/0S+   16:15   0:00 grep 
nm-applet

2024-04-15 16:15:12 dpchrist@laalaa ~
$ nm-applet

2024-04-15 16:15:31 dpchrist@laalaa ~
$ ps aux | grep nm-applet
dpchrist1518  0.1  0.2 426500 35380 ?Sl   16:06   0:00 nm-applet
dpchrist1952  0.0  0.0   3240   644 pts/0S+   16:15   0:00 grep 
nm-applet


That seems to show it's running from the outset, just not being displayed on 
the panel.

Does rebooting (or logging out and in again) bring it back?



No.


David

Fwd: Debian 11 Xfce panel Network Manager applet has disappeared

[David Christensen]

 Forwarded Message 
Subject: Re: Debian 11 Xfce panel Network Manager applet has disappeared
Date: Wed, 17 Apr 2024 11:38:49 -0700
From: David Christensen 
To: Gareth Evans 

On 4/17/24 03:47, Gareth Evans wrote:

On Wed 17/04/2024 at 09:18, David Christensen  wrote:

On 4/16/24 08:56, Gareth Evans wrote:

On 16 Apr 2024, at 00:18, David Christensen  wrote:

On 4/15/24 09:21, Gareth Evans wrote:

On Sun 14/04/2024 at 13:29, David Christensen wrote:
...
I have used the Xfce panel Network Manager applet for many years.
Tonight, I noticed that it has disappeared (!).
...

2024-04-15 16:08:23 dpchrist@laalaa ~
$ ps aux | grep nm-applet
dpchrist1518  0.1  0.2 426500 35380 ?Sl   16:06   0:00 nm-applet
dpchrist1940  0.0  0.0   3240   644 pts/0S+   16:15   0:00 grep 
nm-applet

2024-04-15 16:15:12 dpchrist@laalaa ~
$ nm-applet

2024-04-15 16:15:31 dpchrist@laalaa ~
$ ps aux | grep nm-applet
dpchrist1518  0.1  0.2 426500 35380 ?Sl   16:06   0:00 nm-applet
dpchrist1952  0.0  0.0   3240   644 pts/0S+   16:15   0:00 grep 
nm-applet


That seems to show it's running from the outset, just not being displayed on 
the panel.

Does rebooting (or logging out and in again) bring it back?



No.


OK.  You may have checked this already, but in case not, if I install XFCE and 
go to

Settings > Session and Startup > Application Autostart

there is an entry in the list called

"Network (Manage your network connections)"



It is checked.



which shows a tooltip of "command: nm-applet"



Command: nm-applet



Might this somehow have become unset?

I'm not sure if it's possible for GUI config helpers to become detached from 
actual settings - this seems to describe the relevant locations:

https://unix.stackexchange.com/questions/669372/xfce4-session-and-startup-where-are-autostart-items-saved



2024-04-17 11:21:06 dpchrist@laalaa ~
$ grep nm-applet ~/.config/autostart
grep: /home/dpchrist/.config/autostart: No such file or directory

2024-04-17 11:33:11 dpchrist@laalaa ~
$ find .config -name autostart

2024-04-17 11:33:22 dpchrist@laalaa ~
$


2024-04-17 11:34:14 dpchrist@laalaa ~
$ grep -r nm-applet /etc/xdg/autostart
/etc/xdg/autostart/nm-applet.desktop:Exec=nm-applet
/etc/xdg/autostart/nm-applet.desktop:X-GNOME-Bugzilla-Component=nm-applet


My WAG is that nm-applet is failing to start, but I have been unable to 
find if and where any error message is reported.



David

Re: Debian 11 Xfce panel Network Manager applet has disappeared

[David Christensen]

On 4/15/24 09:21, Gareth Evans wrote:

On Sun 14/04/2024 at 13:29, David Christensen wrote:

...
I have used the Xfce panel Network Manager applet for many years.
Tonight, I noticed that it has disappeared (!).
...

Hi David,

I can't speak for XFCE, but certainly for Mate there was a time when multiple 
notification area panel widgets were available, not all of which would show 
everything to be expected.  Is that a possibility?

Does

$ ps aux |grep nm-applet

show anything?  Before or after

$ nm-applet

?



2024-04-15 16:08:23 dpchrist@laalaa ~
$ ps aux | grep nm-applet
dpchrist1518  0.1  0.2 426500 35380 ?Sl   16:06   0:00 nm-applet
dpchrist1940  0.0  0.0   3240   644 pts/0S+   16:15   0:00 grep 
nm-applet


2024-04-15 16:15:12 dpchrist@laalaa ~
$ nm-applet

2024-04-15 16:15:31 dpchrist@laalaa ~
$ ps aux | grep nm-applet
dpchrist1518  0.1  0.2 426500 35380 ?Sl   16:06   0:00 nm-applet
dpchrist1952  0.0  0.0   3240   644 pts/0S+   16:15   0:00 grep 
nm-applet



David

Debian 11 Xfce panel Network Manager applet has disappeared

[David Christensen]

debian-user:

I have a Dell Latitude E6520:

2024-04-14 04:28:39 dpchrist@laalaa ~
$ cat /etc/debian_version ; uname -a
11.9
Linux laalaa 5.10.0-28-amd64 #1 SMP Debian 5.10.209-2 (2024-01-31) 
x86_64 GNU/Linux


2024-04-14 04:34:40 dpchrist@laalaa ~
$ dpkg-query -l xfce4 network-manager network-manager-gnome
Desired=Unknown/Install/Remove/Purge/Hold
| 
Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend

|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name  Version  Architecture Description
+++-=---=
ii  network-manager   1.30.6-1+deb11u1 amd64network 
management framework (daemon and userspace tools)
ii  network-manager-gnome 1.20.0-3 amd64network 
management framework (GNOME frontend)
ii  xfce4 4.16 all  Meta-package for 
the Xfce Lightweight Desktop Environment



I have used the Xfce panel Network Manager applet for many years. 
Tonight, I noticed that it has disappeared (!).



But, the machine is connected to my LAN:

2024-04-14 05:24:10 root@laalaa ~
# ifconfig wlp3s0
wlp3s0: flags=4163  mtu 1500
inet 192.168.REDACTED  netmask 255.255.255.0  broadcast 
192.168.REDACTED

inet6 REDACTED  prefixlen 64  scopeid 0x20
ether REDACTED  txqueuelen 1000  (Ethernet)
RX packets 5786  bytes 2830592 (2.6 MiB)
RX errors 0  dropped 119  overruns 0  frame 0
TX packets 3897  bytes 518278 (506.1 KiB)
TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0


Looking in the the Xfce panel Application Menu, I am unable to find 
Network Manager.



Looking at the Debian WIKI page "NetworkManager":

https://wiki.debian.org/NetworkManager


It looks like the Network Manager daemon is running:

2024-04-14 04:32:49 dpchrist@laalaa ~
$ ps -A | grep -i network
828 ?00:00:00 NetworkManager


nm-applet(1) looks like the program I want (?).  Attempting to start it 
via a terminal has no effect:


2024-04-14 04:40:25 dpchrist@laalaa ~
$ which nm-applet
/usr/bin/nm-applet

2024-04-14 05:27:05 dpchrist@laalaa ~
$ nm-applet

2024-04-14 05:27:08 dpchrist@laalaa ~
$


RTFM nm-applet(1), it seems the desktop session manager is failing to 
start nm-applet(1) (?):


2024-04-14 04:58:49 dpchrist@laalaa ~
$ man nm-applet | cat
...
DESCRIPTION
   nm-applet  is  a GTK-based GUI applet to monitor network status
   and devices and to start and stop network  connections  managed
   by  NetworkManager.   nm-applet is normally started at login by
   the desktop session manager and does not need to be  run  manu-
   ally.   nm-applet conforms to the XDG System Tray specification
   and requires that the desktop environment provide a System Tray
   implementation in which the applet will be embedded.


I am unable to find relevant error messages under /var/log.


The network Connection Editor can be run via a terminal:

2024-04-14 04:55:29 dpchrist@laalaa ~
$ which nm-connection-editor
/usr/bin/nm-connection-editor

2024-04-14 04:55:38 dpchrist@laalaa ~
$ nm-connection-editor


Does anyone know why the Network Manager Xfce panel applet is missing, 
how to get it back, and/or how to start it some other way?



David

Re: HDD long-term data storage with ensured integrity

[David Christensen]

On 4/12/24 08:14, piorunz wrote:

On 10/04/2024 12:10, David Christensen wrote:

Those sound like some compelling features.


I believe the last time I tried Btrfs was Debian 9 (?).  I ran into
problems because I did not do the required manual maintenance
(rebalancing).  Does the Btrfs in Debian 11 or Debian 12 still require
manual maintenance?  If so, what and how often?


I don't do balance at all, it's not required.

Scrub is recommended, because it will detect any bit-rot due to hardware
errors on HDD media. It scans the entire surface of allocated sectors on
all drives. I do scrub usually monthly.



Thank you for the information.


David

Re: HDD long-term data storage with ensured integrity

[David Christensen]

On 4/10/24 08:49, Paul Leiber wrote:

Am 10.04.2024 um 13:10 schrieb David Christensen:
Does the Btrfs in Debian 11 or Debian 12 still require 
manual maintenance?  If so, what and how often?


Scrub and balance are actions which have been recommended. I am using 
btrfsmaintenance scripts [1][2] to automate this. I am doing a weekly 
balance and a monthly scrub. After some reading today, I am getting 
unsure if this is approach is correct, especially if balance is 
necessary anymore (it usually doesn't find anything to do anyway), so 
please take these periods with caution. My main message is that such 
operations can be automated using the linked scripts.


Best regards,

Paul

[1] https://packages.debian.org/bookworm/btrfsmaintenance
[2] https://github.com/kdave/btrfsmaintenance



Thank you.  Those scripts should be useful.


David

Re: HDD long-term data storage with ensured integrity

[David Christensen]

On 4/9/24 17:08, piorunz wrote:

On 02/04/2024 13:53, David Christensen wrote:


Does anyone have any comments or suggestions regarding how to use
magnetic hard disk drives, commodity x86 computers, and Debian for
long-term data storage with ensured integrity?


I use Btrfs, on all my systems, including some servers, with soft Raid1
and Raid10 modes (because these modes are considered stable and
production ready). I decided on Btrfs not ZFS, because Btrfs allows to
migrate drives on the fly while partition is live and heavily used,
replace them with different sizes and types, mixed capacities, change
Raid levels, change amount of drives too. I could go from single drive
to Raid10 on 4 drives and back while my data is 100% available at all 
times.

It saved my bacon many times, including hard checksum corruption on NVMe
drive which otherwise I would never know about. Thanks to Btrfs I
located the corrupted files, fixed them, got hardware replaced under
warranty.
Also helped with corrupted RAM: Btrfs just refused to save file because
saved copy couldn't match read checksum from the source due to RAM bit
flips. Diagnosed, then replaced memory, all good.
I like a lot when one of the drives get ATA reset for whatever reason,
and all other drives continue to read and write, I can continue using
the system for hours, if I even notice. Not possible in normal
circumstances without Raid. Once the problematic drive is back, or after
reboot if it's more serious, then I do "scrub" command and everything is
resynced again. If I don't do that, then Btrfs dynamically correct
checksum errors on the fly anyway.
And list goes on - I've been using Btrfs for last 5 years, not a single
problem to date, it survived hard resets, power losses, drive failures,
countless migrations.



Those sound like some compelling features.


I believe the last time I tried Btrfs was Debian 9 (?).  I ran into 
problems because I did not do the required manual maintenance 
(rebalancing).  Does the Btrfs in Debian 11 or Debian 12 still require 
manual maintenance?  If so, what and how often?




[1] https://github.com/openzfs/zfs/issues/15526

[2] https://github.com/openzfs/zfs/issues/15933


Problems reported here are from Linux kernel 6.5 and 6.7 on Gentoo
system. Does this even affects Debian Stable with 6.1 LTS?



I do not know.



--
With kindest regards, Piotr.

⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system
⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org/
⠈⠳⣄



David

Re: Why LVM

[David Christensen]

On 4/8/24 16:54, Stefan Monnier wrote:

If I have a hot-pluggable device (SD card, USB drive, hot-plug SATA/SAS
drive and rack, etc.), can I put LVM on it such that when the device is
connected to a Debian system with a graphical desktop (I use Xfce) an icon
is displayed on the desktop that I can interact with to display the file
systems in my file manager (Thunar)?


In the past: definitely not.  Currently: no idea.
I suspect not, because I think the behavior on disconnection is still
poor (you want to be extra careful to deactivate all the volumes on the
drive *before* removing it, otherwise they tend to linger "for ever").

I guess that's one area where partitions are still significantly better
than LVM.


 Stefan "who doesn't use much hot-plugging of mass storage"



Thank you for the clarification.  :-)


David

Re: Why LVM

[David Christensen]

On 4/8/24 14:08, Stefan Monnier wrote:

David Christensen [2024-04-08 11:28:04] wrote:

Why LVM?


Personally, I've been using LVM everywhere I can (i.e. everywhere
except on my OpenWRT router, tho I've also used LVM there back when my
router had an HDD.  I also use LVM on my 2GB USB rescue image).

To me the question is rather the reverse: why not?
I basically see it as a more flexible form of partitioning.

Even in the worst cases where I have a single LV volume, I appreciate
the fact that it forces me to name things, isolating me from issue
linked to predicting the name of the device and the issues that plague
UUIDs (the fact they're hard to remember, and that they're a bit too
magical/hidden for my taste, so they sometimes change when I don't want
them to and vice versa).


 Stefan



If I have a hot-pluggable device (SD card, USB drive, hot-plug SATA/SAS 
drive and rack, etc.), can I put LVM on it such that when the device is 
connected to a Debian system with a graphical desktop (I use Xfce) an 
icon is displayed on the desktop that I can interact with to display the 
file systems in my file manager (Thunar)?



David

Re: HDD long-term data storage with ensured integrity

[David Christensen]

On 4/8/24 13:04, Marc SCHAEFER wrote:

Hello,

On Mon, Apr 08, 2024 at 11:28:04AM -0700, David Christensen wrote:

So, an ext4 file system on an LVM logical volume?

Why LVM?  Are you implementing redundancy (RAID)?  Is your data larger than
a single disk (concatenation/ JBOD)?  Something else?


For off-site long-term offline archiving, no, I am not using RAID.

No, it's not LVM+md, just plain LVM for flexibility.

Typically I use 16 TB hard drives, and I tend to use one LV per data
source, the LV name being the data source and the date of the copy.
Or sometimes I just copy a raw volume (ext4 or something else)
to a LV.

With smaller drives (4 TB) I tend to not use LVM, just plain ext4 on the
raw disk.

I almost never use partitionning.

However, I tend to use luks encryption (per ext4 filesystem) when the
drives are stored off-site.  So it's either LVM -> LV -> LUKS -> ext4
or raw disk -> LUKS -> ext4.

You can find some of the scripts I use to automate this off-site
long-term archiving here:

https://git.alphanet.ch/gitweb/?p=various;a=tree;f=offsite-archival/LVM-LUKS



Thank you for the clarification.  :-)


David

Re: HDD long-term data storage with ensured integrity

[David Christensen]

On 4/8/24 02:38, Marc SCHAEFER wrote:

For offline storage:

On Tue, Apr 02, 2024 at 05:53:15AM -0700, David Christensen wrote:

Does anyone have any comments or suggestions regarding how to use magnetic
hard disk drives, commodity x86 computers, and Debian for long-term data
storage with ensured integrity?


I use LVM on ext4, and I add a MD5SUMS file at the root.

I then power up the drives at least once a year and check the MD5SUMS.

A simple CRC could also work, obviously.

So far, I have not detected MORE corruption with this method than the
drive ECC itself (current drives & buses are much better than they
used to be).  When I have errors detected, I replace the file with
another copy (I usually have multiple off-site copies, and sometimes
even on-site online copies, but not always).  When the errors add
up, it is time to buy another drive, usually after 5+ years or
even sometimes 10+ years.

So, just re-reading the content might be enough, once a year or so.

This is for HDD (for SDD I have no offline storage experience, it
could be shorter).



Thank you for the reply.


So, an ext4 file system on an LVM logical volume?


Why LVM?  Are you implementing redundancy (RAID)?  Is your data larger 
than a single disk (concatenation/ JBOD)?  Something else?



David

Re: readonly installer, (SOLVED)

[David Christensen]

On 4/3/24 19:05, Stefan Monnier wrote:

I have a 128 MB USB flash drive from back in the day that includes a write
protect switch.  There are few products today that offer that feature.


Side note: AFAIK this "write protect switch" doesn't prevent writing.
It just tells your card reader that you'd like to avoid writing to it.
Whether it ends up doing what you want depends on the hardware exposing
that info to the driver and the driver paying attention to it.


 Stefan



No, it is a USB flash drive with a write protect switch.  Here is a 
modern example:


https://www.amazon.com/dp/B00JJIE95G


David

Re: Debian ISOs on USB stick

[David Christensen]

On 4/3/24 05:56, Thomas Schmitt wrote:

Hi,

i read from bytes 2085412 to 2085479:
   "Info rrmation Syste rm VolumeSYSTEM~"
which is similar to the alterations of one of the USB sticks shown in
   https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1056998#35

The web knows about a Microsoft folder named "System Volume Information".
   
https://answers.microsoft.com/en-us/windows/forum/all/system-volume-information-what-is-it-and-what-is/3bc81844-0baa-46bd-9949-4efb4678b677
   "whenever I put my flash-drive or my micro sd adapter and sd card into
my windows 8.1 something called "System Volume Information" is always
getting added on."

So did you perhaps show this USB stick to a running MS-Windows system ?


Have a nice day :)

Thomas



It is possible the drive was inserted into a Windows computer.


If and when I need a newer d-i, perhaps I will put the ISO onto a USB 
flash drive, conduct more experiments, and post the results.



I apologize for blaming d-i for what might be Dell, Intel, BIOS/UEFI, 
Microsoft, and/or other bugs.



David

Re: readonly installer, (SOLVED)

[David Christensen]

On 4/3/24 08:16, David Wright wrote:

On Tue 02 Apr 2024 at 05:54:06 (-0700), David Christensen wrote:

On 4/1/24 11:35, DdB wrote:

Am 01.04.2024 um 18:52 schrieb David Christensen:

A bad USB flash drive would explain why you cannot boot the Debian
installer.  Please buy a good quality USB 3.0+ flash drive and try again.


A friend of mine just let me use an external CD-Drive with the netboot
image.


I thought about suggesting that in my last post, but did not want to
complicate things.  A key advantage of using a CD-R disc is that you
can verify the disc contents and/or checksum against the ISO and/or
checksum now and in the future.  This is not true for a USB flash
drive, because the Debian installer modifies the contents of the USB
flash drive when it runs.


If this troubles you, you can also use an SD card with a write-lock,
or a µSD card with a lock on the SD adaptor.

Check that the write-lock works with the logs when you plug it in,
or run fdisk/gdisk and immediately quit.

Cheers,
David.



I have a 128 MB USB flash drive from back in the day that includes a 
write protect switch.  There are few products today that offer that feature.



David

Re: HDD long-term data storage with ensured integrity

[David Christensen]

On 4/2/24 14:57, David Christensen wrote:
AIUI neither LVM nor ext4 have data and metadata checksum and correction 
features.  But, it should be possible to achieve such by including 
dm-integrity (for checksumming) and some form of RAID (for correction) 
in the storage stack.  I need to explore that possibility further.



I have RTFM dm-integrity before and it is still experimental.  I need 
something that is production ready:


https://manpages.debian.org/bookworm/cryptsetup-bin/cryptsetup.8.en.html

Authenticated disk encryption (EXPERIMENTAL)


David

Re: Debian ISOs on USB stick, was: SOLVED

[David Christensen]

On 4/3/24 03:36, David Christensen wrote:

On 4/3/24 00:30, Thomas Schmitt wrote:

Hi,

David Christensen wrote:
It's a relatively simple experiment to confirm that a USB flash drive 
with

d-i changes after the first boot.


This could still be
   https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1056998
where Lenovo BIOS and/or MS-Windows altered the USB stick.



Same for finding which bytes change.


I fail to find this particular info in
   Date: Tue, 2 Apr 2024 14:46:42 -0700
   From: David Christensen 
   Message-ID: 

If we have the exact ISO name (i.e. URL from where it stems) and the
byte address of the alteration, xorriso can find the affected file, if
any.

In case of bug #1056998 it was the EFI partition image 
/boot/grub/efi.img.

Mounting the altered and unaltered image files showed changes in the
FAT filesystem which point to the culprits Lenovo and Microsoft.


The other plausible way of altering the ISO image on the stick would be
adding a new partition.
The MBR partition table is part of the Debian ISO and thus part of the
checksummed area. Even if all other alterations happen after the end
of the checksummed ISO image, the changed partition table will cause the
Debian checksum to become invalid.

(I am not aware that Debian installer changes the table. If it does 
indeed

then this might be worth a new bug discussion.)


Have a nice day :)

Thomas



2024-04-03 03:29:18 root@laalaa /samba/dpchrist/iso/debian/11.3.0
# cmp --verbose debian-11.3.0-amd64-netinst.iso /dev/sdb
   2083201   0 377
   2083202   0 377
   2083203   0 377
   2085249   0 377
   2085250   0 377
   2085251   0 377
   2085409   0 102
   2085410   0  40
   2085412   0 111
   2085414   0 156
   2085416   0 146
   2085418   0 157
   2085420   0  17
   2085422   0 162
   2085423   0 162
   2085425   0 155
   2085427   0 141
   2085429   0 164
   2085431   0 151
   2085433   0 157
   2085437   0 156
   2085441   0   1
   2085442   0 123
   2085444   0 171
   2085446   0 163
   2085448   0 164
   2085450   0 145
   2085452   0  17
   2085454   0 162
   2085455   0 155
   2085457   0  40
   2085459   0 126
   2085461   0 157
   2085463   0 154
   2085465   0 165
   2085469   0 155
   2085471   0 145
   2085473   0 123
   2085474   0 131
   2085475   0 123
   2085476   0 124
   2085477   0 105
   2085478   0 115
   2085479   0 176
   2085480   0  61
   2085481   0  40
   2085482   0  40
   2085483   0  40
   2085484   0  26
   2085486   0 167
   2085487   0 174
   2085488   0 277
   2085489   0 235
   2085490   0 124
   2085491   0 235
   2085492   0 124
   2085495   0 175
   2085496   0 277
   2085497   0 235
   2085498   0 124
   2085500   0   5
   4719105   0  56
   4719106   0  40
   4719107   0  40
   4719108   0  40
   4719109   0  40
   4719110   0  40
   4719111   0  40
   4719112   0  40
   4719113   0  40
   4719114   0  40
   4719115   0  40
   4719116   0  20
   4719118   0 167
   4719119   0 174
   4719120   0 277
   4719121   0 235
   4719122   0 124
   4719123   0 235
   4719124   0 124
   4719127   0 175
   4719128   0 277
   4719129   0 235
   4719130   0 124
   4719132   0   5
   4719137   0  56
   4719138   0  56
   4719139   0  40
   4719140   0  40
   4719141   0  40
   4719142   0  40
   4719143   0  40
   4719144   0  40
   4719145   0  40
   4719146   0  40
   4719147   0  40
   4719148   0  20
   4719150   0 167
   4719151   0 174
   4719152   0 277
   4719153   0 235
   4719154   0 124
   4719155   0 235
   4719156   0 124
   4719159   0 175
   4719160   0 277
   4719161   0 235
   4719162   0 124
   4719169   0 102
   4719170   0 107
   4719172   0 165
   4719174   0 151
   4719176   0 144
   4719180   0  17
   4719182   0 377
   4719183   0 377
   4719184   0 377
   4719185   0 377
   4719186   0 377
   4719187   0 377
   4719188   0 377
   4719189   0 377
   4719190   0 377
   4719191   0 377
   4719192   0 377
   4719193   0 377
   4719194   0 377
   4719197   0 377
   4719198   0 377
   4719199   0 377
   4719200   0 377
   4719201   0   1
   4719202   0 111
   4719204   0 156
   4719206   0 144
   4719208   0 145
   4719210   0 170
   4719212   0  17
   4719214   0 377
   4719215   0 145
   4719217   0 162
   4719219   0 126
   4719221   0 157
   4719223   0 154
   4719225   0 165
   4719229   0 155
   4719231   0 145
   4719233   0 111
   4719234   0 116
   4719235   0 104
   4719236   0 105
   4719237   0 130
   4719238   0 105
   4719239   0 176
   4719240   0  61
   4719241   0  40
   4719242   0  40
   4719243   0  40
   4719244   0  40
   4719246   0 171
   4719247   0 174
   4719248   0 277
   4719249   0 235
   4719250   0 124
   4719251   0 235
   4719252   0 124
   4719255   0 175
   4719256   0 277
   4719257   0 235
   4719258   0 124
   4719259   0   1
   4719260   0   5
   4719261   0 114
   4721153   0 173
   4721155   0  71
   4721157   0 101
   4721159   0  65
   4721161   0 104
   4721163   0 101
   4721165   0 106
   4721167   0  65
   4721169   0  67
   4721171   0  55

Re: Debian ISOs on USB stick, was: SOLVED

[David Christensen]

On 4/3/24 00:30, Thomas Schmitt wrote:

Hi,

David Christensen wrote:

It's a relatively simple experiment to confirm that a USB flash drive with
d-i changes after the first boot.


This could still be
   https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1056998
where Lenovo BIOS and/or MS-Windows altered the USB stick.



Same for finding which bytes change.


I fail to find this particular info in
   Date: Tue, 2 Apr 2024 14:46:42 -0700
   From: David Christensen 
   Message-ID: 

If we have the exact ISO name (i.e. URL from where it stems) and the
byte address of the alteration, xorriso can find the affected file, if
any.

In case of bug #1056998 it was the EFI partition image /boot/grub/efi.img.
Mounting the altered and unaltered image files showed changes in the
FAT filesystem which point to the culprits Lenovo and Microsoft.


The other plausible way of altering the ISO image on the stick would be
adding a new partition.
The MBR partition table is part of the Debian ISO and thus part of the
checksummed area. Even if all other alterations happen after the end
of the checksummed ISO image, the changed partition table will cause the
Debian checksum to become invalid.

(I am not aware that Debian installer changes the table. If it does indeed
then this might be worth a new bug discussion.)


Have a nice day :)

Thomas



2024-04-03 03:29:18 root@laalaa /samba/dpchrist/iso/debian/11.3.0
# cmp --verbose debian-11.3.0-amd64-netinst.iso /dev/sdb
  2083201   0 377
  2083202   0 377
  2083203   0 377
  2085249   0 377
  2085250   0 377
  2085251   0 377
  2085409   0 102
  2085410   0  40
  2085412   0 111
  2085414   0 156
  2085416   0 146
  2085418   0 157
  2085420   0  17
  2085422   0 162
  2085423   0 162
  2085425   0 155
  2085427   0 141
  2085429   0 164
  2085431   0 151
  2085433   0 157
  2085437   0 156
  2085441   0   1
  2085442   0 123
  2085444   0 171
  2085446   0 163
  2085448   0 164
  2085450   0 145
  2085452   0  17
  2085454   0 162
  2085455   0 155
  2085457   0  40
  2085459   0 126
  2085461   0 157
  2085463   0 154
  2085465   0 165
  2085469   0 155
  2085471   0 145
  2085473   0 123
  2085474   0 131
  2085475   0 123
  2085476   0 124
  2085477   0 105
  2085478   0 115
  2085479   0 176
  2085480   0  61
  2085481   0  40
  2085482   0  40
  2085483   0  40
  2085484   0  26
  2085486   0 167
  2085487   0 174
  2085488   0 277
  2085489   0 235
  2085490   0 124
  2085491   0 235
  2085492   0 124
  2085495   0 175
  2085496   0 277
  2085497   0 235
  2085498   0 124
  2085500   0   5
  4719105   0  56
  4719106   0  40
  4719107   0  40
  4719108   0  40
  4719109   0  40
  4719110   0  40
  4719111   0  40
  4719112   0  40
  4719113   0  40
  4719114   0  40
  4719115   0  40
  4719116   0  20
  4719118   0 167
  4719119   0 174
  4719120   0 277
  4719121   0 235
  4719122   0 124
  4719123   0 235
  4719124   0 124
  4719127   0 175
  4719128   0 277
  4719129   0 235
  4719130   0 124
  4719132   0   5
  4719137   0  56
  4719138   0  56
  4719139   0  40
  4719140   0  40
  4719141   0  40
  4719142   0  40
  4719143   0  40
  4719144   0  40
  4719145   0  40
  4719146   0  40
  4719147   0  40
  4719148   0  20
  4719150   0 167
  4719151   0 174
  4719152   0 277
  4719153   0 235
  4719154   0 124
  4719155   0 235
  4719156   0 124
  4719159   0 175
  4719160   0 277
  4719161   0 235
  4719162   0 124
  4719169   0 102
  4719170   0 107
  4719172   0 165
  4719174   0 151
  4719176   0 144
  4719180   0  17
  4719182   0 377
  4719183   0 377
  4719184   0 377
  4719185   0 377
  4719186   0 377
  4719187   0 377
  4719188   0 377
  4719189   0 377
  4719190   0 377
  4719191   0 377
  4719192   0 377
  4719193   0 377
  4719194   0 377
  4719197   0 377
  4719198   0 377
  4719199   0 377
  4719200   0 377
  4719201   0   1
  4719202   0 111
  4719204   0 156
  4719206   0 144
  4719208   0 145
  4719210   0 170
  4719212   0  17
  4719214   0 377
  4719215   0 145
  4719217   0 162
  4719219   0 126
  4719221   0 157
  4719223   0 154
  4719225   0 165
  4719229   0 155
  4719231   0 145
  4719233   0 111
  4719234   0 116
  4719235   0 104
  4719236   0 105
  4719237   0 130
  4719238   0 105
  4719239   0 176
  4719240   0  61
  4719241   0  40
  4719242   0  40
  4719243   0  40
  4719244   0  40
  4719246   0 171
  4719247   0 174
  4719248   0 277
  4719249   0 235
  4719250   0 124
  4719251   0 235
  4719252   0 124
  4719255   0 175
  4719256   0 277
  4719257   0 235
  4719258   0 124
  4719259   0   1
  4719260   0   5
  4719261   0 114
  4721153   0 173
  4721155   0  71
  4721157   0 101
  4721159   0  65
  4721161   0 104
  4721163   0 101
  4721165   0 106
  4721167   0  65
  4721169   0  67
  4721171   0  55
  4721173   0  71
  4721175   0 102
  4721177   0  63
  4721179   0  61
  4721181   0  55
  4721183   0  64
  4721185   0  71
  4721187   0  62
  4721189   0 105
  4721191   0  55
  4721193   0 102
  4721195   0  66
  4721197   0 105

Re: Debian ISOs on USB stick, was: SOLVED

[David Christensen]

On 4/2/24 08:56, Thomas Schmitt wrote:

Hi,

David Christensen wrote:

the Debian installer modifies the contents of the USB flash drive when
it runs.


Do you mean inside the range of the ISO image or outside by creating a
new partition ?


songbird wrote:

if it is an iso image copied to the USB stick it should not
be modified if you haven't somehow told the installer to
install the system to that USB stick (somehow).


There are other parties which feel entitled to operate on the EFI System
Partition of a USB stick.
In
   https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1056998
we found that Lenovo Thinkpad firmware created directories for storing
an empty file named "/efi/Lenovo/BIOS/SelfHealing.fd" and that MS-Windows
created a 12-byte file named "/System Volume Information/WPSettings.dat"
when it had contact with the USB stick.



i guess if you wanted to be really sure you could mount it read-only.


I think it's the installer which mounts the ISO 9660 filesystem.
Whatever, the Linux kernel has no regular means to alter an ISO 9660
filesystem. Neither kernel nor Debain installer will be so daring to
operate with byte level commands on that filesystem.

But the FAT filesystem in file /boot/grub/efi.img of the ISO 9660
filesystem in debian-12.*-amd64-netinst.iso is advertised by the partition
table of the image and thus attracts vermin.


Have a nice day :)

Thomas



Please see my reply to songbird.


It's a relatively simple experiment to confirm that a USB flash drive 
with d-i changes after the first boot.  Same for finding which bytes 
change.  The challenge is figuring out what performed the change(s) and 
why.  I assumed it was d-i, but no longer own 64-bit BIOS-only computers 
to confirm.



David

Re: HDD long-term data storage with ensured integrity

[David Christensen]

On 4/2/24 06:55, Stefan Monnier wrote:

The most obvious alternative to ZFS on Debian would be Btrfs.  Does anyone
have any comments or suggestions regarding Btrfs and data corruption bugs,
concurrency, CMM level, PSP, etc.?


If you're worried about such things, I'd think "the most obvious
alternative" is LVM+ext4.  Both Btrfs and ZFS share the same underlying
problem: more features => more code => more bugs.


 Stefan



AIUI neither LVM nor ext4 have data and metadata checksum and correction 
features.  But, it should be possible to achieve such by including 
dm-integrity (for checksumming) and some form of RAID (for correction) 
in the storage stack.  I need to explore that possibility further.



David

Re: SOLVED

[David Christensen]

On 4/2/24 07:55, songbird wrote:

David Christensen wrote:

I thought about suggesting that in my last post, but did not want to
complicate things.  A key advantage of using a CD-R disc is that you can
verify the disc contents and/or checksum against the ISO and/or checksum
now and in the future.  This is not true for a USB flash drive, because
the Debian installer modifies the contents of the USB flash drive when
it runs.


   if it is an iso image copied to the USB stick it should not
be modified if you haven't somehow told the installer to
install the system to that USB stick (somehow).

   i guess if you wanted to be really sure you could mount it
read-only.


   songbird



I used to think that the d-i ran in memory when booted from read-write 
media, but discovered otherwise several years ago.



I previously downloaded debian-11.3.0-amd64-netinst.iso and SHA512SUMS:

2022-04-29 22:16:19 dpchrist@tinkywinky 
~/samba/dpchrist/iso/debian/11.3.0

$ ls -l
total 380414
-rwxr-xr-x 1 dpchrist dpchrist   494 Apr 28 21:04 SHA512SUMS
-rwxr-xr-x 1 dpchrist dpchrist   833 Apr 28 21:04 
SHA512SUMS.sign
-rwxr-xr-x 1 dpchrist dpchrist 396361728 Apr 28 21:05 
debian-11.3.0-amd64-netinst.iso



I verified the checksum of the ISO file:

2022-04-29 22:17:17 dpchrist@tinkywinky 
~/samba/dpchrist/iso/debian/11.3.0

$ grep debian-11.3.0-amd64-netinst.iso SHA512SUMS | sha512sum -c
debian-11.3.0-amd64-netinst.iso: OK


I burned the ISO to a zeroed USB flash drive:

2022-04-29 22:39:25 root@tinkywinky 
~/hardware/adata/usb-flash-drive/REDACTED
# time dd 
if=/home/dpchrist/samba/dpchrist/iso/debian/11.3.0/debian-11.3.0-amd64-netinst.iso 
of=/dev/disk/by-id/usb-ADATA_USB_Flash_Drive_1392303332110024-0\:0 bs=1M 
iflag=fullblock oflag=sync,noatime status=progress

394264576 bytes (394 MB, 376 MiB) copied, 76.1204 s, 5.2 MB/s
378+0 records in
378+0 records out
396361728 bytes (396 MB, 378 MiB) copied, 76.5701 s, 5.2 MB/s

real1m16.582s
user0m0.012s
sys 0m0.584s


I computed the checksum of the relevant blocks of the USB flash drive:

2022-04-29 22:43:56 root@tinkywinky 
~/hardware/adata/usb-flash-drive/REDACTED
# time dd 
if=/dev/disk/by-id/usb-ADATA_USB_Flash_Drive_REDACTED-0\:0 bs=1M 
count=378 iflag=fullblock | sha512sum

378+0 records in
378+0 records out
396361728 bytes (396 MB, 378 MiB) copied, 25.0641 s, 15.8 MB/s

2810f894afab9ac2631ddd097599761c1481b85e629d6a3197fe1488713af048d37241eb85def681ba86e62b406dd9b891ee1ae7915416335b6bb000d57c1e53 
 -


real0m25.068s
user0m3.468s
sys 0m0.720s


The USB flash drive checksum matched the value stored in SHA512SUMS:

2022-04-29 22:44:58 root@tinkywinky 
~/hardware/adata/usb-flash-drive/REDACTED
# grep 
2810f894afab9ac2631ddd097599761c1481b85e629d6a3197fe1488713af048d37241eb85def681ba86e62b406dd9b891ee1ae7915416335b6bb000d57c1e53 
/home/dpchrist/samba/dpchrist/iso/debian/11.3.0/SHA512SUMS


2810f894afab9ac2631ddd097599761c1481b85e629d6a3197fe1488713af048d37241eb85def681ba86e62b406dd9b891ee1ae7915416335b6bb000d57c1e53 
 debian-11.3.0-amd64-netinst.iso



I have since used the USB flash drive to install Debian onto one or more 
computers.



If I compute the checksum of the relevant blocks of the USB flash drive 
today:


2024-04-02 14:32:43 root@laalaa ~
# time dd if=/dev/disk/by-id/usb-ADATA_USB_Flash_Drive_REDACTED-0\:0 
bs=1M count=378 iflag=fullblock | sha512sum

378+0 records in
378+0 records out
6cbb7e54fccdf550cbb5535d7dd9357513b36b767f0aaa550b1d58b37ef827c881036dcf47b3c377d196c4e77d14c786a6dd975aa558a11a81cf9ae107062abc 
 -

396361728 bytes (396 MB, 378 MiB) copied, 27.7592 s, 14.3 MB/s

real0m27.766s
user0m4.134s
sys 0m1.380s


The checksum has changed because d-i modified the USB flash drive.


I previously confirmed this behavior years ago after booting a d-i USB 
flash drive only once.



You must use read-only media (e.g. CD-R) if you want to be able to 
verify d-i after use.



David

Re: SOLVED

[David Christensen]

On 4/1/24 11:35, DdB wrote:

Am 01.04.2024 um 18:52 schrieb David Christensen:

A bad USB flash drive would explain why you cannot boot the Debian
installer.  Please buy a good quality USB 3.0+ flash drive and try again.


A friend of mine just let me use an external CD-Drive with the netboot
image. 



I thought about suggesting that in my last post, but did not want to 
complicate things.  A key advantage of using a CD-R disc is that you can 
verify the disc contents and/or checksum against the ISO and/or checksum 
now and in the future.  This is not true for a USB flash drive, because 
the Debian installer modifies the contents of the USB flash drive when 
it runs.




This is already the third time, i am restarting the installation
process, due to my false assumptions about the intelligence within the
installer.

The last time, i was quite happy until i came to notice, that partitions
were not aligned with physical sector boundaries, which i assumed would
be elementary best practice.



I chose manual partitioning and the Debian installer aligned the 
partitions to 2**20 byte boundaries:


2024-04-02 04:07:16 root@laalaa ~
# cat /etc/debian_version ; uname -a
11.9
Linux laalaa 5.10.0-28-amd64 #1 SMP Debian 5.10.209-2 (2024-01-31) 
x86_64 GNU/Linux


2024-04-02 04:08:18 root@laalaa ~
# fdisk -l /dev/sda
Disk /dev/sda: 55.9 GiB, 60022480896 bytes, 117231408 sectors
Disk model: INTEL SSDSC2CW06
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x544032f5

Device BootStart   End  Sectors  Size Id Type
/dev/sda1  *2048   1953791  1951744  953M 83 Linux
/dev/sda21953792   3907583  1953792  954M 83 Linux
/dev/sda33907584  29298687 25391104 12.1G 83 Linux
/dev/sda4   29298688 117229567 87930880 41.9G 83 Linux




But apart from losing some of my illusions the hard way, all is well.
A big thank you to all the crowd offering suggestions and encouragement.

so long, DdB



I'm glad you were able to install Debian.  :-)


David

HDD long-term data storage with ensured integrity

[David Christensen]

On 3/31/24 02:18, DdB wrote:
> i intend to create a huge backup server from some oldish hardware.
> Hardware has been partly refurbished and offers 1 SSD + 8 HDD on a
> 6core Intel with 64 GB RAM. ... the [Debian] installer ... aborts.

On 4/1/24 11:35, DdB wrote:
> A friend of mine just let me use an external CD-Drive with the netboot
> image. ... all is well.


Now you get to solve the same problem I have been stuck on since last 
November -- how to use those HDD's.



ZFS has been my bulk storage solution of choice for the past ~4 years, 
but the recent data corruption bugs [1, 2] have me worried.  From a 
technical perspective, it's about incorrect concurrent execution of GNU 
cp(1), Linux, and/or OpenZFS.  From a management perspective, it's about 
Capability Maturity Model (CMM) [3] and Programming Systems Product 
(PSP) [4].



The most obvious alternative to ZFS on Debian would be Btrfs.  Does 
anyone have any comments or suggestions regarding Btrfs and data 
corruption bugs, concurrency, CMM level, PSP, etc.?



Does anyone have any comments or suggestions regarding how to use 
magnetic hard disk drives, commodity x86 computers, and Debian for 
long-term data storage with ensured integrity?



David


[1] https://github.com/openzfs/zfs/issues/15526

[2] https://github.com/openzfs/zfs/issues/15933

[3] https://en.wikipedia.org/wiki/Capability_maturity_model

[4] https://en.wikipedia.org/wiki/The_Mythical_Man-Month

Re: help needed to get a bookworm install to succeed

[David Christensen]

On 4/1/24 03:10, DdB wrote:

Am 01.04.2024 um 07:44 schrieb David Christensen:

Please post a console session that identifies the ISO you are using,
verifies the checksum, burns the ISO to a USB flash drive, and compares
the ISO against the flash drive.


Ok, in the meantime, i came to similar conclusions and found that the
USB-stick i was using, had consistent read errors at the first 2
gigabytes after having been used for years as memory extension in my
router. Fixed that and will replace the stick.



A bad USB flash drive would explain why you cannot boot the Debian 
installer.  Please buy a good quality USB 3.0+ flash drive and try again.



David

Re: help needed to get a bookworm install to succeed

[David Christensen]

On 3/31/24 02:18, DdB wrote:

Hello list,

i intend to create a huge backup server from some oldish hardware.
Hardware has been partly refurbished and offers 1 SSD + 8 HDD on a 6core
Intel with 64 GB RAM.
Already before assembling the hardware, grub was working from the SSD,
which got lvm partitioning and is basically empty. As i have no working
CD drive nor can this old machine boot from USB, i put an ISO for
bookworm onto an lvm-LV. Using grub, i can manually boot from that ISO
and see the first installer screens. But after asking some questions,
the installer wants to mount the external media (ISO), and does not find
it on sd[a-z], then aborts.
By switching to Desktop 4, i can see the attempt to search for the
"CD"-drive, which is bound to fail.
I am not familiar with the very restricted shell, that is available from
the installer (busybox) and have not yet found an approach to circumvent
my problems. i would like to use the installer, as debootstrapping would
necessitate alot more knowledge than mine.

Suggestions are welcome :-)
DdB




A computer with a 6-core processor, 64 GB memory, and 9 drive bays/ 
ports that cannot boot USB?  That does not make sense.



Please post a console session that identifies the ISO you are using, 
verifies the checksum, burns the ISO to a USB flash drive, and compares 
the ISO against the flash drive.



Then insert the USB flash drive into a USB port on the the target 
computer, power up and enter Setup, reset the settings to factory 
defaults, enable USB booting, set the USB flash drive as the first boot 
device, save, and exit.  The Debian installer should then boot.



David

Re: Debian 12.5 up-to-date Xfce, Firefox clings to USB stick

[David Christensen]

On 3/30/24 08:17, Antti-Pekka Känsälä wrote:

What could be the deal, when Firefox tries to stop me from unmounting a
stick, after I've accessed files on it through Firefox?  I worry about my
stick security.  Thanks.



Linux knows what files are open on each file system.  If you try to 
unmount a file system with open files or eject a mounted USB drive with 
open files, Linux will refuse and your desktop environment will display 
a suitable error dialog.  This is a feature, not a bug.



The solution is to close all the files on the file system, and then 
unmount it.



David

Re: Debian 12.5.0 amd64 and OpenZFS bug #15526

[David Christensen]

On 3/25/24 15:05, Gareth Evans wrote:

On Fri 22/03/2024 at 21:01, Gareth Evans  wrote:

As anyone interested can see from the ref to #15933 in the below, there seems 
to have been considerable effort in getting to grips with this bug (actually 
multiple bugs), and it looks like a fix may be forthcoming, though not sure at 
the time of writing if there may be some further polishing first

https://github.com/openzfs/zfs/pull/16019


https://github.com/openzfs/zfs/issues/15933

is now closed as completed with fix

https://github.com/openzfs/zfs/commit/102b468b5e190973fbaee6fe682727eb33079811

which for the moment necessarily adds synchronous writes.

FYI.
Gareth



Thank you for keeping an eye on this.


Looking at the github commit, the C code makes me worry -- it does not 
appear to use traditional C/C++ thread-safe programming techniques such 
as I learned in CS and used when I did systems programming (e.g. guard 
functions, critical sections, locks, semaphores, etc.).  Do I need to 
look at more enclosing code to see such, are those techniques missing, 
are there some newer techniques I do not understand, or something else?



David

Re: trying to parse lines from an awkwardly formatted HAR file ...

[David Christensen]

On 3/22/24 22:53, Albretch Mueller wrote:

out of a HAR file containing lots of obfuscating js cr@p and all kinds of
nonsense I was able to extract line looking like:

var00='{\"index\":\"prod-h-006\",\"fields\":{\"identifier\":\"bub_gb_O2EAMAAJ\",\"title\":\"Die
Wissenschaft vom subjectiven Geist\",\"creator\":[\"Karl Rosenkranz\",
\"Mr. ABC123\"],\"collection\":[\"europeanlibraries\",
\"americana\"],\"year\":1843,\"language\":[\"German\"],\"item_size\":797368506},\"_score\":[50.629513]}'
echo "// __ \$var00: |$var00|"

The final result that I need would look like:
o
var02='bub_gb_O2EAMAAJ|Die Wissenschaft vom subjectiven Geist|["Karl
Rosenkranz", "Mr. ABC123"]|["europeanlibraries",
"americana"]|1843|["German"]|797368506|[50.629513]'
echo "// __ \$var02: |$var02|"

I have tried substring substitution, sed et tr to no avail.

lbrtchx



My daily driver:

2024-03-23 04:02:27 dpchrist@laalaa 
~/sandbox/perl/debian-users/20240322-2253-albretch-mueller

$ cat /etc/debian_version; uname -a; perl -v | head -n 2 | grep .
11.9
Linux laalaa 5.10.0-28-amd64 #1 SMP Debian 5.10.209-2 (2024-01-31) 
x86_64 GNU/Linux
This is perl 5, version 32, subversion 1 (v5.32.1) built for 
x86_64-linux-gnu-thread-multi



Put the JSON into a data file, one record per line (my mailer is 
line-wrapping data.json -- it contains two lines):


2024-03-23 04:22:20 dpchrist@laalaa 
~/sandbox/perl/debian-users/20240322-2253-albretch-mueller

$ cat data.json
{"index":"prod-h-006","fields":{"identifier":"bub_gb_O2EAMAAJ","title":"Die 
Wissenschaft vom subjectiven Geist","creator":["Karl Rosenkranz", "Mr. 
ABC123"],"collection":["europeanlibraries", 
"americana"],"year":1843,"language":["German"],"item_size":797368506},"_score":[50.629513]}
{"index":"prod-h-007","fields":{"identifier":"abc_de_12FGHIJKLMNO","title":"My 
Title","creator":["Some Body", "Somebody 
Else"],"collection":["europeanlibraries", 
"americana"],"year":2024,"language":["English"],"item_size":1234567890},"_score":[12.345678]}



A Perl script to read newline-delimited JSON records and pretty print each:

2024-03-23 04:28:59 dpchrist@laalaa 
~/sandbox/perl/debian-users/20240322-2253-albretch-mueller

$ cat munge-json
#!/usr/bin/perl
# $Id: munge-json,v 1.3 2024/03/23 11:28:58 dpchrist Exp $
# Refer to debian-user 3/22/24 22:53 Albretch Mueller
# "trying to parse lines from an awkwardly formatted HAR file"
# by David Paul Christensen dpchr...@holgerdanske.com
# Public Domain
use strict;
use warnings;
use Data::Dumper;
use JSON;
use Getopt::Long;
$Data::Dumper::Sortkeys = 1;
my $debug;
GetOptions('debug|d' => \$debug) or die;
while (<>) {
my $rh = decode_json $_;
print Data::Dumper->Dump([$rh], [qw(rh)]) if $debug;
print
join('|',
$rh->{fields}{identifier},
$rh->{fields}{title},
'["' .  join('", "', @{$rh->{fields}{creator}}) . '"]',
'["' .  join('", "', @{$rh->{fields}{collection}}) . '"]',
$rh->{fields}{year},
'["' .  join('", "', @{$rh->{fields}{language}}) . '"]',
$rh->{fields}{item_size},
'[' .  join(', ', @{$rh->{_score}}) . ']',
), "\n";
}   


Run the script as a Unix filter:

2024-03-23 04:30:16 dpchrist@laalaa 
~/sandbox/perl/debian-users/20240322-2253-albretch-mueller

$ ./munge-json data.json
bub_gb_O2EAMAAJ|Die Wissenschaft vom subjectiven Geist|["Karl 
Rosenkranz", "Mr. ABC123"]|["europeanlibraries", 
"americana"]|1843|["German"]|797368506|[50.629513]
abc_de_12FGHIJKLMNO|My Title|["Some Body", "Somebody 
Else"]|["europeanlibraries", 
"americana"]|2024|["English"]|1234567890|[12.345678]


2024-03-23 04:30:18 dpchrist@laalaa 
~/sandbox/perl/debian-users/20240322-2253-albretch-mueller

$ cat data.json | ./munge-json
bub_gb_O2EAMAAJ|Die Wissenschaft vom subjectiven Geist|["Karl 
Rosenkranz", "Mr. ABC123"]|["europeanlibraries", 
"americana"]|1843|["German"]|797368506|[50.629513]
abc_de_12FGHIJKLMNO|My Title|["Some Body", "Somebody 
Else"]|["europeanlibraries", 
"americana"]|2024|["English"]|1234567890|[12.345678]



David

Re: electrons/the Internet doesn't like … that I like to eat raw garlic, ...

[David Christensen]

On 3/4/24 16:06, David Wright wrote:

On Mon 04 Mar 2024 at 12:36:54 (-0800), David Christensen wrote:

On 3/4/24 08:37, Albretch Mueller wrote:



_LINK="https://christuniversity.in/uploads/course/E_21-25_Lateral
Entry(1)_20210618043317.pdf"


I ignored the filename, and pasted https://christuniversity.in/uploads/course/
into FF. Here's the text copy/pasted off the page that was displayed.
It was accompanied by the image that is displayed at
https://christuniversity.in/images/cour-btch-bnnr.jpg
...
Just a data point.



Testing ping again:

2024-03-04 17:31:14 dpchrist@laalaa ~
$ ping -c 3 christuniversity.in
PING christuniversity.in (111.93.136.229) 56(84) bytes of data.
64 bytes from 111.93.136.229 (111.93.136.229): icmp_seq=1 ttl=49 time=273 ms
64 bytes from 111.93.136.229 (111.93.136.229): icmp_seq=2 ttl=49 time=272 ms
64 bytes from 111.93.136.229 (111.93.136.229): icmp_seq=3 ttl=49 time=273 ms

--- christuniversity.in ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 271.914/272.406/272.725/0.352 ms

2024-03-04 17:31:23 dpchrist@laalaa ~
$ ping -c 3 103.105.225.131
PING 103.105.225.131 (103.105.225.131) 56(84) bytes of data.

--- 103.105.225.131 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 2060ms

2024-03-04 17:31:41 dpchrist@laalaa ~
$ ping -c 3 111.93.136.229
PING 111.93.136.229 (111.93.136.229) 56(84) bytes of data.
64 bytes from 111.93.136.229: icmp_seq=1 ttl=49 time=277 ms
64 bytes from 111.93.136.229: icmp_seq=2 ttl=49 time=280 ms
64 bytes from 111.93.136.229: icmp_seq=3 ttl=49 time=276 ms

--- 111.93.136.229 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 276.118/277.636/280.041/1.719 ms


So, same results as before -- one IP works, the other does not, and I 
got lucky with the FQDN (a previous test got the wrong IPv4 address and 
timed out).



Testing Firefox:

https://christuniversity.in/uploads/course/

https://103.105.225.131/uploads/course/

https://111.93.136.229/uploads/course/


All three time out.


Doing whois searches on the A record IP addresses:

1.  https://www.whois.com/whois/103.105.225.131


The IPv4 address holder appears to be a small ISP with 4 @ IPv4 class C 
ranges (1,024 addresses).  It appears nothing is connected to the 
christuniversity.in IPv4 address.



2. https://www.whois.com/whois/111.93.136.229


The IPv4 address holder appears to be a larger ISP with 1 @ IPv4 class B 
range (65,535 addresses).  It appears there is a host connected to the 
christuniversity.in IPv4 address, but I cannot connect to its web server.



STFW for information about DNS A (Address) records, I see:

https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/

What is a DNS A record?
...
The vast majority of websites only have one A record, but it is possible 
to have several. Some higher profile websites will have several 
different A records as part of a technique called round robin load 
balancing, which can distribute request traffic to one of several IP 
addresses, each hosting identical content.



So, two DNS A records for the same FQDN is allowed and can be useful.


Searching for all DNS records for christuniversity.in :

https://www.whatsmydns.net/dns-lookup?query=christuniversity.in=opendns


I see the two A (address) records that we have been discussing:

id 21430, opcode QUERY, rcode NOERROR, flags QR RD RA
;QUESTION
christuniversity.in. IN A
;ANSWER
christuniversity.in. 60 IN A 111.93.136.229
christuniversity.in. 60 IN A 103.105.225.131
;AUTHORITY
;ADDITIONAL


I see five MX (mail exchanger) records:

id 52477, opcode QUERY, rcode NOERROR, flags QR RD RA
;QUESTION
christuniversity.in. IN MX
;ANSWER
christuniversity.in. 60 IN MX 1 aspmx.l.google.com.
christuniversity.in. 60 IN MX 5 alt1.aspmx.l.google.com.
christuniversity.in. 60 IN MX 5 alt2.aspmx.l.google.com.
christuniversity.in. 60 IN MX 10 alt3.aspmx.l.google.com.
christuniversity.in. 60 IN MX 10 alt4.aspmx.l.google.com.
;AUTHORITY
;ADDITIONAL


I see two NS (nameserver) records:

id 57399, opcode QUERY, rcode NOERROR, flags QR RD RA
;QUESTION
christuniversity.in. IN NS
;ANSWER
christuniversity.in. 60 IN NS ns1.christuniversity.in.
christuniversity.in. 60 IN NS ns2.christuniversity.in.
;AUTHORITY
;ADDITIONAL


I find it strange that there are no A records for:

ns1.christuniversity.in
ns2.christuniversity.in


And yet dig(1) can find them:

2024-03-04 17:56:59 dpchrist@laalaa ~
$ dig @9.9.9.9 ns1.christuniversity.in

; <<>> DiG 9.16.48-Debian <<>> @9.9.9.9 ns1.christuniversity.in
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 40331
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;ns1.christuniversity.in.   IN  A

;; ANSWER SECTION:
ns1.christuniver

Re: resolv.conf (was Re: electrons/the Internet [racism redacted])

[David Christensen]

On 3/4/24 13:11, Greg Wooledge wrote:

On Mon, Mar 04, 2024 at 12:36:54PM -0800, David Christensen wrote:

I believe Debian rewrites /etc/resolv.conf on every boot.


This is not correct.  It's *partly* correct if you ignore a lot of
complicating factors.

Short version: read <https://wiki.debian.org/resolv.conf>.

Long version follows:

If you have a static network interface configuration, defined in
/etc/network/interfaces, with no DHCP client, no VPN stuff going on,
etc. then your /etc/resolv.conf will not be changed.  You can edit the
file and put whatever you want in it, and it'll remain as you wish it
to be.

Unfortunately, almost *nobody* has a setup like this any more.

In a more typical environment, you get your IP address via DHCP, which
means you're running a DHCP client daemon.  Most DHCP client daemons
will rewrite the /etc/resolv.conf file every time they refresh their
DHCP lease.  This may indeed happen at boot time, but it'll also happen
a couple times a day during normal operations.

So, a simple instruction like "edit /etc/resolv.conf" is no longer
possible.  Even worse, there's no single *alternative* either.  You can't
even say "do ___ instead".

To put the correct values into your /etc/resolv.conf file nowdays, you
have to select a *strategy*.  You need to find an indirect way to put
the right content into some *other* place, in such a way that it will
eventually find its way into /etc/resolv.conf every time the file is
rewritten.  And there are *lots* of strategies that will work, so you
can't even say "obviously this one is best".  Life is not that simple.

Or, you could use chattr +i to make the /etc/resolv.conf file immutable,
so DHCP clients and other programs cannot overwrite it.

Either way, you take ownership of whatever strategy you decide to use,
together with its pros and cons.  You'll have to understand that on *this*
system, you went with *this* strategy, and remember where to put your
changes, and how to make them.  Or at the very least, you'll need to be
*aware* of all the strategies you've got in play on all of your systems,
and know how to identify which one is in use on any given system.



Thank you for the clarification.  Thankfully, my gateway DHCP server and 
my Debian instances work together.



David

Re: electrons/the Internet doesn't like question authority niggahs?, or is it that I like to eat raw garlic, ...

[David Christensen]

On 3/4/24 08:37, Albretch Mueller wrote:





Yes, networking problems are infuriating.



Something that shouldn't be happening at all is that after I use
traceroute once, it doesn't work again and my Internet access speed
describes like a sinus curve which amplitude remains for the most part
under 16KiB per second and for more than one second as 0B per second.

_LINK="https://christuniversity.in/uploads/course/E_21-25_Lateral
Entry(1)_20210618043317.pdf"



I have AT Internet service in Tracy, California.


My daily driver is:

2024-03-04 10:47:12 dpchrist@laalaa ~
$ cat /etc/debian_version ; uname -a
11.9
Linux laalaa 5.10.0-28-amd64 #1 SMP Debian 5.10.209-2 (2024-01-31) 
x86_64 GNU/Linux



When I click the above link in my mail client (Thunderbird), my browser 
(Firefox) attempts to open the URL.  But, the URL is mangled by mail 
client line wrap and/or indentation (?), and the connection times out:


https://christuniversity.in/uploads/course/E_21-25_Lateral

An error occurred during a connection to christuniversity.in.

The site could be temporarily unavailable or too busy. Try again in 
a few moments.
If you are unable to load any pages, check your computer’s network 
connection.
If your computer or network is protected by a firewall or proxy, 
make sure that Firefox is permitted to access the web.



The following URL's also time out (see DNS comments, below):

http://christuniversity.in/
http://103.105.225.131/
http://111.93.136.229/

https://christuniversity.in/
https://103.105.225.131/
https://111.93.136.229/



  1) is the file actually there?:

wget -q --spider "${_LINK}"; _WGETQ=$?



I refrain from spidering web sites -- being blackholed is not good.



$ ls -l /etc/resolv.conf
-rw-r--r-- 1 root root 204 Mar  3 18:59 /etc/resolv.conf



2024-03-04 10:50:49 dpchrist@laalaa ~
$ ls -l /etc/resolv.conf
-rw-r--r-- 1 root root 83 Mar  4 09:50 /etc/resolv.conf



$ cat /etc/resolv.conf
# Generated by NetworkManager
# nameserver 192.168.1.254
# nameserver 192.168.68.1

# https://serverfault.com/questions/76421/wget-cant-resolve-host
# RED 2013-03-31
nameserver 8.8.8.8
nameserver 8.8.4.4



2024-03-04 10:51:32 dpchrist@laalaa ~
$ cat /etc/resolv.conf
# Generated by NetworkManager
search tracy.holgerdanske.com
nameserver 192.168.5.1


I believe Debian rewrites /etc/resolv.conf on every boot.


Hard coding Google Public DNS servers should work, but letting your 
gateway do it for your LAN is easier to manage, is faster, and conserves 
WAN bandwidth.  I would revert your changes.



And, Google is watching you.


STFW for DNS privacy:

https://avoidthehack.com/best-dns-privacy


I think I will configure my gateway to use Quad9:

https://www.quad9.net/service/locations/


The next level up would be DNS over TLS (DoT), DNS over HTTPS (DoH), 
DNSCrypt, etc.:


https://en.wikipedia.org/wiki/DNS_over_TLS



$ ls -l /etc/nsswitch.conf
-rw-r--r-- 1 root root 613 Mar  3 12:57 /etc/nsswitch.conf



2024-03-04 10:52:04 dpchrist@laalaa ~
$ ls -l /etc/nsswitch.conf
-rw-r--r-- 1 root root 542 Jan  9  2022 /etc/nsswitch.conf



$ cat /etc/nsswitch.conf
# /etc/nsswitch.conf
#
# Example configuration of GNU Name Service Switch functionality.
# If you have the `glibc-doc-reference' and `info' packages installed, try:
# `info libc "Name Service Switch"' for information about this file.

passwd: files
group:  files
shadow: files
gshadow:files

#hosts:  files mdns4_minimal [NOTFOUND=return] dns myhostname

hosts:  files dns mdns4_minimal [NOTFOUND=return] dns myhostname

networks:   files

protocols:  db files
services:   db files
ethers: db files
rpc:db files

netgroup:   nis



2024-03-04 10:56:37 dpchrist@laalaa ~
$ cat /etc/nsswitch.conf
# /etc/nsswitch.conf
#
# Example configuration of GNU Name Service Switch functionality.
# If you have the `glibc-doc-reference' and `info' packages installed, try:
# `info libc "Name Service Switch"' for information about this file.

passwd: files systemd
group:  files systemd
shadow: files
gshadow:files

hosts:  files mdns4_minimal [NOTFOUND=return] dns
networks:   files

protocols:  db files
services:   db files
ethers: db files
rpc:db files

netgroup:   nis


It appears my /etc/nsswitch.conf has not been touched since 
installation.  I would revert your changes.




; <<>> DiG 9.18.19-1~deb12u1-Debian <<>> +time christuniversity.in
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49715
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;christuniversity.in.   IN  A

;; ANSWER SECTION:
christuniversity.in.60  IN  A   103.105.225.131
christuniversity.in.60  IN  A   111.93.136.229

;; Query time: 327 msec
;; SERVER: 

Re: Debian 12.5.0 amd64 and OpenZFS bug #15526

[David Christensen]

On 2/26/24 20:52, Gareth Evans wrote:

Replied to OP by mistake, reposting to list.

On Sun 25/02/2024 at 05:34, David Christensen  wrote:

debian-user:

Is Debian 12.5.0 amd64 affected by OpenZFS bug #15526?

https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-12.5.0-amd64-netinst.iso

https://packages.debian.org/bookworm/zfs-dkms

https://github.com/openzfs/zfs/issues/15526


Hi David,

Given the complexity of the issues, I'm not sure if this truly answers your 
question, but

https://github.com/openzfs/zfs/issues/15933

seems to suggest that or a similar issue is still ongoing with Open ZFS 2.2.3, 
which is later than the version currently available from bookworm or 
bookworm-backports.

It seems bookworm-backports might eventually provide the solution, if at all, 
per the Debian wiki on ZFS:

"it is recommended by Debian ZFS on Linux Team to install ZFS related packages from 
Backports archive. Upstream stable patches will be tracked and compatibility is always 
maintained."
https://wiki.debian.org/ZFS

Currently:

$ apt policy zfs-dkms
zfs-dkms:
   Installed: 2.2.2-4~bpo12+1
   Candidate: 2.2.2-4~bpo12+1
   Version table:
  *** 2.2.2-4~bpo12+1 100
 100 https://deb.debian.org/debian bookworm-backports/contrib amd64 
Packages
 100 https://deb.debian.org/debian bookworm-backports/contrib i386 
Packages
 100 /var/lib/dpkg/status
  2.1.11-1 500
 500 https://deb.debian.org/debian bookworm/contrib amd64 Packages
 500 https://deb.debian.org/debian bookworm/contrib i386 Packages

Hope that helps.
Gareth



That you for citing OpenZFS bug #15933.


These appear to be the ZFS packages for the available Debian releases:

https://packages.debian.org/buster/zfs-dkms

buster  zfs-dkms (0.7.12-2+deb10u2)
buster-backportszfs-dkms (2.0.3-9~bpo10+1)
bullseyezfs-dkms (2.0.3-9+deb11u1)
bullseye-backports  zfs-dkms (2.1.11-1~bpo11+1)
bookwormzfs-dkms (2.1.11-1)
bookworm-backports  zfs-dkms (2.2.2-4~bpo12+1)
trixie  zfs-dkms (2.2.2-4)


The question is, how far back to go?  Is OpenZFS 2.1.x buggy?  OpenZFS 
2.0.x?  What is 0.7.12 -- OpenZFS, ZFS-on-Linux, or something else -- 
and is it buggy?



David

Debian 12.5.0 amd64 and OpenZFS bug #15526

[David Christensen]

debian-user:

Is Debian 12.5.0 amd64 affected by OpenZFS bug #15526?

https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-12.5.0-amd64-netinst.iso

https://packages.debian.org/bookworm/zfs-dkms

https://github.com/openzfs/zfs/issues/15526


David

Re: Orphaned Inode Problem

[David Christensen]

On 2/21/24 03:00, Jörg-Volker Peetz wrote:

Hi,

did you take a look at the smartctl output?

Somewhere I read, for maintainance of an SSD all it's cells should be 
read from time to time like this


sudo dd if=/dev/DEVICE of=/dev/null bs=8M status=progress

where device is something like sda or nvme0n1, especially if it was 
switched off for a longer period. At least, it shows the current read 
performance of the device.
An SSD should regularly be trimmed, if in use. This is to assist it's 
wear leveling process.


What's your opinion?

Regards,
Jörg.



I prefer to run a SMART long test periodically.  This should read every 
cell, including those that are reserved and not visible to the OS.



AIUI So long as the SSD can maintain a supply of erased cells via 
manufacturer over-provisioning, trim is not required to maintain 
performance.  If you have workload that does a lot of writes in a short 
period of time and exhausts the manufacturer over-provisioning, leaving 
free space on the SSD and trimming can be a work-around.



If you are using strong encryption, not trimming will leave crypttext on 
disk that creates more work for an attacker.  If you are using weak 
encryption, not trimming will leave crypttext on disk that an attacker 
can recover.



For imaging/ cloning, trimming will zero blocks freed by the OS and 
facilitate compression of the image file.



I have a SOHO network with about two dozen disks.  Running smartctl by 
hand is a PITA.  Running fstrim(8) by hand is easy enough.  I try to do 
both once a month.  I need to figure out smartd(8).



David

Re: HDD error: Current_Pending_Sector

[David Christensen]

On 2/20/24 09:51, Default User wrote:

Hi guys!

I am running Debian 12 Stable, up to date, on a low-spec Dell Inspiron
15 3000 Model 3511. Firmware is also up to date.

I have a 4 Gb Western Digital external usb SATA HDD, Model WDC
WD40NDZW-11A8JS1.  It has only one partition, formatted as ext4. The
filesystem is labeled MSD00012.

Every night, I use rsync to copy all contents of a (theoretically)
identical drive, which has filesystem label MSD00014, to the drive with
MSD00012.

Two nights ago, I could not do the copy correctly.  Apparently, as a
safety measure, MSD00012 was automatically re-mounted as read only, due
to a filesystem error.

I used the gnome-disks utility to unmount and then remount it. It was
remounted as read-write.

Now it "works", BUT . . .  I ran:

sudo smartctl --test=long /dev/sdb on it,

and it reports a Current_Pending_Sector error, at LBA 325904690.


From sudo smartctl --all /dev/sdb > backup_drive_b_test.txt:




smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.1.0-18-amd64] (local
build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke,
www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family: Western Digital Elements / My Passport (USB, AF)
Device Model: WDC WD40NDZW-11A8JS1
Serial Number:WD-
LU WWN Device Id: 5 0014ee 269112168
Firmware Version: 01.01A01
User Capacity:4,000,753,475,584 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate:5400 rpm
Form Factor:  2.5 inches
TRIM Command: Available, deterministic
Device is:In smartctl database 7.3/5319
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:Tue Feb 20 11:32:04 2024 EST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection
activity
was never started.
Auto Offline Data Collection:
Disabled.
Self-test execution status:  ( 121) The previous self-test
completed having
the read element of the test
failed.
Total time to complete Offline
data collection:(12240) seconds.
Offline data collection
capabilities:(0x1b) SMART execute Offline
immediate.
Auto Offline data collection
on/off support.
Suspend Offline collection
upon new
command.
Offline surface scan
supported.
Self-test supported.
No Conveyance Self-test
supported.
No Selective Self-test
supported.
SMART capabilities:(0x0003) Saves SMART data before
entering
power-saving mode.
Supports SMART auto save
timer.
Error logging capability:(0x01) Error logging supported.
General Purpose Logging
supported.
Short self-test routine
recommended polling time:(   2) minutes.
Extended self-test routine
recommended polling time:(  24) minutes.
SCT capabilities:  (0x30b5) SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME  FLAG VALUE WORST THRESH TYPE
UPDATED  WHEN_FAILED RAW_VALUE
   1 Raw_Read_Error_Rate 0x002f   200   198   051Pre-fail
Always   -   41
   3 Spin_Up_Time0x0027   253   253   021Pre-fail
Always   -   4741
   4 Start_Stop_Count0x0032   099   099   000Old_age
Always   -   1175
   5 Reallocated_Sector_Ct   0x0033   200   200   140Pre-fail
Always   -   0
   7 Seek_Error_Rate 0x002e   200   200   000Old_age
Always   -   0
   9 Power_On_Hours  0x0032   099   099   000Old_age
Always   -   1311
  10 Spin_Retry_Count0x0032   100   100   000Old_age
Always   -   0
  11 Calibration_Retry_Count 0x0032   100   100   000Old_age
Always   -   0
  12 Power_Cycle_Count   0x0032   100   100   000Old_age
Always   -   693
192 Power-Off_Retract_Count 0x0032   200   200   000Old_age
Always   -   23
193 Load_Cycle_Count0x0032   199   199   000Old_age
Always   -   3045
194 Temperature_Celsius 0x0022   114   102   000Old_age

Re: red SATA cables "notoriously bad"?

[David Christensen]

On 2/19/24 18:07, Felix Miata wrote:

My experience with that particular color cables matches Gene's. Cut one open, 
and
out comes a powdery substance instead of clean copper strands. I think most for
gen 1.0 SATA 2 decades ago, so there shouldn't be many still around bogging down
3.0 drives.



About 10 (?) years ago, I seem to recall trouble-shooting a SATA 
connection problem and coming to the conclusion that the (red) SATA 
cable was the problem.  I cannot recall if I had heard Gene's story at 
the time.  I believe I decided to cut off one end, taking a 50% chance 
of getting something I could use as a break-out/ pig tail.  To my 
surprise, there was no copper within the cable, just brownish dust! 
Unfortunately, I did not photograph the cable and it is long gone.



4 or more years ago, I was plagued with SATA III connection issues; 
likely due to old SATA I and SATA II cables and mobile racks.  I bought 
a bunch of black SATA cables marked "6 Gbps" with locking connectors and 
got rid of all of my existing cables (most of which were red).  I later 
retired all of my SATA I and SATA II mobile racks, moved most of my 
drives internal, and bought a few SATA III mobile racks for off-site 
backup drives.  My SATA connection problems are finally resolved.



David

Re: partition reporting full, but not

[David Christensen]

On 2/18/24 19:20, Keith Bainbridge wrote:

I am convinced that the missing space is used by btrfs snapshot process.



Perhaps.  But, are you re-balancing your btrfs file systems regularly?

https://manpages.debian.org/bookworm/btrfs-progs/btrfs-balance.8.en.html


Doing it by hand was not practical for me.  I wrote a Perl script to 
automate the process.  On SSD's, the results were decent.  On USB flash 
drives, not so much.



Searching for a power tool today, I see:

2024-02-18 23:27:43 dpchrist@laalaa ~/stretch-amd64
$ apt-cache search btrfs | grep mainten
btrfsmaintenance - automate btrfs maintenance tasks on mountpoints or 
directories


https://packages.debian.org/bookworm/btrfsmaintenance


I suggest installing and trying the btrfsmaintenance package.


David

Re: partition reporting full, but not

[David Christensen]

Keith Bainbridge composed on 2024-02-17 15:44 (UTC+1100):


Yes the / partitions are btrfs



Several years ago, I installed Debian (9?) using btrfs for root (and 
boot?).  I failed to understand that btrfs required regular maintenance 
and/or I was too lazy to figure it out and do it.  After a few months, 
the systems started running slowly and I seem to recall conflicting 
reports of storage usage.  I STFW, RTFM, etc., and tried doing the 
maintenance by hand.  I quickly came to the conclusion that I needed to 
run `btrfs balance start ...` many, many times.  So, I wrote Perl script 
and let it hammer on the file systems for hours at a time (!).  I was 
able to rescue most of the disks to decent performance, but one was 
especially bad and I was only able to rescue it to marginal performance. 
 I continued running btrfs for a while and running the Perl script 
periodically.  Ultimatedly, I backed up, put in fresh OS disks, and 
installed using ext4.  This was one of my reasons to use FreeBSD and ZFS 
for my SOHO servers.



David

Re: f3tools vs Silicon Power 4T drive

[David Christensen]

On 2/16/24 12:46, Stefan Monnier wrote:

One of the 1T samsungs in the md raid10 isn't entirely happy but mdadm has
not fussed about it, and smartctl seems to say its ok after testing.
  Other than that the gui access delay (30+ seconds) problems I have did
NOT go away when I moved /home off the raid to another SSD, so I may move
it back. One of the reasons I ma rsync'ing this /home back to it every
other day or so, takes < 5 minutes.

Please get a small SSD, do a fresh install, and test for the access delay.
If the delay is not present, incrementally add and test applications.
If you encounter the delay, please stop and post the details; console
sessions are best.  If not, then connect the disks with /home and test.
If you encounter the delay, then please stop and post the details.  If you
do not encounter the delay, then your system is fixed.
Take a Clonezilla image.


FWIW, my crystal ball says "30s => software timeout rather than hardware
problem"



+1


David

Re: f3tools vs Silicon Power 4T drive

[David Christensen]

On 2/15/24 22:16, gene heskett wrote:
I want to know with absolute certainty, with of the 4 drives in that 
raid10, actually has a belly ache. When it has a belly ache. I can't see 
any reason on this ball of rock and water, why I should be expected to 
replace a drive at a time until the belly ache goes away.



I seem to recall the Samsung 1 TB SSD's in your /home RAID10 were worn 
out.  I suggest installing the 2 TB M.2 WD Black, partitioning it with 
GPT, creating one large partition, mounting it at /data, and copying all 
of the data from /home to /data before the SSD's and RAID fail completely.



I recently had an Intel SSD 520 Series 180 GB go from operational to 
toast, with nothing in between.  If that happens to one of those Samsung 
1 TB SSD's, there will be no way for the RAID10 to correct the bad 
blocks on the other other SSD.  You will corrupt and lose data.



I leave /home on my root partition.  My working directories are in CVS. 
The only ephemeral data is in $HOME/.thunderbird.  I have a mail filter 
that copies incoming mail to a second folder on the IMAP server.  I Bcc 
outgoing mail to another mail account.  If my OS disk dies, I restore 
the image from last month, update Debian, check out my work, reconnect 
Thunderbird to the various e-mail servers, and clean up the Thunderbird 
folders as required.  No data is lost.



David

Re: f3tools vs Silicon Power 4T drive

[David Christensen]

On 2/15/24 17:44, gene heskett wrote:
One of the 1T samsungs in the md raid10 isn't entirely happy but mdadm 
has not fussed about it, and smartctl seems to say its ok after testing. 
  Other than that the gui access delay (30+ seconds) problems I have did 
NOT go away when I moved /home off the raid to another SSD, so I may 
move it back. One of the reasons I ma rsync'ing this /home back to it 
every other day or so, takes < 5 minutes.



Please get a small SSD, do a fresh install, and test for the access 
delay.  If the delay is not present, incrementally add and test 
applications.  If you encounter the delay, please stop and post the 
details; console sessions are best.  If not, then connect the disks with 
/home and test.  If you encounter the delay, then please stop and post 
the details.  If you do not encounter the delay, then your system is 
fixed.  Take a Clonezilla image.



David

Re: f3tools vs Silicon Power 4T drive

[David Christensen]

On 2/15/24 12:59, gene heskett wrote:

...  gigastones, I 5 of them but when all
are plugged in there are only 3 becauae there are 2 pairs of matching 
serial numbers ...



I recall 2 pairs of SSD's with matching serial numbers.  Please remove 
one SSD of each pair so that the remaining SSD's all have unique serial 
numbers.  Return them for a refund while you still can.  If you cannot, 
put them in another computer or put them on the shelf as spares.



David

Re: SMART Uncorrectable_Error_Cnt rising - should I be worried?

[David Christensen]

On 2/16/24 10:56, Roy J. Tellason, Sr. wrote:

On Friday 16 February 2024 04:52:22 am David Christensen wrote:

I think the Raspberry Pi, etc., users on this list live with USB storage
and have found it to be reliable enough for personal and SOHO network use.
  
I have one,  haven't done much with it.  Are there any alternative ways to interface storage?  Maybe add SATA ports or something?



In general, there are many combinations of storage interfaces offered in 
the marketplace.  For your specific single-board computer (SBC), I 
suggest checking the manual and checking the manufacturer sales and/or 
support web pages.



David

Re: f3tools vs Silicon Power 4T drive

[David Christensen]

On 2/15/24 12:19, gene heskett wrote:

On 2/15/24 11:21, Andy Smith wrote:

... redundancy plans ...

Like which version of a raid is the best at tolerating a failed drive, 
which give he best balance between redundancy and capacity.



Given a small number of disks, N (say, 4 to 8), the obvious choices are 
RAID5, RAID6, and RAID10.



Regarding redundancy:

* RAID5 can tolerate the loss of any one disk.

* RAID6 can tolerate the loss of any two disks.

* RAID10 can tolerate the loss of any one disk.  If you get lucky, 
RAID10 can tolerate the loss of multiple disks if each lost disk is in a 
different mirror.



Regarding capacity, if each disk stores B bytes:

* RAID5 gives you (N-1) * B capacity.

* RAID6 gives you (N-2) * B capacity.

* RAID10 gives you (N/2) * B capacity.


If each disk has performance P:

* RAID5 has performance ranging from P to (N-1) * P.

* RAID6 has performance ranging from P to (N-2) * P.

* RAID10 with M mirrors of D disks each has write performance M * P and 
read performance M * D * P.



Other factors to consider:

* All of the above needs to be reconsidered when one or more disks fail 
-- e.g. the array is operating in degraded mode.


* All of the above needs to be reconsidered when a failed disk has been 
replaced -- e.g. the array is resilvering.


* All of the above needs to be reconsidered when disk(s) fail during 
resilvering (!).


* RAID5 and RAID6 typically do not allow changes to topology -- e.g. the 
number of disks in the array and the number of bytes used in each disk.


* RAID0, RAID1, and JBOD may allow some changes to topology.  What is 
allowed depends upon implementation.


* With more disks, you may be able to create hierarchies -- e.g. stripe 
of mirrors (RAID10).  Redundancy, capacity, and/or performance under 
operational, degraded, resilvering, etc., modes all need to be reconsidered.


* Hot spares can be added.  Again, reconsider everything.

* And more.


So, it's a multi-dimensional problem and there are many combinations and 
permutations.  The more disks you have, the more possibilities you have. 
 I suggest picking two or three, and exploring them using a dedicated 
computer, a snapshot of your data, and your workload.



I am currently using ZFS and a stripe of 2 mirrors with 2 @ 3 TB HDD's 
each and SSD read cache.  I expect the same could be implemented with 
mdadm(8), lvm(8), bcache, dm-cache, btrfs, and others.



David

Re: SMART Uncorrectable_Error_Cnt rising - should I be worried?

[David Christensen]

On 2/15/24 07:41, The Wanderer wrote:

On 2024-02-15 at 03:09, David Christensen wrote:

On 2/14/24 18:54, The Wanderer wrote:

On 2024-01-09 at 14:22, The Wanderer wrote:

On 2024-01-09 at 14:01, Michael Kjörling wrote:

On 9 Jan 2024 13:25 -0500, from The Wanderer

I've ordered a 22TB external drive


Make?  Model?  How it is interfaced to your computer?


It's a WD Elements 20TB drive (I'm not sure where I got the 22 from);
the back of the case has the part number WDBWLG0200HBK-X8 (or possibly
-XB, the font is kind of ambiguous). The connection, per the packaging
label, is USB-3.



Okay.


STFW it seems that drive uses CMR, which is good:

https://nascompares.com/answer/list-of-wd-cmr-and-smr-hard-drives-hdd/



The big change of plans in the middle of my month-plus process was the
decision to replace the entire 8-drive array with a 6-drive array, and
the reason for that was because the 8-drive array left me with no open
SATA ports to be able to connect spare drives in order to do drive
replacements without needing to rebuild the whole shaboozle.



Having spare drive bays for RAID drive replacement is smart.



If you have a processor, memory, PCIe slot, and HBA to match those
SSD's, the performance of those SSD's should be very nice.


The CPU is a Ryxen 5 5600X. The RAM is G-Skill DDR4 2666MHz, in two 32GB
DIMMs. I don't know how to assess PCIe slots and HBA, but the
motherboard is an Asus ROG Crosshair VIII Dark Hero, which I think was
the top-of-the-line enthusiast motherboard (with the port set my
criteria called for) the year I built this machine.

I'm pretty sure my performance bottleneck for most things is the CPU (or
the GPU, where that comes into play, which here it doesn't);
storage-wise this seems so far to be at least as fast as what I had
before, but it's hard to tell if it's faster.



It would not surprise me if the Intel D3-S4510 server drives are 
somewhat slower than the Samsung EVO 870 desktop drives.  But the Intel 
disks are designed to pull a heavy load all day for years on end.



Do you have a tool to monitor disk throughput and utilization?  I use 
Xfce panel Disk Performance Monitor applets and nmon(1) in a Terminal. 
Those plus CPU and memory monitoring tools should allow you to determine 
if your workload is CPU bound, memory bound, or I/O bound.




The key concept is "data lifetime". (Or alternatively, "destruction
policy".)


I can see that for when you have a tiered backup structure, and are
looking at the lifetimes of each backup copy. For my live system, my
intended data lifetime (outside of caches and data kept in /tmp) is
basically "forever".



I try to group my data in anticipation of backup, etc., requirements. 
When I get it right, disaster preparedness and disaster recovery are easier.




I believe ZFS can do more hard links. (Much more?  Limited by
available storage space?)


I'm not sure, but I'll have to look into that, when I get to the point
of trying to set up that tiered backup.
...

... without [rsnapshot hard link]
deduplication there wouldn't have been enough space on the drive
for more than the single copy, ...


ZFS provides similarly useful results with built-in compression and
de-duplication.


I have the impression that there are risk and/or complexity aspects to
it ...



Of course.  ZFS is sophisticated storage technology.  It looks 
deceptively simple when you are window shopping, but becomes non-trivial 
once you put real data on it, have to live with it 24x7, have to prepare 
for disasters, and have to recover from disasters.  There is a lot to 
learn and "more than enough rope to shoot yourself in the foot".




That sounds like an N-way merge problem ...


It does sound like that, yes. I'm already aware of jdupes, and of a few
other tools (part of the work I already did in getting this far was
rdfind, which is what I used to set up much of the hardlink
deduplication that wound up biting me in the butt), but have not
investigated LVM snapshot - and the idea of trying to script something
like this, without an existing known-safe copy of the data to fall back
on, leaves me *very* nervous.

Figuring out how to be prepared to roll back is the other uncertain and
nervous-making part. In some cases it's straightforward enough, but
doing it at the scale of the size of those copies is at best daunting.



https://html.duckduckgo.com/html?q=lvm%20snapshot%20restore


Use another computer or a VM to learn and practice LVM snapshots and 
restores, then use those skills when doing the N-way merge.




out of the way, shutting down all parts of the system that might be
writing to the affected filesystems, and manually copying out the
final state of the *other* parts of those filesystems via rsync,
bypassing rsnapshot. That was on Saturday the 10th.

Then I grabbed copies of various metadata about the filesystems,
the LVM, and the mdraid config; modified /etc/fstab to not mount
them; deactivated the md

Re: SMART Uncorrectable_Error_Cnt rising - should I be worried?

[David Christensen]

On 2/14/24 18:54, The Wanderer wrote:

TL;DR: It worked! I'm back up and running, with what appears to be all
my data safely recovered from the failing storage stack!



That is good to hear.  :-)



On 2024-01-09 at 14:22, The Wanderer wrote:


On 2024-01-09 at 14:01, Michael Kjörling wrote:


On 9 Jan 2024 13:25 -0500, from wande...@fastmail.fm (The
Wanderer):


I've ordered a 22TB external drive 



Make?  Model?  How it is interfaced to your computer?



for the purpose of creating
such a backup. Fingers crossed that things last long enough for
it to get here and get the backup created.


I suggest selecting, installing and configuring (as much as
possible) whatever software you will use to actually perform the
backup while you wait for the drive to arrive. It might save you a
little time later. Opinions differ but I like rsnapshot myself;
it's really just a front-end for rsync, so the copy is simply
files, making partial or full restoration easy without any special
tools.


My intention was to shut down everything that normally runs, log out
as the user who normally runs it, log in as root (whose home
directory, like the main installed system, is on a different RAID
array with different backing drives), and use rsync from that point.
My understanding is that in that arrangement, the only thing
accessing the RAID-6 array should be the rsync process itself.

For additional clarity: the RAID-6 array is backing a pair of
logical volumes, which are backing the /home and /opt partitions. The
entire rest of the system is on a series of other logical volumes
which are backed by a RAID-1 array, which is based on entirely
different drives (different model, different form factor, different
capacity, I think even different connection technology) and which has
not seen any warnings arise.


dmesg does have what appears to be an error entry for each of
the events reported in the alert mails, correlated with the
devices in question. I can provide a sample of one of those, if
desired.


As long as the drive is being honest about failures and is
reporting failures rapidly, the RAID array can do its work. What
you absolutely don't want to see is I/O errors relating to the RAID
array device (for example, with mdraid, /dev/md*), because that
would presumably mean that the redundancy was insufficient to
correct for the failure. If that happens, you are falling off a
proverbial cliff.


Yeah, *that* would be indicative of current catastrophic failure. I
have not seen any messages related to the RAID array itself.


In the time since this, I continued mostly-normal but somewhat-curtailed
use of the system, and saw few messages about these matters that did not
arise from attempts to back up the data for later recovery purposes.



Migrating large amounts of data from one storage configuration to 
another storage configuration is non-trivial.  Anticipating problems and 
preparing for them ahead of time (e.g. backups) makes it even less 
trivial.  The last time I lost data was during a migration when I had 
barely enough hardware.  I made a conscious decision to always have a 
surplus of hardware.




(For awareness: this is all a source of considerable psychological
stress to me, to an extent that is leaving me on the edge of
physically ill, and I am managing to remain on the good side of that
line only by minimizing my mental engagement with the issue as much
as possible. I am currently able to read and respond to these mails
without pressing that line, but that may change at any moment, and if
so I will stop replying without notice until things change again.)


This need to stop reading wound up happening almost immediately after I
sent the message to which I am replying.



I remember reading your comment and then noticing you went silent.  I 
apologize if I pushed your button.




I now, however, have good news to report back: after more than a month,
at least one change of plans, nearly $2200 in replacement hard drives,



Ouch.


If you have a processor, memory, PCIe slot, and HBA to match those 
SSD's, the performance of those SSD's should be very nice.




much nervous stress, several days of running data copies to and from a
20+-terabyte mechanical hard drive over USB, and a complete manual
removal of my old 8-drive RAID-6 array and build of a new 6-drive RAID-6
array (and of the LVM structure on top of it), I now appear to have
complete success.

I am now running on a restored copy of the data on the affected
partitions, taken from a nearly-fully-shut-down system state, which is
sitting on a new RAID-6 array built on what I understand to be
data-center-class SSDs (which should, therefore, be more suitable to the
24/7-uptime read-mostly workload I expect of my storage). The current
filesystems involved are roughly the same size as the ones previously in
use, but the underlying drives are nearly 2x the size; I decided to
leave the extra capacity for later allocation via LVM, if and when I may
need it.



When I was 

Re: f3tools vs Silicon Power 4T drive

[David Christensen]

On 2/14/24 18:06, gene heskett wrote:
Will the by-id string fit in the space reserved for a label?That IF 
there was a connection between the /dev/sdc that udev assigns and 
anything in this list:


root@coyote:~# ls /dev/disk/by-id
ata-ATAPI_iHAS424_B_3524253_327133504865 
ata-Samsung_SSD_870_EVO_1TB_S626NF0R302509W-part1    wwn-0x5002538f413394a5
ata-Gigastone_SSD_GST02TBG221146 
ata-Samsung_SSD_870_EVO_1TB_S626NF0R302509W-part2 
wwn-0x5002538f413394a5-part1
ata-Gigastone_SSD_GST02TBG221146-part1 
ata-Samsung_SSD_870_EVO_1TB_S626NF0R302509W-part3 
wwn-0x5002538f413394a5-part2
ata-Gigastone_SSD_GSTD02TB230102 
ata-Samsung_SSD_870_QVO_1TB_S5RRNF0T201730V wwn-0x5002538f413394a5-part3
ata-Gigastone_SSD_GSTD02TB230102-part1 
ata-Samsung_SSD_870_QVO_1TB_S5RRNF0T201730V-part1    wwn-0x5002538f413394a9
ata-Gigastone_SSD_GSTG02TB230206 
ata-Samsung_SSD_870_QVO_1TB_S5RRNF0T201730V-part2 
wwn-0x5002538f413394a9-part1
ata-Gigastone_SSD_GSTG02TB230206-part1 
ata-Samsung_SSD_870_QVO_1TB_S5RRNF0T201730V-part3 
wwn-0x5002538f413394a9-part2
ata-Samsung_SSD_870_EVO_1TB_S626NF0R302498T 
ata-SPCC_Solid_State_Disk_AA231107S304KG00080 wwn-0x5002538f413394a9-part3
ata-Samsung_SSD_870_EVO_1TB_S626NF0R302498T-part1 
ata-SPCC_Solid_State_Disk_AA231107S304KG00080-part1  wwn-0x5002538f413394ae
ata-Samsung_SSD_870_EVO_1TB_S626NF0R302498T-part2  md-name-coyote:0 
    wwn-0x5002538f413394ae-part1
ata-Samsung_SSD_870_EVO_1TB_S626NF0R302498T-part3 md-name-coyote:0-part1 
wwn-0x5002538f413394ae-part2
ata-Samsung_SSD_870_EVO_1TB_S626NF0R302502E    md-name-coyote:2 
    wwn-0x5002538f413394ae-part3
ata-Samsung_SSD_870_EVO_1TB_S626NF0R302502E-part1  md-name-_none_:1 
    wwn-0x5002538f413394b0
ata-Samsung_SSD_870_EVO_1TB_S626NF0R302502E-part2 
md-uuid-3d5a3621:c0e32c8a:e3f7ebb3:318edbfb wwn-0x5002538f413394b0-part1
ata-Samsung_SSD_870_EVO_1TB_S626NF0R302502E-part3 
md-uuid-3d5a3621:c0e32c8a:e3f7ebb3:318edbfb-part1 
wwn-0x5002538f413394b0-part2
ata-Samsung_SSD_870_EVO_1TB_S626NF0R302507V 
md-uuid-57a88605:27f5a773:5be347c1:7c5e7342 wwn-0x5002538f413394b0-part3
ata-Samsung_SSD_870_EVO_1TB_S626NF0R302507V-part1 
md-uuid-bb6e03ce:19d290c8:5171004f:0127a392  wwn-0x5002538f42205e8e
ata-Samsung_SSD_870_EVO_1TB_S626NF0R302507V-part2 
usb-SPCC_Sol_id_State_Disk_1234567897E6-0:0 wwn-0x5002538f42205e8e-part1
ata-Samsung_SSD_870_EVO_1TB_S626NF0R302507V-part3 
usb-SPCC_Sol_id_State_Disk_1234567897E6-0:0-part1 
wwn-0x5002538f42205e8e-part2
ata-Samsung_SSD_870_EVO_1TB_S626NF0R302509W 
usb-USB_Mass_Storage_Device_816820130806-0:0 wwn-0x5002538f42205e8e-part3

root@coyote:~#

I dare you to find the disk that udev calls sdc in the above wall of text.

Why can't you understand that I want a unique label for all of this 
stuff that is NOT a wall of HEX numbers no one can remember.  Its not 
mounted, so blkid does NOT see it.



For labeled disk partitions, use /dev/disk/by-label/* paths:

2024-02-14 18:22:34 root@taz ~
# ls -1 /dev/disk/by-label/
sda3_crypt
taz_boot
taz_root


David

Re: f3tools vs Silicon Power 4T drive

[David Christensen]

On 2/14/24 17:48, gene heskett wrote:

On 2/14/24 19:48, Andy Smith wrote:

On Wed, Feb 14, 2024 at 05:09:02PM -0500, gene heskett wrote:

I have made 1 full partiton om each one, a labeled those partitions  as
SiPwr_0 and SiPwr_1


Please show us the command you used¹ to do that, so we know what
exactly you are talking about, because as previously discussed
there's a lot of different things that you like to call "partition
labels".


This is what gparted calls a "partition label" and certainly does not 
need a 4.5 megabyte camera image to see. or even a 50k screen snap.
Taking this screenshot was a pita, because the gparted window disappears 
behind the terminal screen when you click on take another shot, so you 
have to quit, then find the gparted on the tool bar to bring it back to 
the front, then move it and the terminal so its not totally hidden. Then 
rerun spectacle again waste a click bringing it fwd, then 30 seconds 
later the spectacal instructions finally show up and after 5 minutes of 
screwing around, finally get the screen shot attached to prove I'm not 
lieing.



The easy and accurate answer is to use a root console, fdisk(8) with 
--list-details, select the console session, and paste into a mail reply:


2024-02-14 18:09:26 root@taz ~
# fdisk --list-details /dev/sda
Disk /dev/sda: 55.9 GiB, 60022480896 bytes, 117231408 sectors
Disk model: INTEL SSDSC2CW06
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 816CF78F-AFAD-4F70-AAA0-B08C6CE95AE7
First LBA: 34
Last LBA: 117231374
Alternative LBA: 117231407
Partition entries LBA: 2
Allocated partition entries: 128

DeviceStart   End  Sectors Type-UUID 
   UUID Name  Attrs
/dev/sda1  2048   1953791  1951744 
C12A7328-F81F-11D2-BA4B-00A0C93EC93B 
5A1358F4-23A2-4CF6-A4E2-0A30A0FFC904 ESP
/dev/sda2   1953792   3907583  1953792 
0FC63DAF-8483-4772-8E79-3D69D8477DE4 
B429D984-E32D-4BAE-A7AE-137168B0F0F3 taz_boot
/dev/sda3   3907584   5861375  1953792 
0FC63DAF-8483-4772-8E79-3D69D8477DE4 
83862E6A-7B89-4AB9-A21D-BAEF3AD0F7A3 taz_swap_crypt
/dev/sda4   5861376  29298687 23437312 
0FC63DAF-8483-4772-8E79-3D69D8477DE4 
2A708FD7-F6EE-49D7-8E23-65905BCD6512 taz_root_crypt
/dev/sda5  29298688 117229567 87930880 
0FC63DAF-8483-4772-8E79-3D69D8477DE4 
B8468EA2-B66D-4D13-9FD1-E46AEDA58067 taz_scratch_crypt



David

Re: shred bug? [was: Unidentified subject!]

[David Christensen]

On 2/13/24 09:40, debian-u...@howorth.org.uk wrote:

Greg Wooledge  wrote:


Shred will determine the size of the file, then write data to the
file, rewind, write data again, etc.  On a traditional hard drive,
that will overwrite the original private information.  On modern
devices, it may not.


Thanks for the excellent explanation :)

One nitpick. You say "On a traditional hard drive, that will overwrite
the original private information" but that's not quite true. It also
needs to be a "traditional" file system! That is, not journalled or COW.

So nowadays I would expect shred not to work unless you got very
lucky, or planned carefully.



Perhaps zerofree(8)?


David

Re: shred bug? [was: Unidentified subject!]

[David Christensen]

On 2/13/24 11:31, gene heskett wrote:
Next experiment is a pair of 4T 
Silicon Power SSD's When they & the startech usb3 adapters arrive.  I'll 
get that NAS built for amanda yet.



2.5" SATA SSD's and SATA to USB adapter cables for $187.97 + $10.99 = 
$198.96 each set?


https://www.amazon.com/dp/B0BVLRFFWQ

https://www.amazon.com/dp/B00HJZJI84


Why not external USB drives for $192.99?

https://www.amazon.com/dp/B0C6XVZS4K


For $7 more, you can get the "Pro edition" in black with two cables:

https://www.amazon.com/dp/B0C69QD5NK


You could plug those into the two USB-C 3.1 Gen 2 ports on your Asus 
PRIME Z370-A II motherboard.



David

Re: Fast Random Data Generation (Was: Re: Unidentified subject!)

[David Christensen]

On 2/12/24 08:30, Linux-Fan wrote:

David Christensen writes:


On 2/11/24 02:26, Linux-Fan wrote:
I wrote a program to automatically generate random bytes in multiple 
threads:

https://masysma.net/32/big4.xhtml


What algorithm did you implement?


I copied the algorithm from here:
https://www.javamex.com/tutorials/random_numbers/numerical_recipes.shtml



That Java code uses locks, which implies it uses global state and cannot 
be run multi-threaded (?).  (E.g. one process with one JVM.)



Is it possible to obtain parallel operation on an SMP machine with 
multiple virtual processors?  (Other than multiple OS processes with one 
PRNG on one JVM each?)



I found it during the development of another application where I needed 
a lot of random data for simulation purposes :)


My implementation code is here:
https://github.com/m7a/bo-big/blob/master/latest/Big4.java

If I were to do it again today, I'd probably switch to any of these PRNGS:

* https://burtleburtle.net/bob/rand/smallprng.html
* https://www.pcg-random.org/



Hard core.  I'll let the experts figure it out; and then I will use 
their libraries and programs.



David

Re: shred bug? [was: Unidentified subject!]

[David Christensen]

On 2/12/24 08:50, Curt wrote:

On 2024-02-11,   wrote:



On Sun, Feb 11, 2024 at 09:54:24AM -0500, Greg Wooledge wrote:

[...]


If FILE is -, shred standard output.
=20
In every sentence, the word FILE appears.  There's nothing in there
which says "you can operate on a non-file".


Point taken, yes.


I thought everything was a file.



"Everything is a file" is a design feature of the Unix operating system:

https://en.wikipedia.org/wiki/Everything_is_a_file


But, there is more than one kind of file.


And, not every program supports every kind of file.


The manual page for find(1) provides a shopping list of file types it 
supports:


2024-02-12 12:32:13 dpchrist@laalaa ~
$ man find | egrep -A 20 '^   .type c'
   -type c
  File is of type c:

  b  block (buffered) special

  c  character (unbuffered) special

  d  directory

  p  named pipe (FIFO)

  f  regular file

  l  symbolic link; this is never true if  the
 -L option or the -follow option is in ef-
 fect, unless the symbolic link is broken.
 If  you want to search for symbolic links
 when -L is in effect, use -xtype.

  s  socket


As for shred(1), the argument FILE is conventionally a regular file.  We 
are discussing the special case described in the manual page:


   If FILE is -, shred standard output.


David

Re: Fast Random Data Generation (Was: Re: Unidentified subject!)

[David Christensen]

On 2/11/24 02:26, Linux-Fan wrote:
I wrote a program to automatically generate random bytes in multiple 
threads:

https://masysma.net/32/big4.xhtml

Before knowing about `fio` this way my way to benchmark SSDs :)

Example:

| $ big4 -b /dev/null 100 GiB
| Ma_Sys.ma Big 4.0.2, Copyright (c) 2014, 2019, 2020 Ma_Sys.ma.
| For further info send an e-mail to ma_sys...@web.de.
|| 0.00% +0 MiB 0 MiB/s 0/102400 MiB
| 3.48% +3562 MiB 3255 MiB/s 3562/102400 MiB
| 11.06% +7764 MiB 5407 MiB/s 11329/102400 MiB
| 19.31% +8436 MiB 6387 MiB/s 19768/102400 MiB
| 27.71% +8605 MiB 6928 MiB/s 28378/102400 MiB
| 35.16% +7616 MiB 7062 MiB/s 35999/102400 MiB
| 42.58% +7595 MiB 7150 MiB/s 43598/102400 MiB
| 50.12% +7720 MiB 7230 MiB/s 51321/102400 MiB
| 58.57% +8648 MiB 7405 MiB/s 59975/102400 MiB
| 66.96% +8588 MiB 7535 MiB/s 68569/102400 MiB
| 75.11% +8343 MiB 7615 MiB/s 76916/102400 MiB
| 83.38% +8463 MiB 7691 MiB/s 85383/102400 MiB
| 91.74% +8551 MiB 7762 MiB/s 93937/102400 MiB
| 99.97% +8426 MiB 7813 MiB/s 102368/102400 MiB
|| Wrote 102400 MiB in 13 s @ 7812.023 MiB/s



What algorithm did you implement?



Secure Random can be obtained from OpenSSL:

| $ time for i in `seq 1 100`; do openssl rand -out /dev/null $((1024 * 
1024 * 1024)); done

|
| real    0m49.288s
| user    0m44.710s
| sys    0m4.579s

Effectively 2078 MiB/s (quite OK for single-threaded operation). It is 
not designed to generate large amounts of random data as the size is 
limited by integer range...



Thank you for posting the openssl(1) incantation.


Benchmarking my daily driver laptop without Intel Secure Key:

2024-02-11 21:54:04 dpchrist@laalaa ~
$ lscpu | grep 'Model name'
Model name: Intel(R) Core(TM) i7-2720QM CPU @ 
2.20GHz


2024-02-11 21:54:09 dpchrist@laalaa ~
$ time for i in `seq 1 100`; do openssl rand -out /dev/null $((1024 * 
1024 * 1024)); done


real1m40.149s
user1m25.174s
sys 0m14.952s


So, ~1.072E+9 bytes per second.


Benchmarking a workstation with Intel Secure Key:

2024-02-11 21:54:40 dpchrist@taz ~
$ lscpu | grep 'Model name'
Model name: Intel(R) Xeon(R) E-2174G CPU @ 3.80GHz

2024-02-11 21:54:46 dpchrist@taz ~
$ time for i in `seq 1 100`; do openssl rand -out /dev/null $((1024 * 
1024 * 1024)); done


real1m14.696s
user1m0.338s
sys 0m14.353s


So, ~1.437E+09 bytes per second.


David

Re: shred bug? [was: Unidentified subject!]

[David Christensen]

On 2/11/24 06:54, Greg Wooledge wrote:

On Sun, Feb 11, 2024 at 03:45:21PM +0100, to...@tuxteam.de wrote:

On Sun, Feb 11, 2024 at 09:37:31AM -0500, Greg Wooledge wrote:

On Sun, Feb 11, 2024 at 08:02:12AM +0100, to...@tuxteam.de wrote:


[...]


What Thomas was trying to do is to get a cheap, fast random number
generator. Shred seems to have such.


Well... I certainly wouldn't call it a bug.  Maybe a feature request.


Still there's the discrepancy between doc and behaviour.


There isn't.  The documentation says:

SYNOPSIS
shred [OPTION]... FILE...



I interpret the above line to be a prototype for invoking the shred(1) 
program:


* "shred" is the program name

* "[OPTION]..." is one or more option specifiers that may be omitted. 
Each should be described below.


* "FILE..." is one or more argument specifies that should be file system 
paths (strings).




DESCRIPTION
Overwrite  the specified FILE(s) repeatedly, in order to make it harder
for even very expensive hardware probing to recover the data.

If FILE is -, shred standard output.



I interpret the above line at face value -- if the caller provides a 
dash as the argument, shred(1) will operate on standard output.




In every sentence, the word FILE appears.  There's nothing in there
which says "you can operate on a non-file".



Dash is not a file, yet the above sentence says shred(1) can operate on it.



Once you grasp what the command is *intended* to do (rewind and overwrite
a file repeatedly), it makes absolutely perfect sense that it should
only operate on rewindable file system objects.



An expert may infer what you have stated, but I prefer manual pages that 
are explicit.



The GNU project must have found a need for the FILE='-' feature, 
otherwise it would not exist.  The manual page should describe that need 
(e.g. why) and how to properly use shred(1) to solve the need.




If you want it to write a stream of data instead of performing its normal
operation (rewinding and rewriting), that's a new feature.



Humans are (in)famous for doing unexpected things.



If you'd prefer the documentation to say explicitly "only regular files
and block devices are allowed", that would be an upstream documentation
*clarification* request.



Apparently, shred(1) has both an info(1) page (?) and a man(1) page. 
The obvious solution is to write one document that is complete and 
correct, and use it everywhere -- e.g. DRY.



David

Re: Unidentified subject!

[David Christensen]

On 2/11/24 03:13, Thomas Schmitt wrote:

Hi,

David Christensen wrote:

Concurrency:
threads throughput
8   205+198+180+195+205+184+184+189=1,540 MB/s


There remains the question how to join these streams without losing speed
in order to produce a single checksum. (Or one would have to divide the
target into 8 areas which get checked separately.)



I had similar thoughts.  A FIFO should be able to join the streams. 
But, dividing the device by the number of virtual cores and putting a 
thread on each makes more sense.  Either done right should fill the 
drive I/O capacity.




Does this 8 thread generator cause any problems with the usability of
the rest of the system ? Sluggish program behavior or so ?



CPU Graph shows all eight virtual cores at 100%, so everything else on 
the system would be sluggish (unless you use nice(1)).



Here is a processor with Intel Secure Key and otherwise unloaded:

2024-02-11 11:48:21 dpchrist@taz ~
$ lscpu | grep 'Model name'
Model name: Intel(R) Xeon(R) E-2174G CPU @ 3.80GHz

2024-02-11 11:59:55 dpchrist@taz ~
$ cat /etc/debian_version ; uname -a
11.8
Linux taz 5.10.0-27-amd64 #1 SMP Debian 5.10.205-2 (2023-12-31) x86_64 
GNU/Linux


2024-02-11 12:02:52 dpchrist@taz ~
$ dd if=/dev/urandom of=/dev/null bs=1M count=10K
10240+0 records in
10240+0 records out
10737418240 bytes (11 GB, 10 GiB) copied, 20.0469 s, 536 MB/s

threads throughput
1   536 MB/s
2   512+512 = 1,024 MB/s
3   502+503+503 = 1,508 MB/s
4   492+491+492+492 = 1,967 MB/s
5   492+384+491+385+491 = 2,243 MB/s
6   379+491+492+379+379+379 = 2,499 MB/s
7   352+491+356+388+352+357+388 = 2,684 MB/s
8   355+354+344+348+344+354+353+349 = 2,801 MB/s



I have to correct my previous measurement on the 4 GHz Xeon, which was
made with a debuggable version of the program that produced the stream.
The production binary which is compiled with -O2 can write 2500 MB/s into
a pipe with a pacifier program which counts the data:

   $ time $(scdbackup -where bin)/cd_backup_planer -write_random - 100g 
2s62gss463ar46492bni | $(scdbackup -where bin)/raedchen -step 100m -no_output 
-print_count
   100.0g bytes

   real0m39.884s
   user0m30.629s
   sys 0m41.013s

(One would have to install scdbackup to reproduce this and to see raedchen
  count the bytes while spinning the classic SunOS boot wheel: |/-\|/-\|/-\
http://scdbackup.webframe.org/main_eng.html
http://scdbackup.webframe.org/examples.html
  Oh nostalgy ...
)

Totally useless but yielding nearly 4000 MB/s:

   $ time $(scdbackup -where bin)/cd_backup_planer -write_random - 100g 
2s62gss463ar46492bni >/dev/null

   real0m27.064s
   user0m23.433s
   sys 0m3.646s

The main bottleneck in my proposal would be the checksummer:

   $ time $(scdbackup -where bin)/cd_backup_planer -write_random - 100g 
2s62gss463ar46492bni | md5sum
   5a6ba41c2c18423fa33355005445c183  -

   real2m8.160s
   user2m25.599s
   sys 0m22.663s

That's quite exactly 800 MiB/s ~= 6.7 Gbps.
Still good enough for vanilla USB-3 with a fast SSD, i'd say.



Yes -- more than enough throughput.


Before I knew of fdupes(1) and jdupes(1), I wrote a Perl script to find 
duplicate files.  It uses the Digest module, and supports any algorithm 
supported by that module.  Here are some runs against a local ext4 on 
LUKS (with AES-NI) on Intel SSD 520 Series 60 GB and check summing whole 
files:


2024-02-11 13:32:47 dpchrist@taz ~
$ time finddups --filter w --digest MD4 .thunderbird/ >/dev/null

real0m0.878s
user0m0.741s
sys 0m0.137s

2024-02-11 13:33:14 dpchrist@taz ~
$ time finddups --filter w --digest MD5 .thunderbird/ >/dev/null

real0m1.110s
user0m0.977s
sys 0m0.132s

2024-02-11 13:33:19 dpchrist@taz ~
$ time finddups --filter w --digest SHA-1 .thunderbird/ >/dev/null

real0m1.306s
user0m1.151s
sys 0m0.156s

2024-02-11 13:36:40 dpchrist@taz ~
$ time finddups --filter w --digest SHA-256 .thunderbird/ >/dev/null

real0m2.545s
user0m2.424s
sys 0m0.121s

2024-02-11 13:36:51 dpchrist@taz ~
$ time finddups --filter w --digest SHA-384 .thunderbird/ >/dev/null

real0m1.808s
user0m1.652s
sys 0m0.157s

2024-02-11 13:37:00 dpchrist@taz ~
$ time finddups --filter w --digest SHA-512 .thunderbird/ >/dev/null

real0m1.814s
user0m1.673s
sys 0m0.141s


It is curious that SHA-384 and SHA-512 are faster than SHA-256.  I can 
confirm similar results on:


2024-02-11 13:39:58 dpchrist@laalaa ~
$ lscpu | grep 'Model name'
Model name: Intel(R) Core(TM) i7-2720QM CPU @ 
2.20GHz



David

Re: Unidentified subject!

[David Christensen]

On 2/11/24 00:07, Thomas Schmitt wrote:

In the other thread about the /dev/sdm test:

Gene Heskett wrote:

Creating file 39.h2w ... 1.98% -- 1.90 MB/s -- 257:11:32
[...]
$ sudo f3probe --destructive --time-ops /dev/sdm
Bad news: The device `/dev/sdm' is a counterfeit of type limbo
Device geometry:
  *Usable* size: 59.15 GB (124050944 blocks)
 Announced size: 1.91 TB (409600 blocks)


David Christensen wrote:

Which other thread?  Please provide a URL to archived post.


https://lists.debian.org/msgid-search/e7a0c1a1-f973-4007-a86d-8d91d8d91...@shentel.net
https://lists.debian.org/msgid-search/b566504b-5677-4d84-b5b3-b0de63230...@shentel.net



Thank you.  :-)


David

Re: Unidentified subject!

[David Christensen]

On 2/11/24 00:11, Thomas Schmitt wrote:

Hi,

David Christensen wrote:

$ time dd if=/dev/urandom bs=8K count=128K | wc -c
[...]
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.30652 s, 249 MB/s


This looks good enough for practical use on spinning rust and slow SSD.



Yes.



Maybe the "wc" pipe slows it down ?
... not much on 4 GHz Xeon with Debian 11:

   $ time dd if=/dev/urandom bs=8K count=128K | wc -c
   ...
   1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.13074 s, 260 MB/s
   $ time dd if=/dev/urandom bs=8K count=128K of=/dev/null
   ...
   1073741824 bytes (1.1 GB, 1.0 GiB) copied, 3.95569 s, 271 MB/s



My CPU has a Max Turbo Frequency of 3.3 GHz.  I would expect a 4 GHz 
processor to be ~21% faster, but apparently not.



Baseline with pipeline, wc(1), and bs=8K due to unknown Bash pipeline 
bottleneck (?):


2024-02-11 01:18:33 dpchrist@laalaa ~
$ dd if=/dev/urandom bs=8K count=128K | wc -c
1073741824
131072+0 records in
131072+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.27283 s, 251 MB/s


Eliminate pipeline and wc(1):

2024-02-11 01:18:44 dpchrist@laalaa ~
$ dd if=/dev/urandom of=/dev/null bs=8K count=128K
131072+0 records in
131072+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 3.75946 s, 286 MB/s


Increase block size:

2024-02-11 01:18:51 dpchrist@laalaa ~
$ dd if=/dev/urandom of=/dev/null bs=1M count=1K
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 3.62874 s, 296 MB/s


Concurrency:

threads throughput
1   296 MB/s
2   285+286=571 MB/s
3   271+264+266=801 MB/s
4   249+250+241+262=1,002 MB/s
5   225+214+210+224+225=1,098 MB/s
6   223+199+199+204+213+205=1,243 MB/s
7   191+209+210+204+213+201+197=1,425 MB/s
8   205+198+180+195+205+184+184+189=1,540 MB/s



Last time i tested /dev/urandom it was much slower on comparable machines
and also became slower as the amount grew.



Did you figure out why the Linux random number subsystem slowed, and at 
what amount?




Therefore i still have my amateur RNG which works with a little bit of
MD5 and a lot of EXOR. It produces about 1100 MiB/s on the 4 GHz machine.
No cryptographical strength, but chaotic enough to avoid any systematic
pattern which could be recognized by a cheater and represented with
some high compression factor.
The original purpose was to avoid any systematic interference with the
encoding of data blocks on optical media.

I am sure there are faster RNGs around with better random quality.



I assume the Linux kernel in Debian 11 is new enough to support RDRAND (?):

https://en.wikipedia.org/wiki/RdRand


But, my processor is too old to have Secure Key.



$ time perl -MMath::Random::ISAAC::XS -e 
'$i=Math::Random::ISAAC::XS->new(12345678); print pack 'L',$i->irand while 1' | 
dd bs=8K count=128K | wc -c
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 82.6523 s, 13.0 MB/s


Now that's merely sufficient for shredding the content of a BD-RE medium
or a slow USB stick.



Okay.



I suggest using /dev/urandom and tee(1) to write the same CSPRN
stream to all of the devices and using cmp(1) to validate.


I'd propose to use a checksummer like md5sum or sha256sum instead of cmp:

  $random_generator | tee $target | $checksummer
  dd if=$target bs=... count=... | $checksummer

This way one can use unreproducible random streams and does not have to
store the whole stream on a reliable device for comparison.



TIMTOWTDI.  :-)


David

Re: shred bug? [was: Unidentified subject!]

[David Christensen]

On 2/10/24 16:10, Greg Wooledge wrote:

On Sat, Feb 10, 2024 at 04:05:21PM -0800, David Christensen wrote:

2024-02-10 16:03:50 dpchrist@laalaa ~
$ shred -s 1K - | wc -c
shred: -: invalid file type
0


It looks like a shred(1) needs a bug report.


I'm confused what you expected this command to do.  You wanted to
"destroy" (by overwriting with random data) a pipe to wc?  What
would that even look like?

The basic premise of shred is that it determines the size of the file,
then writes data over it, rewinds it, and repeats this a few times.
A pipe to a process has no size, and can't be rewound.

Declaring a pipe to be an "invalid file type" for shredding sounds
pretty reasonable to me.



The documentation is confusing:

On 2/10/24 16:05, David Christensen wrote:
> 2024-02-10 16:03:42 dpchrist@laalaa ~
> $ man shred | grep 'If FILE is -'
> If FILE is -, shred standard output.


David

Re: shred bug? [was: Unidentified subject!]

[David Christensen]

On 2/10/24 04:40, to...@tuxteam.de wrote:

On Sat, Feb 10, 2024 at 11:38:21AM +0100, Thomas Schmitt wrote:

[...]


But shred(1) on Debian 11 refuses on "-" contrary to its documentation:
   shred: -: invalid file type
A non-existing file path causes "No such file or directory".


Hmm. This looks like a genuine bug: the man page mentions it.

Also, /dev/stdout as target runs into the very same problem.

Cheers



Testing:

2024-02-10 16:01:54 dpchrist@laalaa ~
$ cat /etc/debian_version ; uname -a
11.8
Linux laalaa 5.10.0-27-amd64 #1 SMP Debian 5.10.205-2 (2023-12-31) 
x86_64 GNU/Linux


2024-02-10 16:02:34 dpchrist@laalaa ~
$ bash --version | head -n 1
GNU bash, version 5.1.4(1)-release (x86_64-pc-linux-gnu)

2024-02-10 16:02:48 dpchrist@laalaa ~
$ shred --version | head -n 1
shred (GNU coreutils) 8.32

2024-02-10 16:03:42 dpchrist@laalaa ~
$ man shred | grep 'If FILE is -'
   If FILE is -, shred standard output.

2024-02-10 16:03:50 dpchrist@laalaa ~
$ shred -s 1K - | wc -c
shred: -: invalid file type
0


It looks like a shred(1) needs a bug report.


David

Re: testing new sdm drive continued

[David Christensen]

On 2/10/24 08:25, gene heskett wrote:

I managed to kill f3write, so f3probe could access it:
ene@coyote:/mnt/disktest$ sudo f3probe --destructive --time-ops /dev/sdm
F3 probe 8.0
Copyright (C) 2010 Digirati Internet LTDA.
This is free software; see the source for copying conditions.

WARNING: Probing normally takes from a few seconds to 15 minutes, but
  it can take longer. Please be patient.

Bad news: The device `/dev/sdm' is a counterfeit of type limbo

You can "fix" this device using the following command:
f3fix --last-sec=124050943 /dev/sdm

Device geometry:
  *Usable* size: 59.15 GB (124050944 blocks)
     Announced size: 1.91 TB (409600 blocks)
     Module: 2.00 TB (2^41 Bytes)
     Approximate cache size: 1.00 MB (2048 blocks), need-reset=no
    Physical block size: 512.00 Byte (2^9 Bytes)

Probe time: 2.07s
  Operation: total time / count = avg time
   Read: 311.9ms / 4212 = 74us
  Write: 1.75s / 24740 = 70us
  Reset: 1us / 1 = 1us
gene@coyote:/mnt/disktest$
No faster than it is, its not worth the f3fix effort, I can buy 
reputable, much faster sd cards at 1/3rd the cost.



Okay -- here is the thread Thomas referenced.  It has not hit the 
archive servers yet.



The f3probe results are bad news, but not unexpected.  At least you have 
a definite answer that you can send to the seller and/or Amazon (?).



David

Re: Unidentified subject!

[David Christensen]

On 2/10/24 10:28, Thomas Schmitt wrote:

In the other thread about the /dev/sdm test:

Creating file 39.h2w ... 1.98% -- 1.90 MB/s -- 257:11:32
but is taking a few bytes now and then.
[...]
$ ls -l
total 40627044
[...]
$ sudo f3probe --destructive --time-ops /dev/sdm
Bad news: The device `/dev/sdm' is a counterfeit of type limbo
Device geometry:
 *Usable* size: 59.15 GB (124050944 blocks)
Announced size: 1.91 TB (409600 blocks)
...
Probe time: 2.07s



Which other thread?  Please provide a URL to archived post.


So, the 2 TB USB SSD's are really scam 64 GB devices?


David

Re: Unidentified subject!

[David Christensen]

On 2/10/24 02:38, Thomas Schmitt wrote:

I have an own weak-random generator, but shred beats it by a factor of 10
when writing to /dev/null.



As a baseline, here is a 2011 Dell Latitude E6520 with Debian generating 
a non-repeatable 1 GiB stream of cryptographically secure pseudo-random 
numbers:


2024-02-10 14:10:27 dpchrist@laalaa ~
$ lscpu | grep 'Model name'
Model name: Intel(R) Core(TM) i7-2720QM CPU @ 
2.20GHz


2024-02-10 14:01:25 dpchrist@laalaa ~
$ cat /etc/debian_version ; uname -a
11.8
Linux laalaa 5.10.0-27-amd64 #1 SMP Debian 5.10.205-2 (2023-12-31) 
x86_64 GNU/Linux


2024-02-10 14:01:47 dpchrist@laalaa ~
$ bash --version | head -n 1
GNU bash, version 5.1.4(1)-release (x86_64-pc-linux-gnu)

2024-02-10 14:13:08 dpchrist@laalaa ~
$ time dd if=/dev/urandom bs=8K count=128K | wc -c
1073741824
131072+0 records in
131072+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.30652 s, 249 MB/s

real0m4.311s
user0m0.107s
sys 0m4.695s


If the OP has a known good storage device of equal or larger size than 
the UUT(s), I suggest using /dev/urandom and tee(1) to write the same 
CSPRN stream to all of the devices and using cmp(1) to validate.



I use Perl.  When I needed a CSPRNG, I searched and found:

https://metacpan.org/pod/Math::Random::ISAAC::XS


Using Perl and Math::Random::ISAAC::XS to generate a repeatable stream 
of cryptographically secure pseudo-random numbers:


2024-02-10 14:09:12 dpchrist@laalaa ~
$ perl -v | head -n 2 | grep .
This is perl 5, version 32, subversion 1 (v5.32.1) built for 
x86_64-linux-gnu-thread-multi


2024-02-10 14:09:53 dpchrist@laalaa ~
$ perl -MMath::Random::ISAAC::XS -e 'print 
$Math::Random::ISAAC::XS::VERSION, $/'

1.004

2024-02-10 14:10:32 dpchrist@laalaa ~
$ time perl -MMath::Random::ISAAC::XS -e 
'$i=Math::Random::ISAAC::XS->new(12345678); print pack 'L',$i->irand 
while 1' | dd bs=8K count=128K | wc -c

1073741824
131072+0 records in
131072+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 82.6523 s, 13.0 MB/s

real1m22.657s
user1m22.892s
sys 0m5.810s


The 'perl ... | dd ...' pipeline appears to be limited to a block size 
of 8,192 bytes.  I do not know if this is due to bash(1), perl(1), or 
dd(1) (?).



The repeatability of the ISAAC stream would substitute for the known 
good storage device, but the throughput is ~19x slower than 
/dev/urandom.  A drive capacity validation tool would want to feature 
concurrent threads and I/O on SMP machines.



David

Re: Things I don't touch with a 3.048m barge pole: USB storage (WasRe: Unidentified subject!)

[David Christensen]

On 2/9/24 04:53, gene heskett wrote:

Interesting report from gdisk however:
GPT fdisk (gdisk) version 1.0.9

Partition table scan:
   MBR: MBR only
   BSD: not present
   APM: not present
   GPT: not present


***
Found invalid GPT and valid MBR; converting MBR to GPT format
in memory. THIS OPERATION IS POTENTIALLY DESTRUCTIVE! Exit by
typing 'q' if you don't want to convert your MBR partitions
to GPT format!
***


Warning! Secondary partition table overlaps the last partition by
33 blocks!
You will need to delete this partition or resize it in another utility.

Command (? for help):
Command (? for help): p
Disk /dev/sdm: 409600 sectors, 1.9 TiB
Model: SSD 3.0
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): 3230045D-589D-4601-8C4D-E9C4684B9657
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 409566
Partitions will be aligned on 64-sector boundaries
Total free space is 30 sectors (15.0 KiB)

Number  Start (sector)    End (sector)  Size   Code  Name
    1  64  409599   1.9 TiB 0700  Microsoft 
basic data


Command (? for help): q

What do we make of that?  Some sort of NTFS?



Do these commands produce any clues or error messages?

# fdisk -l /dev/sdm

# tail /var/log/messages

# dmesg | tail


David

Re: testing new sdm drive

[David Christensen]

On 2/9/24 00:51, gene heskett wrote:

On 2/8/24 13:25, David Christensen wrote:

On 2/7/24 23:14, gene heskett wrote:

gene@coyote:/etc$ sudo smartctl --all -dscsi /dev/sdm
...
scsiModePageOffset: response length too short, resp_len=4 offset=4 
bd_len=0
scsiModePageOffset: response length too short, resp_len=4 offset=4 
bd_len=0

 >> Terminate command early due to bad response to IEC mode page
A mandatory SMART command failed: exiting. To continue, add one or 
more '-T permissive' options.

gene@coyote:/etc$

And then again, it worked, sorta
...
Please try again with the drive connected directly to a motherboard 
USB 3.0 port.



That is where it still is, on a blue usb3.0 port on a 3 yo ASUS mobo.



Does smartctl(8) fail when you connect the USB SSD to other USB ports?


David

Re: testing new sdm drive

[David Christensen]

On 2/8/24 12:36, Linux-Fan wrote:

Alexander V. Makartsev writes:

From here on I'd suggest trying the tools from package `f3`.



Thank you for the suggestion -- I was hoping somebody knew of a FOSS 
Debian package that can validate drive capacity:


https://packages.debian.org/bookworm/f3

https://oss.digirati.com.br/f3/

https://fight-flash-fraud.readthedocs.io/en/stable/


David

Re: testing new sdm drive

[David Christensen]

On 2/8/24 11:23, gene heskett wrote:

On 2/8/24 07:22, Alexander V. Makartsev wrote:

This is how I would test it.
...
Looks neat. Any chance this will crash my machine? I have other design 
work going on, and I'd hate to have to start from scratch.



Do not use a production computer for drive maintenance; especially not 
your primary workstation.  Use a spare computer.



I thought you were going to hook up all the new USB SSD's to a USB hub 
to a SBC, and turn it into a file server, backup server, or some such?



David

Re: testing new sdm drive

[David Christensen]

On 2/8/24 10:24, David Christensen wrote:

On 2/7/24 23:14, gene heskett wrote:

gene@coyote:/etc$ sudo smartctl --all -dscsi /dev/sdm
...
scsiModePageOffset: response length too short, resp_len=4 offset=4 
bd_len=0
scsiModePageOffset: response length too short, resp_len=4 offset=4 
bd_len=0

 >> Terminate command early due to bad response to IEC mode page
A mandatory SMART command failed: exiting. To continue, add one or 
more '-T permissive' options.

...
Please try again with the drive connected directly to a motherboard USB 
3.0 port.



Error:  The motherboard has no USB 3.0 ports.


Correction:  Please try connecting the drive to each and every 
motherboard USB port to see if and which ports work with the drive and 
smartctl(8).



David

Re: testing new sdm drive

[David Christensen]

On 2/7/24 23:14, gene heskett wrote:

gene@coyote:/etc$ sudo smartctl --all -dscsi /dev/sdm
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-6.1.0-17-rt-amd64] (local 
build)

Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:
Product:  SSD 3.0
Revision: 2.00
Compliance:   SPC-2
User Capacity:    2,097,152,000,000 bytes [2.09 TB]
Logical block size:   512 bytes
scsiModePageOffset: response length too short, resp_len=4 offset=4 bd_len=0
scsiModePageOffset: response length too short, resp_len=4 offset=4 bd_len=0
 >> Terminate command early due to bad response to IEC mode page
A mandatory SMART command failed: exiting. To continue, add one or more 
'-T permissive' options.

gene@coyote:/etc$

And then again, it worked, sorta

Cheers, Gene Heskett, CET.



Please try again with the drive connected directly to a motherboard USB 
3.0 port.



I seem to recall that you have a lot of USB devices connected to your 
computer(s).  The Asus PRIME Z370-A II Series manual page ix states:


Intel ® Z370 Chipset
- 6 x USB 3.1 Gen 1 ports (4 ports @mid-board, 2 ports @back panel)
USB
- 6 x USB 2.0/1.1 ports (4 ports @mid-board, 2 ports @back panel)
Asmedia ® USB 3.1 Gen 2 controller
- 1 x USB 3.1 Gen 2 port @back panel (teal blue, Type-A)
- 1 x USB 3.1 Gen 2 port @back panel (USB Type CTM)


Page 1-16 states:

USB 3.1 Gen 1 connectors (20-1 pin U31G1_12; U31G1_34)

This connector allows you to connect a USB 3.1 Gen 1 module for 
additional USB 3.1 Gen 1 front or rear panel ports. With an installed 
USB 3.1 Gen 1 module, you can enjoy all the benefits of USB 3.1 Gen 
1including faster data transfer speeds of up to 5 Gb/s, faster charging 
time for USB-chargeable devices, optimized power efficiency, and 
backward compatibility with USB 2.0.


The USB 3.1 Gen 1 module is purchased separately.


Page 1-17 states:

USB 2.0 connectors (10-1 pin USB910; USB1112)

These connectors are for USB 2.0 ports. Connect the USB module cable to 
these connectors, then install the module to a slot opening at the back 
of the system chassis. This USB connector complies with USB 2.0 
specification that supports up to 480 Mb/s connection speed.


The USB 2.0 module is purchased separately.


STFW including asus.com, I am unable to find "USB 3.1 Gen 1 module" or 
"USB 2.0 module" (?).



Does your chassis have front panel USB 2.0 and/or USB 3.1 Gen 1 ports 
with cables and matching connectors?  Have you connected them to the 
motherboard headers?  Do they work?



Alternatively, if you have an available chassis expansion slot:

https://www.startech.com/en-us/cables/usbplate4


I am unable to find a similar part for the motherboard USB 3.1 Gen 1 
20-pin headers.  Perhaps a USB 3.0 will work (?):


https://www.amazon.com/RIITOP-Female-Connector-Adapter-Bracket/dp/B01KJPUI5W


Or, if you have an available motherboard PCIe slot:

https://www.startech.com/en-us/cards-adapters/usb-30/cards?filter_bustype=pci%2520express


David

Portable External Hard Drive 2TB (was: Unidentified subject!)

[David Christensen]

On 1/22/24 19:55, gene heskett wrote:
> 2T ssd's:
>
> 




It appears Amazon took down the above web page.  Using Amazon search, I see:

https://www.amazon.com/Portable-External-USB3-1-Desktop-Chromebook/dp/B0CSNGML52/


On 2/7/24 12:30, gene heskett wrote:
> Well the 2T memory everybody was curious about 3 weeks ago got here 
early.

>
>  From dmesg after plugging one in:
> [629240.916163] usb 1-2: new high-speed USB device number 39 using 
xhci_hcd

> [629241.066221] usb 1-2: New USB device found, idVendor=048d,
> idProduct=1234, bcdDevice= 2.00
> [629241.066234] usb 1-2: New USB device strings: Mfr=1, Product=2,
> SerialNumber=3
> [629241.066239] usb 1-2: Product: Disk 3.0
> [629241.066242] usb 1-2: Manufacturer: USB
> [629241.066246] usb 1-2: SerialNumber: 2697241127107725123
> [629241.069485] usb-storage 1-2:1.0: USB Mass Storage device detected
> [629241.074187] scsi host37: usb-storage 1-2:1.0
> [629242.100738] scsi 37:0:0:0: Direct-Access  SSD 3.0   2.00
> PQ: 0 ANSI: 4
> [629242.100959] sd 37:0:0:0: Attached scsi generic sg13 type 0
> [629242.101190] sd 37:0:0:0: [sdm] 409600 512-byte logical blocks:
> (2.10 TB/1.91 TiB)
> [629242.101289] sd 37:0:0:0: [sdm] Write Protect is off
> [629242.101290] sd 37:0:0:0: [sdm] Mode Sense: 03 00 00 00
> [629242.101409] sd 37:0:0:0: [sdm] No Caching mode page found
> [629242.101410] sd 37:0:0:0: [sdm] Assuming drive cache: write through
> [629242.103927]  sdm: sdm1
> [629242.104047] sd 37:0:0:0: [sdm] Attached SCSI disk
> gene@coyote:
>
> Looks like a reasonable facsimile of a 2T disk to me.
>
> Cheers, Gene Heskett, CET.


I would be curious to see a smartctl(8) full report:

# smartctl -x /dev/sdm


Have you figured out how to verify it actually holds 2 TB?


David

Re: Copy from Firefox and paste into Terminal with Vim

[David Christensen]

On 2/6/24 06:25, Max Nikulin wrote:

On 06/02/2024 13:28, David Christensen wrote:

On 2/5/24 19:03, Max Nikulin wrote:

    xclip -o -selection PRIMARY
    xclip -o -selection CLIPBOARD


That is useful.


I expected that you would try both commands when vim is unable to paste. 
It would allow to discriminate whether it is Firefox or Vim issue.



I tried using those commands, but was getting confusing results.  If and 
when my Debian, X, Xfce, Terminal, and/or Vim misbehave again, I will 
see if those commands produce any clues.




If I start Firefox, browse to https://www.toyota.com/dealers, select the
the first dealer contents, start Vim, press and release the 
double-quote key, press and release the asterisk key, and press and 
release the p key, Vim inserts a blank line.


What is the effect of ["], [+], [p] keystrokes?



https://lists.debian.org/debian-user/2024/02/msg00277.html


or C-R* in insert mode. 


I do not understand your notation:

 C-R*


[Ctrl+R], [*] or [Ctrl+R], [+]

     :help i_CTRL-R



In command mode:

* Pressing Ctrl+R causes the bottom line to say:

Already at newest change

* Pressing Ctrl+R then + causes the cursor to move down one line.

* Pressing Ctrl+R then * causes the bottom line to say:

search hit BOTTOM, continuing at TOP


In insert mode:

* Pressing Ctrl+R causes the following two characters to be inserted 
into columns 53 and 54 of the bottom line:


^R

* Pressing Ctrl+R then + causes the two characters to be removed from 
the bottom line


* Pressing Ctrl+R then * causes the two characters to be removed from 
the bottom line




... Ctrl+Shift+V


I was trying to recommend against it, despite currently bracketed paste 
is enabled by default in BASH.



https://lists.debian.org/debian-user/2024/02/msg00276.html



https://security.stackexchange.com/questions/39118/how-can-i-protect-myself-from-this-kind-of-clipboard-abuse



Yuck.


I wonder if the https://www.toyota.com/dealers content I was attempting 
to paste into Vim contained exploits (?).  I have rebooted several 
times; perhaps the effects were not persistent (?).  Toyota's web site 
is different today; perhaps Toyota removed the exploits (?).



I wonder if the following will display exploits (?):

$ xclip -o -selection PRIMARY

$ xclip -o -selection CLIPBOARD


I wonder if my Vim has blocked "+p and "*p as a means to protect against 
clipboard attacks?




As to vim docs

     :help gui-selection
     :help quoteplus



https://lists.debian.org/debian-user/2024/02/msg00277.html



and for completeness

     :help registers



It seems that I have been conflating the terms and concepts "buffer" and 
"register".




Perhaps

     :help user-manual

contains description as well



Vim has more features than I know or use.


David

Re: Copy from Firefox and paste into Terminal with Vim

[David Christensen]

On 2/6/24 05:48, John Hasler wrote:

My .vimrc contains

syntax on
set mouse-=a

And pasting works.



Thank you for the reply.  :-)


If and when Firefox, Debian, X, Xfce, Terminal, and/or Vim misbehave 
again, I will try your suggestions.




VIM - Vi IMproved 9.0 (2022 Jun 28, compiled Nov 20 2023 16:05:25)
Included patches: 1-2116



2024-02-06 23:39:21 dpchrist@laalaa ~
$ cat /etc/debian_version ; uname -a ; vim --version | head -n 2
11.8
Linux laalaa 5.10.0-27-amd64 #1 SMP Debian 5.10.205-2 (2023-12-31) 
x86_64 GNU/Linux

VIM - Vi IMproved 8.2 (2019 Dec 12, compiled Oct 01 2021 01:51:08)
Included patches: 1-2434


David

Re: Copy from Firefox and paste into Terminal with Vim

[David Christensen]

On 2/6/24 03:33, Ralph Aichinger wrote:

On Mon, 2024-02-05 at 15:14 -0800, David Christensen wrote:

I am unable to determine if the problem is Firefox, Vim, or something
else.

Comments or suggestions?


As others have written, vim has changed copy+paste defaults some time
ago. Some even call this changing defaults "they broke copy+paste" ;).
One easy thing you might want to check is if this also happens in
neovim. Neovim did not make this copy+paste change, and it might
behave subtly different. It's quick to install and check.

/ralph



Thank you for the reply.  :-)


I prefer to use Vim, as it seems to be available and/or installed on all 
of the various hosts I use (including those I do not administer).



David

Re: Copy from Firefox and paste into Terminal with Vim

[David Christensen]

On 2/6/24 00:12, Klaus Singvogel wrote:

David Christensen wrote:

On 2/5/24 21:45, to...@tuxteam.de wrote:

Try ":set mouse=" and see whether it helps. Perhaps it's that.


That's the way. That's the fix for the root cause.



Thank you for the reply.  :-)


Currently, Firefox, Vim, select, copy, and/or paste seem to be behaving. 
 If and when the gremlins wake up, I'll try your suggestion.



David

Re: Copy from Firefox and paste into Terminal with Vim

[David Christensen]

On 2/6/24 04:28, Greg Wooledge wrote:

On Mon, Feb 05, 2024 at 10:28:53PM -0800, David Christensen wrote:

Continuing from above in Vim in Insert mode, if I then simultaneously press
the Ctrl, Shift, and v keys, and then release all keys, Vim inserts the
contents of the clipboard; as confirmed by:

 xclip -o -selection CLIPBOARD


How's that possible?



Thank you for the reply.  :-)


Apparently, something in the 920M bytes of my boot file system and/or in 
the 8009M bytes of my root file system make it possible.




Are you running a GUI version of vim (gvim?)



No.



instead of running vim in a terminal?



I am running Vim in Terminal.



Or are you using an exotic terminal?



I do not believe so.



If you're using a terminal that isn't xterm, please specify which.



I used the Xfce Panel Preferences dialog to create a Launcher item with 
one Terminal Emulator item:


Name:   Terminal Emulator
Comment:Use the command line
Command:exo-open --launch TerminalEmulator
Working Directory:
Icon:   org.xfce.terminalemulator
Options:
checked Use startup notification
unchecked   Run in terminal


I click on the panel icon to start Terminal.



In xterm and urxvt, Ctrl-Shift-v is identical to Ctrl-v ("literal
next"), so there's no way vim can distinguish the two.  And yes, I
tested it just to be sure.  In both xterm and urxvt, vim, insert > mode, 
Ctrl-Shift-v acts exactly like Ctrl-v.



I am seeing similar behavior:

https://lists.debian.org/debian-user/2024/02/msg00276.html



This applies in general to *any* issue that involves exotic key
combinations, because different terminals handle them differently.



Okay.


David