Re: live-build: what do live-task-* do?

[Thomas Schmitt]

Hi,

Hans wrote:
> I am playing around with live-build.

This package has its own specialized mailing list:
  debian-l...@lists.debian.org


Have a nice day :)

Thomas

Re: Will te UUID or blkid of a device change?

[Thomas Schmitt]

Hi,

Hans wrote:
> > I want to make sure, that the correct USB-stick is used.
> > Thus I can do by using the UUID of the target stick like
> > dd if=/path/to/myfile.iso of=UUID="123456-abcd-"

David Wright wrote:
>   # dd of=/dev/disk/by-id/JetFlash_Transcend_4GB_JKNB2FYG-0:0  …  …

This is indeed a good solution if the device ID is known and systemd does
not change its way of composing the "by-id" name.
But if you first have to find out the right "by-id" name, then there is
again the risk of user error, especially when in a hurry.


There is a different, interactive approach which depends on the fact that
the Linux kernel creates a new device file when a USB stick is plugged in:

  https://packages.debian.org/stable/xorriso-dd-target
  https://wiki.debian.org/XorrisoDdTarget

The man page of xorriso-dd-target demonstrates more use cases, like:
  - List all devices with reasoning
  - Evaluate particular given devices


Have a nice day :)

Thomas

Re: How to create a custom Debian ISO

[Thomas Schmitt]

Hi,

Aditya Garg wrote:
> This one is gonna be interesting.
> Wish me luck.

Fingers are crossed ...

(But everything in the procedure is supposed to be deterministic. So there
is few room for luck, good or bad. We rather have to navigate the chaos.)


Have a nice day :)

Thomas

Re: How to create a custom Debian ISO

[Thomas Schmitt]

Hi,

to...@tuxteam.de wrote:
> Not the OP, but thanks, Thomas.

Well, ISO 9660 is known to be my hobby. So i can hardly resist trying
to acquire new users for xorriso.


Have a nice day :)

Thomas

Re: How to create a custom Debian ISO

[Thomas Schmitt]

Hi,

Aditya Garg wrote:
> I would prefer making the ISO as similar to the official Debian ISO and just
> replace the Debian kernel with the customised kernel.

In that case, i'd go along
  https://wiki.debian.org/RepackBootableISO
Either by using the xorrisofs options in /.disk/mkisofs of the ISO :

  
https://wiki.debian.org/RepackBootableISO#Learn_about_the_actually_used_ISO_production_command

or by relying on the capability of xorriso to determine the commands
which will reproduce the boot equipmemt of the loaded ISO :

  
https://wiki.debian.org/RepackBootableISO#In_xorriso_load_ISO_tree_and_write_modified_new_ISO

If you need help with finding the appropriate xorriso commands, ask me
in private or in public at bug-xorr...@gnu.org .


What remains is to find out whether this works out of the box or whether
the kernel has to be announced in some files of the ISO or even
cryptographically signed in some way.

--

Just in case your adventure goes beyond replacing the kernel and possibly
the boot loader menu files, i warn of a bug in xorriso-1.5.6 and older:
Don't overwrite the El Torito boot image files in a xorriso run that uses
  -boot_image "any" "replay"
The boot image files in Debian amd64 ISOs are /isolinux/isolinux.bin
and /boot/grub/efi.img .
If you need to replace them, then we have to talk.


Have a nice day :)

Thomas

[off topic] High Sierra, was: Cindex

[Thomas Schmitt]

Hi,

Brad Rogers quoted:
> macOS 10.13 (High Sierra)

Hah ! Do they think that ISO 9660 is dead enough so they can highjack
its birth name ?

  https://en.wikipedia.org/wiki/ISO_9660
  "ISO 9660 traces its roots to the High Sierra Format, [...]
   In November 1985, representatives of computer hardware manufacturers
   gathered at the High Sierra Hotel and Casino [...] in Stateline, Nevada.
   This group became known as the High Sierra Group (HSG).
   Present at the meeting were representatives from Apple Computer [...]"

A company now run by cultureless bleeps suffering from historical amnesia.


Have a nice day :)

Thomas

Re: How to create a custom Debian ISO

[Thomas Schmitt]

Hi,

Aditya Garg wrote to debian-devel:
> > I wanted to create a custom ISO of Debian, with the following 
> > customisations:
> > 1. I want to add a custom kernel that supports my Hardware.
> > 2. I want to add my own Apt repo which hosts various software packages to
support my hardware.
> > I am not able to get any good documentation for the same. Please help.

Marvin Renich wrote:
> The package live-build from the Debian Live project might help you do
> what you want.

Indeed the live-build package seems to be in use outside Debian's own
ISO production. Mailing list is
  debian-l...@lists.debian.org
There exists a manual
  https://live-team.pages.debian.net/live-manual/html/live-manual/index.en.html

Installation ISOs are made by package debian-cd, of which i am not aware
that it would have have users outside the official ISO production.i
Mailing list is
  debian...@lists.debian.org
Your impression about lack of documentation is not wrong as far as this
project is concerned. :))


Nevertheless the production step of packing up the ISO from a prepared
file tree is documented together with methods to use a Debian installation
ISO as base for the preparation:
  https://wiki.debian.org/RepackBootableISO

Packages may probably be added at the appropriate place in the directory
tree under /pool. (Managing a Debian repo is not my turf. Sorry for
being vague here.)

Changing the content of a Debian ISO might need some follow-up work in
administrative files of the ISO.
When merging Debian ISOs, my script
  
https://dev.lovelyhq.com/libburnia/libisoburn/raw/branch/master/test/merge_debian_isos
manipulates:
  /README.txt
  /dists/*/Release
and merges the files listed in /dists/*/Release.
You would have to explore whether these files are affected by your
changes.


Have a nice day :)

Thomas

Re: Adding package to Debian Distro

[Thomas Schmitt]

Hi,

kiruthikaanbusuresh wrote:
> Hi Debian Team,

Standard disclaimer: We are the users. A team only by coincidence.

(And you seem not to be subscribed to the mailing list.
Thus i CC: your mail address.)


> There is a package by name rsct which is specific to IBM. I
> would like to know the process to get this added to the Debian Distro.

If it complies to
  https://wiki.debian.org/DebianFreeSoftwareGuidelines
and is of some general use, then have a look at
  https://wiki.debian.org/RFP


> Should I have to get sponsorship for getting it added to Debian ?

If you want to contribute own work to this packaging endeavor: Yes.
It will probably increase your chances for success.

See
  https://mentors.debian.net/
  https://wiki.debian.org/ITP


Have a nice day :)

Thomas

Re: bootable pendrive from zip file

[Thomas Schmitt]

Hi,

Max Nikulin wrote:
> I was not aware that partition type might be an issue.

Thanks to the normative power of the facts a "may" in the specs becomes
a reason to return a mainboard with an EFI that chooses to join the
"may not" side.


Have a nice day :)

Thomas

Re: bootable pendrive from zip file

[Thomas Schmitt]

Hi,

i wrote:
> > It is disputed, whether the specs say that the partitions must be marked
> > by 0xEF in legacy MBR tables and by C12A7328-F81F-11D2-BA4B-00A0C93EC93B
> > in GPT.

Max Nikulin wrote:
> It happened so that I had locally a file with UEFI spec Version 2.3.1,
> Errata C June 27, 2012.
> [...]
> "12.3.3 Number and Location of System Partitions
> ... Further, UEFI implementations may allow the use of conforming FAT
> partitions which do not use the ESP GUID."
> [...]
> From my point of view it is opposed to "must be" for strict partition type
> checks.

It is not forbidden, indeed. But it is not guaranteed that it works.
So if the boot success were not covered by tradition, it would be a
case of "your mileage may vary".

Another problem with the statement is that it only talks of GUID and
thus of GPT partitioning, while the specs allow MBR partitioning too.
It needs a bit of semantical stretching to let it cover the whole specs.


> Later versions may have some updates.

The statement is still in UEFI 2.8 as 13.3.3.


> Some details are in 12.3.4.2 Diskette
> USB pen drive is not a diskette, but it increases probability that
> superfloppy formatting style is supported. Of course, singe FAT partition is
> more portable.

If that's a quote from the specs, then it isn't in UEFI 2.8 any more.
If it's a statement by you, then i agree that there is a chance for
unpartitioned USB sticks to work.
You may also see implementations which boot via the El Torito boot
catalog from USB stick and via a partition table from optical media.


Have a nice day :)

Thomas

Re: bootable pendrive from zip file

[Thomas Schmitt]

Hi,

Max Nikulin wrote:
> Out of curiosity, does the requirement of specific GUID exist for removable
> drives?

It is disputed, whether the specs say that the partitions must be marked
by 0xEF in legacy MBR tables and by C12A7328-F81F-11D2-BA4B-00A0C93EC93B
in GPT.
In practice there seems to be no such demand. A zillion Microsoft users
would complain if the specs were interpreted narrowly.


> A USB drive may be formatted without partition table.

The specs only talk of partitions. Whether the real implementations would
look into the FAT filesystem of an unpartitioned device would have to be
tested.

(Boot firmware is a bitch. The hunchbacked partition layout of Debian ISOs
is still the one which boots on most machines. Other distros decided to
clean up at the cost of losing some old laptops.)


> 7z and bsdtar can extract content of ISO files without mounting images.

But mounting needs no special software and gives you the opportunity
to use many different ways of copying, which may be decisive when the
target is a heavily restricted filesystem like FAT.

(I still wonder which software in the Debian ISO needs the symbolic link
"/debian -> ." and which parts of the file tree are accessed via this
link. Probably one can avoid to duplicate the whole tree under /debian.)


Have a nice day :)

Thomas

Re: bootable pendrive from zip file

[Thomas Schmitt]

Hi,

Luis Muñoz Fuente wrote:
> I assume the problem is the debian link, which points to the same directory:
> $ ls -l tmp/debian
> lrwxrwxrwx 1 user user 1 Apr 22 20:47 tmp/debian -> .
> and creates a loop,

That's not a link loop, because "." is not a symbolic link.
But if a tree traversal is instructed to follow symbolic links, then the
traversal becomes endless recursion which is quite equivalent to an
endless loop.

I guess it would work better with zip option --symlinks.
But FAT cannot represent symbolic links. So in the FAT filesystem you will
possibly need at least one layer of duplication to emulate the link.
I.e. a copy of the whole file tree which is accessible via /debian. In
that copy there will probably be no need for another tree under
/debian/debian.


Have a nice day :)

Thomas

Re: bootable pendrive from zip file

[Thomas Schmitt]

Hi,

Luis Muñoz Fuente wrote:
> why does extracting the files from the debian iso increase the
> size so much?

Hard to say if you do not show what you do in particular.


In general an increase of about 120 MB is to be expected because of
expansion of hardlinks:

  $ du debian-12.2.0-amd64-netinst.iso
  643076  debian-12.2.0-amd64-netinst.iso
  $ sudo mount debian-12.2.0-amd64-netinst.iso /mnt/iso
  mount: /dev/loop0 is write-protected, mounting read-only
  $ du -s /mnt/iso
  762497  /mnt/iso

The bulk of duplication is with directory trees /firmware and
/pool/non-free-firmware . Some is with kernels and initrds.


> When I take a folder that occupies 5 GiB and with mkisofs I create an iso
> file, it still occupies 5 GiB.

Not if your disk filesystem supports sparse files and your files contain
substantial areas of unwritten bytes. In that case the ISO will be larger.

> And if I later extract the files it takes up 5 GiB again,

Not if there was a substantial amount of hardlink siblings among your
input files. mkisofs will store them with shared content but Linux will
represent them as independent files.

But an increase of an amd64 netinst ISO from 659 to 1500 MB cannot be
explained by hardlinks alone. Maybe you put it into the ZIP archive
twice ?


Have a nice day :)

Thomas

Re: bootable pendrive from zip file

[Thomas Schmitt]

Hi,

Luis Muñoz Fuente wrote:
> I recently used clonezilla and followed these instructions:
> https://clonezilla.org/liveusb.php#linux-setup

The variation for "uEFI", i assume.

This aims at an undocumented habit of EFI implementations to look in
any FAT filesystem for a \EFI\BOOT directory with a suitable BOOT*.EFI
file and to start it, if found.
(Officially documented is to look in FAT filesystems of partitions with
MBR type 0xEF or GPT type GUID C12A7328-F81F-11D2-BA4B-00A0C93EC93B.)


> I have tried to transform the debian-12.5.0-amd64-netinst.iso file into zip
> but the size has gone from 659 MiB to 1.5 GiB, [...]
> Does anyone know why this happens.

To get an answer you will have to show what you did and where you
measured the resulting size (as ZIP archive file or on the USB stick ?).


But i don't think that intermediate storage as ZIP is needed at all.

The make-bootable-by-copy trick depends on /EFI/BOOT and BOOT*.EFI files
being in the ISO or ZIP archive. A Debian netinst ISO filesystem for amd64
contains an unpacked copy of its EFI boot partition files.
(Others do too, thanks to the relentless proselytization of Pete Batard,
 the developer of program Rufus.)

So just mount the ISO:

  $ sudo mount debian-12.5.0-amd64-netinst.iso /mnt/iso
  mount: /dev/loop0 is write-protected, mounting read-only

and copy its content to the mounted USB stick.
You will perceive about 100 MB increase in size, because Linux does not
represent the hardlinks in the ISO which save some space.

The result is supposed to boot where a Clonezilla stick boots after it
was made by unpacking the ZIP archive.
Another question is how far the programs in a Debian netinst ISO are
prepared to run from a FAT filesystem and to find their files in it.

Your mileage may vary.


Illustration: The two copies of the \EFI\BOOT directory in a netinst ISO

  $ sudo mount debian-12.2.0-amd64-netinst.iso /mnt/iso
  $ find /mnt/iso/EFI/boot
  /mnt/iso/EFI/boot
  /mnt/iso/EFI/boot/bootia32.efi
  /mnt/iso/EFI/boot/bootx64.efi
  /mnt/iso/EFI/boot/grubia32.efi
  /mnt/iso/EFI/boot/grubx64.efi
  $ mount /mnt/iso/boot/grub/efi.img /mnt/fat
  $ find /mnt/fat/efi/boot | sort
  /mnt/fat/efi/boot
  /mnt/fat/efi/boot/bootia32.efi
  /mnt/fat/efi/boot/bootx64.efi
  /mnt/fat/efi/boot/grubia32.efi
  /mnt/fat/efi/boot/grubx64.efi
  $


Have a nice day :)

Thomas

Re: LibreOffice removed from Debian

[Thomas Schmitt]

Hi,

Vincent Lefevre wrote:
> Is there any reason why LibreOffice has been removed from Debian???

  https://tracker.debian.org/pkg/libreoffice

says

  The dependencies of libuno-cppuhelpergcc3-3=4:24.2.0-1 cannot be
  satisfied in unstable on arm64, s390x, i386, ppc64el, armel, amd64,
  and armhf
  [...]
  Depends on packages which need a new maintainer
  [...]
  This package has been requested to be removed. [...]
  Please see bug number #1069123 for more information.

This bug looks somewhat like a misunderstanding between the maintainer
and the Debian FTP Master.
The maintainer declares a lot of packages to be "cruft"
(https://en.wikipedia.org/wiki/File:Harvard_Cruft_Hall.png)
referring to various arches, but not to amd64.
The Masters seem to react by marking the source package "libreoffice"
for removal. (It is in the maintainer's cruft list, indeed.)


Have a nice day :)

Thomas

Re: NextGov: Linux XZ Utils Backdoor Was Long Con, Possibly With Support

[Thomas Schmitt]

Hi,

Nicholas Geovanis wrote:
> But what if next time the back-doored software _does_ build without error?

The initial build problems did not cause suspicion.
It was the CPU load of sshd and an obscure complaint by valgrind which
caused the discovery.
  https://boehs.org/node/everything-i-know-about-the-xz-backdoor
quotes the discoverer Andres Freund:
  "I was doing some micro-benchmarking at the time, needed to quiesce
   the system to reduce noise. Saw sshd processes were using a surprising
   amount of CPU, despite immediately failing because of wrong usernames
   etc. Profiled sshd, showing lots of cpu time in liblzma, with perf
   unable to attribute it to a symbol. Got suspicious. Recalled that I had
   seen an odd valgrind complaint in automated testing of postgres, a few
   weeks earlier, after package updates.
   Really required a lot of coincidences."


gene heskett wrote:
> In light of that its worth noting that an M$ employee was the first to
> spot it.

Indeed.
Thus we should also praise the peace between Microsoft and free software
which broke out a few years ago.


There remains the question, whom a good citizen should contact when
spotting something that could be a backdoor (or a subtenant ?) of
Debian's content or infrastructure.

It seems unwise for a non-expert to do this in public, unless one wants
to accuse the innocent or to warn the hoodlums.


Have a nice day :)

Thomas

Re: Debian ISOs on USB stick

[Thomas Schmitt]

Hi,

i read from bytes 2085412 to 2085479:
  "Info rrmation Syste rm VolumeSYSTEM~"
which is similar to the alterations of one of the USB sticks shown in
  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1056998#35

The web knows about a Microsoft folder named "System Volume Information".
  
https://answers.microsoft.com/en-us/windows/forum/all/system-volume-information-what-is-it-and-what-is/3bc81844-0baa-46bd-9949-4efb4678b677
  "whenever I put my flash-drive or my micro sd adapter and sd card into
   my windows 8.1 something called "System Volume Information" is always
   getting added on."

So did you perhaps show this USB stick to a running MS-Windows system ?


Have a nice day :)

Thomas

Re: Debian ISOs on USB stick

[Thomas Schmitt]

Hi,

David Christensen wrote:
> # cmp --verbose debian-11.3.0-amd64-netinst.iso /dev/sdb

I got my copy from
  
https://get.debian.org/images/archive/11.3.0/amd64/iso-cd/debian-11.3.0-amd64-netinst.iso
SHA256 matches:
  7892981e1da216e79fb3a1536ce5ebab157afdd20048fe458f2ae34fbc26c19b

In a further mail:
> https://cdimage.debian.org/cdimage/archive/11.3.0/amd64/iso-cd/

Same SHA256 there.


>   2083201   0 377

Byte counting of cmp is decimal and starts at 1. xorriso can search for
files which have their data in a block range. 2083201 / 2048 = block 1017.
Range size in this case is just 1 block:

  $ xorriso -indev debian-11.3.0-amd64-netinst.iso -find / -lba_range 1017 1 
-exec report_lba --
  ...
  Report layout: xt , Startlba ,   Blocks , Filesize , ISO image path
  File data lba:  0 , 1016 , 1296 ,  2654208 , '/boot/grub/efi.img'

So it's indeed occupied by the FAT filesystem image which contains the
EFI-specific boot equipment.

>   4719105   0  56

Byte 4719105 is in block 2304, i.e. still in /boot/grub/efi.img, which
has bytes up to the end of block 2311.

I guess the bytes with the 2xx numbers are the directory change and
the 4xx numbers are content of new files.


You could mount both ISOs (e.g. at /mnt/iso1 and /mnt/iso2) and then the
two FAT image files (e.g. /mnt/iso1/boot/grub/efi.img and
/mnt/iso2/boot/grub/efi.img) in order to learn which files have emerged
or changed in the USB stick's mounted FAT filesystem.

Maybe we find a new ESP groper additionaly to Lenovo and Microsoft.
Usually they leave traces for which one can search in the web.


Have a nice day :)

Thomas

Debian ISOs on USB stick, was: SOLVED

[Thomas Schmitt]

Hi,

David Christensen wrote:
> It's a relatively simple experiment to confirm that a USB flash drive with
> d-i changes after the first boot.

This could still be
  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1056998
where Lenovo BIOS and/or MS-Windows altered the USB stick.


> Same for finding which bytes change.

I fail to find this particular info in
  Date: Tue, 2 Apr 2024 14:46:42 -0700
  From: David Christensen 
  Message-ID: 

If we have the exact ISO name (i.e. URL from where it stems) and the
byte address of the alteration, xorriso can find the affected file, if
any.

In case of bug #1056998 it was the EFI partition image /boot/grub/efi.img.
Mounting the altered and unaltered image files showed changes in the
FAT filesystem which point to the culprits Lenovo and Microsoft.


The other plausible way of altering the ISO image on the stick would be
adding a new partition.
The MBR partition table is part of the Debian ISO and thus part of the
checksummed area. Even if all other alterations happen after the end
of the checksummed ISO image, the changed partition table will cause the
Debian checksum to become invalid.

(I am not aware that Debian installer changes the table. If it does indeed
then this might be worth a new bug discussion.)


Have a nice day :)

Thomas

Re: Debian 12.5: pigz 2.6-1 fails with error message (Upstream issue 111)

[Thomas Schmitt]

Hi,

Chung  Jonathan wrote:
> Yes, I think the local fix is the way to go.

I wrote:
> > (You forgot to Cc: debian-user@lists.debian.org.
> > Consider to send your mail to the list address, too. I too would then
> > resend my following reply to the list.)

Since my "following reply" is quoted in Jonathan Chung's reply to the list
i don't have to resend it. (I gave my opinion that the problem is not a
bug in the context of Debian 12 or 13 and pointed to
https://wiki.debian.org/BuildingTutorial for a private fix of the problem.)


Jeffrey Walton wrote:
> Your problem is one that plagues Linux. You compile and link against
> one version of a library, and then you runtime link against another
> version.

This should not be a problem with a well maintained library which cares
to stay ABI compatible with its older releases.
In the present case it was a bug in the loading program pigz which
prevented zlib from being usable.


> I consider it a
> security bug since essentially random libraries are being loaded at
> runtime.
> To fix the problem yourself, add an RPATH to your LDFLAGS when
> building your program:
> -Wl,-rpath=/path/to/expected/libz -Wl,--enable-new-dtags

Well, this is nearly as unflexible as static compilation but does not
seem to prevent the use of a replaced library at the given path.

Using .so files has its advantages and disadvantages. For a distro the
advantage (without the pigz bug) is that customers of different versions
of a library can be consolidated to using the newest available version.
An advantage for the user is that bugs in a library can be fixed without
the need for re-building all its customers.


Have a nice day :)

Thomas

Debian ISOs on USB stick, was: SOLVED

[Thomas Schmitt]

Hi,

David Christensen wrote:
> > the Debian installer modifies the contents of the USB flash drive when
> > it runs.

Do you mean inside the range of the ISO image or outside by creating a
new partition ?


songbird wrote:
> if it is an iso image copied to the USB stick it should not
> be modified if you haven't somehow told the installer to
> install the system to that USB stick (somehow).

There are other parties which feel entitled to operate on the EFI System
Partition of a USB stick.
In
  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1056998
we found that Lenovo Thinkpad firmware created directories for storing
an empty file named "/efi/Lenovo/BIOS/SelfHealing.fd" and that MS-Windows
created a 12-byte file named "/System Volume Information/WPSettings.dat"
when it had contact with the USB stick.


> i guess if you wanted to be really sure you could mount it read-only.

I think it's the installer which mounts the ISO 9660 filesystem.
Whatever, the Linux kernel has no regular means to alter an ISO 9660
filesystem. Neither kernel nor Debain installer will be so daring to
operate with byte level commands on that filesystem.

But the FAT filesystem in file /boot/grub/efi.img of the ISO 9660
filesystem in debian-12.*-amd64-netinst.iso is advertised by the partition
table of the image and thus attracts vermin.


Have a nice day :)

Thomas

Re: Debian 12.5: pigz 2.6-1 fails with error message (Upstream issue 111)

[Thomas Schmitt]

Hi,

Jonathan Chung wrote:
> > pigz 2.6-1 on Debian 12.5 fails to execute due to a fixed bug on
> > upstream https://github.com/madler/pigz/issues/111
> > Installing the version from sid resolves the issue which is clearly not
> > optimal. I think the fix should be backported.
> > Can someone help me to file a bug report?

Michael Kjörling wrote:
> https://www.debian.org/Bugs/Reporting

This covers the question "how", but not "what".
So my two cents:

"Backported" is probably the wrong wish.
  https://backports.debian.org/Contribute/
says:
  "Backports are about additional features that are only offered in a
   new version, not a replacement for getting fixes into stable - use
   stable-updates for that."

That would probably mean to patch the existing pigz package in Debian 12
by the upstream remedy
  "Make pigz compatible with two-component zlib version numbers. "
  https://github.com/madler/pigz/commit/907ca0763be4547a9b0cce8c105721748814974

pigz.c:
@@ -1333,7 +1333,7 @@ local long zlib_vernum(void) {
}
ver++;
} while (left);
-   return left < 2 ? num << (left << 2) : -1;
+   return left < 3 ? num << (left << 2) : -1;
 }

 // -- check value combination routines for parallel calculation --


The new state can already be seen in
  https://sources.debian.org/src/pigz/2.8-1/pigz.c/#L1339
The old state is in
  https://sources.debian.org/src/pigz/2.6-1/pigz.c/#L1317
But there the "// -- check ..." comment is missing in favor of a line
  #ifndef NOTHREAD
So a patch proposal would need some minor hand work.


There remains the question, though, why you run into zlib-1.3 on a
Debian "stable" system where the zlib version is "1:1.2.13.dfsg-1":
  https://tracker.debian.org/pkg/zlib


Have a nice day :)

Thomas

Re: printing QR-codes on labels with 300dpi label printers with LaTeX

[Thomas Schmitt]

Hi,

Max Nikulin wrote:
> I admit "dithering" may be incorrect term, [...]
> Consider 2 squares having size of 2.5×2.5 pixels. Non-even sizes and fuzzy
> lines variants:
> █████
> ██████
> ████ ██
>██ ██
>█████
> Second variant might have sense if an image is treated as a photo unlikely
> having regular patterns with horizontal and vertical lines.

I see ...
The second rendering style is probably best described as "Bad Distortion".


Have a nice day :)

Thomas

Re: How does the 64bits time_t transition work?

[Thomas Schmitt]

Hi,

Marco Moock wrote:
> The libs will have a suffix of t64

I wonder whether those suffixes will go away at some stage of this effort.

(Further i wonder when the package tracker appearance of libisoburn
 will become less ugly than currently:
   https://tracker.debian.org/pkg/libisoburn
 and how i shall deal with a bug report which complains about the
 inconsistent state of the control file in respect to libburn and
 libisofs:
   https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1067103
)


Have a nice day :)

Thomas

Re: printing QR-codes on labels with 300dpi label printers with LaTeX

[Thomas Schmitt]

Hi,

Max Nikulin wrote:
> When vector graphics, that does not match device resolution, is rasterized,
> the result is either non-even sizes of similar elements or fuzzy lines due
> to dithering.

Nitpicking:

"Dithering" in raster graphics is emulation of color resolution at the
expense of space resolution. Multiple coarsly colored pixels together
create the impression of a finer color tone. A certain random aspect is
added to prevent unwanted patters.
  https://en.wikipedia.org/wiki/Dither

The fuzzy lines are rather the opposite. They use surplus color
resolution to emulate ibetter spacial resolution. Over here the usual
term is "Anti-aliasing".
  https://en.wikipedia.org/wiki/Line_drawing_algorithm
  https://www.geeksforgeeks.org/antialiasing/
I get a suspiciously high share of german language results when
searching this term in the web. But "Computer Grapics" by Foley, Van Dam,
Feiner, Hughes of 1990 has a dozen occurences in its Index.


Have a nice day :)

Thomas

Re: OT: End the Phone-Based Childhood Now

[Thomas Schmitt]

Hi,

Curt wrote:
> as I believe Paul Valéry once noted, even the past isn't what it used
> to be.

That's why i want everything back exactly the way as it never was.


Have a nice day :)

Thomas

Re: OT: End the Phone-Based Childhood Now

[Thomas Schmitt]

Hi,

when i was a young Babyboomer in the late 1970s we were accused of
destroying society by a "twin culture of sexual license and cannabis".
The big Decline of the West was a sure thing, accelerated by excessive
tv consumption.
Other future threats were drying up oil wells, a comming ice age,
and the unstoppable fertility of the asian peoples.

Meanwhile the future is past and the pundits of the 70s are dead and
ridiculed.
Predictions are difficult - especially when it's about the future.


Have a nice day :)

Thomas

Re: Spam from the list?

[Thomas Schmitt]

Hi,

Andy Smith wrote:
> [...] I argue that at present it
> isn't a good idea to just reject all DKIM failures like OP's mailbox
> provider appears to be doing.

Just for the records:
The mails in question don't get rejected but rather marked as spam
and then get delivered.

The currently best theory is that megamailservers.eu adds a header
  X-Spam-Flag: YES
if it perceives DKIM problems, and that the local anti-spam software
of the receiver takes this header as reason to alter the subject by
the prefix "*SPAM*".

Whether it is a good idea to map DKIM failure to a spam marking header
is another interesting topic.

Original post of this thread with an example of all headers of a mail:
  https://lists.debian.org/msgid-search/3371640.PXJkl210th@protheus2


Have a nice day :)

Thomas

Re: Spam from the list?

[Thomas Schmitt]

Hi,

Hans wrote:
> Re: *SPAM* Re: Spam from the list?
> In-Reply-To: <20240306112253.55e25...@earth.stargate.org.uk>

referring the mail

> > Date: Wed, 6 Mar 2024 11:22:53 +
> > From: Brad Rogers 
> > Message-ID: <20240306112253.55e25...@earth.stargate.org.uk>

I assume that this mail appeared with the "*SPAM*" marker in
your mailbox.
(The currently most plausible theory is that megamailservers.eu adds
"X-Spam-Flag: YES" and your local mail processing takes this header as
reason to change the subject.)

So what does the mail which you received from Brad via debian-user
say about "Authentication-Results:" ?
Your initial post quoted three such headers. Expect more than one.


Have a nice day :)

Thomas

Re: Spam from the list?

[Thomas Schmitt]

Hi,

Hans wrote:
> I changed nothing and suddenly many mails from debian-user
> (but not all, only some) are recognized as spam.

But the one you posted here did not come from debian-user.

So maybe what changed is an inadverted subscription to one of
  debian-bugs-d...@lists.debian.org
  debian-de...@lists.debian.org
  debian-pyt...@lists.debian.org,
  w...@debian.org
This might have broadened the set of mail senders and thus gives your
mail provider opportunities to complain like spotted by Dan Ritter:

> Authentication-Results: mail104c50.megamailservers.eu;
>   dkim=fail reason="signature verification failed" (4096-bit key)
> header.d=4angle.com header.i=@4angle.com header.b="bS+3bWmq"

"4angle.com" matches the mail address of the bug submitter
"Edward Betts ".


The shown message headers offer unsubscription from debian-devel:

> List-Unsubscribe: 
> 

I.e. to send a mail to debian-devel-requ...@lists.debian.org with the
subject line
  unsubscribe


Have a nice day :)

Thomas

Re: Spam from the list?

[Thomas Schmitt]

Hi,

Hans wrote:
> during the last moonths I get more mails from the debian-user list marked as
> spam than before.
> [...]
> Below I send the header of an example of such a mail, maybe you can see the
> reason?

The message does not look like it came to you via debian-user:

> X-Original-To: lists-debian-de...@bendel.debian.org
> Delivered-To: lists-debian-de...@bendel.debian.org
> Received: from localhost (localhost [127.0.0.1])
> by bendel.debian.org (Postfix) with ESMTP id B720220598
> for ; Wed,  6 Mar 2024
> [...]
> Resent-To: debian-bugs-d...@lists.debian.org
> Resent-CC: debian-de...@lists.debian.org, debian-pyt...@lists.debian.org,
> w...@debian.org

Are you perhaps subscribed to one of the "Resent-*" lists ?


> Subject: *SPAM* Bug#1065537: ITP: bleak-retry-connector -- Connector
> for Bleak Clients that handles transient connection failures

The mark "*SPAM*" does not appear in the archive

  https://lists.debian.org/debian-devel/2024/03/msg00076.html

All in all it looks like a legit message, not like spam.
So the suspect would sit after Debian's mail servers.


The only Received header i see between Debian and you is:

> Received: from bendel.debian.org (bendel.debian.org [82.195.75.100])
> by mail104c50.megamailservers.eu (8.14.9/8.13.1) with ESMTP
> id 4269vZOl098298
> for ; Wed, 6 Mar 2024 09:57:37 +

It looks like either megamailservers.eu or your own processing added
the spam mark to the subject.


Have a nice day :)

Thomas

Re: partition reporting full, but not

[Thomas Schmitt]

Hi,

> when cfdisk reports:
> Device  Start   End  Sectors   Size  Type
> /dev/sda2   1785522176  1786245119 722944  353M  EFI System
> /dev/sda3   1786245120  1933045759  146800640  70G   EFI System
> I don't understand the 'EFI System' note /dev/sda3 is /

The partition type does not necessarily match the type of filesystem
which is currently in that partition. It's just a field in the
partition table which must have some value.


Have a nice day :)

Thomas

Re: Contact Name...

[Thomas Schmitt]

Hi,

Clayton Penn wrote:
> I have attempted to register for the Debian Forums, but have not received a
> verification email

Did you try wether your new account is already working ?
(Sorry, i'm not familiar with the current registration procedure.)

If not:
There seems to be some problem with GMail:

  https://forums.debian.net/viewtopic.php?t=158230

Are you perhaps directly or indirectly using GMail ?


> Do you happen to have another email address contact?

I have an account at forums.debian.net, although not used in years.
So if you want to add information to above topic, i could try to do so
on your behalf.

But first consider to try registering with a completely different mail
provider which is surely not using GMail's software.


Have a nice day :)

Thomas

Re: "I: update-initramfs is disabled (live system is running on read-only media)" ...

[Thomas Schmitt]

Hi,

Albretch Mueller wrote:
> > How can you update the initramfs on read-only media?

to...@tuxteam.de wrote:
> You can't. Initramfs resides in the boot medium. To update it,
> you have to write to said medium.

One will have to create a new read-only medium.


In case the original is a Debian Live ISO:

One would have to extract the initramfs file out of the ISO. If its name
is not known, then the boot loader configuration file should tell. Like
in /boot/grub/grub.cfg of debian-live-12.0.0-amd64-standard.iso:
   initrd  /live/initrd.img-6.1.0-9-amd64
or in its /isolinux/live.cfg:
   initrd /live/initrd.img

Next one would modify the extracted initramfs.
(This is an adventure on its own. Other will know more about it than me.)

Finally one would pack up a new ISO, taking all files from the old ISO but
replacing the initramfs file by the modified one from hard disk.
Roughly like in
  
https://wiki.debian.org/RepackBootableISO#In_xorriso_load_ISO_tree_and_write_modified_new_ISO

Details could be determined when the name of ISO and initramfs file is
known. If it's about DVD media, it would be interesting to learn about
the DVD drives at the computer which shall do the modification.


Have a nice day :)

Thomas

Re: shred bug? [was: Unidentified subject!]

[Thomas Schmitt]

Hi,

Gene Heskett wrote:
> Next experiment is a pair of 4T Silicon Power SSD's

When f3 has (hopefully) given its OK, the topic of a full write-and-read
test will come up again. I'm looking forward to all the spin-off topics.


Have a nice day :)

Thomas

Re: shred bug? [was: Unidentified subject!]

[Thomas Schmitt]

Hi,

Greg Wooledge wrote:
> Heh.  Don't forget your own attempts to use a shredder as a PRNG stream.

My original idea was to watch a minimal shred run by teeing its work into
a checksummer.

But then topic drift came in. So we got a farm show of random generators
and a discussion about what exactly is a bug in shred's documentation.
Plus the shell programming webinar. And a diagnosis about a rightously
failed attempt to change the partition table type from MBR/DOS to GPT.

And this all because Gene Heskett was adventurous enough to buy a cheap
fake USB disk.


Have a nice day :)

Thomas

Re: shred bug? [was: Unidentified subject!]

[Thomas Schmitt]

Hi,

Greg Wooledge wrote:
> Let me write out the example again, but with the bug fixed, and then
> explain what each line does, [... lecture about advanced shell
> programming ...]

And this all because Gene Heskett was adventurous enough to buy a cheap
fake USB disk. :))


Have a nice day :)

Thomas

Re: shred bug? [was: Unidentified subject!]

[Thomas Schmitt]

Hi,

"info shred" says:
> > >   i=$(mktemp)
> > >   exec 3<>"$i"
> > >   rm -- "$i"
> > >   echo "Hello, world" >&3
> > >   shred - >&3
> > >   exec 3>-

Greg Wooledge wrote:
> In fact, that last line is
> written incorrectly.  It should say "exec 3>&-" and what that does
> is close file descriptor 3, which was previously opened on line 2.
> [...]
> This is an obvious bug in the info page.  I wonder how many years
> this has gone unnoticed.

  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=155175#36
of 22 Dec 2005 states:

  "I'll assume that this is now adequately explained in the info page
   (below).  If not then please reopen.  // Thomas Hood
   [...]
   exec 3>-"

The bug report is from 02 Aug 2002 and states that the info page contains
the short and broad promise which we can still see in "man shred".

So we can assume a bug age of 18 to 22 years.


Have a nice day :)

Thomas

Re: shred bug? [was: Unidentified subject!]

[Thomas Schmitt]

Hi,

> https://en.wikipedia.org/wiki/Everything_is_a_file
> But, there is more than one kind of file.

"All files are equal.
 But some files are more equal than others."

(George Orwell in his dystopic novel "Server Farm".)


Have a nice day :)

Thomas

Re: Combining Distro DVD's

[Thomas Schmitt]

Hi,

Steve Matzura wrote:
> I thought it'd be a nice idea to combine any and all distribution media for
> a release into a single medium--a USB drive, of course.

The initial situation will depend much on the distro ...
But given that Debian is about the last one i know with all its packages
in DVD images, i assume you mean the Debian installer ISO sets. Like
  https://cdimage.debian.org/debian-cd/current/amd64/jigdo-dvd/


> I'd start by
> creating my USB drive by extracting the first DVD to it, thereby ensuring
> the boot block and boot material is where it should be.

There is the dilemma that unpacking the first ISO into the USB stick's
filesystem will not enable the copied boot stuff, and that putting the
first ISO plainly onto the stick (e.g. by dd) will leave you with a
read-only filesystem on the stick.

So you'd need to invest some extra expertise.
Either for making a bootable USB stick with read-write filesystem from
the first ISO, or for merging the ISOs into a single ISO. In the former
case you need knowledge about booting Debian (by GRUB, i guess). In the
latter case you need xorriso.


> But then what do I
> do with the additional media? Surely there will be some files with the same
> name among the individual pieces of media, and some will probably contain
> the same info while others probably will not.

They should be either either merged or discarded, depending on their
meaning and on their importance for your final result. I.e. whether you
merged the ISOs or you extracted to a writable filesystem and installed
a bootloader yourself.

The way of merging multiple Debian DVD ISOs of the same release and
CPU architecture to a single ISO by xorriso is described in

  https://wiki.debian.org/MergeDebianIsos

(The script merge_debian_isos would also be the source to learn about
the proper handling of files with expected name collision.)


I leave it to others to elaborate the ways of creating a bootable USB
stick with writable filesystem and the content of Debian DVD 1 with
work-ready installer software. Then it's an adventure on its own to
add all the packages from the other DVD images and to properly care
for their non-package files.


Have a nice day :)

Thomas

Re: grub-pc error when upgrading from buster to bullseye

[Thomas Schmitt]

Hi,

John Boxall wrote:
> I am aware that the label and uuid (drive and partition) are replicated on
> the cloned drive, but I can't find the model number (in text format) stored
> anywhere on the drive.

Maybe the grub-pc package takes its configuration from a different drive
which is attached to the system ?

Somewhat wayward idea:
Does the initrd contain the inappropirate address ?
(I don't see much connection between initrd and grub-pc. But initrd is a
classic hideout for obsolete paths after modification of boot procedures.)


Have a nice day :)

Thomas

Re: grub-pc error when upgrading from buster to bullseye

[Thomas Schmitt]

Hi,

John Boxall wrote:
>   Setting up grub-pc (2.06-3~deb11u6) ...
>   /dev/disk/by-id/ata-WDC_WDS100T2B0A-00SM50_21185R801540 does not
> exist, so cannot grub-install to it!
> What is confusing to me is that the error indicates the source SDD even
> though I have updated the boot images and installed grub on the cloned HDD.

The disk/by-id file names are made up from hardware properties.
I believe to see in the name at least: Manufacturer, Model, Serial Number.

So you will have to find the configuration file which knows that
/dev/disk/by-id address and change it either to the new hardware id or
to a /dev/disk/by-uuid address, which refers to the cloned disk content.


Have a nice day :)

Thomas

Re: shred bug? [was: Unidentified subject!]

[Thomas Schmitt]

Hi,

to...@tuxteam.de wrote:
> Still there's the discrepancy between doc and behaviour.

Depends at which documentation you look. Obviously stemming from
  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=155175#36
i read in
  https://www.gnu.org/software/coreutils/manual/html_node/shred-invocation.html

 "A file of ‘-’ denotes standard output. The intended use of this is to
  shred a removed temporary file. For example: [shell wizzardry]"

It works as long as stdout is connected to a data file, or block device,
or directory, or symbolic link, or to a character device that is not a
terminal.
(Maybe it refuses later on some of these types, but not at the location
with the message "invalid file type". I wonder if i can connect stdout
to a symbolic link instead of its target.)

The bug would thus have to be filed against the man page
  https://sources.debian.org/src/coreutils/9.4-3/man/shred.1/
which says only

  "If FILE is \-, shred standard output."

The info empire of coreutils says what above web manual says.
  https://sources.debian.org/src/coreutils/9.4-3/doc/coreutils.texi/#L10705


Have a nice day :)

Thomas

Re: shred bug?

[Thomas Schmitt]

Hi,

debian-u...@howorth.org.uk wrote:
> Maybe it is unstated but mandatory to use -n 1 as well?
> And optionally -s N?

Naw. It just doesn't want to work pipes.

Initially i tried with these options:

  shred -n 1 -s 1K -v - | sha256sum

as preparation for a proposal to Gene Heskett, like:
  shred -n 1 -s 204768K -v - | tee /dev/sdm1 | sha256sum


> I expect reading the code would tell.

My code analysis is in
  
https://lists.debian.org/msgid-search/1162291656137153...@scdbackup.webframe.org

to...@tuxteam.de found bug 155175 from 2002, which explains why. See
  https://lists.debian.org/msgid-search/zcdxzya0ayrmp...@tuxteam.de


Have a nice day :)

Thomas

Re: Fast Random Data Generation

[Thomas Schmitt]

Hi,

Linux-Fan wrote:
> I wrote a program to automatically generate random bytes in multiple
> threads:
> https://masysma.net/32/big4.xhtml
> ...
> || Wrote 102400 MiB in 13 s @ 7812.023 MiB/s

That's impressive.


> Secure Random can be obtained from OpenSSL:
>
> | $ time for i in `seq 1 100`; do openssl rand -out /dev/null $((1024 * 1024 
> * 1024)); done
> |
> | real  0m49.288s
> | user  0m44.710s
> | sys   0m4.579s
> Effectively 2078 MiB/s (quite OK for single-threaded operation).

Probably the best choice for sceptics who critizise a reproducible random
stream or mistrust random people's aptness to produce random.
(If the number of desired loops gets larger, then one could do arithmetik
 in a "while" loop instead of "for" and "seq".)


Have a nice day :)

Thomas

Re: Unidentified subject!

[Thomas Schmitt]

Hi,

David Christensen wrote:
> Concurrency:
> threads throughput
> 8   205+198+180+195+205+184+184+189=1,540 MB/s

There remains the question how to join these streams without losing speed
in order to produce a single checksum. (Or one would have to divide the
target into 8 areas which get checked separately.)

Does this 8 thread generator cause any problems with the usability of
the rest of the system ? Sluggish program behavior or so ?

The main reason to have my own low-quality implementation of a random
stream was that /dev/urandom was too slow for 12x speed (= 1.8 MB/s) CD-RW
media and that higher random quality still put too much load on a
single-core 600 MHz Pentium system. That was nearly 25 years ago.


I wrote:
> > Last time i tested /dev/urandom it was much slower on comparable machines
> > and also became slower as the amount grew.

> Did you figure out why the Linux random number subsystem slowed, and at what
> amount?

No. I cannot even remember when and why i had reason to compare it with
my own stream generator. Maybe 5 or 10 years ago. The throughput was
more than 100 times better.


I have to correct my previous measurement on the 4 GHz Xeon, which was
made with a debuggable version of the program that produced the stream.
The production binary which is compiled with -O2 can write 2500 MB/s into
a pipe with a pacifier program which counts the data:

  $ time $(scdbackup -where bin)/cd_backup_planer -write_random - 100g 
2s62gss463ar46492bni | $(scdbackup -where bin)/raedchen -step 100m -no_output 
-print_count
  100.0g bytes

  real0m39.884s
  user0m30.629s
  sys 0m41.013s

(One would have to install scdbackup to reproduce this and to see raedchen
 count the bytes while spinning the classic SunOS boot wheel: |/-\|/-\|/-\
   http://scdbackup.webframe.org/main_eng.html
   http://scdbackup.webframe.org/examples.html
 Oh nostalgy ...
)

Totally useless but yielding nearly 4000 MB/s:

  $ time $(scdbackup -where bin)/cd_backup_planer -write_random - 100g 
2s62gss463ar46492bni >/dev/null

  real0m27.064s
  user0m23.433s
  sys 0m3.646s

The main bottleneck in my proposal would be the checksummer:

  $ time $(scdbackup -where bin)/cd_backup_planer -write_random - 100g 
2s62gss463ar46492bni | md5sum
  5a6ba41c2c18423fa33355005445c183  -

  real2m8.160s
  user2m25.599s
  sys 0m22.663s

That's quite exactly 800 MiB/s ~= 6.7 Gbps.
Still good enough for vanilla USB-3 with a fast SSD, i'd say.


> TIMTOWTDI.  :-)

Looks like another example of a weak random stream. :))


Have a nice day :)

Thomas

Re: Unidentified subject!

[Thomas Schmitt]

Hi,

David Christensen wrote:
> $ time dd if=/dev/urandom bs=8K count=128K | wc -c
> [...]
> 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.30652 s, 249 MB/s

This looks good enough for practical use on spinning rust and slow SSD.

Maybe the "wc" pipe slows it down ?
... not much on 4 GHz Xeon with Debian 11:

  $ time dd if=/dev/urandom bs=8K count=128K | wc -c
  ...
  1073741824 bytes (1.1 GB, 1.0 GiB) copied, 4.13074 s, 260 MB/s
  $ time dd if=/dev/urandom bs=8K count=128K of=/dev/null
  ...
  1073741824 bytes (1.1 GB, 1.0 GiB) copied, 3.95569 s, 271 MB/s

Last time i tested /dev/urandom it was much slower on comparable machines
and also became slower as the amount grew.

Therefore i still have my amateur RNG which works with a little bit of
MD5 and a lot of EXOR. It produces about 1100 MiB/s on the 4 GHz machine.
No cryptographical strength, but chaotic enough to avoid any systematic
pattern which could be recognized by a cheater and represented with
some high compression factor.
The original purpose was to avoid any systematic interference with the
encoding of data blocks on optical media.

I am sure there are faster RNGs around with better random quality.


> $ time perl -MMath::Random::ISAAC::XS -e 
> '$i=Math::Random::ISAAC::XS->new(12345678); print pack 'L',$i->irand while 1' 
> | dd bs=8K count=128K | wc -c
> 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 82.6523 s, 13.0 MB/s

Now that's merely sufficient for shredding the content of a BD-RE medium
or a slow USB stick.


> I suggest using /dev/urandom and tee(1) to write the same CSPRN
> stream to all of the devices and using cmp(1) to validate.

I'd propose to use a checksummer like md5sum or sha256sum instead of cmp:

 $random_generator | tee $target | $checksummer
 dd if=$target bs=... count=... | $checksummer

This way one can use unreproducible random streams and does not have to
store the whole stream on a reliable device for comparison.


Have a nice day :)

Thomas

Re: Unidentified subject!

[Thomas Schmitt]

Hi,

i wrote:
> > In the other thread about the /dev/sdm test:
Gene Heskett wrote:
> > > Creating file 39.h2w ... 1.98% -- 1.90 MB/s -- 257:11:32
> > > [...]
> > > $ sudo f3probe --destructive --time-ops /dev/sdm
> > > Bad news: The device `/dev/sdm' is a counterfeit of type limbo
> > > Device geometry:
> > >  *Usable* size: 59.15 GB (124050944 blocks)
> > > Announced size: 1.91 TB (409600 blocks)

David Christensen wrote:
> Which other thread?  Please provide a URL to archived post.

https://lists.debian.org/msgid-search/e7a0c1a1-f973-4007-a86d-8d91d8d91...@shentel.net
https://lists.debian.org/msgid-search/b566504b-5677-4d84-b5b3-b0de63230...@shentel.net


> So, the 2 TB USB SSD's are really scam 64 GB devices?

The manufacturer would probably state that it's no intentional scam but
just poor product quality. (Exploiting Hanlon's Razor.)

It might be intentional that the write speed gets slower and slower as
the end of the usable area is approached. This avoids the need for
confessing the failure to the operating system.
But it might also be an honest attempt of the disk's firmware to find
some blocks which can take and give back the arriving data.


Have a nice day :)

Thomas

Re: Unidentified subject!

[Thomas Schmitt]

Hi,

gene heskett wrote:
> my fading eyesight couldn't see
> the diffs between () and {} in a 6 point font.  I need a bigger, more
> legible font in t-bird.

That's why i propose to copy+paste problematic command lines.

Your mouse can read it, your mail client can send it, and we have
youngsters here of merely 60 years who will be glad to tell our
most senior regular why the lines don't work.

With some luck this creates nostalgic tangents about how the used features
evolved since the early 1980s.


In the other thread about the /dev/sdm test:
> Creating file 39.h2w ... 1.98% -- 1.90 MB/s -- 257:11:32
> but is taking a few bytes now and then.
> [...]
> $ ls -l
> total 40627044
> [...]
> $ sudo f3probe --destructive --time-ops /dev/sdm
> Bad news: The device `/dev/sdm' is a counterfeit of type limbo
> Device geometry:
> *Usable* size: 59.15 GB (124050944 blocks)
>Announced size: 1.91 TB (409600 blocks)

That's really barefaced.
How can the seller believe to get away with a problem which will show
already after a few dozen GB were written ?


> Probe time: 2.07s

That's indeed a quick diagnosis. Congrats to the developers of f3probe.


Have a nice day :)

Thomas

Re: shred bug?

[Thomas Schmitt]

Hi,

to...@tuxteam.de wrote:
> Ah, it seems to be this one, from 2002:
>  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=155175

So it's not a bug but a feature. :(

I'm riddling over the code about the connection to an old graphics
algorithm (Bresenham's Algorithm) and how shred produces a random pattern
at all.


Have a nice day :)

Thomas

Re: shred bug?

[Thomas Schmitt]

Hi,

i wrote:
> >   shred: -: invalid file type

to...@tuxteam.de wrote:
> Hmm. This looks like a genuine bug: the man page mentions it.

Even the help text in
  https://sources.debian.org/src/coreutils/9.4-3/src/shred.c/
says
  If FILE is -, shred standard output.

The name "-" is recognized in line 1257 and leads to a call of wipefd()
in line 958. The error messages there look different from above.
So i hop from line 973 to do_wipefd() and find the message in line 845.
fd is supposed to be 1 (= stdout). fstat(2) was successful but now this
condition snaps:

  if ((S_ISCHR (st.st_mode) && isatty (fd))
  || S_ISFIFO (st.st_mode)
  || S_ISSOCK (st.st_mode))

The problem seems to be in the S_ISFIFO part.
In a little test program fstat() reports about fd==1:
  st.st_mode= 010600 , S_ISFIFO(st.st_mode)= 1
(st_mode shown in octal as in man 2 fstat.)

It looks like the test expects a pipe(2) file descriptor to be
classified as S_ISCHR and !isatty().
Without redirection through a pipe, fd 1 has st.st_mode 20620, S_ISCHR,
and isatty()==1. The isatty() result is indeed a good reason, not to
flood stdout by a zillion random bytes.


Does anybody have an old GNU/Linux system where a file descriptor from
pipe(2) is classified as character device (S_IFCHR, S_ISCHR) ?

Does anybody have an idea why shred would want to exclude fifos ?
(What ciould shred do with a non-tty character device that cannot be done
 with a fifo ?)


Have a nice day :)

Thomas

Re: Unidentified subject!

[Thomas Schmitt]

Hi,

Gene Heskett wrote:
> Is bash not actually bash these days? It is not doing for loops for me.

Come on Gene, be no sophie. Copy+paste your failing line here. :))


IIRC the for-loop in question writes several copies of the same file.
( https://lists.debian.org/debian-user/2024/02/msg00318.html )
Others already pointed out that this invites for firmware scams like
deduplication or silent overwriting of older data.

I would vote for a filesystem killer like
  shred -n 1 -s 204768K -v - | tee /dev/sdm1 | sha256sum
  dd if=/dev/sdm1 bs=32K count=6399 skip=32 | sha256sum

But shred(1) on Debian 11 refuses on "-" contrary to its documentation:
  shred: -: invalid file type
A non-existing file path causes "No such file or directory".


The filesystem killer aspect could be removed by creating large data files
in the readily partitioned and formatted filesystems of the disk.
(Replace "/dev/sdm1" by "/where/mounted/random_test_file" and reduce the
 numbers 204768K and bs=32K count=6399 to what you expect to
 fit into the filesystem. Like 20K and bs=32K count=6250.
 Ask df for storage capacities.)

But there needs to be a fast pseudo-random byte stream (which shred would
provide if i could talk it into writing to stdout) which you can split
for the device and for sha256sum.
I have an own weak-random generator, but shred beats it by a factor of 10
when writing to /dev/null.


Have a nice day :)

Thomas

Re: Generic Linux / clib question

[Thomas Schmitt]

Hi,

Greg Wooledge wrote:
> > The readline library is released under the full GPL, not the LGPL.  If
> > you dynamically link it with a program, then you can only release that
> > program under terms compatible with the GPL.  This is an intentional
> > choice.

to...@tuxteam.de wrote:
> Practically it doesn't change much, but for some lawyer at Intel it might.

A substantial problem can be that libreadline became "GPLv3 or later" a
few years ago and that GPLv3 is not compatible with GPLv2-only (i.e. not
"or later"). The main reason is in the patent-fighting restrictions in
GPLv3, which would be forbidden extra restrictions under GPLv2.

So at best this causes "GPLv2 or later" programs to become "GPLv3 or later",
like the Debian binary of xorriso. At worst you would have a license clash
and would have to resort to e.g. libedit. (Not what the FSF would like.)


> The gist is that, if the user is the one doing the last linking step,
> all is fine. She's allowed anything.

She's allowed to use it but not to give it to others without obeying the
demands and restrictions of GPL.

Personally i would be ok with a more liberal use of my GPL'ed software.
But the official stance of the FSF is that the license applies on first
contact. In case of libreadline this would the line
  #include 
and as next occasion the preparations of the compiler and linker to employ
libreadline.so at run time.


Have a nice day :)

Thomas

Re: Generic Linux / clib question

[Thomas Schmitt]

Hi,

Van Snyder wrote:
> Years ago, I knew the name of the routines one could use to have some stdin
> history and be able to edit it, like you can do in XTerm or gnuplot or 

Sounds like readline:
  https://en.wikipedia.org/wiki/GNU_Readline
  https://tracker.debian.org/pkg/readline

An alternative with BSD license is libedit:
  https://tracker.debian.org/pkg/libedit


Have a nice day :)

Thomas

Re: Things I don't touch with a 3.048m barge pole

[Thomas Schmitt]

Hi,

gene heskett wrote:
> GPT fdisk (gdisk) version 1.0.9
>
> Partition table scan:
>   MBR: MBR only
> [...]
> Found invalid GPT and valid MBR; converting MBR to GPT format
> in memory.
> [...]
> Warning! Secondary partition table overlaps the last partition by 33 blocks!
> You will need to delete this partition or resize it in another utility.
> [...]
> Disk /dev/sdm: 409600 sectors, 1.9 TiB
> Number  Start (sector)End (sector)  Size   Code  Name
>1  64  409599   1.9 TiB 0700  Microsoft basic data

It wants to say "found no GPT at all and cannot easily convert the MBR
partition table".


> What do we make of that?  Some sort of NTFS?

NTFS would be a possible candidate for the filesystem inside the
partition. What does this command report:

  file -s /dev/sdm1


But the complaint from gdisk is just about an MBR partitioned stick with
no unclaimed room at the end for storing the GPT backup table.
Indeed if you want to convert the MBR partition table to GPT you will have
to delete or shrink the single partition number 1. (Shrinking would
of course depend on what kind of filesystem is in the partition and
whether Debian has shrink software for it.)
The same will be necessary if you want more than one partition.

Whatever, there is nothing suspicious about the gdisk behavior with that
particular disk.


Have a nice day :)

Thomas

Re: How can we change the keyboard layout?

[Thomas Schmitt]

Hi,

Loris Bennett wrote:
> As many have pointed out, it is short for 'Steuerung', but I have met
> many Germans who refer to this key as 'String'.  I am not sure why

BASIC ?

Or the popular bundle theory:
[Strg] (= [Ctrl]) means "String",
[AltGr] (= right side [Alt]) means "Altgriechisch" (= ancient greek),
[Entf] (= [Delete]) means "Entfetten" (= degrease).


Have a nice day :)

Thomas

Re: about 64bits time_t transition and deborphan

[Thomas Schmitt]

Hi,

Patrice Duroux wrote:
> Out of curiosity, I started this transition on some packages from
> experimental and I observed that deborphan is not without
> «disruption».
> [...]
> Is this something to be reported to deborphan as it could also be in
> some other cases than this time_t transition? or wait and see... ;-)

I guess "wait and see" for now.

The whole operation seems to be still in progress and leaves traces
in the package tracker. See e.g. section "testing migrations" in
  https://tracker.debian.org/pkg/libisoburn

The associated mailing list got notifications as bug reports 1062380 for
libisoburn and 1062381 for libisofs on february 1st. Yesterday came
bug report 1063123 for libburn.
I am curious how this will unfold further. Especially whether the new
binary package name "libisoburn1t64" will persist for future releases.


Have a nice day :)

Thomas

Re: install Kernel and GRUB in chroot.

[Thomas Schmitt]

Hi,

Dmitry wrote:
> Yep. `dd` copy partitions table. Amazing.

Not so amazing after you realize that a partition table is just data on
the storage medium and not some special property of the storage device.
dd copies data. If these data contain a partition table and get copied to
the right place on the storage medium, the partition table will be
recognized by EFI and Linux.


> applies no 'intelligence' to the operation.

This describes it very well. Sometimes dumb is good. Sometimes not.


Initially you stated in
  https://lists.debian.org/debian-user/2024/02/msg8.html
>...> I need to prepare that system for booting.
>...> 1. Install Kernel.
>...> 2. Install GRUB and Configure.
>...> 3. Add changes to UEFI to start booting.

dd-ing a bootable Debian ISO will not do what you describe.
Assumed the ISO is prepared for booting from USB stick, you will get a
bootable Live or an installer system. At least ISOs for i386, amd64, and
arm64 should be prepared for that.

If it is not ready for booting from USB stick, it will be just a storage
with a mountable filesystem and Debian files in it.


Max Nikulin wrote:
> > Just copy files from LiveCD (it should have EFI/Boot/bootx64.efi)
> > to the ESP partition on the USB stick.

The /EFI/boot directory of a bootable Debian ISO usually does not contain
the full GRUB equipment for EFI. Important parts of an amd64 Live ISO are
in /boot/grub.
The programs in /EFI/boot are specialized on convincing Secure Boot and
on finding the ISO filesystem with /boot/grub in it. (Actually those are
copies of the EFI boot partition files. The boot partition is a FAT
filesystem image file inside the ISO named /boot/grub/efi.img .)


Tim Woodall wrote:
> > I'm not exactly sure what you're doing.

I join this statement. :))

Do you want a normal changeable Debian system installation or do you want
a Live system with its immutable core and maybe some partition where you
can store files ?
(Just curiosity of mine. Possibly i could not help much with chroot
questions anyway.)


Have a nice day :)

Thomas

Re: AW: AW: su su- sudo dont work

[Thomas Schmitt]

Hi,

Andy Smith wrote:
> It is hard to understand how what Michael/Sophie/Tobias does can in
> any way be "fun" for them, though maybe that is just our lack of
> understanding.

I expressed my suspicion of a "Hurz" performance in
  https://lists.debian.org/debian-user/2023/05/msg00100.html


Have a nice day :)

Thomas

Re: Automatically installing GRUB on multiple drives

[Thomas Schmitt]

Hi,

Nicolas George wrote:
> You seem to be assuming that the system will first check sector 0 to
> parse the MBR and then, if the MBR declares a GPT sector try to use the
> GPT.

That's what the UEFI specs prescribe. GPT is defined by UEFI-2.8 in
chapter 5 "GUID Partition Table (GPT) Disk Layout". Especially:

  5.2.3 Protective MBR
  For a bootable disk, a Protective MBR must be located at LBA 0 (i.e.,
  the first logical block) of the disk if it is using the GPT disk layout.
  The Protective MBR precedes the GUID Partition Table Header to maintain
  compatibility with existing tools that do not understand GPT partition
  structures.


> I think it is the other way around on modern systems: it will first
> check sector 1 for a GPT header, and only if it fails check sector 0.

Given the creativity of firmware programmers, this is not impossible.
At least the programmers of older versions of OVMF took the presence of
a GPT header as reason to boot, whereas without it did not boot.
Meanwhile this demand for GPT debris has vanished and OVMF boots from
media with only MBR partitions, too.


I wrote:
> > This layout [used by Debian installation ISOs] was invented by Matthew
> > J. Garrett for Fedora

> I think I invented independently something similar.
> https://www.normalesup.org/~george/comp/live_iso_usb/grub_hybrid.html

Not to forget Vladimir Serbinenko who specified how a grub-mkrescue ISO
shall present its lures for BIOS and EFI on optical media and USB stick.
The ISO has a pure GPT partition table, where the ISO filesystem is not
mountable as partition but only by the base device (like /dev/sdc) or
possibly by its HFS+ directory tree via the Apple Partition Map, if
present.

(To create such an ISO, install grub-common, grub-efi-amd64, grub-efi-ia32,
and grub-pc. Then run grub-mkrescue with some dummy directory as payload.
The ISO will boot to a GRUB prompt.)


Have a nice day :)

Thomas

Re: Automatically installing GRUB on multiple drives

[Thomas Schmitt]

Hi,

i hate to put in question the benefit of my proposal, but:

Nicolas George wrote:
> The firmware would never write in parts of the
> drive that might contain data.

See

  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1056998
  "cdrom: Installation media changes after booting it"

Two occasions were shown in this bug where the EFI system partition of
a Debian installation ISO on USB stick changed. One was caused by a
Microsoft operating system, writing a file named WPSettings.dat. But the
other was from Lenovo firmware writing /efi/Lenovo/BIOS/SelfHealing.fd .

One may doubt that the success of these operations is desirable at all.
The ISO was also tested with a not-anymore-writable DVD. In that case the
Lenovo firmware did not raise protest over the fact that it was not
possible to write to the EFI partition.


Have a nice day :)

Thomas

Re: Automatically installing GRUB on multiple drives

[Thomas Schmitt]

Hi,

Nicolas George wrote:
> Interesting. Indeed, “table-length: 4” causes sfdisk to only write 3
> sectors at the beginning and 2 at the end. I checked it really does not
> write elsewhere.
> That makes it possible to use full-disk RAID on a UEFI boot drive. Very
> good news.

\o/

(Nearly as good as Stefan Monnier's crystal ball. And that without
understanding the dirty details which cause the need for a small partition
table.)


> More and more firmwares will only boot with GPT. I think I met
> only once a firmware that booted UEFI, 32 bits, with a MBR

The Debian installation and live ISOs have MBR partitions with only a
flimsy echo of GPT. There is a GPT header block and an entries array.
But it does not get announced by a Protective MBR. Rather they have two
partitions of which one is meant to be invisible to EFI ("Empty") and
one is advertised as EFI partition:

$ /sbin/fdisk -l debian-12.2.0-amd64-netinst.iso
...
Disklabel type: dos
...
Device   Boot Start End Sectors  Size Id Type
debian-12.2.0-amd64-netinst.iso1 *0 1286143 1286144  628M  0 Empty
debian-12.2.0-amd64-netinst.iso2   4476   23451   18976  9.3M ef EFI (FAT-12

So any system which boots this ISO from USB stick does not rely on
the presence of a valid GPT.
(The only particular example of GPT addiction i know of are old versions
of OVMF, the EFI used by qemu, which wanted to see the GPT header block,
even without Protective MBR.)

This layout was invented by Matthew J. Garrett for Fedora and is still
the most bootable of all possible weird ways to present boot stuff for
legacy BIOS and EFI on USB stick in the same image. (There are mad legacy
BIOSes which hate EFI's demand for no MBR boot flag. Several distros
abandoned above layout in favor of plain MBR or plain GPT. The price
is that they have to leave behind some of the existing machines.)


> GPT
>  ├─EFI
>  └─RAID
> └─LVM (of course)
>
> Now, thanks to you, I know I can do:
>
> GPT
>  ┊  RAID
>  └───┤
>  ├─EFI
>  └─LVM

Ah. Now i understand how accidentially useful my technical nitpicking was.
(A consequence of me playing Dr. Pol with the arm in the ISO 9660 cow
up to my shoulder.)


Have a nice day :)

Thomas

Re: Automatically installing GRUB on multiple drives

[Thomas Schmitt]

Hi,

i cannot make qualified proposals for the GRUB question, but stumble over
your technical statements.

Nicolas George wrote:
> Since mdadm can only put its superblock at the end of the device (1.0),
> at the beginning of the device (1.1) and 4 Ko from the beginning (1.2),
> but they still have not invented 1.3 to have the metadata 17 Ko from the
> beginning or the end, which would be necessary to be compatible with
> GPT,

Although it would be unusually small, it is possible to have a GPT of
only 4 KiB of size:
- 512 bytes for Protective MBR (the magic number of GPT)
- 512 bytes for the GPT header block
- 3 KiB for an array of 24 partition entries.

Question is of course, whether any partition editor is willing to create
such a small GPT. The internet says that sfdisk has "table-length" among
its input "Header lines". So it would be a matter of learning and
experimenting.
(Possibly i would be faster with writing the first header blocks by
hand, following UEFI specs or my cheat sheet in libisofs.)


> we have to partition them and put the EFI system partition outside
> them.

Do you mean you partition them DOS-style ?
If so, then a partition of type 0xEF could be used as system partition.
Probably any partition type will do, because EFI is very eager to look
into any partition with FAT filesystem.


Have a nice day :)

Thomas

Re: Looking for archive management system for backups burned to optical discs

[Thomas Schmitt]

Hi,

David Christensen wrote:
> I have a SOHO file server with ~1 TB of data.  I would like archive the data
> by burning it to a series of optical discs organized by time (e.g. mtime).
> I expect to periodically burn additional discs in the future, each covering
> a span of time from the previous last disc to the then-current time.

I use my own software for making incremental multi-volume backups, based
on file timestamps (m and c), inode numbers, and content checksums.
  http://scdbackup.webframe.org/main_eng.html
  http://scdbackup.webframe.org/examples.html#incremental

The software and the texts are quite old. The proposed backup scheme
is not in use here any more.
Instead i have four independent backup families, each comprised of
level 0 to N with no repetitions below the current level N.
Further i have backups of the configuration and memorized file lists
on 4 CDs.

Level 0 fills dozens of BD-RE discs. The other levels fill at most one
BD-RE. Level N of each family exists in three copies which get larger
with each backup run of that level. Whenever this level BD threatens to
overflow, i archive the latest BD of that level and start level N+1.
That step is a bit bumpy, because i have to restore the file lists of
level N from CD after a backup has been planned but was not performed.
When overflow is foreseeable, i make a copy of the file lists on disk
before i start the planning run, or i simply start level N+1 without
waiting for the overflow.

I use scdbackup for the slowly growing bulk of my file collection.
The agile parts of my hard disk are only about 5 GB and get covered by
incremental multi-session backups of xorriso (which learned a lot about
incrementality from scdbackup). With zisofs compression i can put about
30 incremental backups of one DVD+RW or 250 backups on one BD-RE.
Day by day.


> The term "archive management system" comes to mind.

I would not attribute this title to scdbackup. It was created to scratch
my itch when hard disks grew much faster in capacity than the backup
media. (Also it was the motivation to start programming on ISO 9660
producers and burn programs.)

So it might be that you are better off with a more professional backup
system. :))

(Else we will probably have to read together
  http://scdbackup.webframe.org/cd_backup_planer_help
and my backup configurations to compose configurations for you.)

--
About timestamps and incremental backup:

If you only go for mtime, them you miss changes of file attributes
which are indicated by ctime.
Even more, timestamps alone are not a reliable way to determine which
files are new at their current location in the directory tree.
If you move a file from one directory to the other, then the timestamps
of the file do _not_ get updated. Only the two involved directories get
new timestamps.
So when the backup tool encounters directories with young timestamps
it has to use other means to determine whether their data files were
moved. scdbackup uses recorded device and inode numbers, and as last
resort recorded MD5 sums for that purpose.

(Of course, content MD5 comparison is slow and causes high disk load,
compared to simple directory tree traversal with timestamps and inode
numbers. So scdbackup tries to avoid this when possible and allowed
by the -changetest_options in the backup configuration file.)


Have a nice day :)

Thomas

Re: OT: Is there any size limit for ISO's?

[Thomas Schmitt]

Hi,

Hans wrote:
> does anyone know, if there is a limit of the size an iso may have?

With xorriso: 4 TiB = 4096 GiB.

An ISO 9660 filesystem may have 2 exp 32 data blocks. The usual block
size is 2 exp 11 = 2048 bytes. That would be 2 exp 43 = 8 TiB in total.
But xorriso uses libburn for writing and a poor design decision of
about 20 years ago lets it use signed int for block addresses.
So only 2 exp 31 blocks = 4 TiB are possible with libburn and xorriso.


> Background: I have a selfmade lifefile iso image, which was created by
> bootcdwrite. The size of the imagefile is about 32GB.

Looks like bootcd uses genisoimage for producing the ISO,
(https://tracker.debian.org/media/packages/b/bootcd/control-6.7)

Anyways, 32 GB of size should not be a problem.


> When the stick is booting, first appear the usual text messages,

Can you determine the origin of the last visible messages ?
BIOS/EFI ?
Boot loader (GRUB or ISOLINUX/SYSLINUX) ?
Linux ?
Debian installation software ?


> but then the screen goes blank and the further boot hangs.
> However, when I am booting this file (this iso-image) with Virtualbox,
> everything is working well.

Is Virtualbox using the same BIOS type as the real machine ?
I mean EFI versus Legacy BIOS aka CSM mode.


> Note: In earlier times I had a smaller image (about 8 GB), which booted
> perfectly.

Is the USB stick large enough for the image ?

Can dd read the whole image back from the USB stick ?

  # Set variable blocks to image file size divided by 2048
  blocks=...
  # Set variable stick to device file address of the USB stick
  stick=...

  dd if="$stick" bs=2048 count="$blocks" | md5sum

If no i/o error is reported by dd, compute the MD5 sum of the image file
and compare both.


> So I believe, the size of the image is, what matters. But maybe I am wrong.

It's not impossible. Especially if the stick hardware has problems.
But depending on the origin of the last visible message there is also
a chance that boot loader or Linux derail because of buggy file content
in the elsewise healthy ISO.


Have a nice day :)

Thomas

Re: counting commas

[Thomas Schmitt]

Hi,

Greg Wooledge wrote:
> unicorn:~$ string="apple,banana,cherry,date"
> unicorn:~$ commas=${string//[!,]/}
> unicorn:~$ echo "${#commas}"
> 3

Always astonishing what a good bashism can do.


> But at this point, we have to wonder what the *actual* goal is.

Up to now we only know about the astonishment of fxkl4...@protonmail.com
that grep -c does not count characters.

For a more complicated use case i would write a little C program where
i'd be in control of every single bit of throughput.
(Ok, C causes scars on the programmer's self esteem. But what does not
kill me makes me just stronger. I'm a vim user.)


Have a nice day :)

Thomas

Re: smartctl cannot access my storage, need syntax help

[Thomas Schmitt]

Hi,

Anssi Saari
> It does seem strange to me, even in MS-DOS era I was able to set a
> terminal scrollback to 5000 lines without issue, when RAM was maybe 4 MB
> and a DOS terminal program probably had access to way less than that.

I have no problems with 130 xterms of 10,000 lines each.


> So does rsync really generate gigabytes of verbose output?

rsync can be extremely verbose when the number of transferred files is
very high.


> Or is xfce-terminal storing the scrollback in a very inefficient way?

I would not be astonished to learn that the luxury ornamented terminals
of the various desktops waste many extra bytes when memorizing plain text.
But the real bug is the fact that the scroll back memory is unlimited and
can summon the OOM killer. (I imagine it like the Discworld Death of Rats.)

If i were a user of Xfce i would report this as bug to its Debian
maintainer. Bug title "xfce-terminal: A landmine on the kids' playground".


Have a nice day :)

Thomas

Re: normally start new xterms

[Thomas Schmitt]

Hi,

to...@tuxteam.de wrote:
> > > I coopted the otherwise useless "Windows" key (aka "Left Super" for
> > > WM things: Super-L makes an xterm:
> > >  # Terminal
> > >  Key   "t"   A   4  Exec exec xterm

i wrote:
> > For me the Flying Windows keys pop up or push down the affected window:
> >   Key Super_L A N RaiseLower
> >   Key Super_R A N RaiseLower
> > What i don't understand in your example is the Keyname "t".

to...@tuxteam.de wrote:
> It's Modifier (that Flying Window thing), aka "4" and the regular "t"
> (my mnemonics: "terminal").

Ah. You use it as modifier for other keys, i use it as plain key.
The quotation marks around "t" are only ornamental ?


Have a nice day :)

Thomas

Re: counting commas

[Thomas Schmitt]

Hi,

fxkl4...@protonmail.com wrote:
> why doesn't grep count 2 commas
> echo 'Kích thước máy xay cỏ, giá máy thế nào , phụ tùng máy mua ở đâu' |
> grep -c ,
> 1

Happens with much simpler text too:
  $ echo ',,,' | grep -c ','
  1

The man page explains it:

 -c, --count
Suppress  normal output; instead print a count of matching lines
for each input file.  With the -v,  --invert-match  option  (see
below), count non-matching lines.  (-c is specified by POSIX.)

Note that it counts lines, not characters.

Counting characters by grep seems somewhat tricky:
  
https://www.unix.com/shell-programming-and-scripting/128033-grep-command-count-characters.html

An answer to this question
  
https://stackoverflow.com/questions/16679369/count-occurrences-of-a-char-in-a-string-using-bash
proposes
echo "referee" | tr -cd 'e' | wc -c

  $ echo ',,,' | tr -cd ',' | wc -c
  3
  $ echo 'Kích thước máy xay cỏ, giá máy thế nào , phụ tùng máy mua ở đâu' | tr 
-cd ',' | wc -c
  2

(I hope i did not quote something that's against the code of conduct. :))


Have a nice day :)

Thomas

Re: normally start new xterms

[Thomas Schmitt]

Hi,

to...@tuxteam.de wrote:
> I coopted the otherwise useless "Windows" key (aka "Left Super" for
> WM things: Super-L makes an xterm:
>  # Terminal
>  Key   "t"   A   4  Exec exec xterm

For me the Flying Windows keys pop up or push down the affected window:

  Key Super_L A N RaiseLower
  Key Super_R A N RaiseLower

What i don't understand in your example is the Keyname "t".
man fvwm points me to /usr/include/X11/keysymdef.h or
/usr/X11R6/lib/X11/XKeysymDB (which seems not exist in Debian except
a file "XKeysymDB" of ipackage "xemacs21-support").

Do you know documentation which describes your "t" ?

(I see in my configuration that in the past i bound a pseudo-key F34 to
keycodes 115 and 116 by xmodmap(1) and used the Keyname F34 with fvwm
command "Key". After some system change i had to google and learned that
"Super_L" and "Super_R" work without help of xmodmap.)


Have a nice day :)

Thomas

Re: normally start new xterms

[Thomas Schmitt]

Hi,

i wrote:
> > *FvwmButtons xterm_ts5 linuxterm.xpm Exec xterm -ls -geometry 80x24 -bg 
> > wheat -fg black -sl 1 +sb

John Conover wrote:
>
>Action 'Exec exec xterm ...'

The framework of this line probably stems from a SuSE Linux of 1999.
It still works with the fvwm of Debian 11.

I now read man 1 FvwmButtons. All applicable descriptions and examples
demand the keyword "Action".

The "exec" before "xterm" is used in some examples of the man page and
mentioned already in my S.R.Bourne of 1983 (without "xterm", of course).
As reasoning i found
  
https://www.fvwm.org/Wiki/Tips/FvwmStartup/#33-use-exec-exec-to-prevent-unnecessary-dead-shell-processes
I have 63 such processes lingering around. But there are 131 xterms.
Many of them have process 1 as parent. Probably they stem from a script
which i use to populate the 8 "desktops" by a handful of xterms when the
system comes up.


Have a nice day :)

Thomas

Re: normally start new xterms

[Thomas Schmitt]

Hi,

i wrote:
> >xterm -ls -geometry 80x24 -bg wheat -fg black -sl 1 +sb &

Max Nikulin wrote:
> Options may be put into ~/.Xresources
> xterm*vt100.saveLines: 1
> xterm*VT100.background: wheat
> xterm*VT100.foreground: black

I have it in ~/.fvwm2rc as:

*FvwmButtons xterm_ts5 linuxterm.xpm Exec xterm -ls -geometry 80x24 -bg wheat 
-fg black -sl 1 +sb

This causes a button in the button box which creates a new xterm when
clicked.


Have a nice day :)

Thomas

Re: smartctl cannot access my storage, need syntax help

[Thomas Schmitt]

Hi,

gene heskett wrote:
> > where did the extra 19.4G's come from? Can filesystem
> > ext4's overhead account for that?

In an earlier mail:

> > > command line: rsync -a --bwlimit=10m --fsync --progress /home/ 
> > > /mnt/homevol

David Christensen wrote:
> Please RTFM rsync(1) to choose your options.  These look
> useful:
>--archive, -a   (-rlptgoD)
>--delete
>--hard-links, -H
>--one-file-system, -x
>--sparse, -S

I bet on --hard-links and --sparse as means to avoid the extra disk space
consumption. (--archive is important for other reasons, but it was
already in use as -a with your successful rsync run. --delete will be
of importance if the rsync run gets repeated on the already filled target
directory tree.)

man rsync:

 -H, --hard-links
This tells rsync to look for hard-linked files in the source and
link together the corresponding files on the destination.  With‐
out  this option, hard-linked files in the source are treated as
though they were separate files.
[...]
 -S, --sparse
Try  to  handle  sparse  files  efficiently so they take up less
space on the destination. [...]

One can observe a similar inflation effect when copying the files of a
Debian installation ISO to hard disk. In the original disk directory
on the machine which created the ISO there were hardlinked kernels and
firmware packages. In the ISO these link siblings share the same file
content storage.
But when mounted, the siblings get treated as separate files with
different inode numbers. So the 8,135,584 bytes of the hardlink siblings
  /install.amd/gtk/vmlinuz
  /install.amd/vmlinuz
  /install.amd/xen/vmlinuz
get triplicated when these three files get copied out of the ISO.

I am somewhat astonished that --hard-links is not default in rsync,
as it is quite important for backup fidelity.
(On the other hand it is some effort to find all siblings on the disk.)

Sparse files are files with large areas of 0-bytes. Many filesystems
don't store the zeros but rather an instruction to hand out the given
number of 0-bytes when requested by a reader.


If i were you, i'd let rsync make a complete new copy with --hard-links
--sparse, and --delete, but without --bwlimit= in order to get a higher
copy fidelity and also to check whether the transfer speed really was not
to blame for the appearance of the OOM killer.


Have a nice day :)

Thomas

Re: smartctl cannot access my storage, need syntax help

[Thomas Schmitt]

Hi,

i wrote:
> > What did finally help ? Just the shorter terminal scroll back memory ?

gene heskett wrote:
> That, and possibly the --bwlimit=10m, giving the SSD time to keep their
> stuff in one sock.

Then i place my bet on the terminal alone.
Linux is able to handle disk-to-disk copies that are larger than the
available memory. This is a standard use case.


> > How large was it set when your runs caused the OOM killer to act ?

> different terminal, xfce4's is apparently unlimited but can't find it in the
> config prefs.

I normally start new xterms by

  xterm -ls -geometry 80x24 -bg wheat -fg black -sl 1 +sb &

The -sl option gives the number of lines to be memorized for scrollback.
Black-on-wheat is a calmative color combination which does not overwork
the eyes.


Have a nice day :)

Thomas

Re: smartctl cannot access my storage, need syntax help

[Thomas Schmitt]

Hi,

i see that i messed up "h" and "k" in my explanation of the fight over
the link targets in /dev/disk/by-id. So another attempt:

sdh has a unique serial number GSTD02TB230102. Thus we see in
  https://lists.debian.org/debian-user/2024/01/msg00667.html
these two links:

  /dev/sdh/dev/disk/by-id/ata-Gigastone_SSD_GSTD02TB230102
  /dev/sdh1   /dev/disk/by-id/ata-Gigastone_SSD_GSTD02TB230102-part1

sdi and sdj share the serial number GST02TBG221146. So the concurrent
attempts to create the links let only these two survive:

  /dev/sdi/dev/disk/by-id/ata-Gigastone_SSD_GST02TBG221146
  /dev/sdj1   /dev/disk/by-id/ata-Gigastone_SSD_GST02TBG221146-part1

sdk and sdl share GSTG02TB230206. The survivors are:

  /dev/sdk/dev/disk/by-id/ata-Gigastone_SSD_GSTG02TB230206
  /dev/sdk1   /dev/disk/by-id/ata-Gigastone_SSD_GSTG02TB230206-part1

The next system startup might yield other survivors.


Have a nice day :)

Thomas

Re: smartctl cannot access my storage, need syntax help

[Thomas Schmitt]

Hi,

David Christensen wrote:
> I suspect the conflicting serial numbers are causing problems in the kernel,
> as indicated by the /dev/disk/by-id/* problems.

That's not in the kernel but in udev/systemd's process of creating the
symbolic links in /dev/disk/by-id/.
It gets /dev/sd[h-l] and /dev/sd[h-l]1 as kernel generated device files.
But sd[ij] and also sd[hl] show pair-wise the same serial numbers.
In case of sd[ij] the outcome is mixed: links to sdi and sdj1 survive.
In case of sd[hl] we see a less strange outcome: sdh and sdh1, while
sdl and sdl1 are missing.

The open question (at least to me) is whether it's the disks or the
controllers or the drivers which cause the duplication.


Have a nice day :)

Thomas

Re: smartctl cannot access my storage, need syntax help

[Thomas Schmitt]

Hi,

Curt wrote:
> I discovered a couple of discussions of the phenomenon, the upshot of which
> were:
> 1) That's what you get when you purchase cheap SSDs.
> https://www.reddit.com/r/truenas/comments/s0rrpo/two_sata_ssds_with_identical_serial_numbers/
> 2) SSDs belonging to the same software RAID show identical serial numbers
> in software, but these numbers don't match the serial numbers printed on the
> SSDs themselves.
> https://www.reddit.com/r/truenas/comments/s0rrpo/two_sata_ssds_with_identical_serial_numbers/

Those URLs are identical. (OMG ! Is it contageous ?)

Number 2 would match my suspicion that some layer in the disk driving
gets confused and mixes up the serial numbers.


> But you said *similar*.

By "colliding serial numbers" i mean indeed "identical serial numbers".

How cheap the disks may ever be, that would be no excuse for not making
them individually distinguishable.


> As Gene's threads have too many movable parts
> for me to follow, on that point I couldn't say.

This one begins to gain presence in the web. So one can use search engines
and AI to untangle its sub-threads. I meanwhile participate in two of them:
serial number collision, rsync caused OOM killer (solved now, but how ?).


Have a nice day :)

Thomas

Re: smartctl cannot access my storage, need syntax help

[Thomas Schmitt]

Hi,

after i began enumerating suspects, gene heskett wrote:
> terminals scroll back memory, I purposely set this
> particular terminals scrollback to 200 lines with that in mind.

How large was it set when your runs caused the OOM killer to act ?

I have a good number of xterms with 10,000 lines each. No tabs, no KDE,
but 8 fvwm "desktops" (virtual screens) full of terminal windows.


> > [Request to test the disks one-by-one on some other computer, whether
> >  they bear the same serial number at all controllers in all machines.]

> Not as easily tried, the other 4 are in twin mounts in another portion of
> the drive cages in this 30" tall tiger direct cage and not too readily
> accessible w/o tipping the mobo out on its hinged mount.

One should raise protest at Gigastone if the disks really have the same
serial numbers. But before doing so, one would have to make sure that
it is not some weird effect of them all being plugged into that machine
at the same time.


> One of you made the remark that seems to be the secret password.

What did finally help ? Just the shorter terminal scroll back memory ?

It would explain why a verbose rsync could summon the OOM killer always
around the same stage of progress. But what waste of memory would have
to happen with each of the rsync messages ?

(You mentioned LABEL as a possibility. But not as actually used.)


> Its still, slowly at 10 megs a second, working.

I see in your previous mail rsync option --bwlimit=10m . But in the
same mail there is an older quote from you that --bwlimit=3m only
prolonged the time until the OOM killer appeared.
So i wonder whether it would work at a more contemporary speed.



Self-incrimination: The rest of this mail is off topic.

> they gave all 7nth graders the Iowa
> test in 1947, similar to the S/B IQ test but not copyrighted, there fore a
> lot cheaper, and I came out of that with an equivalent of 147.

I was tested in the 1960s but they did not tell the results to kids or
parents. We only got recommendations at which of our three types of school
we should continue at the age of 10 or 11 years.
(So it was not to avoid discrimination of the dumb but rather to avoid
that pupils feel more intelligent than their teachers.)


Have a nice day :)

Thomas

Re: smartctl cannot access my storage, need syntax help

[Thomas Schmitt]

Hi,

Gene Heskett wrote:
> lsblk, which I've published several times, shows 5 drives.

Duh. Obviously this thread overstretches my mental capacity.


> And I've since tried cp in addition to rsync, does the same thing, killing
> the sysytem with the OOM but much quicker. cp using all system memory (32Gb)
> in 1 minute, another 500K into swap adds another 15 secs, and the OOM kills
> the system. So both cp and rsync act broken.

I get the suspicion that your disk set overstretches the mental capacity
of the hardware or the operating system.
Both "cp" and "rsync" are heavily tested by the GNU/Linux community and
quite independently developed. A common memory leak would have to sit
deeper in the software stack, i.e. in kernel or firmware.


> rsync, with a --bwlimit=3m set, takes much longer to kill the system but the
> amount of data moved is very similar, 13.5G from clean disk to system freeze
> for rsync, 13.4G for cp.

This observation might be significant. But i fail to make up a theory.


> gene@coyote:~/src/klipper-docs$  lsblk -d -o NAME,MAJ:MIN,MODEL,SERIAL,WWN 
> /dev/sd[hijkl]
> NAME MAJ:MIN MODEL SERIAL WWN
> sdh8:112 Gigastone SSD GSTD02TB230102
> sdi8:128 Gigastone SSD GST02TBG221146
> sdj8:144 Gigastone SSD GST02TBG221146
> sdk8:160 Gigastone SSD GSTG02TB230206
> sdl8:176 Gigastone SSD GSTG02TB230206

This is just weird.
I still have difficulties to believe that any disk manufacturer would
hand out disks with colliding serial numbers. I googled for this
phenomenon, but except two mails of Gene nothing similar popped up.

One of these mails from a thread in december reveals that the three
unique serial numbers GSTD02TB230102, GST02TBG221146, GSTG02TB230206
each come with a different version of "1C0", "7A0", "5A0", respectively.
  https://www.mail-archive.com/debian-user@lists.debian.org/msg799307.html
That's unexpected, too, as the disk properties look identical elsewise.

I guess that it is not possible to identify which disk came with which
of the two separate purchases ?
How many days were these purchases apart ?


David Christensen wrote:
> I suggest removing one GST02TBG221146 and one GSTG02TB230206.  Put them on
> the shelf, in other computer(s), or sell them.  Then perhaps copying the
> /home RAID10 2 TB to one Gigastone 2 TB SSD would work.

I join this proposal.
... and dimly remember to have seen the proposal to attach the disks
one by one without the other four, in order to see whether the serial
numbers are the same as with all five together.

Since you got quite some hardware zoo:
Consider to try the Gigastone disks with a different machine.
Do the serial numbers show up as with the machine where you experience
all those difficulties.


Have a nice day :)

Thomas

Re: smartctl cannot access my storage, need syntax help

[Thomas Schmitt]

Hi,

i, too, wondered where there should be a duplicate serial number.
But indeed:

David Wright wrote:
> > /dev/sdi53  /dev/disk/by-id/ata-Gigastone_SSD_GST02TBG221146
> > /dev/sdj1   54  /dev/disk/by-id/ata-Gigastone_SSD_GST02TBG221146-part1
>   ↑ that is /really/ bad!

Does the number of 4 device files /dev/sd[h-k] match the number of
installed ata-Gigastone_SSD devices ? Gene talked of
"5, ordered in 2 separate orders".
(Looking at https://lists.debian.org/debian-user/2024/01/msg00667.html)
Now we see 3 to 4, depending on what one wants to believe.

Wild ideas:
One possible reason could be that a device is mapped to both, /dev/sdi
and /dev/sdj. udev would then suffer a race condition when creating the
/dev/disk/by-id.
Another could be that udev's assessment of the drives derails and that
serial number information spilled from the assessment of /dev/sdi to
the assessment of /dev/sdj*.

It would be interesting to see the output of

  ls -l /dev/sd[ij]*

in order to learn about the existence of /dev/sdj and the the device
numbers of sdi* and sdj*.

Further one should inquire the serial numbers by

  lsblk -d -o NAME,MAJ:MIN,MODEL,SERIAL,WWN /dev/sd[hijk]


Have a nice day :)

Thomas

Re: 512e vs 4K sector confusion

[Thomas Schmitt]

Hi,

Andy Smith wrote:
> what I meant was that fdisk showed a single partition of
> 3.2TB size, while the entire disk being only the 400G

Then it's what i would expect from fdisk.


> I did try using fdisk on the destination to delete the partition and
> recreate it with the correct numbers, but the ext4 driver and fsck
> and other filesystem tools were still totally unable to work with
> it. Yet, a sha256sum of the disk image file at both sides showed the
> same value, so it is the same data.

I would also expect complaints about read attempts after the end of
the device.


> Is it actually possible to create an LVM volume that thinks it has 512b
> logical sectors, on an underlying device that is 4Kn?

Interesting question.
Googling for LVM instructions ... pvcreate(8), vgcreate(8), lvcreate(8).
Their man pages show no options for selecting block sizes.

losetup(8) has an option which looks helpful:
  -b, --sector-size size
 Set the logical sector size of the loop device in bytes (since Linux
 4.14).
But that would be another software layer between filesystem and physical
device.
(losetup -b 4096 would possibly help to mount an image file from a disk
with logical block size 4K.)


Have a nice day :)

Thomas

Re: 512e vs 4K sector confusion

[Thomas Schmitt]

Hi,

Andy Smith wrote:
> I've got a disk image that sits on top of an LVM logical volume
> that is on top of an mdadm RAID-1 that is on top of a pair of:
> Device Model: Samsung SSD 870 EVO 4TB
> Sector Size:  512 bytes logical/physical
> so let';s say that is at /dev/foo/disk_image (where /dev/foo is the
> name of the LVM VG and disk_image is the LV)
> So
> # fdisk -ul /dev/foo/disk_image
> Disk /dev/foo/disk_image: 400 GiB, 429496729600 bytes, 838860800 sectors
> Units: sectors of 1 * 512 = 512 bytes

fdisk thinks that this device has a logical block size of 512.

This information is not stored in MBR partition table (or in GPT) but
rather it is a property of the storage device.


> Disklabel type: dos
> So it's a 400G disk image with an MBR and a single partition, right?

Yes.

(But nitpicking:
It has an MBR partition table with entries other than the protective
entry which would indicate the presence of GPT. I.e. GPT contains an
MBR too, but its partition table entries are stored outside of the MBR.)


> Now, I dd that disk image across the network to another machine
> which has a similar setup, except here the "foo" volume group is on
> a pair of
> Device Model: HGST HUS726T6TALN6L4
> Sector Size:  4096 bytes logical/physical

So the logical volume was probably created with the logical block size
of the involved physical disk devices.


> fdisk
> thinks it is 8 times bigger than it really is, and thinks it has 4K
> sectors.

fdisk asks the device (unless it is a regular file, i guess) for its
logical size. Since the partition table gives no hint about its own
idea of the bytes/block ratio, you end up with vastely inflated byte
numbers.

I wonder, though, why fdisk would misrepresent the total disk size,
which i would expect to come from the storage device.
Maybe fdisk is more confused than i expect.

Can you show the whole output of fdisk from the new logical device ?


Whatever:
Not only the partition table might get misinterpreted like the partition
table. Depending on the filesystem and its Linux driver the same wrong
address computations can happen inside the filesystem.

(ISO 9660 has a field to announce the intended logical block size.
But the Linux driver "iso9660" ignores it and assumes 2048 bytes/block.
It is nice enough to convert the LBA values of the filesystem to
512 bytes per logical block addresses if needed. I guess it does the same
for disk devices with 4096 bytes per logical block, but never tested this.)


> Is there any way to make this work?
> DeviceBoot Start   End   Sectors  Size Id Type
> /dev/foo_disk_image2048 838858751 838856704  400G 83 Linux

The start address and the number of sectors is integer divisible by 8.
You could change the partition start and size so that the block numbers
address the correct bytes with the 8 fold larger logical block size.

But given that this could still create an unusual situation in the
filesystem driver, i would try to create a device with a logical block
size of 512.
(... or i would create and mount a new filesystem on the new device and
copy the files over from the mounted image file.)


Have a nice day :)

Thomas

Re: VAX emulation/simulation (was Re: systemd-timesyncd)

[Thomas Schmitt]

Hi,

Bret Busby wrote:
> Whilst, as I previously made the point, this is all off-topic for a Debian
> operating system users mailing list

But i found a premium excuse in the debian-cd and debian-live ISOs. :o)


> the last seven versions of a file, being retained, was a
> useful tool for software developers, but, responsible software development,
> and, especially, the teaching of responsible software development, have been
> abandoned, over the last decades.

This has obviously been replaced by Source Code Control System, which
now has its final successor under the name Git.
(A coarse substitute are frequently taken incremental backups.)


Have a nice day :)

Thomas

Re: VAX emulation/simulation (was Re: systemd-timesyncd)

[Thomas Schmitt]

Hi,

Nicholas Geovanis wrote:
> You ruined my day :-)

It was not my fault. Send complaints to the people who convened as
"High Sierra Group" in 1986.


> Something similar to IBM's kludgiest relic of the early 1960s has appeared
> in linux?

The unixoid community added System Use Protocol and Rock Ridge Interchange
Protocol in the early 1990s in order to get X/Open functionality on top
of ISO 9660. That's POSIX with long file names (up to 255 bytes) and
paths (up to 1024 bytes) where only 0-byte and '/' have special meanings.

A company Who Must Not Be Named introduced Joliet to store names of up
to 64 characters in a 16 bit character set (while still ignoring the
difference between uppercase and lowercase).

Linux mount(8) introduced a character mapping from the uppercase character
set of ISO 9660 to lowercase. This mapping also removes the version part
of the file names.


> The idea that we need version numbers embedded in filenames
> involuntarily may be "natural" to somebody.

I have never seen any version other than ";1" (and ISOs which simply
ignore the specs about file names). It's a non-functional relic, which
in Linux can only be uncovered if you suppress Rock Ridge, Joliet, and
name mapping during the mount command.


> And I've been an IBM mainframe admin and developer too.

In the times when a full scale mainframe came with a female discus
thrower ?
  http://www.ibmsystem3.nl/5444/images/5444DISK.jpg


Have a nice day :)

Thomas

CD-DA sector addressing, was Re: playing CDROM music questions

[Thomas Schmitt]

Hi,

i cannot contribute much to the practical issues with playing music.
But i'd like to clarify technical properties of CD-DA media:

Nicolas George wrote:
> compared to data CDs, audio CDs lack one layer of error-correcting code

True.

Another drawback is that CD-DA sectors cannot be read by the usual SCSI
commands for hard disk reading, but only by the special command READ CD.

> and [lack] the synchronization necessary to tell the
> position of a particular sector.

Not true. It is possible to read CD-DA with sector granularity.

Random addressing may be somewhat cumbersome for the drive, because the
exact sector address is part of the low level data of the sector.
See MMC-5 4.2.3.5.2 "Mode-1 Q" for how it is implemented on medium and
6.20 "READ CD Command" for how it is to be operated by software of the
hosting computer.

> This is why tools like cdparanoia exist: between one read and the next
> they seek back a few sectors and check that the overlap matches.

If ever it is the drive's firmware which has to guess the position and
then to pick the sectors with the desired address in Mode-1 Q (or in a
neighbor if one of the 10% sectors without Mode-1 Q is desired).

The main reason for specialized CD readers like cdparanoia is rather in
the fact that data readers like mount(8) want to use the general SCSI disk
read commands. There are more reasons, as the man page of cdparanoia
indicates, including workarounds for bugs of antique drives.


Have a nice day :)

Thomas

Re: VAX emulation/simulation (was Re: systemd-timesyncd)

[Thomas Schmitt]

Hi,

Bret Busby wrote:
> > .;

Jeremy Nicoll wrote:
> IBM's MVS & its successors, most recently z/OS, have something
> similar called a GDG (or Generation Data Group).

The principle made it into ISO 9660 specifications.

To make this thread relevant for Debian, let's assume that somebody
asked about the peculiar filenames in the netinst ISO when mounting
its plain ISO 9660 personality:

  $ sudo mount -o norock,nojoliet,map=off debian-12.2.0-amd64-netinst.iso 
/mnt/iso
  mount: /dev/loop0 is write-protected, mounting read-only
  $ find /mnt/iso
  /mnt/iso
  /mnt/iso/BOOT
  /mnt/iso/BOOT/GRUB
  /mnt/iso/BOOT/GRUB/EFI.IMG;1
  /mnt/iso/BOOT/GRUB/FONT.PF2;1
  ...
  /mnt/iso/_DISK/MKISOFS.;1
  /mnt/iso/_DISK/UDEB_INC.;1


Have a nice day :)

Thomas

Re: difference in seconds between two formatted dates ...

[Thomas Schmitt]

Hi,

Albretch Mueller wrote:
> But how do you strace a program saving the output (of the stracing)
> in a logfile while you also save that program's output without making
> it part of the stracing?

man strace says:

  -o filename Write the trace output to the file  filename  rather
  than  to  stderr.

So strace normally directs its output to stderr. That's file descriptor 2.
You can redirect it by the "2>file" gesture of the shell:

  $ strace echo hello 2>file
  hello
  $ cat file
  execve("/bin/echo", ["echo"], [/* 35 vars */]) = 0
  brk(0)  = 0x13ba000
  ...
  exit_group(0)   = ?
  +++ exited with 0 +++

Or you may use the mentioned -o option which will keep the tracee's stderr
out of the file:

  $ strace -o file echo hello
  hello
  $ cat file
  ...


Have a nice day :)

Thomas

Re: Dealing with SPAM.

[Thomas Schmitt]

Hi,

to...@tuxteam.de wrote:
> [...] (it's actually a logistic function [1]).
> [1] https://en.wikipedia.org/wiki/Logistic_function
> Looking forward to Yet Another Of Those Nerdy Monster Threads ;-)

Since it's happening periodically with about the same participants,
shouldn't we rather try to model it as hysteresis ?
  https://en.wikipedia.org/wiki/Hysteresis
  ("Not to be confused with Hysteria.")


Have a nice day :)

Thomas

Re: lists

[Thomas Schmitt]

Hi,

(Cc-ing poc...@columbus.rr.com just in case the unsubscription strikes
again.)

Pocket wrote:
> I spend several days trying to subscribed to the list, with the web signup
> ALWAYS time out with a gateway error.

The mails from the list have these informational headers:

  List-Subscribe:

  List-Unsubscribe:


So to subscribe, send a mail to
  debian-user-requ...@lists.debian.org
with the Subject: text "subscribe".


> I get/have a kick rate of 2% (one bounce in the last 60 days), then I am
> kicked and no longer receive anything from the user list.

It is possible that these warning mails and the final unsubscription
are not directly related.

A while ago this list had a problem with fake bounces. A troll managed
to convince the list server that some of the subscribers would reject the
list mails. I am not familiar with the exact technical properties of that
attack and how it was finally disabled by the listmasters.

It was in october 2021. See the threads under:
  https://lists.debian.org/debian-user/2021/10/msg00248.html
  https://lists.debian.org/debian-user/2021/10/msg00337.html
I became target after complaining that i was not deemed hateworthy enough
to become a victim (and showing an example of a real bounce):
  https://lists.debian.org/debian-user/2021/10/msg00434.html
The problem ended after a preliminary change:
  https://lists.debian.org/debian-user/2021/10/msg00567.html
  "We now disabled bounce-detection if it is related to that.
   We'll check how to be more strict in accepting these kind of mails
   now.
   Yours, Cord, Debian Listmaster of the day"
I guess that this change was later revoked after means were found to
recognize the fake bounce mails.

Maybe now the attack was refined ? Maybe i will be the next victim ?


Have a nice day :)

Thomas

Re: Monthly FAQ for Debian-user mailing list (modified 16th December 2023)

[Thomas Schmitt]

Hi,

Andrew M.A. Cater wrote:
> * It may also be useful to for someone to post a summary email from time to
>   time to explain long threads.

You did not move the old "to" but rather added a new one during the change
from the text in 2023/12/msg00045.html to the new one:

> > * It may also be useful to post a summary email from time to time to
> >   explain long threads.

As we are at it, i think the term "for someone" does not add much of
information. Do you have in mind future readers of the archive ?
Or casual readers who got lost in forth and back ? Or contributors
who deviated from the original topic of the thread ?

Further:

I see a surplus leading blank in the line
  "   and scan it for redundancy, duplicate words and redundancy."
Other lines have only two of those blanks.

I see an inner double blank in
  "  Any advice given accordingly  may be inaccurate but is given in good 
faith."


Have a nice day :)

Thomas

Re: On file systems

[Thomas Schmitt]

Hi,

to...@tuxteam.de wrote:
> Remember
> Apple's "fat binaries", which contained a binary for 68K and another
> for PowerPC? Those were made with "forks", which was Apple's variant
> of "several streams in one file". And so on.

The most extreme example i know is Solaris:

  https://docs.oracle.com/cd/E36784_01/html/E36883/fsattr-5.html

  fsattr - extended file attributes

  Description
  Attributes are logically supported as files within the file system.
  The file system is therefore augmented with an orthogonal name space of
  file attributes. Any file (including attribute files) can have an
  arbitrarily deep attribute tree associated with it. Attribute values
  are accessed by file descriptors obtained through a special attribute
  interface.

So every file can have a second job as directory ... in theory.
In practice, though:

  Implementation are [...] permitted to reject operations that are not
  supported. For example, the implementation for the UFS file system
  allows only regular files as attributes (for example, no
  sub-directories) and rejects attempts to place attributes on attributes.


Have a nice day :)

Thomas

Re: ntpsec as server questions

[Thomas Schmitt]

Hi,

gene heskett wrote:
> > In the FWIW dept this time formula is pretty accurate back to the
> > middle of 4713 BC.

Greg Wooledge wrote:
> Even the *Julian* calendar used in ancient Rome wouldn't have been in
> use in 4713 BC.  Any calendar would have been locally defined, if one
> existed at all.

What Gene describes is the Julian Date, an astronomical time counting.
I first met it in HP BASIC but it is also used by database systems.
0.00 is 1st Januar 4713 BC 12:00 UTC (year -4712 because the is
no year 0 in BC/AD counting).

It must not be confused with the Julian Calendar, which invented the
the 4-year leap year rule.
This worked until pope Gregor saw the need for a finer adjustment to the
day fractions of the astronomical year. This yielded the rules about
100 and 400 years.

Afaik, they all suffer from being day-oriented which makes trouble because
the astronomical days get longer over time and thus cause leap seconds.
For long term accuracy in the range of seconds one needs to work with time
countings close to International Atomic Time (TAI) which is nearly perfect
but becomes only available up to a month too late.


Have a nice day :)

Thomas

Re: Mailing List

[Thomas Schmitt]

Hi,

David Wright wrote:
> I'm subscribed, but I don't receive that badge of honour.
> This is from my other post in this thread—no LDOSUBSCRIBER:
>
> >   X-Spam-Status: No, score=-4.9 required=4.0 tests=CAPINIT,FOURLA,
> > HEADER_FROM_DIFFERENT_DOMAINS,LDO_WHITELIST,RCVD_IN_DNSWL_LOW,
> > T_SCC_BODY_TEXT_LINE autolearn=unavailable autolearn_force=no
> > version=3.4.2

This is known to happen if the mail is sent from an address that is different
from the subscribed mail address. Maybe you discovered a new cause.


> I'm guessing your last example is Curt's.
> >   X-Spam-Status: No,
> > score=-10.5 required=4.0 tests=FREEMAIL_FORGED_FROMDOMAIN,
> > FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,LDOSUBSCRIBER,
> > LDO_WHITELIST,T_SCC_BODY_TEXT_LINE autolearn=unavailable
> > autolearn_force=no version=3.4.2

Yes. It's from:

  Subject: Re: rhs time out error?
  Date: Sun, 26 Nov 2023 16:30:02 - (UTC)
  Message-ID: 

> The only occurrence of
> the From: address in the entire email is in the From: line.
> That's no different from my own post, except for the lines at the
> very top, which show my post being delivered to me.
>
> I had thought the server was using the envelope-from in order to
> identify subscribers, yet Curt's posts, like mine, have different
> envelope-from and From: addresses, which is presumably the reason
> behind HEADER_FROM_DIFFERENT_DOMAINS.

HEADER_FROM_DIFFERENT_DOMAINS would have been my first suspicion, too.
But i see no "envelope-from" Curt's mail and yours.
Only
  Envelope-To: 
(If "envelope-from" is a typo and "Envelope-To:" differs from "From:",
then we'd probably have the situation of different sender and receiver.)


It would be interesting to see all "Received:" headers of your own
mail when it arrives back to you. I see in your mail to which i now reply:

  Received: from bendel.debian.org ([82.195.75.100]) by mx-ha.gmx.net
(mxgmx109
 [212.227.17.5]) with ESMTPS (Nemesis) id 1MZzsi-1qmfcQ0kRo-00R0wN for
 ; Sun, 03 Dec 2023 05:24:06 +0100
  Received: from localhost (localhost [127.0.0.1])
by bendel.debian.org (Postfix) with QMQP
id 0F8C9209D5; Sun,  3 Dec 2023 04:23:55 + (UTC)
  Received: from localhost (localhost [127.0.0.1])
by bendel.debian.org (Postfix) with ESMTP id 0413D20837
for ; Sun,
3 Dec 2023 04:23:42 + (UTC)
  Received: from bendel.debian.org ([127.0.0.1])
by localhost (lists.debian.org [127.0.0.1]) (amavisd-new, port 2525)
with ESMTP id 1Ym_tsLfWzf6 for ;
Sun,  3 Dec 2023 04:23:33 + (UTC)
  Received: from omta012.uswest2.a.cloudfilter.net
(omta012.uswest2.a.cloudfilter.net [35.164.127.235])
(using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
(Client CN "Client", Issuer "CA" (not verified))
by bendel.debian.org (Postfix) with ESMTPS id 1A7BD209C6
for ; Sun,
3 Dec 2023 04:23:32 + (UTC)
  Received: from cxr.smtp.a.cloudfilter.net ([10.0.16.145])
by cmsmtp with ESMTP
id 9deErTpHpaga99e0vrCJA8; Sun, 03 Dec 2023 04:23:29 +
  Received: from axis.corp ([68.102.133.185])
by cmsmtp with ESMTPSA
id 9e0srwhHYTywR9e0urtWnm; Sun, 03 Dec 2023 04:23:29 +

All these headers except the first were added on the sender side, i.e
on the list server bendel.debian.org and in your mail provider's realm.

In a mail of mine to this list i see:

  Received: from bendel.debian.org ([82.195.75.100]) by mx-ha.gmx.net
(mxgmx009
[212.227.15.9]) with ESMTPS (Nemesis) id 1MMYsv-1qsRpG2aBT-00SPIm for
; Fri, 01 Dec 2023 21:11:45 +0100
  Received: from localhost (localhost [127.0.0.1])
by bendel.debian.org (Postfix) with QMQP
id 869AC20BB3; Fri,  1 Dec 2023 20:11:33 + (UTC)
  Received: from localhost (localhost [127.0.0.1])
by bendel.debian.org (Postfix) with ESMTP id 920DB20B9A
for ; Fri,
1 Dec 2023 20:11:22 + (UTC)
  Received: from bendel.debian.org ([127.0.0.1])
by localhost (lists.debian.org [127.0.0.1]) (amavisd-new, port 2525)
with ESMTP id g1fSuxvi8ZX9 for ;
Fri,  1 Dec 2023 20:11:19 + (UTC)
  Received: from mout.gmx.net (mout.gmx.net [212.227.15.15])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
 key-exchange ECDHE (P-256) server-signature RSA-PSS (2048 bits)
server-digest SHA256)
(Client did not present a certificate)
by bendel.debian.org (Postfix) with ESMTPS id 14F4520A27
for ; Fri,
1 Dec 2023 20:11:19 + (UTC)
  Received: from scdbackup.webframe.org ([91.8.169.164]) by mail.gmx.net
(mrgmx005 [212.227.17.190]) with ESMTPSA (Nemesis) id
1Mqs0R-1reJbZ24e1-00mwgO for ; Fri,
01 Dec 2023 21:11:16 +0100

So there is one mail relay hop more on your side, before the mail reaches
the Debian list server. I wonder whether this earns you the spam test
attribute HEADER_FROM_DIFFERENT_DOMAINS which i don't get in my mails:

  X-Spam-Status: No, score=-12.0 required=4.0 tests=DKIM_SIGNED,DKIM_VALID,

Re: Monthly FAQ for Debian-user mailing list (modified 1st December2023)

[Thomas Schmitt]

Hi,

Andrew M.A. Cater wrote:
> * Before posting, it may be useful to check your post for spelling mistakes
>   and scan it for redundancy, duplicate words and redundancy.

Some wisdom cannot be repeated enough.


Have a nice day :)

Thomas

Re: Monthly FAQ for Debian-user mailing list (modified 1st December2023)

[Thomas Schmitt]

Hi,

there is a new surplus word "private" in these lines:

> * Please post answers back to the list so others can benefit: private
>   private conversations don't benefit people who may only be following


Have a nice day :)

Thomas

Re: Mailing List

[Thomas Schmitt]

Hi,

> Anyone one else having trouble with the mailing list?

I got your message via the list.

> Have received any messages since Nov 30

Normal traffic yesterday and today, i'd say.

> I can not tell if I am still subscribed

The "From:" address poc...@columbus.rr.com seems not to be recognized as
being subscribed.
The message to which i reply bears no "LDOSUBSCRIBER":

  X-Spam-Status: No, score=2.7 required=4.0 tests=CAPINIT,HTML_MESSAGE,
KHOP_HELO_FCRDNS,LDO_WHITELIST,META_ATTENDEES_DBSPAM1,
RCVD_IN_MSPIKE_H2,T_SCC_BODY_TEXT_LINE autolearn=no autolearn_force=no
version=3.4.2

Unlike examples of other messages from other sender to the list:

  X-Spam-Status: No, score=-10.7 required=4.0 tests=DKIM_SIGNED,DKIM_VALID,
FOURLA,LDOSUBSCRIBER,LDO_WHITELIST,RCVD_IN_DNSWL_LOW,
T_SCC_BODY_TEXT_LINE,WORD_WITHOUT_VOWELS autolearn=unavailable
autolearn_force=no version=3.4.2

  X-Spam-Status: No, score=-8.7 required=4.0 tests=DKIM_SIGNED,DKIM_VALID,
DKIM_VALID_AU,DKIM_VALID_EF,FORGED_HOTMAIL_RCVD2,FOURLA,FREEMAIL_FROM,
HTML_MESSAGE,LDOSUBSCRIBER,LDO_WHITELIST,RCVD_IN_DNSWL_NONE,
RCVD_IN_MSPIKE_H2,T_SCC_BODY_TEXT_LINE autolearn=unavailable
autolearn_force=no version=3.4.2

  X-Spam-Status: No,
score=-10.5 required=4.0 tests=FREEMAIL_FORGED_FROMDOMAIN,
FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,LDOSUBSCRIBER,
LDO_WHITELIST,T_SCC_BODY_TEXT_LINE autolearn=unavailable
autolearn_force=no version=3.4.2

> I get
>   Error: Overload
> On the https://lists.debian.org/users.html [lists.debian.org] page

That's quite a long page.
Try
  https://lists.debian.org/debian-user/
  https://lists.debian.org/debian-user/2023/12/threads.html
and your message at
  https://lists.debian.org/debian-user/2023/12/msg00028.html


Have a nice day :)

Thomas

Re: rhs time out error?

[Thomas Schmitt]

Hi,

Karen Lewellen wrote:
> ..ah, typo indeed.
> it should be rsh.

Quite a while ago rsh has been put in the pillory for not encrypting the
connection. The town crier urged everybody to use ssh instead.

Found in the web:

https://fossies.org/linux/alpine/imap/docs/FAQ.txt

  _4.3 How does the c-client library choose which of its several
  mechanisms to use to establish an IMAP connection to the server? I
  noticed that it can connect on port 143, port 993, via rsh, and via
  ssh._
c-client chooses how to establish an IMAP connection via the
following rules:

+ If /ssl is specified, use an SSL connection. Fail otherwise.
+ Else if client is a UNIX system and "ssh server exec
  /etc/rimapd" works, use that
+ Else if /tryssl is specified and an SSL connection works, use
  that.
+ Else if client is a UNIX system and "rsh server exec
  /etc/rimapd" works, use that.
+ Else use a non-SSL connection.

  _7.25 Why is there a long delay in Alpine or any other c-client based
  application call before I get connected to the IMAP server?
  [...]
It must be emphasized that this is a bug in rsh. It is _not_ a
bug in the IMAP toolkit.

The use of rsh can be disabled in any the following ways:

+ You can disable it for this particular session by either:
 o setting an explicit port number in the mailbox name, e.g.
  {imapserver.foo.com:143}INBOX
 o using SSL (the /ssl switch)
+ You can disable rsh globally by setting the rsh timeout value
  to 0 with the call:
   mail_parameters (NIL,SET_RSHTIMEOUT,0);

The instructions "/ssl switch" and
"mail_parameters (NIL,SET_RSHTIMEOUT,0);" riddle me.
There is
  https://ratfactor.com/slackware/alpine-config.html
with lots of explanations. Maybe the desired answers are in there.


I get my mail by POP and send by SMTP. Both via stunnel(8) because GMX
insists in encryption. (Possibly one can meanwhile configure alpine to do
the SSL encryption of SMTP on its own. My POP client runs externally.)
In my ~/.pinrc i see:

  # Sets the format of the command used to open a UNIX remote
  # shell connection.  The default is "%s %s -l %s exec /etc/r%sd"
  # NOTE: the 4 (four) "%s" entries MUST exist in the provided command
  # where the first is for the command's path, the second is for the
  # host to connect to, the third is for the user to connect as, and the
  # fourth is for the connection method (typically "imap")
  rsh-command=

If i'd find a non-empty content there, i'd try whether making it empty
and setting something senseful to ssh-command= and ssh-path= will
avoid the rsh problems.

(The proposed content of that value looks like a sprintf(3) format
 string. Let's hope that alpine applies some checks before using it.)


Have a nice day :)

Thomas

Re: rhs time out error?

[Thomas Schmitt]

Hi,

Karen Lewellen wrote:
> I cannot find out what rhs means.
> Getting an rhs to imap server timeout with one of the new office emails.

Can it be that "rhs" is a typo and should rather be "rsh" ?

The web has old messages which resemble what you describe:

  Subject:
  "[Alpine-info] Why do I get "rsh to IMAP server timed out?"
  https://marc.info/?l=alpine-info=134929183126515=2

  Body:
  "IMAP 17:04:27 5/27 mm_log warning: rsh to IMAP server timed out"
  
https://alpine-alpha.u.washington.narkive.com/LCq961ri/remote-pinerc-and-rsh-timeout


Have a nice day :)

Thomas

Re: debian-12.2.0-amd64-netinst.iso usb grub boot error "invalid buffer alignment"

[Thomas Schmitt]

Hi,

Zenaan Harkness wrote:
> > > all of the grub menu options (Graphical
> > > install, Graphical expert install, Expert install, etc) give 2 errors:
> > > 1) "..invalid buffer alignment... " with some long number beginning with
> > > minus.
> > > 2) kernel fail to load error, presumably due to error 1

Before discussing the tanget about copying bootable ISOs to USB stick:

I suspect that something is wrong with the software in the ISO or with
the hardware or with the involved firmware.
The mailing list for such problems would be debian...@lists.debian.org
but some maintainers of debian-cd are subscribed to debian-user, too.


Tom Furie wrote:
> > You can't just copy the iso to the stick and get it to boot.

to...@tuxteam.de wrote:
> Actually... if you copy to the whole device (not to a partition), as
> OP seems to have done, it should work.

Indeed.
The only pitfall would be a GPT backup block at the end of the device's
writable capacity. Some software could take it as indication that a
clueless user damaged the GPT. Perky software might try to re-create
the old GPT and thus hide the EFI partiton of the ISO.

But since Zenaan Harkness reports GRUB to have started and offering the
expected installation menu, it's improbable that such a mishap occured.


> all blocks of the block device get overwritten,

Not all. Only as far as the new image reaches. That's how the GPT backup
can survive (xorriso-dd-target would explicitely overwrite the last block
by zeros).
EFI firmwares are not known to react on GPT backups, although this cannot
be ruled out completely.


Have a nice day :)

Thomas

Re: UUID permanent at removable device?

[Thomas Schmitt]

Hi,

Hans wrote:
> I want to automatically create a bootable USB-stick using dd from an
> ISO-file.

The landscape of ISO files is wide and varied.
An URL for getting the ISO would help to make more specific statements.


> However, after generating the stick the UUID of the first partition
> (/dev/sdc1) is changing, so next time, the script will not work again,
> of course.
> But does the UUID also change of the device itself? Does the UUID of
> /dev/sdc change, too?

The UUID as of program lsblk is a property of the data which is stored
on the device, not of the device itself.
For example after dd'ing debian-12.0.0-amd64-netinst.iso :

  $ lsblk -o NAME,UUID /dev/sdc
  NAME   UUID
  sdc2023-06-10-10-25-55-00
  ├─sdc1 2023-06-10-10-25-55-00
  └─sdc2 DEB0-0001

Because there is no valid GUID Partition Table in this ISO, the UUIDs are
fabricated and don't deserve the attribute "Universially Unique".
sdc and sdc1 get the creation date of the ISO 9660 filesystem as Id.
sdc2 gets the Volume Serial Number of the FAT filesystem (just 4 bytes)
as UUID.

It does not look better with PARTUUID instead of UUID:

  $ lsblk -o NAME,PARTUUID /dev/sdc1
  NAME   PARTUUID
  sdc1 23261683-01

The shown id "23261683" is the hex representation of the MBR Disc
Signature which has 4 bytes.


> I want to dd it automatically like
> dd if=/path/to/my/image.iso of=UUID=34567890-afde-.-1234

Consider refering to hardware properties, e.g. by using the symbolic links
in /dev/disk/by-id .

  dd if=/path/to/my/image.iso \
 of=/dev/disk/by-id/usb-INTENSO_USB_AA0401287699-0:0

Device link name found by knowing that this time the USB stick is sdc:

  $ ls -l /dev/disk/by-id
  ...
  lrwxrwxrwx 1 root root  9 Nov 12 19:14 usb-INTENSO_USB_AA0401287699-0:0 
-> ../../sdc
  ...


Have a nice day :)

Thomas