Re: Proposal to augment CD/faq/#verify, version 2

[Thomas Schmitt]

Hi,

i wrote:
> > - The text points to the authenticity verification page
> > https://www.debian.org/CD/verify
> >   which gives no tangible example how to verify *SUMS files by *SUMS.sign.

Max Nikulin wrote:
> I do not mind that there is a page which purpose is solely to specify key
> IDs and fingerprints since it is most sensitive info. What I do not like
> are descriptions of links to this page:
> - "verification guide"
>   - "Detailed information on how to authenticate the signed checksum [...]
>    > $ gpg --keyserver keyring.debian.org --recv-keys 64E6EA7D 6294BE9B 09EA8AC3

> Despite I do not expect keys having collisions for 32 bit identifiers
> uploaded to keyring.debian.org, I think, it is better to use 64 bit
> identifiers here

I understand from the web that in case of identical short key ids all
matching keys are received from the server.
This is not an additional security problem in the proposed instructions
because they prescribe to compare the fingerprint, not the subset of the
fingerprint which is the key id.


> I am unsure if there are drawback of the following recipe. Debian users may
> try:
> sudo apt install debian-keyring

Wouldn't that import all keys ?
If so, then if the short ids impose any problem, downloading all keys
would be even more of a problem.


> Nowadays SUMS files may be obtained using https: protocol from
> cdimage.debian.org even if image file is downloaded from a local mirror. It
> is secure enough.

It is not. Most obviously because if you do not trust the download of
the ISO image, then you cannot trust the SUMS files from the same
directory and via the same internet connection.
On the other hand, if you would trust download directory and connection,
then MD5 would be fully sufficient to detect non-malicious transport
damage. But Debian decomissioned MD5SUMS for a reason.


Have a nice day :)

Thomas

Proposal to augment CD/faq/#verify, version 2

[Thomas Schmitt]

   debian-12.7.0-amd64-netinst.iso: OK

* Checksums of already written optical media or USB keys match those found
  in the checksum files if you curb the data stream from the medium to
  the size of the ISO image.
  This is necessary because nearly all media would return more bytes
  after the end of the written ISO image. The checksums will only match
  if the number of bytes is exactly the same as in the image.

There are several ways to achieve this exactness:

* The "isosize" program can be used to find out the appropriate amount
  of bytes to be read from the medium. It shows the "sector count" and the
  "sector size" of the ISO filesystem on the medium. The latter is not
  necessarily the storage block size of the medium, but always 2048.
  Optical media are presented by GNU/Linux as /dev/srN, USB keys as
  /dev/sdX. If the ISO is on a USB key presented as /dev/sdc:
$ sudo /sbin/isosize -x /dev/sdc
sector count: 323072, sector size: 2048
  If the ISO is on a CD, DVD or BD presented as /dev/sr0:
$ /sbin/isosize -x /dev/sr0
sector count: 323072, sector size: 2048

* Then "sector count" and "sector size" are passed to "dd" to read the
  appropriate amount of bytes from the medium. The byte stream is then
  piped to the appropriate checksum tool "sha512sum" or "sha256sum"
  and memorized in a shell variable:
$ computed=$(sudo dd if=/dev/sdc count=323072 bs=2048 | sha512sum | awk 
'{print $1}')

* The computed checksum is to be compared against the corresponding
  checksum found in the appropriate checksum file SHA512SUMS or
  SHA256SUMS:
$ recorded=$(grep ' debian-12.7.0-amd64-netinst.iso$' SHA512SUMS | awk 
'{print $1}')
$ test "$computed" = "$recorded" && echo "OK. MATCH."
OK. MATCH.
  If the latter command puts out nothing instead of "OK. MATCH.", then the
  verification check has failed and the ISO filesystem on the medium was
  altered.
  There are harmless reasons for this failure but also dangerous ones.
  See the next section for an attempt to find altered files in the ISO.

This three-step procedure for media and also the two-step procedure for
image files may each be replaced by a run of the script
https://people.debian.org/~danchev/debian-iso/check_debian_iso .

* Download the script, verify it, and give yourself x-permission:

TODO: Move check_debian_iso to an official Debian location and sign
  it by a Debian CD signing key.

$ wget 
https://dev.lovelyhq.com/libburnia/libisoburn/raw/branch/master/xorriso-dd-target/check_debian_iso
...
    2024-09-03 09:24:57 (144 MB/s) - ‘check_debian_iso’ saved [5373/5373]

TODO: Describe download of signature file and verification.
  (Currently it is signed by check_debian_iso.{sig,asc} with the
   key of GNU xorriso by Thomas Schmitt. For key fingerprint see:
 https://www.gnu.org/software/xorriso/#download )

$ chmod u+x ./check_debian_iso

* ISO image file verification is done by:
$ ./check_debian_iso SHA512SUMS debian-12.7.0-amd64-netinst.iso

* Media verification is done by:
$ sudo ./check_debian_iso SHA512SUMS debian-12.7.0-amd64-netinst.iso 
/dev/sdc
  or
$ ./check_debian_iso SHA512SUMS debian-12.7.0-amd64-netinst.iso /dev/sr0
  The ISO image itself is not needed for these two runs with media.
  Only its name is needed for looking it up in the checksum file.

The last output line of the ./check_debian_iso runs must then be like
  Ok: '/dev/sdc' matches 'debian-12.7.0-amd64-netinst.iso' in 'SHA512SUMS'
A mismatch would yield
  MISMATCH: '/dev/sdc' checksum differs from 'debian-12.7.0-amd64-netinst.iso' 
in 'SHA512SUMS'

If you are curious about the script's options, run
  $ ./check_debian_iso -help
(The mentioned checksum files MD5SUMS and SHA1SUMS are not offered any
more, because these checksums are meanwhile deemed insufficient as
protection against malicious changes.)


New section. (Note that i know about the opportinity to run cdrom-checker.
But i deem it unwise to boot a possibly manipulated ISO just for
inspection.):


How to find the altered files in case of no match ?

If the verification attempt yields a non-matching checksum although
you are quite sure that you got an original Debian ISO, it is possible
to look for files in the ISO which got hit by the alteration. This works
only if the alteration does not hamper mountability of the ISO filesystem.

Mount the damaged ISO with a mountpoint of your choice
  $ path_to_image_or_usb_device=/dev/sdc
  $ mountpoint=/mnt/iso
  $ sudo mount "$path_to_image_or_usb_device" "$mountpoint"

Let program "md5sum" verify 

Re: Proposal to augment CD/faq/#verify

[Thomas Schmitt]

Hi,

i have posted my proposal as
  https://wiki.debian.org/VerifyISOImage
for testing by debian-user and it turned out that the examples need
some correction.

I will post a new version of the FAQ proposal and after two days of
no new problem reports.

Further shellcheck found a minor problem in script check_debian_iso
which i fixed in a new version.


Have a nice day :)

Thomas

Proposal to augment CD/faq/#verify

[Thomas Schmitt]

sosize -x /dev/sr0
sector count: 323072, sector size: 2048

* Then "sector count" and "sector size" are passed to "dd" to read the
  appropriate amount of bytes from the medium. The byte stream is then
  piped to the appropriate checksum tool "sha512sum" or "sha256sum"
  and memorized in a shell variable:
$ computed=$(dd if=/dev/sdc count=323072 bs=2048 | sha512sum | awk '{print 
$1}')

* The computed checksum is to be compared against the corresponding
  checksum found in the appropriate checksum file SHA512SUMS or
  SHA256SUMS:
$ recorded=$(grep ' debian-12.7.0-amd64-netinst.iso$' SHA512SUMS | awk 
'{print $1}')
$ test "$computed" = "$recorded" && echo "OK. MATCH."
OK. MATCH.
  If the latter command puts out nothing instead of "OK. MATCH.", then the
  verification check has failed and the ISO filesystem on the medium was
  altered.
  There are harmless reasons for this failure but also dangerous ones.
  See the next section for an attempt to find altered files in the ISO.

This three-step procedure for media and also the two-step procedure for
image files may each be replaced by a run of the script
https://people.debian.org/~danchev/debian-iso/check_debian_iso .

* Download the script and give yourself x-permission:
$ wget https://people.debian.org/~danchev/debian-iso/check_debian_iso
...
2024-09-03 09:24:57 (144 MB/s) - ‘check_debian_iso’ saved [5373/5373]
$ chmod u+x ./check_debian_iso

* ISO image file verification is done by:
$ ./check_debian_iso SHA512SUMS debian-12.7.0-amd64-netinst.iso

* Media verification is done by:
$ ./check_debian_iso SHA512SUMS debian-12.7.0-amd64-netinst.iso /dev/sdc
  or
$ ./check_debian_iso SHA512SUMS debian-12.7.0-amd64-netinst.iso /dev/sr0
  The ISO image itself is not needed for these two runs with media.
  Only its name is needed for looking it up in the checksum file.

The last output line of the ./check_debian_iso runs must then be like
  Ok: '/dev/sdc' matches 'debian-12.7.0-amd64-netinst.iso' in 'SHA512SUMS'
A mismatch would yield
  MISMATCH: '/dev/sdc' checksum differs from 'debian-12.7.0-amd64-netinst.iso' 
in 'SHA512SUMS'

If you are curious about the script's options, run
  $ ./check_debian_iso -help
(The mentioned checksum files MD5SUMS and SHA1SUMS are not offered any
more, because these checksums are meanwhile deemed insufficient as
protection against malicious changes.)

>>> TODO: Move check_debian_iso to an official Debian location and sign
  it by a Debian CD signing key. Then describe its verification.
  (Currently it is signed by a key of George Danchev and by the
   key of GNU xorriso by Thomas Schmitt.)


New section. (Note that i know about the opportinity to run cdrom-checker.
But i deem it unwise to boot a possibly manipulated ISO just for
inspection.):


How to find the altered files in case of no match

If the verification attempt yields a non-matching checksum although
you are quite sure that you got an original Debian ISO, it is possible
to look for files in the ISO which got hit by the alteration. This works
only if the alteration does not hamper mountability of the ISO filesystem.

Mount the damaged ISO with a mountpoint of your choice
  $ path_to_image_or_usb_device=/dev/sdc
  $ mountpoint=/mnt/iso
  $ sudo mount "$path_to_image_or_usb_device" "$mountpoint"

Let program "md5sum" verify the files listed in "$mountpoint"/md5sum.txt
  $ cd "$mountpoint"
  $ md5sum -c ./md5sum.txt | grep ': FAILED$'
  ./EFI/debian/grub.cfg: FAILED
  md5sum: WARNING: 1 computed checksum did NOT match

The altered file in this example is the image file which serves as
EFI System Partition when the ISO is presented to EFI on a USB key.
It often gets altered by proprietary software when you plug the USB key
into a running MS-Windows system. Normally this alteration is harmless and
just indicates that some small files were added to the FAT filesystem in
the system partition image. But such alteration of course weakens the
credibility of your download and copy efforts.

As noted in the previous section, MD5 is not a cryptographically secure
checksum any more. But it still is a good check against non-malicious
alterations.




Have a nice day :)

Thomas

Re: Request: more images in the weekly build -> arm64

[Thomas Schmitt]

Hi,

Roland Clobus wrote:
> The latest gnome image took 3 hours according to the log [2].
> [2] 
> https://get.debian.org/images/weekly-live-builds/amd64/iso-hybrid/debian-live-testing-amd64-gnome.log
>  (from 03:12:32 to 06:04:48)

Interesting log. There are two time zones used in two kinds of time stamp.
The more frequently given stamps say "[2024-06-17 04:12:33]" to
"[2024-06-17 07:04:19]".

The first timestamp after ISO production is
  [2024-06-17 04:58:02] lb binary_onie

It seems that most of the two hours after building the ISO are spent with
activities which have nothing to do with the ISO production.
More than half of the overall 3 hours get spent with "lb chroot_archives"
runs. (I assume that those don't happen in debian-cd.)

In particular there is a strange time span of more than 1 hour from
  [2024-06-17 04:58:32] lb chroot_archives source install
  P: Configuring file /etc/apt/sources.list
to
  [2024-06-17 06:03:35] lb source_live
  P: Begin copying live-build configuration...
with not much reported inbetween.
Less than a minute before this time the ISO was produced.

Further i see a potential slow network connection at the very end.
  [2024-06-17 06:32:48] lb chroot_archives source remove
  P: Deconfiguring file /etc/apt/sources.list
  Get:1 http://deb.debian.org/debian trixie InRelease [169 kB]
  ...
  Get:13 http://deb.debian.org/debian trixie/non-free-firmware Icons (64x64) 
[29 B]
  Fetched 40.9 MB in 31min 26s (21.7 kB/s)
  Reading package lists...
  [2024-06-17 07:04:19] lb chroot_hostname remove


Have a nice day :)

Thomas

Re: verify debian12 signature

[Thomas Schmitt]

Hi,

Juan Manuel wrote:
> I have downloaded the latest version of debian12. I
> have the GNU4win program with kleopatra  but I don't know how to check the
> digital signature.

Do i get it right that you have an MS-Windows system ?
(And that by "GNU4win" you mean "GPG4win" ?)

If so, you need a program to compute a SHA256 or SHA512 checksum from
the downloaded .iso file and a program to verify the corresponding
SHA*SUMS file by its SHA*SUMS.sign file.

I'm not a user of MS-Windows, so i cannot recommend any software for those
tasks. If no other way would be to see, i'd consider to install WSL
and to do the verification by its help.
  https://wiki.debian.org/InstallingDebianOn/Microsoft/Windows/SubsystemForLinux

In a Debian provided shell i would then do:
  gpg --verify SHA256SUMS.sign SHA256SUMS
which should yield one of the key fingerprints as listed on
  https://www.debian.org/CD/verify
Important will be these result statements:
  "Good signature from" ... "Debian" ...
  "Primary key fingerprint:" ... one of the listed fingerprints ...

Then i'd compute the SHA256 of the .iso file
  sha256sum debian-12.5.0-amd64-netinst.iso
and compare it with the checksum string which is listed for the .iso file
in SHA256SUMS.


Alternative ideas:

Maybe you can perform the .sign check similar to what
  
https://docs.oracle.com/cd/E17952_01/mysql-5.7-en/checking-gpg-signature-windows.html
proposes for "mysql-installer-community-5.7.44.msi" as payload file
(yours would be SHA256SUMS) and "mysql-installer-community-5.7.44.msi.asc"
as detached signature file (yours: SHA256SUMS.sign).

This page
  https://3d-imaging.co.uk/blog/verifying-sig-files-with-gpgp4win/
states that GPG4win would offer the command line tool to run
  gpg --verify gpg4win*.exe.sig gpg4win*.exe
(You'd just use file names SHA256SUMS.sign SHA256SUMS instead.)

As for computing the SHA256 sum of the .iso, i find on
  
https://www.pctipp.ch/praxis/windows-10/windows-10-sha256-hash-bordmitteln-pruefen-2507915.html
a proposal for PowerShell, which in your case would look like:
  Get-Filehash debian-12.5.0-amd64-netinst.iso -Algorithm SHA256


Have a nice day :)

Thomas

Re: Where to download cd iso?

[Thomas Schmitt]

Hi,

Tatsu Takamaro wrote:
> I've received my own message to you. But I don't see your answer...

Well, i have't seen that message and the mailing list archive doesn't
show it either:
  https://lists.debian.org/debian-cd/2024/03/threads.html
(The one i reply to is your follow-up:
  https://lists.debian.org/debian-cd/2024/03/msg00022.html
)


> > I don't see the cd iso on the iso-cd page.
> > I mean a usual one, not a netinst.

I guess they have been discontinued because of the insane number of ISO
images with 650 MB which would emerge. (I estimate ~ 130 pieces.)

The smallest granularity seems to be the ~4.5 GB sized DVD images which
are available via Jigdo download:
  https://cdimage.debian.org/debian-cd/current/amd64/jigdo-dvd/
21 pieces.


Have a nice day :)

Thomas

Re: Problem with jigdo files

[Thomas Schmitt]

Hi,

Thomas Lange wrote:
> Maybe you can check if there's a problem with DVD17.

What's the exact URL of .jigdo and .template file which was tried ?

For now i tested the offer for amd64 debian-12.4.0 (demonstrating the wget
workaround for a long fixed bug in older jigdo-lite about https):

  $ wget 
https://cdimage.debian.org/debian-cd/current/amd64/jigdo-dvd/debian-12.4.0-amd64-DVD-17.jigdo
  ...
  $ wget 
https://cdimage.debian.org/debian-cd/current/amd64/jigdo-dvd/debian-12.4.0-amd64-DVD-17.template
  ...
  $ jigdo-lite
  ...
  jigdo [...]: debian-12.4.0-amd64-DVD-17.jigdo
  ...
  Files to scan:
  ...
  Debian mirror [http://ftp.de.debian.org/debian/]:
  ...
  Finished!
  The fact that you got this far is a strong indication that 
`debian-12.4.0-amd64-DVD-17.iso'
  was generated correctly. I will perform an additional, final check,
  which you can interrupt safely with Ctrl-C if you do not want to wait.

  OK: Checksums match, image is good!
  $

Download via the cheapest Telekom DSL lasted 21 minutes.

At https://cdimage.debian.org/debian-cd/current/amd64/jigdo-dvd/SHA256SUMS
i read the line:

  025abfe804857c88ee52545540e6bfa241306ade21a73d69cbd92173a0355820  
debian-12.4.0-amd64-DVD-17.iso

The downloaded ISO matches:

  $ sha256sum debian-12.4.0-amd64-DVD-17.iso
  025abfe804857c88ee52545540e6bfa241306ade21a73d69cbd92173a0355820

So it seems possible to get DVD 17 via jigdo and a german Debian mirror.

Ask Hannes to try my above example or try it yourself.
(If nothing helps i would be willing to create the ISOs and to upload
them at some given URL. For sending two DVDs by mail i'd demand an Aldi
Panettone as reward ... or a 5-pack BD-R media ...)


Have a nice day :)

Thomas

Bug#1056998: cdrom: Installation media changes after booting it

[Thomas Schmitt]

Hi,

i wrote:
> >Does any of the bystanders know from what package this message text stems ?

Steve McIntyre wrote:
> That's cdrom-checker:

So that would be indirectly put into the code at

  https://sources.debian.org/src/cdrom-checker/1.62/main.c/?hl=152#L152

by "Template: cdrom-checker/mismatch" in:

  
https://sources.debian.org/src/cdrom-checker/1.62/debian/cdrom-checker.templates/?hl=43#L43

Using my micking instincts i guess my proposal would be to create a
new template like:

   Template: cdrom-checker/mismatch-efi
   Type: error
   # :sl3:
   _Description: Integrity test failed on EFI boot image
   The EFI boot image file ${FILE} failed the MD5 checksum verification.
   Possibly some software or firmware made changes in the EFI System
   Partition. This may be considered harmless unless booting of this
   Debian ISO encounters difficulties or other files show checksum
   mismatches, too.

and to add a case in main.c, line 151, like (shown with reduced
indentation):

   if(status != 0) {
 if(strcmp(filename, "./boot/grub/efi.img") == 0) {
   debconf_subst(debconf, "cdrom-checker/mismatch-efi", "FILE", filename);
   debconf_input(debconf, "high", "cdrom-checker/mismatch-efi");
 } else {
   debconf_subst(debconf, "cdrom-checker/mismatch", "FILE", filename);
   debconf_input(debconf, "high", "cdrom-checker/mismatch");
 }
   } else {
 debconf_input(debconf, "critical", "cdrom-checker/passed");
   }

A native speaker can probably find a better text and my guessing of the
inner code relations may be wrong, of course. Even the path
"./boot/grub/efi.img" is guessed from Ram Reddy's reported error message.


Have a nice day :)

Thomas

Bug#1056998: cdrom: Installation media changes after booting it

[Thomas Schmitt]

Hi,

trying to get our mails in sync again. We seem to have reached consensus
in our understanding of the problem.

i wrote:
> > might possibly be "WPSettings.dat", which causes questions in the
> > internet.

Ram Reddy wrote:
> I think that this could possibly be explained by flaws in my testing. Some
> of the computers were using Windows, and for those computers I plugged in
> the USB drive, shut down the computer, and booted the USB drive from the
> BIOS.

This explains the MS-Windows file.

> the ones where I shut down windows first had no modifications.

So the Thinkpad firmwares are exonerated.

> it seems to be that the only modifications of the installer caused by
> firmware is with the Legion 7i gen 5's self healing BIOS backup.

This suspect can be considered to have confessed.

--

As for a better message from Integrity Check:

I find in the ISO in file /install.amd/initrd.gz the lines:
  "The file ${FILE} failed the MD5 checksum verification. Your installation"
  "media or this file may have been corrupted."

But i failed to find these texts in debian-cd or debian-installer
Even searching for just "corrupted" yields only a remark in debian-cd.

Does any of the bystanders know from what package this message text stems ?


Have a nice day :)

Thomas

Bug#1056998: cdrom: Installation media changes after booting it

[Thomas Schmitt]

Hi,

Ram Reddy wrote:
> Thomas Schmitt wrote:
> > FWIW check the BIOS L[123] cache settings [...]

This was posted by Nicholas Geovanis , not by me.

My suspicion is that the Lenovo Legion EFI BIOS intentionally creates
directories and empty file "/efi/Lenovo/BIOS/SelfHealing.fd".
I do not see much benefit in investigating this further.

More riddling is the directory and file found after using the USB stick
on the Lenovo Thinkpad machines. The internet says that
"/System Volume Information/WPSettings.dat" is part of Microsoft services.
So it might be that the USB stick got in contact with MS-Windows while
or before it was plugged into the Thinkpads.
If this suspicion can be excluded then i'd say that Thinkpad has some
strange BIOS habits.

What remains for Debian in this bug report is the observation that the
integrity check of the Debian installation ISO says "Your installation
media or this file may have been corrupted" when just a harmless change
has been made in the filesystem of the EFI partition.
Some special handling of /boot/grub/efi.img during the integrity check
would cause less surprise to the user.
Like:

  "Possibly some software or firmware made changes in the EFI System
   Partition. This may be considered harmless unless booting of this
   Debian ISO encounters difficulties or other files show checksum
   mismatches, too."


Have a nice day :)

Thomas

Bug#1056998: cdrom: Installation media changes after booting it

[Thomas Schmitt]

Hi,

Ram Reddy wrote:
> https://drive.google.com/file/d/1Zd6iufVRsfIu-qzC-tJx4FEvCOESOz4_/view?usp=sharing

I downloaded the tarball and compared the original FAT filesystem with the
various modified filesystem images.

--

In Legion7iG5-*_modified.esp the suspect lost its ID card at the crime
scene:
At byte 39072 (0x98a0) the changes go from 0-bytes to the text "LENOVO".
At byte 9711680 (0x943040) i see a change from 0-bytes to "BIOS".

Diffing the result of "find" on the mounted unmodified.esp filesystem and
Legion7iG5-*_modified.esp shows that a new branch of directoriies with a
new file is in each of the modified filesystems:
  > ./efi/Lenovo
  > ./efi/Lenovo/BIOS
  > ./efi/Lenovo/BIOS/SelfHealing.fd
The file is empty.

--

In ThinkpadX1CarbonG5-0_modified.esp there is no company name to see in
the changed bytes. I see UTF-16 strings "mation", "System", and
"Volum\000me". ASCII texts "SYSTEM~1", "WPSETT~1DAT". The latter might
possibly be "WPSettings.dat", which causes questions in the internet.
Most plausible seems an answer in the course of
  
https://answers.microsoft.com/en-us/insider/forum/all/whats-wpsettingsdat-generated-by/e11bca97-8c76-4662-8897-774ea3d5691a
  "The WPSettings.dat file is generated by the Storage Service (StorSvc).
   It seems that WPSettings.dat means the data files of Windows Phone's
   Store Settings saved on the drives, [...]"

Diffing the result of "find" on the mounted unmodified.esp filesystem and
ThinkpadX1CarbonG5-0_modified.esp shows that a new directory with a new
file is in the modified filesystem:
  ./System Volume Information
  ./System Volume Information/WPSettings.dat
The file has 12 bytes of binary salad:
  Hex:   0c  00  00  00  2e  42  6b  82  5d  88  0e  c5
  Char:   .   B   k   ]
  Dec:   12   0   0   0  46  66 107 130  93 136  14 197

--

While it makes some sense to me that Lenovo Legion BIOS adds some Lenovo
stuff to the EFI System Partition, i really wonder why Lenovo Thinkpad
BIOS adds a Microsoft directory and file.

Whatever, i'd say that the software in the ISO and especially Debian
Installer are not suspicious to create directories with such names.


Have a nice day :)

Thomas

Bug#1056998: cdrom: Installation media changes after booting it

[Thomas Schmitt]

Hi,

Ram Reddy wrote:
> I tested the installer on one of my laptops, and found that its contents
> didn't change.

Good to know that the software in the ISO still works from read-only DVDs.

> On the Lenovo Legion 7i Gen 5*, Lenovo Thinkpad X1 Carbon Gen 2*, Lenovo
> Thinkpad X1 Carbon Gen 5 Intel*, and one Lenovo Thinkpad E14 Gen 5 all
> showed the corruption error. 
> However, on the Lenovo Yoga C740 and a different Lenovo Thinkpad E14 Gen 5
> Intel showed no corruption issues. 

This seems to indicate that the firmware has a stake in the problem ...

> Both the Thinkpad E14 Gen 5s had the same specifications and type number,
> differing only in that the one with corruption of the installer has 24GB of
> memory (16GB installed in the slot, 8GB soldered) and the other only has 8GB
> soldered. They both have the same BIOS version, R2AET32W(1.07).

... but the trigger would have to be very subtle.

> This seems to be really interesting because the corruption only happened on
> certain computers, and it would stay that way on repeated attempts.

I ponder how to distinguish a change made by the firmware itself from a
change made by the software in the ISO image.

> Again, the
> ones that had corruption all had it at the same location. (byte 2303211,
> line 21165) 

Maybe we can learn more by comparing the files /boot/grub/efi.img of the
original ISO and of an altered USB stick.

Mount both and copy out the files, or use dd to extract them from the
partition interval that is marked in the ISO (ask by: fdisk -l).
With debian-12.2.0-amd64-netinst.iso that would be

  iso=debian-12.2.0-amd64-netinst.iso

or (with the approriate letter instead of "X")

  iso=/dev/sdX

and then

  dd bs=512 if="$iso" of="$iso".esp skip=4476 count=18976

Both images will have about 9.5 MiB of size. I would be interested in
seeing the altered one, as i can extract the original on my own.
So if you have a way to upload the altered EFI partition image to a
publicly accessible place i would be happy to investigate it.

Much smaller and thus more bugreport-friendly would be a binary diff.
You would need something more detailed than plain cmp. cmp -l gives the
differing byte values one by line (i.e. lengthy if many bytes differ).
I made an own willful alteration (subtracting 4476 * 512 from 2303211 yields
the new problem byte address 11499):

  cp debian-12.2.0-amd64-netinst.iso.esp 
debian-12.2.0-amd64-netinst.iso.esp.altered
  echo ABCDE | dd conv=notrunc of=debian-12.2.0-amd64-netinst.iso.esp.altered 
bs=1 seek=11499 count=5

Then

  cmp -l debian-12.2.0-amd64-netinst.iso.esp 
debian-12.2.0-amd64-netinst.iso.esp.altered

yields

  11500   0 101
  11501   0 102
  11502   0 103
  11503   0 104
  11504   0 105

(Odd enough: Addresses are decimal starting at 1, not at 0. Values are
octal.)

Less useful is cmp -b:

  debian-12.2.0-amd64-netinst.iso.esp 
debian-12.2.0-amd64-netinst.iso.esp.altered differ: byte 11500, line 278 is   0 
^@ 101 A

(Only the "A" is from my change. There seem to be pairs of octal value and
character meaning.)

I have my own unpublished differ for analyzing ISO entrails. So only the
big battleship "diffoscope" comes to my mind as public binary differ.
Probably there are leaner programs for that job.


Have a nice day :)

Thomas

Bug#1056998: cdrom: Installation media changes after booting it

[Thomas Schmitt]

Hi,

Ram Reddy wrote:
> [...] usb drive [...] debian-12.2.0-amd64-netinst.iso [...]
> [...] Integrity Test section [...]
> ./boot/grub/efi.img failed the MD5 checksum verification. Your installation
> media or this file may have been corrupted.
> [...]
> It was always byte 2303211, line 21165 which changed.

I'd say that corruption of ISO image or USB is quite unlikely.

It rather looks like some change was made to the FAT filesystem in the
EFI partition by the parts of the installer which ran in your experiments.
(I expect EFI firmware not to manipulate that filesystem ...)
The EFI partition of this particular ISO begins at byte 2291712. The
alteration thus happened 11499 bytes inside that partition and its
filesystem. In the original ISO i see a sequence of 0-bytes there.

It would be interesting to check whether any errors get reported if the
ISO is presented on a read-only CD or DVD instead of a writable USB stick.
(To be absolutely read-only it would have to be CD-R, unformatted CD-RW,
DVD+R, DVD-R, or unformatted DVD-RW. The others can be overwritten without
burn program.)

Explanation why the file in the read-only ISO filesystem is reported as
being altered:
The partition shares its block range with the ISO file /boot/grub/efi.img
which is mentioned in the ISO file /md5sum.txt . So the integrity check
detects the difference between recorded MD5 and the MD5 of the altered
FAT filesystem image file /boot/grub/efi.img.


Have a nice day :)

Thomas

Bug#1051445: syslinux.efi crashes on isohybrid boot as cdrom

[Thomas Schmitt]

Hi,

congrats for solving this long standing problem.

A few years ago this would have been great news for Knoppix, which staid
for quite a while with SYSLINUX for EFI. But somewhere between versions
8.6 (2019) and 9.3 (2022) it switched to GRUB.

Although the most recent commit on https://repo.or.cz/syslinux.git is
4.5 years old, consider to post your patches to
  sysli...@syslinux.org
Maybe it helps some puzzled SYSLINUX user in the future.


Have a nice day :)

Thomas

Re: Free Software DVD contains non-free firmware

[Thomas Schmitt]

Hi,

Philip Hands wrote:
> While looking at the FSF site, I noticed this somewhat amusing method
> for reconciling these two stances:
>  https://www.gnu.org/philosophy/install-fest-devil.html

Typically Stallman:

  "My new idea is that the install fest could allow the devil to hang
   around, off in a corner of the hall, or the next room. (Actually, a
   human being wearing sign saying 'The Devil,' and maybe a toy mask or
   horns.) The devil would offer to install nonfree drivers [...]"


> but I'm afraid I've no idea how one could implement something equivalent
> in the medium of downloadable images.

Andrew M.A. Cater wrote:
> > You can pass various parameters to the installer: you can also uninstall
> > the non-free firmware after installation -

So one could explain and advertise to organizers of install fests how
to instruct the installer to ignore non-free firmware.
Or an imposter of archangel Michael could offer to eradicate any impurity
as last step of the installation (possibly after making a first backup).


Philip Hands wrote:
> I'm sure if we had a tool for converting "+firmware" to "pure" images,

As GNU maintainer of xorriso i am probably obliged to help with the
purification of Debian ISOs, if there is real interest.
The main open problem which i see in this regard is the list of develish
packages which shall be removed.

Would /firmware/Contents-firmware with the third word in each line tell
what is non-free ? In debian-12.0.0-amd64-netinst.iso i see lines like:
  /lib/firmware/ath9k_htc/htc_9271-1.4.0.fw   
firmware-ath9k-htc_1.4.0-108-gd856466+dfsg1-1.3_all.deb main
  /lib/firmware/amd/amd_sev_fam17h_model0xh.sbin  
amd64-microcode_3.20230414.1_amd64.deb non-free-firmware

Is all non-free firmware under /pool confined in /pool/non-free-firmware ?
(Is all firmware under /firmare mirrored under /pool ?)

If so, then one could remove the develish files from the various content
lists and re-pack the ISO with updated content lists and removed files.
This is probably not more complicated than merging debian-cd ISOs by
  
https://dev.lovelyhq.com/libburnia/libisoburn/raw/master/test/merge_debian_isos
as described in
  https://wiki.debian.org/MergeDebianIsos

(It seems that /firmware/Contents-firmware should be merged, too.
Now i wonder why this did not hamper the use of the resulting amd64 ISOs.
Is it not used by any program ?)


> I would expect the FSF to be
> almost exactly as grumpy even if "pure" images were easily available.

A do-it-yourself purifier program would not bring Debian on the FSF list
of free distros. But it would help with posing as install fest devil.

People who want to organize such a fest are entitled to ask me for
support.


Have a nice day :)

Thomas

Re: UHD-BD

[Thomas Schmitt]

Hi,

griffin tucker wrote:
> not just useful for debian or linux distros, but any consecutively
> named  .iso9660 files

I assume that the script merge_debian_isos is mainly usable with a set of
ISOs made by the debian-cd package. The files of which the content gets
manipulated are probably specific to debian-installer.

But of course its mount operations and its xorriso run demonstrate how to
merge multiple ISO 9660 filesystems without refering to Debian specific
files.
In the general case i would expect the need for some kind of file
collision handling. I.e. what to do with differing data files which have
the same path in different ISOs.

(Actually the names of the input ISOs can be quite arbitrary. The script
expects a list of paths. I'm not sure whether anybody tested input names
which are not single words.)


Have a nice day :)

Thomas

Re: UHD-BD

[Thomas Schmitt]

Hi,

griffin tucker wrote:
> are there any plans to introduce triple-layer blu-rays for debian?

I guess not:
  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1011343

But there is a workaround:
  https://wiki.debian.org/MergeDebianIsos

Possibly it is not yet tested with Debian 12. So reports about success or
failure would be welcome.


Have a nice day :)

Thomas

Bug#1038440: debian-cd: debian-12.0.0-amd64-netinst.iso is too big for a CD

[Thomas Schmitt]

Hi,

when comparing 12.0.0 with 11.5.0 i see three file trees which have grown
substantially:

du of mounted debian-11.5.0-amd64-netinst.iso :
  5   /firmware
  126611  /install.amd
  308479  /pool

du of mounted debian-12.0.0-amd64-netinst.iso :
  221108  /firmware
  156509  /install.amd
  589416  /pool

In pool of 12.0.0 i see another firmware tree
  220570  /pool/non-free-firmware
but this does not cost much media capacity, because the file content is
shared with the files in the /firmware tree:

  $ xorriso -indev debian-12.0.0-amd64-netinst.iso \
-find / -sort_lba -exec report_lba --
  ...
  Report layout: xt , Startlba ,   Blocks , Filesize , ISO image path
  ...
  File data lba:  0 ,18420 , 8217 , 16827872 , 
'/firmware/firmware-atheros_20230210-5_all.deb'
  File data lba:  0 ,18420 , 8217 , 16827872 , 
'/pool/non-free-firmware/f/firmware-nonfree/firmware-atheros_20230210-5_all.deb'
  ...

Such deduplication of hard links also happens with initrd.gz and vmlinuz
files in /install.amd. The real storage sizes as measured in above -find
output are:
   11.5.0   12.0.0  Growth
  /firmware 0 MiB  216 MiB 216 MiB
  /install.amd 67 MiB  138 MiB  71 MiB
  /pool   301 MiB  360 MiB  59 MiB

To fit on a regular "700 MB" (~ 702 MiB) CD-R/CD-RW these would have to
shrink by at least 36 MiB.


Have a nice day :)

Thomas

Re: missing files from different formats of the .iso files

[Thomas Schmitt]

Hi,

there is already a bug report about a similar problem

  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=988120

It points to file
  jigdo-file-cache.db
and mentions a positive one-time effect, which would match the observation
that the first medium nearly always succeeds.
It works if jigdo-file-cache.db is deleted before the jigdo-lite run.

If you do so at the beginning of your script, then this would nicely
explain why further ISOs fail and what is their connection at all.
(I began to wonder how first and further ISOs are different to jigdo-lite.)


The bug report mentions SHA256 as checksums in the .jigdo file, which
i only see for the overall image but not for the single data files.
But maybe it is about SHA256 in the jigdo-file-cache.db.

This comment proposes a patch:
  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=988120#20
and gives a code analysis.


Have a nice day :)

Thomas

Re: missing files from different formats of the .iso files

[Thomas Schmitt]

Hi,

(Possibly your attachments did not make it to the mailing list.
I got only the mail which was sent directly to me.)

griffin tucker wrote:
> img-debian-12.0.0-source-bd_to_dvd_1_of_2-77429.asc
> ...
> Found 2840 of the 2840 files required by the template
> Successfully created `debian-12.0.0-source-DVD-1.iso'

The first DVD was ok. But then:

> ...
> Found 3646 of the 3647 files required by the template
> Copied input files to temporary file `debian-12.0.0-source-DVD-2.iso.tmp'
> ...
> Found 3907 of the 3908 files required by the template
> Copied input files to temporary file `debian-12.0.0-source-DVD-3.iso.tmp'
> ...
> Found 3482 of the 3483 files required by the template
> Found 1268 of the 1269 files required by the template
> Found 2224 of the 2225 files required by the template
> Found 13101 of the 13102 files required by the template
> Found 8045 of the 8046 files required by the template
> Found 14341 of the 14342 files required by the template

This looks highly systematic and not like caused by some glitchy files in
the source pool.
I see the same failures in all second and further ISOs of a jigdo-lite
run. It seems not to be coupled to DVD or BD numbers higher than 1 but
only to the sequence of ISOs in a single jigdo-lite run.

Nevertheless, in the possibly incomplete log file
img-debian-12.0.0-source-dvd_to_dlbd-42245.asc i see already a missing
file with the debian-12.0.0-source-DLBD-1.iso file, which i would expect
to have been first. Was there another download before this in the same
jigdo-lite run ?

Consider to file a bug report towards package "jigdo-file".


---

> >  X-Coil-Desert: 4e292ddc276fab1b_1687051700717_2547565298
> >  X-Obese-Wide-Eyed: 37ca251540d1f045_1687078216293_605541665
> >  X-Ruddy-Name: 05f4b58c06a9aab5_1687078216565_3930339659

> nfi, tbh.
> i'm not a sysadmin and i wouldn't be surprised if it's some sort of
> reference to an inside joke to those that monitor my communications.

Obviously they have room for such shenanigans but not for
  X-Clacks-Overhead: GNU Terry Pratchett


Have a nice day :)

Thomas

Re: missing files from different formats of the .iso files

[Thomas Schmitt]

Hi,

griffin tucker wrote:
> jigdo-lite runs in batch mode, and usually the reconstruction of the
> first .iso is completely reconstructed from a directory of mounted
> .iso's from another format, however, the next few .iso's will
> consistently have 1 file missing that it couldn't find from the
> mounted .iso's, when it should, and i am unable to determine the
> pattern or reason for this

Could be a problem of jigdo-lite with this special file presentation.

Does the missing file have a particular position in the .jigdo file
which you use for composing the new ISO ?
  gunzip < *.jigdo | less

Does it exist in the mounted ISOs and does its checksum match ?
  echo '...char.salad.from.jigdo...'==  | sed -e 's/-/+/g' -e 's/_/\//g' |
base64 -d | od -t x1 | sed -e 's/^...//' -e 's/ //g'


I wrote:
> >   In this case the checksum is MD5. Expect to see SHA256 in newer .jigdo.
> >

> i look forward to the sha256 or sha512 implementation

In this case, the MD5 is not used as verifier but as access key.
When it gets found in the .template file it is looked up in the .jigdo
to get the download URL of the file.
Verification should be done by the overall checksum of the ISO as
published in the SHA256SUMS or SHA512SUMS files which accompany the .jigdo
and .template files.

So i don't see much reason to switch internally to SHA256.
But MD5 has a bad reputation in Debian and so the Jigdo format was
enhanced to use SHA256 instead. Interestingly, even
debian-12.0.0-amd64-netinst.jigdo still uses the shorter MD5s.
(xorriso-1.5.4 in Debian 12 is capable of producing SHA256 in the .jigdo
files. The development version xorriso-1.5.3 can do it since end of 2019.)


> i'm running it in a virtual machine, if that makes a difference
> (afaik, it shouldn't)

That would be indeed astonishing. I rather bet on the mounted ISOs as
trigger. (But i have no stake in jigdo reconstruction, only in its
production as developer of xorriso.)


-
Off topic:

I wonder what those unusual headers with changing names in your mails
mean:
  X-Coil-Desert: 4e292ddc276fab1b_1687051700717_2547565298
  X-Obese-Wide-Eyed: 37ca251540d1f045_1687078216293_605541665
  X-Ruddy-Name: 05f4b58c06a9aab5_1687078216565_3930339659
always between
  X-MailChannels-Auth-Id: ...
and
  X-MC-Loop-Signature: ...


Have a nice day :)

Thomas

Re: missing files from different formats of the .iso files

[Thomas Schmitt]

Hi,

griffin tucker wrote:
> there's a bug in jigdo that makes it occasionally skip files

In that case the missing files woule be present but filled with zeros
when you mount the incompletely composed ISO.
Can you confirm ?


Not picking a file of a jigdo image may have various reasons.
Temporary download problems should become visible by error messages.
Consider to catch the output of jigdo-lite in a file for inspection.

Persistent problems could be (possibly among others):

- File is missing on the chosen Debian mirror.

  In this case it is supposed to be fetched from fallback servers which
  are mentioned at the end of the .jigdo file:

$ gunzip http://us.cdimage.debian.org/cdimage/snapshot/Debian/
Debian=http://snapshot.debian.org/archive/debian/20221218T124218Z/ 
--try-last

  Assuming that  bin-prot_0.14.0-1.debian.tar.xz  would be missing, you
  could check by a web browser whether
 
http://us.cdimage.debian.org/cdimage/snapshot/Debian/pool/main/b/bin-prot/bin-prot_0.14.0-1.debian.tar.xz
  exists.


- The checksum of the file in .jidgo does not match the checksum of the
  file on the mirror server.

  The checksum is listed in .jigdo before the '=', encoded in a variant of
  base64.
  For decoding append enough "=" to get a number of characters which is
  divisible by 4, replace "-" and "_" by their base64 equivalents, decode
  base64, format to hex, and remove od's decorations:

 $ echo 'T__clXB4rmLIbFvoblKNqA'== | sed -e 's/-/+/g' -e 's/_/\//g' |  
base64 -d | od -t x1 | sed -e 's/^...//' -e 's/ //g'
 4fffdc957078ae62c86c5be86e528da8

  In this case the checksum is MD5. Expect to see SHA256 in newer .jigdo.


> from the source
> and then it will request the online source. i'm not sure why.

What is the difference between "the source" and "the online source" ?
I'd normally expect that all jigdo sources of files are online.
Do you have your own local Debian mirror or a set of older ISOs ?


Have a nice day :)

Thomas

Re: Debian 9.7 , Downloading issue

[Thomas Schmitt]

Hi,

(Cc-ing the mailing list debian-cd@lists.debian.org again.)

Muhammad Owais wrote:
> However there is still an issue if I am a windows user how can I use Jigdo.

Steve McIntyre, a Debian Developer who is in charge for debian-cd, offers
binaries for MS-Windows:
  https://www.einval.com/~steve/software/jigdo/download/jigdo-win-0.8.1.zip
If you have PGP, you may verify it by the associated key:
  https://www.einval.com/~steve/software/jigdo/download/jigdo-0.8.1.tar.xz.sig
See also the "Downloads" paragraph on
  https://www.einval.com/~steve/software/jigdo/


Alternatively you may first download a Debian Live ISO via https, put it
on DVD or USB stick or run it in a virtual machine, and then use it to
download the desired debian-cd ISOs.
See
  https://wiki.debian.org/JigdoOnLive

(Lacking experience with non-GNU/Linux systems i cannot well describe how
to get access to a writable filesystem on the hard disk of your normal
operating system. If this imposes difficulties, you will have to ask at
mailing list debian-l...@lists.debian.org or at places where a mix of
other operating systems and GNU/Linux is on topic.)


Have a nice day :)

Thomas

Re: Debian 9.7 , Downloading issue

[Thomas Schmitt]

Hi,

i'm Cc-ing the mailing list debian-cd@lists.debian.org  again.
Please don't expect that i reply to private mails or keep them private,
unless your mail contains a good reason why it shall stay private.

Muhammad Owais wrote directly to my mail address:
> As I found the DVD over here I need the 9.7 version link that I couldn't
> find.
> https://cdimage.debian.org/cdimage/archive/9.13.0/amd64/bt-dvd/

It seems there is no BitTorrent offer for 9.7.0.
You will have to use Jigdo and the files in
  https://cdimage.debian.org/cdimage/archive/9.7.0/amd64/jigdo-dvd/


Fabio Avanzi wrote directly to my mail address, too:
> https://help.ubuntu.com/community/JigdoDownloadHowto

This, or
- the "How to download an image with jigdo" in
https://www.debian.org/CD/jigdo-cd/
  which points to
https://www.einval.com/~steve/software/jigdo/
- the Jigdo part of the wider description how to download Jigdo ISOs if
  you have no system which can provide the "jigdo-lite" program:
https://wiki.debian.org/JigdoOnLive#Install_package_jigdo-file
  up to "Verify the downloaded ISOs".

Of course you will have to enter the appropriate URLs for 9.7.0 amd64.
In any case:
  
https://cdimage.debian.org/cdimage/archive/9.7.0/amd64/jigdo-dvd/debian-9.7.0-amd64-DVD-1.jigdo
The checksum files are at
  https://cdimage.debian.org/cdimage/archive/9.7.0/amd64/jigdo-dvd/SHA512SUMS
  
https://cdimage.debian.org/cdimage/archive/9.7.0/amd64/jigdo-dvd/SHA512SUMS.sign

Depending on the age of your jigdo-lite program, you will have to download
the .jigdo file manually and also the .template file:
  
https://cdimage.debian.org/cdimage/archive/9.7.0/amd64/jigdo-dvd/debian-9.7.0-amd64-DVD-1.template
The reason is given n
  
https://wiki.debian.org/JigdoOnLive#If_needed.2C_work_around_a_shortcoming_of_older_jigdo-lite

The same applies to downloading DVD-2 and DVD-3.


Have a nice day :)

Thomas

Re: Debian 9.7 , Downloading issue

[Thomas Schmitt]

Hi,

Muhammad Owais wrote:
> I couldn't find the
> page from where I can download the Debian 9.7 Complete 3 DVD set.

Choose your CPU type from
  https://cdimage.debian.org/mirror/cdimage/archive/9.7.0/
E.g.
  https://cdimage.debian.org/mirror/cdimage/archive/9.7.0/amd64/jigdo-dvd/


Have a nice day :)

Thomas

Re: How Do You Create ISOs?

[Thomas Schmitt]

Hi,

Ricardo Romanach wrote:
> I am currently attempting to create an ISO file from an existing
> Debian 11 installation.

Google found me
  https://github.com/pieroproietti/penguins-eggs
  "penguins-eggs is a console utility, under continuous development,
   that allows you to remaster your system and redistribute it as iso
   images."

I could not find out by which program the "ISO" is packed up and whether
it is indeed an ISO 9660 filesystem image or some other image which only
gets called "ISO".


> I have not found a tool that
> allows me to properly create an ISO specifically with the ext4 filesystem.

You use the term obviously for any kind of filesystem image.
Debian ISOs are mainly ISO 9660 filesystem images, i.e. read-only, capable
of booting from optical media, readable by about any operating system
(with possibly somewhat dull file names).


> what tools does your team use to create your ISO distributions?

This mailing list is about package "debian-cd"
  https://tracker.debian.org/pkg/debian-cd
  https://packages.debian.org/unstable/debian-cd
  "Tools for building (Official) Debian CD set"

But this package does not create an ISO image so that it is much like the
system on which it produces the image. It rather composes the image
content from Debian package files (.udeb and .deb) and some base software
to give it enough brain to start up, to interact with the user, and to
install the Debian packages to a hard disk (which may be a USB stick).
Very much unlike an installed Debian system.


> I don’t know if my understanding of ISOs is correct, but know this must be
> possible.

Possible: Yes.
Already done and offered: Maybe.
Offered or used by Debian: Not that i am aware.


> tar ... partclone.ext4 ... dd ... e2image ...

Even if you pack up the whole hard disk, it is not guaranteed that copying
the result to another computer's hard disk will work. The packed up system
files might be too much addicted to the hardware of the original computer.

Bootable ISO 9660 images are in most cases prepared to be not specialized
to a particular configuration of hardware. One may expect them to work
with a wide range of machines, if only the processor type matches.


> I don’t know how to use Linux Live Kit since there are very few guides
> on how to use it properly or how to handle errors when they appear.

Nevertheless it promises to do what you desire.
So it seems worth to invest time in learning how to use it.


Have a nice day :)

Thomas

Bug#1031696: Use of symbolic links in non-free ISO images breaks file system transposition support

[Thomas Schmitt]

Hi,

Pete Batard wrote:
> Debian does not use an efi.img.

Oh it does with ISOs for i386 and amd64. There is a data file in the ISO
filesystem named
  /boot/grub/efi.img
advertised as MBR partition of type 0xEF and as El Torito boot image
for EFI:

  $ xorriso -indev debian-11.5.0-amd64-netinst.iso -report_system_area plain 
-report_el_torito plain
  ...
  MBR partition table:   N Status  TypeStart   Blocks
  MBR partition  :   1   0x80  0x000   782336
  MBR partition  :   2   0x00  0xef 4064 5184
  MBR partition path :   2  /boot/grub/efi.img
  ...
  El Torito images   :   N  Pltf  B   Emul  Ld_seg  Hdpt  Ldsiz LBA
  El Torito boot img :   1  BIOS  y   none  0x  0x00  42312
  El Torito boot img :   2  UEFI  y   none  0x  0x00   51841016
  El Torito img path :   1  /isolinux/isolinux.bin
  El Torito img opts :   1  boot-info-table isohybrid-suitable
  El Torito img path :   2  /boot/grub/efi.img

But Debian is nice by having in the ISO filesystem a copy of the file
tree in the FAT filesystem of the EFI partition.

  # mount debian-11.5.0-amd64-netinst.iso /mnt/iso
  ...
  # mount /mnt/iso/boot/grub/efi.img /mnt/fat
  $ (cd /mnt/fat ; find . -type f -exec ls -ld '{}' ';')
  -rwxr-xr-x 1 root root 934240 Sep 10  2022 ./efi/boot/bootx64.efi
  -rwxr-xr-x 1 root root 1684864 Sep 10  2022 ./efi/boot/grubx64.efi
  -rwxr-xr-x 1 root root 101 Sep 10  2022 ./efi/debian/grub.cfg
  $ (cd /mnt/iso ; find ./EFI -type f -exec ls -ld '{}' ';')
  -r--r--r-- 1 root root 934240 Sep 10  2022 ./EFI/boot/bootx64.efi
  -r--r--r-- 1 root root 1684864 Sep 10  2022 ./EFI/boot/grubx64.efi
  -r--r--r-- 1 root root 101 Sep 10  2022 ./EFI/debian/grub.cfg

The differences in the paths become insignificant when copying to FAT.
The tests results indicate that the lack of x-permissions with the ISO
files is not an issue either.

In the arm64 ISOs the /EFI tree is present three times. Once as appended
MBR partition 2, once as FAT image file /boot/grub/efi.img in the ISO
which serves as El Torito boot image, and again as unpacked /EFI tree in
the ISO.
(xorriso could point El Torito to the appended partition, thus eliminating
the need for the /boot/grub/efi.img file.)


> From what I can see, the maximum individual file name length when using
> Joliet extensions is 128 "characters"

It's less. A Joliet directory record has the same maximum size as an ISO
directory record: 255 bytes. Joliet encodes names in UCS-2 which uses
16 bit per character. The fixed header part of a directory record is 34
bytes large. So only 110 UCS-2 characters have room. For some reason
libisofs offers to write 103 characters at most. (It's long ago that this
decision was made not by me.)

In file
  /mnt/iso/.disk/mkisofs
i see the recorded -as mkisofs option:
  -joliet-long
which means according to man xorrisofs:
Allow  103  characters in Joliet file names rather than 64 as is
prescribed by the specification. Allow Joliet paths longer  than
the prescribed limit of 240 characters.


> if you are using Windows' native utilities, you won't be able
> format a partition that is larger than 64 GB to FAT32,

The largest official Debian ISOs are 50 GB.
But i have a script which can merge a complete set to a ~90 GB ISO. :))


Have a nice day :)

Thomas

Bug#1031696: Use of symbolic links in non-free ISO images breaks file system transposition support

[Thomas Schmitt]

Hi,

James Addison wrote:
> The problem, in both cases, was that I hadn't copied the '.disk' dotfile
> directory from the install media ISO filesystem(s) in each case.

Besides such user pitfalls with the produced ISO and the problem of
symbolic links there are other constraints which an ISO has to fulfill
for this use case.
At least:
- The file tree of the FAT filesystem in the EFI partition needs to be
  mirrored by a copy in the ISO filesystem.
- File names must be unique in respect to case-insensitive comparison.
- (I am not sure whether file name length can be a problem.)

I guess Pete Batard can give a more comprehensive list.


Have a nice day :)

Thomas

Bug#1032941: Additional use of Rock Ridge symbolic links in bookwork ISO images

[Thomas Schmitt]

Hi,

the use of xorrisofs/genisoimage option -f would convert those links
into shared content data files, too.


Side note:

Differing from my statement in bug #1031696, the current xorriso version
does not complain about dangling symbolic links when -f is given. It
silently includes them as symbolic links in the ISO.
I will change the behavior so that a SORRY message is emitted in this
case. With default settings of xorriso this will cause a non-zero exit
value of xorriso. But processing will go on as with the current version.

genisoimage -f emits a warning and does not include the dangling link in
the result. I.e. the file is missing in the ISO.


Have a nice day :)

Thomas

Bug#1031696: Use of symbolic links in non-free ISO images breaks file system transposition support

[Thomas Schmitt]

Hi,

James Addison wrote:
> Thinking aloud: as an alternative, would adding the '-f' flag to MKISOFS
> achieve the desired result for both documentation and firmware files,
> without requiring any other changes?

Let's pack up some symbolic links with -f:

  $ sudo mount firmware-bookworm-DI-alpha1-amd64-netinst.iso /mnt/iso
  ...
  $ ls -l /mnt/iso/firmware
  total 4
  lr-xr-xr-x 1 root root   73 Sep 20 14:30 
amd64-microcode_3.20220411.1_amd64.deb -> 
../pool/non-free/a/amd64-microcode/amd64-microcode_3.20220411.1_amd64.deb
  lr-xr-xr-x 1 root root   62 Sep 20 14:30 atmel-firmware_1.3-6_all.deb -> 
../pool/non-free/a/atmel-firmware/atmel-firmware_1.3-6_all.deb
  lr-xr-xr-x 1 root root   62 Sep 20 14:30 bluez-firmware_1.2-7_all.deb -> 
../pool/non-free/b/bluez-firmware/bluez-firmware_1.2-7_all.deb
  ...

So Linux shows the symbolic links of the mounted ISO.
I pick the directory with the links and a tree with most of their target
files and put them into an ISO:

  $ xorriso -as mkisofs -f -R -o test.iso -graft-points 
/firmware=/mnt/iso/firmware /pool/non-free=/mnt/iso/pool/non-free
  ...

Our goal is to have link and target path pointing to the same blocks.
And indeed:

  $ xorriso -indev test.iso -find / -type f -sort_lba -exec report_lba --
  ...
  Report layout: xt , Startlba ,   Blocks , Filesize , ISO image path
  File data lba:  0 ,   49 ,   57 ,   115444 , 
'/firmware/amd64-microcode_3.20220411.1_amd64.deb'
  File data lba:  0 ,   49 ,   57 ,   115444 , 
'/pool/non-free/a/amd64-microcode/amd64-microcode_3.20220411.1_amd64.deb'
  File data lba:  0 ,  106 ,   72 ,   147088 , 
'/firmware/atmel-firmware_1.3-6_all.deb'
  File data lba:  0 ,  106 ,   72 ,   147088 , 
'/pool/non-free/a/atmel-firmware/atmel-firmware_1.3-6_all.deb'
  File data lba:  0 ,  178 ,   89 ,   181620 , 
'/firmware/bluez-firmware_1.2-7_all.deb'
  File data lba:  0 ,  178 ,   89 ,   181620 , 
'/pool/non-free/b/bluez-fir
  ...

With genisoimage i see the same.

The only drawback seems to be that this prevents any symbolic links from
being in the ISO.
Whether the detection of dangling symbolic links at ISO production time is
an advantage or a disadvantage is not so clear.


Have a nice day :)

Thomas

Bug#1031696: Use of symbolic links in non-free ISO images breaks file system transposition support

[Thomas Schmitt]

Hi,

James Addison wrote:
> Please find below a proposed update to the commit message under discussion:

Only one nitpick:

> multiple filenames ("records"). The ISO9660 generation tools
> genisoimage and xorriso can use this to de-duplicate multiple linked
> files from their input, saving space in the output filesystem image.

They do this only with hard links, not with symbolic links.
So i propose: "de-duplicate multiple hard linked files".


Have a nice day :)

Thomas

Bug#1031696: Use of symbolic links in non-free ISO images breaks file system transposition support

[Thomas Schmitt]

Hi,

James Addison wrote:
> I've opened a merge request at
> https://salsa.debian.org/images-team/debian-cd/-/merge_requests/30
> [...]
> The commit message contains an effort to explain what's going on; please
> consider that message as reviewable and open to feedback too.

I'm clumsy with web based development. So my comments by mail:

--

Commit message:
> When creating links to firmware files on the installation media in
> the /firmware directory, want to retain the same content de-duplication
> and resulting space-saving efficiencies as symlinks have provided.

Shouldn't that be ", we want to" ?

Since the symlinks are currently produced, shouldn't it be "as symlinks
currently provide." ?

(Disclaimer: I'm not a native speaker of english.)


> We'd also like the file entries that are created on the disc image
> to be compatible-with and copyable-from as many operating systems
> as possible.

It's rather about being copyable _to_ many filesystems by many operating
systems.

Actually it is mostly about FAT as target filesystem. It is the intended
target for copying because EFI is willing to search for its boot program
in about any partition with a FAT filesystem. (The specs prescribe special
partition types but in practice EFI does not care.)

Pete Batard's use case has two problems with symbolic links:
- The FAT filesystem has no concept of links.
- MS-Windows does not read Rock Ridge and thus ignores the symbolic link
  info anyways. (But this problem is not significant in the end.)

So how about:
  "We'd also like the files of the ISO 9660 filesystem to be copyable to
   as many filesystems by as many operating systems as possible."


> Symlinks have some drawbacks in that regard, since not all operating
> systems fully support symbolic linking.

I propose to write:

  "Links are not supported everywhere. The best substitute is to create
   independent copies of the link target file for every link which is
   encountered."


> Fortunately, the ISO9660 format supports a standardized way to refer
> to a common block of content (by referencing a starting block and a
> block count on-disc) from multiple file paths.
> Both genisofs and xorriso (CD image generation tools) support those kind
> of shared content file references, and are able to replace multiple
> hardlinks (POSIX links) that reference a single on-disk file with a
> single shared content range in the output disc image.

How about:

  "The free ISO 9660 specs ECMA-119 give in 6.5.1 permission to use the
   same file content blocks for multiple files. The ISO 9660 generation
   tools genisoimage and xorriso do this to represent hard link relations
   among their input files and to save space.
   Reading operating systems usually perceive the files as not hardlinked,
   though. So this representation of hard links causes creation of the
   desired independent data files when they get copied out of the ISO
   filesystem."

man genisoimage says that this feature is enabled by default on "Unix-like
operating systems" and can be disabled by option  -no-cache-inodes .
xorriso does not offer to disable it. Its mkisofs emulation ignores the
option -no-cache-inodes .


> Those tools won't do that by default for symlinks, however (for cautious
> and sensible reasons related partly to the potential for
> infinite-link-loops), and so

The tools represent symbolic links in form of Rock Ridge SL entries.
The current Debian ISOs mounted by Linux show the links and Linux follows
them just fine. (Else it won't work for the Debian Installer.)
Handling of infinite link loops is the job of the reader.

So i propose to omit this part of the sentence.


> this changeset switches from the Perl symlink function (that
> creates symbolic links) to the more configurable debian-cd good_link
> function that will create links based on the demands of the
> environment it is configured in.

Well, this aspect is out of my area of expertise.


Have a nice day :)

Thomas

Bug#1031696: Use of symbolic links in non-free ISO images breaks file system transposition support

[Thomas Schmitt]

Hi,

James Addison wrote:
> My interpretation of the commands and output in your comment is that both
> genisoimage and xorriso can translate hardlinks from a source filesystem
> into deduplicated file references in a written ISO filesystem

With genisoimage we only know empirically. With libisofs under xorriso
i can tell that a red-black-tree of device and inode numbers is used to
match hard links. This is well tested by Debian ISOs because already now
the Linux kernels are hard link siblings.
E.g. in firmware-bookworm-DI-alpha1-amd64-netinst.iso 15 MB are won:
  Report layout: xt , Startlba ,   Blocks , Filesize , ISO image path
  File data lba:  0 ,35982 , 3761 ,  7700896 , 
'/install.amd/gtk/vmlinuz'
  File data lba:  0 ,35982 , 3761 ,  7700896 , '/install.amd/vmlinuz'
  File data lba:  0 ,35982 , 3761 ,  7700896 , 
'/install.amd/xen/vmlinuz'

If genisoimage would not deduplicate some hardlinks then the ISO would
simply become larger but stay functional for the boot paths which
debian-cd prepares and also for the file copying method which Pete Batard
wants to get enabled.


This method is intended for booting via EFI from USB stick. In general i
support Pete Batard's goal to have bootable ISOs ready for it, regardless
whether it is a wish or the fix of a regression.
(I looked into firmware-11.6.0-amd64-netinst.iso and found its /firmare
directory filled with symbolic links and one sub directory ./deb11.)

The file extraction method is supposed to be a behavioral pattern of
experienced users of MS-Windows. Let's be inviting to them or else they
might install Debian on WSL.


Have a nice day :)

Thomas

Bug#1031696: Use of symbolic links in non-free ISO images breaks file system transposition support

[Thomas Schmitt]

Hi,

James Addison wrote:
> it looks like the selection of CD image
> creation tool is configured per-architecture here:
> https://salsa.debian.org/images-team/debian-cd/-/blob/5aebb6794a3b8b2393663fb643e35eb8e510c9a4/Makefile#L24

I wish i would understand the clause
  ifneq (,$(filter i386 amd64 arm64 hppa,$(ARCHES)))

Surely i386, amd64, and arm64 get their published Debian ISOs made
by xorriso. hppa seems to have switched to xorriso between Debian 10 and
11. sparc64 10.0 is genisoimage, ppc64el 10.8.0 is xorriso,
armhf 10.1.0 is xorriso ...
powerpc needs HFS and thus its ISO is made by genisoimage.


So let's test hardlink handling in genisoimage:

  $ ls -l /u/test/hardlinks
  total 88
  -rw-r--r-- 2 thomas thomas 42786 Nov 14  2005 hardlink_x
  -rw-r--r-- 2 thomas thomas 42786 Nov 14  2005 x
  $ genisoimage -o test2.iso -R /u/test/hardlinks
  ...
  196 extents written (0 MB)

We inspect the resulting ISO by xorriso:

  $ xorriso -indev test2.iso -find / -type f -exec report_lba --
  ...
  Report layout: xt , Startlba ,   Blocks , Filesize , ISO image path
  File data lba:  0 ,   25 ,   21 ,42786 , '/hardlink_x'
  File data lba:  0 ,   25 ,   21 ,42786 , '/x'
  $

Both files in the ISO refer to the same block range starting at 2048-byte
block 25 up to block 45. So in the ISO, they are deduplicated

Now for a bit of kernel slapstick:

  $ sudo mount test2.iso /mnt/iso
  mount: /mnt/iso: WARNING: source write-protected, mounted read-only.
  $ ls -li /mnt/iso
  1479 -rw-r--r-- 2 thomas thomas 42786 Nov 14  2005 hardlink_x
  1483 -rw-r--r-- 2 thomas thomas 42786 Nov 14  2005 x

Note the link count 2 in combination with the differing inode numbers.
(The link count stems from Rock Ridge field PX. By mistake i mentoned it
as "PN" in my previous mail.)

The false link count does not hamper restoring of the files:

  $ cp -r /mnt/iso /u/test/hardlinks_restored
  $ ls -li  /u/test/hardlinks_restored
  total 88
  8913418 -rw-r--r-- 1 thomas thomas 42786 Mar 11 17:38 hardlink_x
  8913419 -rw-r--r-- 1 thomas thomas 42786 Mar 11 17:38 x

So the files are independent after being restored by Debian 11 to ext4.

The same happens with an ISO made by xorriso:

  $ xorriso -as mkisofs -o test.iso -R /u/test/hardlinks
  ...
  $ xorriso -indev test.iso -find / -type f -exec report_lba
  ...
  Report layout: xt , Startlba ,   Blocks , Filesize , ISO image path
  File data lba:  0 ,   33 ,   21 ,42786 , '/hardlink_x'
  File data lba:  0 ,   33 ,   21 ,42786 , '/x'

  $ sudo mount test.iso /mnt/iso
  ...
  $ cp -r /mnt/iso /u/test/hardlinks_restored
  $ ls -li  /u/test/hardlinks_restored
  total 88
  8913418 -rw-r--r-- 1 thomas thomas 42786 Mar 11 17:47 hardlink_x
  8913419 -rw-r--r-- 1 thomas thomas 42786 Mar 11 17:47 x

Just to prove that the restored files are really not hardlinked:

  $ echo some_tail_bytes >> /u/test/hardlinks_restored/x
  $ ls -li  /u/test/hardlinks_restored
  total 88
  8913418 -rw-r--r-- 1 thomas thomas 42786 Mar 11 17:47 hardlink_x
  8913419 -rw-r--r-- 1 thomas thomas 42802 Mar 11 17:48 x
  $


Have a nice day :)

Thomas

Bug#1031696: Use of symbolic links in non-free ISO images breaks file system transposition support

[Thomas Schmitt]

Hi,

James Addison wrote:
> Something that symlinks can do is to allow image creators to save space by
> by de-duplicating files. [...]

This could be achieved by a bunch of hard links instead of the symbolic
links. In /firmware of firmware-bookworm-DI-alpha1-amd64-netinst.iso
i see only symbolic links to data files. So replacing symlinks by
hardlinks should be technically possible.

xorriso represents hard link siblings in ISO 9660 and Joliet as data files
which use the same data content blocks. So these views of the Debian ISO
would produce separate copies of target and link when both get extracted
to disk.

A Rock Ridge reader could do better by help of the PN entry. But Linux
computes own inode numbers from the byte addresses of the ISO 9660
directory records. So hardlinks do not get represented as such when Linux
mounts an ISO. Thus copying them out of the ISO causes duplication.

This would be a disadvantage of hard links for those people who unpack
Debian ISOs on Linux.


Have a nice day :)

Thomas

Re: sha512sum-error at debian-live-11.5.0-amd64-gnome.iso

[Thomas Schmitt]

Hi,

just for the records i report what Gerd Mühlenbruch and i found out
off list:

The reason for the bad dowload was that two wget processes were writing
to the same file.

Gerd had started the first wget by letting nautilus executing it. For
this it created a terminal window with a running wget, which Gerd soon
after closed by means of the desktop.
Then he wanted to continue downloading by starting wget in terminal
window.

Closing the nautilus-made terminal would normally shut down the program
that is running in it. Not so with wget, where the man page says:

  Wget is non-interactive, meaning that it can work in the background,
  while the user is not logged on.  This allows you to start a retrieval
  and disconnect from the system, letting Wget finish the work.

We did not explore what exactly causes the oversize of the download result.
The situation of two simultanously working wgets on the same file is ill
enough to serve as explanation.


Have a nice day :)

Thomas

Re: sha512sum-error at debian-live-11.5.0-amd64-gnome.iso

[Thomas Schmitt]

Hi,

Gerd Mühlenbruch wrote:
> nautilus is starting a bash terminal.

Do you know which program it runs in there ?
(And with which arguments ?)


> After the break it told me a size of 7.900 Byte.
> ls -l is telling me 150431900 which is huge for the short download time.

I would guess that the download worked multi-threaded in order to get
different pieces of the ISO simultaneously.
But this does not match the fact that the resulting oversized ISO shows
no differences to the correct ISO in the legitimate size interval.


> In the terminal start wget startet at ~250 MB which is disturbing me.

None of these numbers give me ideas in relation to the final oversize
of 480165 bytes.
The only hope for insight seems to be with determining what nautilus
started in that bash window.


Have a nice day :)

Thomas

Re: sha512sum-error at debian-live-11.5.0-amd64-gnome.iso

[Thomas Schmitt]

Hi,

Gerd Mühlenbruch wrote:
> So, it seems to be a problem of nautilus on my computer.

wget -c still has a share in this.
But currently i have no ideas how to find out what makes an incomplete
nautilus download so confusing for wget.


> dd if=debian-11.5.0-amd64-netinst.iso bs=1 skip=400556032 | od -c | less
> >Ausgabe.txt

(My proposal to pipe into the pager "less" was meant to avoid the need
for redirection.)


> 1177740  \0  \0  \0   E   F   I   P   A   R   T  \0  \0 001  \0   \
> 1177760  \0  \0  \0 302 354 241 223  \0  \0  \0  \0 377 357  \v  \0  \0
> 120  \0  \0  \0 001  \0  \0  \0  \0  \0  \0  \0   @  \0  \0  \0  \0

This is a GPT header block signature, but not aligned to a full block
of 512 bytes, as GPT prescribes.
The "E" of "E   F   I" is supposed to be at the start of the block.
24 bytes from there on we have a little-endian 8-byte number which tells
the intended block address:

  (gdb) p 0377 + 0357 * 256 + '\v' * 256 * 256
  782335

This is the address of the last 512-block of the original ISO:

 782335 = 400556032 / 512 - 1

So we have a displaced copy of a GPT backup header with the proper
block address which it would have in the netinst ISO.
Checking this theory:

  dd if=debian-11.5.0-amd64-netinst.iso bs=512 skip=782335 | od -c

yields

  000   E   F   I   P   A   R   T  \0  \0 001  \0   \  \0  \0  \0
  020 302 354 241 223  \0  \0  \0  \0 377 357  \v  \0  \0  \0  \0  \0
  040 001  \0  \0  \0  \0  \0  \0  \0   @  \0  \0  \0  \0  \0  \0  \0
  060 312 357  \v  \0  \0  \0  \0  \0   f 027 253   N 247   r   5   B
  100 215 256   z   9 353   M 314 313 357  \v  \0  \0  \0  \0  \0
  120 320  \0  \0  \0 200  \0  \0  \0   1 231   o 227  \0  \0  \0  \0
  140  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0  \0
  *
  0001000

Both, the existence of the copy 327648 after the original end of the ISO
and its non-alignment by -29 (or 483) bytes are hard to explain.


Have a nice day :)

Thomas

Re: sha512sum-error at debian-live-11.5.0-amd64-gnome.iso

[Thomas Schmitt]

Hi,

Gerd Mühlenbruch wrote:
> I started my batch by doubleclicking the file in nautilus.

Being a fvwm user i can only riddle whether this start by nautilus has
something to do with the problem.
But your post of 3 Oct 2022 18:28:08 + says that you started the
original download by wget -c. So i assume that nautilus is not significant.


> I expected differences in bytes anywhere in the beginning, but comp -l and
> comp -b didn't showed such result.
> ==>>> cmp: EOF on debian-live-11.5.0-amd64-gnome.iso after byte 2910683136

That astonished me already with your original post. There seems to be
a surplus tail appended. What's in that tail ?

E.g. what do you get from

  dd if=Err-debian-live-11.5.0-amd64-gnome.iso bs=1 skip=2910683136 | \
  od -c | less

Is it all 0 ? Is some human readable text to see ?

(It would be more lightweight to inspect Err*-netinst.iso, if you still
 have it
  dd if=Err-debian-11.5.0-amd64-netinst.iso bs=1 skip=400556032 | \
  od -c | less
)

Do you remember how large the ISOs were after their first interrupt ?
Are perhaps the differences between oversize and real size the same
numbers ?
(I compute from your numbers: 4184168 for live and 328163 for netinst.)


There are some bug reports about wget -c on
  https://bugs.debian.org/cgi-bin/pkgreport.cgi?repeatmerged=no&src=wget
but none seems to match what you experience.

> I'm using debian bullseye with gnome 3.38 and wayland.

That's an ideal base for posting a bug report.

> However, I can try your lines the next days.

It would be better for the bug report if the problem could be reproduced
with wget alone.

> If you like, I could add the shell selecting in the first line "#...".

It would be really odd if wget -c would behave differently depending
on the shell which started wget.


Have a nice day :)

Thomas

Re: sha512sum-error at debian-live-11.5.0-amd64-gnome.iso

[Thomas Schmitt]

Hi,

Gerd Mühlenbruch wrote:
> While copying the logfile I remembered that I used wget twice with "-c"
> option.
> [...]
>    cmp debian-live-11.5.0-amd64-gnome.iso
> Err-debian-live-11.5.0-amd64-gnome.iso
>    cmp: EOF on debian-live-11.5.0-amd64-gnome.iso after byte 2910683136,
> [...]
> -rw-r--r-- 1 ghdm ghdm 2911003302 Sep 10 13:49 
> Err-debian-live-11.5.0-amd64-gnome.iso
> -rw-r--r-- 1 ghdm ghdm 2910683136 Sep 10 13:49 
> debian-live-11.5.0-amd64-gnome.iso

That's really strange. Looks like wget -c added 320166 bytes to the
2910683136 bytes of the original ISO.
This collides with the urge to use wget -c rather than a web browser:
  https://www.debian.org/CD/http-ftp/

I fail to reproduce the problem with a netinst CD image (which has only
400 MB):

  $ wget -c 
https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-11.5.0-amd64-netinst.iso
  [...]
  ^C
  $ ls -l debian-11.5.0-amd64-netinst.iso
  [...] 6727900 Oct  3 21:53 debian-11.5.0-amd64-netinst.iso
  $ wget -c 
https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-11.5.0-amd64-netinst.iso
  [...]
  HTTP request sent, awaiting response... 206 Partial Content
  Length: 400556032 (382M), 393828132 (376M) remaining 
[application/x-iso9660-image]
  [...]
  $ ls -l debian-11.5.0-amd64-netinst.iso
  [...]  400556032 Sep 10 14:40 debian-11.5.0-amd64-netinst.iso

In my experiment the correct size was downloaded and sh512sum confirms
that the content is as announced in .../amd64/iso-cd/SHA512SUMS .

It would be interesting to learn under which circumstances wget yields
the result which you observe.


Have a nice day :)

Thomas

Re: sha512sum-error at debian-live-11.5.0-amd64-gnome.iso

[Thomas Schmitt]

Hi,

Gerd Mühlenbruch wrote:
> Pfad="https://cdimage.debian.org/debian-cd/current-live/amd64/iso-hybrid";
> wget $Optionen $Pfad/debian-live-11.5.0-amd64-gnome.iso
> sha512sum --ignore-missing -c SHA512SUMS
>     debian-live-11.5.0-amd64-gnome.iso: FEHLSCHLAG

I did

  wget 
https://cdimage.debian.org/debian-cd/current-live/amd64/iso-hybrid/debian-live-11.5.0-amd64-gnome.iso

and got redirected to

  
https://laotzu.ftp.acc.umu.se/debian-cd/current-live/amd64/iso-hybrid/debian-live-11.5.0-amd64-gnome.iso
  Connecting to laotzu.ftp.acc.umu.se 
(laotzu.ftp.acc.umu.se)|2001:6b0:19::166|:443... connected.

The SHA512SUMS file came directly from

  https://cdimage.debian.org/debian-cd/current-live/amd64/iso-hybrid/SHA512SUMS


The SHA512 of debian-live-11.5.0-amd64-gnome.iso is then

  
450d09f1f1556effce4bc072ad395e45652aa40ed035b8ab35616c54fbef0c6edf803acb483ac25737b12bbae858277c1aa261eee36c4edf505a85c915d73c67

which matches the line in the SHA512SUMS file

  
450d09f1f1556effce4bc072ad395e45652aa40ed035b8ab35616c54fbef0c6edf803acb483ac25737b12bbae858277c1aa261eee36c4edf505a85c915d73c67
  debian-live-11.5.0-amd64-gnome.iso


So do you get a different SHA512 from your downloaded
debian-live-11.5.0-amd64-gnome.iso ?

Or does the downloaded SHA512SUMS file show a different SHA512 ?

---

>     gpg: WARNUNG: Dieser Schlüssel trägt keine vertrauenswürdige Signatur!
>     gpg:  Es gibt keinen Hinweis, daß die Signatur wirklich dem 
> vorgeblichen Besitzer gehört.
>     Haupt-Fingerabdruck  = DF9B 9C49 EAA9 2984 3258  9D76 DA87 E80D 6294 BE9B

The true safety check is not whether the signature is supported by a
chain of trust, but whether it yields one of the fingerprints which are
listed on

  https://www.debian.org/CD/verify

In your case it has the fingerprint of

  pub   rsa4096/DA87E80D6294BE9B 2011-01-05 [SC]
Key fingerprint = DF9B 9C49 EAA9 2984 3258  9D76 DA87 E80D 6294 BE9B
  uid  Debian CD signing key 

(Beware of alternative ways to express space characters when comparing
the strings automatically.)


Have a nice day :)

Thomas

Bug#1011343: WISHLIST: Offical ALL-IN-ONE images?

[Thomas Schmitt]

Hi,

i added to  merge_debian_isos  the capability to burn the resulting
ISO 9660 filesystem directly to optical media and to write it directly
to storage device files. This removes the need for having substantial
free disk space beyond the capacity to store the original ISO images.

While burning to optical media is quite a harmless operation for the
normal user, writing to USB stick devices is not. Therefore
merge_debian_iso can cooperate with xorriso-dd-target, my offer for
safe copying of image files onto USB sticks on GNU/Linux.

I created
  https://wiki.debian.org/MergeDebianIsos
to describe installation, feeding, and usage of merge_debian_isos.
Review and testing are welcome.

(The same applies to the older
  https://wiki.debian.org/XorrisoDdTarget
)


Have a nice day :)

Thomas

Bug#1011343: WISHLIST: Offical ALL-IN-ONE images?

[Thomas Schmitt]

Hi,

Zhang Boyang wrote:
> All OK! Thank you again :)

So as far as we two are concerned, it seems we have a candidate for
release.

Currently it only goes into the GNU xorriso tarball, which is not src of
any Debian package. I could put it into the libisoburn tarball. But as
it is specific to debian-cd ISOs (not even to debian-live ones) i think
it should go into debian-cd (with a copy in GNU xorriso, which already
has libjte as Debian specialty).

Nevertheless, most important would be to offer it as plain text download
for those who have a shell and downloaded ISOs. If it were only available
as Debian package then it would not help those who still have to become
Debian users.
A cool place would be the download pages
  https://cdimage.debian.org/debian-cd/current/{amd64,...}/jigdo-{dvd,bd,dlbd}
and mentioning in their *SUMS files.


Have a nice day :)

Thomas

Bug#1011343: WISHLIST: Offical ALL-IN-ONE images?

[Thomas Schmitt]

Hi,

Zhang Boyang wrote:
> Theoretically if both file is already sorted, we can use the `-m' option

I like this idea. Just in case Debian grows to a million packages.
But i understand that i would need two separate files for sorting.

  grep ' ./pool/' file1
  fgrep -v ' ./pool/' file2
  sort -k 2 -m file1 file2 >temp_file

More temporary files means more need for pre-existence tests and more
cleanup effort.
Currently i just have to concatenate two stdout streams.


> However I don't think O(n * log(n)) is a bottleneck so we may just keep it
> simple and stupid.

Seems to be the best decision for now.
(Unless some shell wizzard shows a way to pipe both streams separately
into sort -m while staying dash compatible and without the need for new
persistent file objects which have to be cleaned up afterwards.)

Actually we did not yet identify a use case where md5sum.txt needs to
be sorted at all. I only sort it because the input files are sorted.


New version:
  
https://dev.lovelyhq.com/libburnia/libisoburn/commit/0e8227e76ae4c4f24097cfac2f415ef8e25ae4e7


Have a nice day :)

Thomas

Bug#1011343: WISHLIST: Offical ALL-IN-ONE images?

[Thomas Schmitt]

Hi,

Zhang Boyang wrote:
> These experiments all succeeded. Thank you very much! Good Job! :)

Thank you for testing and challenging.


I wrote:
> > For now i decided to take the 50 seconds with dash.

> if you really want to reduce runtime I would suggest using
> `sort -s -u -k 2 merged_md5sum.txt' instead of processing each line
> by hand.

The task is to identify those which need newly computed MD5 because they
might have changed. Mostly i know which directories are suspects, because
they are on hard disk and get mapped back into the emerging ISO. Their
MD5s get recomputed from the files on hard disk.
Some other paths in the md5sum.txt may appear multiple times. In this case
it is clear that the data of the file in the emerging ISO stem from iso1.
But it is not clear which of the multiple lines in md5sum.txt stems from
iso1. So the MD5 has to be recomputed from the file in mounted iso1.


> I saw there are some other logic to process md5 records from
> different group of files, so we can use `grep' and `grep -v' to split them,
> process them separately, then merge them finally.

That's a great idea.
The majority of files is in ./pool and surely needs no recomputing, even
if listed multiple times (due to overlapping ISO pools).

This here

  ( fgrep ' ./pool/' temp_file

needs 1.9 seconds instead of 7.2 seconds with the old

  polish_md5sum_txt >temp_file

Times were measured by date '+%s.%N' around the polishing commands.
polish_md5sum_txt and its subordinate were slightly modified for the new
method to read from stdin and to not expect any ./pool file.
The latter brought 0.9 seconds.

The number of lines in md5sum.txt is then the same as with the old method.
My test loop with md5sum -c on the mounted result ISO reports no
mismatches. (It is annoying that gzip inserts a time stamp, so that the
Packages.gz files differ although they bear the same uncompressed
content. So the md5sum.txt file shows differences, too, from run to run.)


> Unfortunately the option `-s' of `sort' is not standard

I understand that it is needed to keep sort -k 2 from distinguishing
lines with differences outside of -k 2 so that sort -u could throw out
surplus lines with duplicate paths.

But with above code sort -u is not needed.
The pool lines have to be identical even if duplicate paths appear at
all. (I only know of one old debian package which existed with different
content but same name, long ago.) So uniq can do its job.
The other lines are made unique by the shell function polish_md5sum_txt.

Complexity-wise this replaces a slow O(n) algorithm by a faster O(n) and
an additional O(n * log(n)) run. At some size of Debian the slow speed
of the linear loop will be compensated by the sorting complexity.
But there is still room: A sort of 11,000 lines lasts about 0.03 seconds.

I will probably commit this change tomorrow. Now it needs cleaning and
handling of the new dependency uniq.


Have a nice day :)

Thomas

Bug#1011343: WISHLIST: Offical ALL-IN-ONE images?

[Thomas Schmitt]

Hi,

i explored two ways to get a correct and complete md5sum.txt after
merging and sorting:

- Generating the whole md5sum.txt from the emerging ISO's file tree is
  possible by help of xorriso, although there is no file tree yet where
  all regular files of the ISO can be found for checksumming.
  But this would work only for MD5 and not for SHA256, because xorriso
  has no API function to compute SHA256. (libjte has SHA256 but not as
  API. There are many xorrisos out there which are not linked to libjte.)
  Run time would be mediocre: 36 seconds for DVD-1+2+3 on a 500MB/s SSD.
  (find+md5sum on mounted result ISO needs 27 seconds. But as said there
   is no result ISO to mount when the script makes md5sum.txt.)

- Removing duplicates from md5sum.txt and identifying those files which
  possibly have a changed MD5 after the merging activities is more error
  prone but signficantly faster.
  Especially it can be easily modified when Debian decides to retire
  md5sum.txt in favor of a sha256sum.txt.
  Run time is still annoying: 8 seconds with dash, 12 seconds with bash.
  As it is now it works with echo | grep.
  I could reduce it to less than a second by using the bashism
  ${Var:Offset:Count} to obtain a substring of the file paths.

Run time for an All-in-one ISO is estimated about 6 to 7 times the time
of DVD-1+2+3.
So i expect ~230 seconds for full MD5 regeneration, ~ 50 seconds for
a loop that runs on dash, and ~6 seconds with a bashism.

For now i decided to take the 50 seconds with dash.

The merged md5sum.txt is not 100% complete. Files in ./firmware which
appear in more than one input ISO will not be listed, because it is not
100% clear from which ISO the one stems which survives the competition.
It appears that the check in
  https://salsa.debian.org/installer-team/cdrom-checker/-/raw/master/main.c
does not insist in a complete list. It only demands that all listed files
exist and yield the listed MD5 when being checksummed.

I tested the correctness of the merged md5sum.txt of DVD-1+2+3 by mounting
the result as /mnt/iso and running what i deem equivalent to the MD5 check
in installer-team/cdrom-checker :

  (cd /mnt/iso
   cat md5sum.txt | while read line
   do
 if echo -n "$line" | md5sum -c 1>/dev/null 2>&1
 then
   dummy=dummy
 else
   echo "BAD: $line"
 fi
   done ) 2>&1 | wc

Result from wc was "0 0 0" (after 47 seconds).

Regrettably i cannot check this with my dummy DLBD-1+2 ISO, because its
data files nearly all have fake content.

Committed changes:
  
https://dev.lovelyhq.com/libburnia/libisoburn/commit/0bc397c02c0ea7c960b59ce92daa267bed23fc07


Have a nice day :)

Thomas

Bug#1011343: WISHLIST: Offical ALL-IN-ONE images?

[Thomas Schmitt]

Hi,

i wrote:
> > > md5sum.txt

Zhang Boyang wrote:
> I think maybe we should just create the md5sum from scratch?

Well, not from scratch, because the paths would come from the merged
md5sum.txt.

But indeed the merged file needs polishing:
- Some files are listed multiple times because they appear in the
  md5sum.txt files of multiple input ISOs.
- The files in ./dists are probably newly generated.

The bulk of packages and support files is not supposed to have changed.

So i will have to promote the md5sum program to hard depenceny and
make a filtering run over the merged and sorted md5sum.txt file to
care for above cases.

I will give a note when this is ready for testing.


Have a nice day :)

Thomas

Bug#1011343: WISHLIST: Offical ALL-IN-ONE images?

[Thomas Schmitt]

Hi,

Zhang Boyang wrote:
> 2) Create my own version of my-DLBD1.iso and my-DLBD2.iso from my private
> mirror.

This explains why a firmware package was missing which is in the official
DLBD-1 but obviously on your my-DLBD2.


With the new script version which merges /firmware trees:
> This time the difference in /firmware is:
> Only in /groundtruth/firmware:
> arm-trusted-firmware-tools_2.4+dfsg-2_amd64.deb
> I don't think this is the merge script's fault. This .deb is not exist in
> both my-DLBD1.iso and my-DLBD2.iso.

Then it's fully ok for me. :))


> There are other differences in filesystem-tree.
> [attachment diff.details.txt]
> .disk/mkisofs

This is just an unchanged copy of the file in your DLBD-1.
(As stated previously i see few reason to overwrite it with the xorriso
arguments which were used for merging.)

> boot/grub/efi.img

Could be about timestamps in the FAT filesystem.

One could try whether exporting SOURCE_DATE_EPOCH=...seconds.since.1970...
with the same value for DLBD and CUSTOM would create the same efi.img
files.
( https://reproducible-builds.org/docs/source-date-epoch/ )

> dists/bullseye/Release

This is a very central file. It would be interesting to see the full diff.

> dists/bullseye/contrib/binary-amd64/Packages.gz

Could be the sequence of packages in this unsorted list of multi-line
records.
Critical are the last package record of DLBD-1 and the first of DLBD-2.
In the merged ISO's Packages.gz they have to be listed with a neat single
empty line between them. (In my tests this was the case.)

> dists/bullseye/contrib/i18n/Translation-cs.gz

At least in the DVD set of 11.2.0 the i18n files which are mentioned in
Release are incomplete counterparts of the respective Packages.gz files
in various languages. Many packages are not listed in them.

> dists/stable

That's a symbolic link to dists/bullseye. So i skip to:

> isolinux/boot.cat

This file contains the block addresses of isolinux/isolinux.bin and of
boot/grub/efi.img. These addresses are subject to many influences during
ISO production. Somewhat random.

> isolinux/f1.txt

Seems to be a production timestamp of d-i.
It would be interesting to see whether it obeys SOURCE_DATE_EPOCH.

> isolinux/isolinux.bin

The -boot-info-table patching at the begin of ISO production writes the
block address of isolinux/isolinux.bin into isolinux/isolinux.bin.

> md5sum.txt

Ouch. My script sorts the merged lines by the MD5 fields rather than by
the file paths.
Further this sorting is subject to locale settings, which is hardly
desirable, if the sequence of lines has a meaning at all.

I committed a new version which fixes both problems in my local tests:
  
https://dev.lovelyhq.com/libburnia/libisoburn/commit/87aab730582cb4268f50062361fec7f13e2b50ab


Have a nice day :)

Thomas

Bug#1011343: WISHLIST: Offical ALL-IN-ONE images?

[Thomas Schmitt]

Hi,

i tested merging of /firmware directories with barely sufficently
complete
  debian-11.0.0-amd64-DLBD-[12].iso.tmp
from aborted jigdo-lite runs.

All files which are reported as being only in CUSTOM(all-in-one) by
  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1011343#115
are listed by xorriso as present in the emerging ISO, of which i
suppressed the actual production out of storage space reasons.

So if the next diff between self-made CUSTOM(all-in-one) and merged.iso
shows again missing files in /firmware, i have to ask for full listings
of the /firmware trees in DLBD-1, DLBD-2, CUSTOM(all-in-one), and
merged.iso .

---
About my shortcuts with download and ISO non-production:

I waited with aborting jigdo-lite just long enough until the first
downloaded packages caused the inflation of the .template file
to .iso.tmp. The directory tree and the management files are then
already present, whereas the packages' content is mostly waiting for
being filled in.

Further i modified the xorriso run to end by
  -find /firmware -- -rollback_end
so that no ISO emerges but only the content of the planned /firmware
directory is shown before the program ends.

(I would go further. But the production of CUSTOM(all-in-one) via
debian-cd is out of my reach, given the section "ABOUT MIRROR" in message
#115. So the effort to complete the download and to produce a merged.iso
would bring no benefit for now.)


Have a nice day :)

Thomas

Bug#1011343: WISHLIST: Offical ALL-IN-ONE images?

[Thomas Schmitt]

Hi,

Zhang Boyang wrote:
> I will definitely try it

Meanwhile i got some insight into the riddle about diffs between merged.iso
and CUSTOM-1.iso like

  Only in /groundtruth/firmware: arm-trusted-firmware-tools_2.4+dfsg-2_amd64.deb
  Only in /groundtruth/firmware: gnome-firmware_3.36.0-1_amd64.deb

Indeed arm-trusted-firmware-tools_2.4+dfsg-2_amd64.deb does not exist in
the /firmware directory of DLBD-1 and thus did not get into merged.iso.

But /firmware/gnome-firmware_3.36.0-1_amd64.deb is in DLBD-1 as symbolic
link and its target is in DLBD-1, too.

Please verify that it is really not in merged.iso.
If so, then please record the messages of the next experiment with the
merge script and send them to me in private.

I now uploaded a new version of the script which merges the /firmware
directories. Just a guess, until i know what's up with /firmware.



To those who are familiar with debian-cd, especially Steve McIntyre:

/firmware is not mentioned in
  https://wiki.debian.org/DebianRepository/Format
So i guess that it is specific to debian-cd or the installer along
  https://wiki.debian.org/Firmware#Firmware_during_the_installation
and that i should merge the /firmware directories.
Correct ?

I am a bit confused by the fact that debian-11.2.0-amd64-DVD-2.iso has
no firmware directory at all. How come ?
Can this happen to a first ISO, too ?
(DVD-1 has a /firmware tree with only inhabitant dep11/README.txt.)


Have a nice day :)

Thomas

Bug#1011343: WISHLIST: Offical ALL-IN-ONE images?

[Thomas Schmitt]

Hi,

i followed David's advise to consider the warnings of shellcheck, to drop
the demand for bash, and to check early whether all needed programs are
available. xorriso gets now checked for being young enough for the job
of replaying the boot related commands as detected with input ISO 1.
(1.3.2 in Debian 8 is too old. 1.4.6 in Debian 9 is ok.)

The warnings yielded no real problem fixes but several improvements in
regard to general shell code quality.
The switch from /bin/bash to /bin/sh was not costly. A printf formatter
had to be changed and a use of "type -p" for a message was dropped.

An test installation with the resulting ISO succeeded and was able install
further packages from all three merged DVD ISOs.
(I still could need proposals for further checking the fitness of the ISO.)


As before, the script is available at
  
https://dev.lovelyhq.com/libburnia/libisoburn/raw/branch/master/test/merge_debian_isos
with
  
https://dev.lovelyhq.com/libburnia/libisoburn/raw/branch/master/test/merge_debian_isos.sig

The changes are to see at
  
https://dev.lovelyhq.com/libburnia/libisoburn/commit/34981b1278610d3f31da90f57fdf1378d6012074


Have a nice day :)

Thomas

Bug#1011343: WISHLIST: Offical ALL-IN-ONE images?

[Thomas Schmitt]

Hi,

i wrote:
> > https://dev.lovelyhq.com/libburnia/libisoburn/raw/branch/master/test/merge_debian_isos

Steve McIntyre wrote:
> That might be a useful thing to include in a package. What do you think?

Best would be if debian-cd would take it, so that it can be adapted when
the repository format in the ISOs gets changed. It could be renamed to
e.g. "debian-cd-merge-isos" and become a separate "binary" package from
the debian-cd source package.
I would of course be willing to help with maintaining it.

But first it needs more testing, especially whether the resulting ISO
lacks anything that an ISO-1 from debian-cd with the same packages has.
Zhang Boyang posted a comparison in
  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1011343#115
  
https://bugs.debian.org/cgi-bin/bugreport.cgi?att=1;bug=1011343;filename=diff.details.txt;msg=115
about which i ponder:

- Shall the script create a new /.disk/mkisofs ?
  The used xorriso run has few similarity with a xorriso run from
  debian-cd. The pool content and package lists have no influence
  on the xorriso run of debian-cd.
  So it might be better to stay with the original.

- There are some /firmware files missing in the merged ISO. Like
/firmware/arm-trusted-firmware-tools_2.4+dfsg-2_amd64.deb
...
/firmware/gnome-firmware_3.36.0-1_amd64.deb
  They are in the pool of DLBD-1. But how did they get into /firmware of
  Zhang Boyang's CUSTOM(all-in-one) ? (Or: Why aren't they in /firmware
  of DLBD-1 ?)


I hope that the script is sufficiently stable against bad arguments and
bad combinations of ISOs. At least i inserted a lot of error messages
and took care to trigger each of them.
Nevertheless, it should be tested independently of me whether it can be
tricked into destroying existing data on disk or leaving temporary files
on disk.


Have a nice day :)

Thomas

Bug#1011343: WISHLIST: Offical ALL-IN-ONE images?

[Thomas Schmitt]

Hi,

although it was not the final solution of this bug report, i beefed up
my merger script for Debian ISOs so that it can combine an arbitrary
number of ISOs (within the limits of /dev/loop* and mount(8)).
Maybe it can serve as answer for the next time this wish comes up.

The script is uploaded as

  
https://dev.lovelyhq.com/libburnia/libisoburn/raw/branch/master/test/merge_debian_isos

with GPG detached signature
  
https://dev.lovelyhq.com/libburnia/libisoburn/raw/branch/master/test/merge_debian_isos.sig
for checking by gpg --verify.

After download, the user probably has to give x-permissions to the script.

When run without arguments it gives this help text:
---

  usage: merge_debian_isos result_iso mount_template iso1 iso2 [... isoN]

  Mounts by sudo the ISO 9660 images iso1 to isoN at directories
  mount_template1 to mount_templateN, if not already mounted that way.
  Then the Debian pools and package lists get merged and a new
  ISO 9660 image result_iso is produced, which must not yet exist.
  If iso1 is bootable then the new image will be bootable by the
  same means.
  At least the parent directory of mount_template must already exist.
  All arguments must be single words without using quotation marks.
  None of the isoN must be equal to another isoM.

  This script creates and finally removes the following temporary tree
  and files which must not yet exist in the current working directory:
./merged_dists , ./merged_md5sum.txt , ./merged_REAMDE.txt
./temp_file
  Further it creates and finally removes directories mount_template*
  if they are needed and do not exist when the script starts.
  It depends on the following programs:
awk, basename, bash, cat, chmod, cp, dirname, expr, fgrep, grep,
gunzip, gzip, head, ls, mkdir, mount, mv, rm, rmdir, sha256sum,
sed, sort, stat, sudo, umount, xorriso
  Recommended are: md5sum, sha1sum, sha512sum

  Exported non-empty variable MERGE_DATE enforces a particular
  date string in the text which gets prepended to /README.txt .
  Exported non-empty variable MERGE_FOR_DIST enforces the use of a
  particular directory in /dists of iso1. Normally only one
  such directory is found and thus no need to set MERGE_FOR_DIST.
  Exported non-empty variable XORRISO overrides command xorriso.
  This may be needed if installed xorriso is older than 1.4.2.

  Example using GNU xorriso-1.5.4 instead of /usr/bin/xorriso:
export XORRISO=$HOME/xorriso-1.5.4/xorriso/xorriso
mkdir merge_mount
merge_debian_isos merged.iso merge_mount/iso \
  debian-11.2.0-amd64-DVD-[12345].iso
rmdir merge_mount

---

The old script merge_2_debian_isos still exists but will refuse to run,
unless the user removes an exit command near the start of the script.
In its downloaded form it tells the user:
---
  THIS SCRIPT IS DEPRECATED ! USE ITS SUCCESSOR: merge_debian_isos

  The script merge_2_debian_isos still exists only because it was mentioned
  in Debian bug 1011343. The successor can merge more than two ISOs.
  So do not edit this script to remove this warning and the 'exit 7' line.
---

I tested the new script by merging the first three ISOs of
  debian-11.2.0-amd64-DVD-*.iso
and installing Debian from the result into a qemu-system-x86_64 VM with
4 GB RAM and 128 GB system disk.

After the installation was completed, i successfuly installed packages
which are in DVDs 1, 2, and 3 and were not installed yet:
  apt-get install apt-file
  apt-get install lame
  apt-get install xorriso

To verify that this success does not come from an unwanted network
connection i tried to install packages from DVD 4
  apt-get install bash-doc
  apt-get install devede
Both attempts failed with the message that the packages are not available.


Have a nice day :)

Thomas

Bug#1011343: WISHLIST: Offical ALL-IN-ONE images?

[Thomas Schmitt]

Hi,

for now it looks like the merged ISO works as fat DVD-1.

I installed a Debian 11.2.0 system from the merged DVD-1+ DVD-2 ISO
in a qemu VM via option -cdrom. Installation went smoothly.
During reboot i aborted the VM to next deface the El Torito boot sector
of the ISO. I removed the -net options from the qemu start command,
hopefully simulating a pulled network adapter.
Then i booted from the virtual -hda, having the -cdrom present too.
I removed from sources.list the deb and deb-src entries with http.

  ping debian.org
gets no replies. So the network is effectively down, although
  ip addr
shows some "ens3" with "altname enp0s3", which looks much like the
"nnp5s0" of the host machine except that the latter has no "altname".

To make sure that i can install stuff, i did as superuser

  dpkg -s apt-file
  dpkg -s lame
  dpkg -s xorriso

which all three indicated that the packages are not yet installed.
Then i tried to install them:

  # From pool of DVD-1
  apt-get install apt-file

  # From pool of DVD-2
  apt-get install lame

  # Not in the two DVD images
  apt-get install xorriso

The first two succeeded and behaved as i know from apt-get.
The third did not succeed, as expected.

--

But
  apt-file update
fails with
  E: The repository 'cdrom://[Debian GNU/Linux 11.2.0 _Bullseye_ - Official 
amd64 DVD Binary-1 20211218-11:13] bullseye Release' does not have a Release 
file
  N: Update [...] can't be done securely [...] disabled by default.
  N: See apt-secure(8) [...]

Suspecting a flaw with my merge procedure, i booted with the original
DVD-1 ISO as -cdrom.
I had to run
  apt-cdrom add
before the repo was accepted (i'm riddling a lot) but then i could prove
that it works by
  apt-get install needrestart
which is on DVD-1.
Nevertheless
  apt-file update
tells the same "E:" and "N:" lines.

So the shortcomming seems already to be in DVD-1.
Anything known about it ?

(I learned that it was not necessary to deface El torito on DVD-1+2.
DVD-1 with El Torito at -cdrom does not keep -hda from booting.)

--

What shall i further test to challenge the merged ISO whether it has
any shorcommings compared to the original DVD-1 ?


Have a nice day :)

Thomas

Bug#1011343: WISHLIST: Offical ALL-IN-ONE images?

[Thomas Schmitt]

Hi,

i uploaded the first version of my merger script as
  
https://dev.lovelyhq.com/libburnia/libisoburn/raw/branch/master/test/merge_2_debian_isos

Please download and give x-permission. The need for sudo can be avoided
by already mounting the two ISOs at the chosen mount points before running
the script.
Review and test reports are appreciated.

As motivation for tests by Debian installation experts:

If this works, then Debian could replace the DLBD sized ISOs by the
BD sized ISOs and instructions how to merge them to DLBD ISOs or a
QLBD/All-in-one ISO. (In that case the script should move into the hands
of debian-cd, i guess.)

===
$ ./merge_2_debian_isos
usage: merge_2_debian_isos \
   boot_iso boot_mount add_iso add_mount result_iso [for_dist]

Mounts by sudo the boot_iso at directory boot_mount and add_iso at
add_mount, if not already mounted that way. Then both Debian pools
and package lists get merged and a new ISO 9660 image result_iso
is produced, which must not yet exist.
If boot_iso is bootable then the new image will be bootable by the
same means.
This script creates the following temporary tree and files which
must not yet exist in the current working directory:
  ./merged_dists , ./merged_md5sum.txt , ./merged_REAMDE.txt
  ./temp_file
The optional sixth argument for_dist should only be given if
this script refuses to work and proposes to give this argument.
Exported non-empty variable MERGE_DATE enforces a particular
date string in the text which gets prepended to /README.txt .
Exported non-empty variable XORRISO overrides command xorriso,
which may be needed if installed xorriso is older than 1.4.2.
Example using GNU xorriso-1.5.4 instead of /usr/bin/xorriso:
  export XORRISO=$HOME/xorriso-1.5.4/xorriso/xorriso
  merge_2_debian_isos debian-11.2.0-amd64-DVD-1.iso /mnt/iso1 \
  debian-11.2.0-amd64-DVD-2.iso /mnt/iso2 merged.iso
===

The /README.txt of the result ISO gets a prefix text:
===
Result of a run of merge_2_debian_isos at 20220523-20:09
Package pools and Packages lists were merged.
The other files stem from the first input ISO.

Input ISO: debian-11.2.0-amd64-DVD-1.iso
  Debian GNU/Linux 11.2.0 "Bullseye" - Official amd64 DVD Binary-1
   20211218-11:13

Input ISO: debian-11.2.0-amd64-DVD-2.iso
  Debian GNU/Linux 11.2.0 "Bullseye" - Official amd64 DVD Binary-2
   20211218-11:13

 --
 [... text of boot_iso's README.txt ...]

===

The result of merging debian-11.2.0-amd64-DVD-[12].iso boots with
  qemu-system-x86_64 -enable-kvm -m 512 -hda merged.iso
to a boot loader menu with Debian logo.

I did not go further with installation yet, mainly because i still lack
ideas and experience how i would verify that the ISO's main repo knows
6788 *.deb files instead of 4753 in DVD-1 and 2035 in DVD-2.

To ease usage and to avoid quadratic behavior (actually triangular), i plan
to beef up the script so that it can merge more than 2 ISOs in one run.
E.g. for downloading all 19 11.3.0 amd64 DVDs and merging them.
(19 /dev/loop* should be no problem, i hope.)


Zhang Boyang wrote:
> I came up with a idea. Maybe you can use 'debian-cd' to create a DLBD set,
> say disc A1 and A2, then create another ALL-IN-ONE set, say disc B. Then
> compare if A1+A2==B. There might be small differences, like the package
> order in Packages.gz, but I think if the overall format is OK, then it will
> be OK.

I fear that several of the tasks in debian-cd are beyond my talents.
(I am about the third worst sysadmin which i know of.)

In the end it is about whether the merged ISO works fully or not.


Have a nice day :)

Thomas

Bug#1011343: WISHLIST: Offical ALL-IN-ONE images?

[Thomas Schmitt]

Hi,

i wrote:
> > I began to ponder about a shortcut in libisofs which would trust the
> > checksum file (-checksum-list , -md5-list) enough to omit the reading of
> > all the package files' content.

Steve McIntyre wrote:
> Cute idea (grin!), but it's a non-started - we wouldn't be able to
> generate the various checksums for the whole image.

Yeah. Ain't it sad ?


> > Would more than one empty line between paragraphs damage the readability ?

> Not sure, to be honest.

So i will have to let my script think more. (It's at 287 lines, meanwhile.)


> Might just work, yeah!

My script does meanwhile:
- Take as arguments the paths of
  boot_iso  boot_mount_directory  add_iso  add_mount_directory  result_iso
- Mount the bootable ISO (e.g. debian-11.3.0-amd64-DLBD-1.iso) and the
  add-on ISO (e.g. debian-11.3.0-amd64-DLBD-2.iso).
- Determine the release name $dist of the bootable ISO (e.g. "bullseye").
- Copy /dists and /md5sum.txt from bootable ISO to local working directory.
- Merge the /md5sum.txt of both ISOs.
- Read the paths from first checksummed file lists of both
  /dists/$dist/Release files, sort and uniq them. (I don't bet that both
  have the same list, although DVD-1 and DVD-2 have the same.)
- Merge the namesake .gz files from that list (if both exist) or copy .gz
  file from the add-on ISO (if not yet in bootable ISO).
- Create a new file /dists/$dist/Release with the header lines from the
  file in the bootable ISO and freshly computed checksums of the merged
  files.
- Produce the bootable result_iso, overwriting /dists and /md5sum.txt
  by the altered copies from the local directory.

This can be repeated by using result_iso as new boot_iso and another add_iso
and a new result_iso name.



How can i verify that the resulting ISO properly announces all its
packages ?
(If i install it to a VM, what must i do to challenge its completeness ?)


Have a nice day :)

Thomas

Bug#1011343: WISHLIST: Offical ALL-IN-ONE images?

[Thomas Schmitt]

Hi,

Steve McIntyre wrote:
> youre initial guess is correct. We don't generate the .iso files
> at all for the larger images [1]. This means we also don't have
> torrent files for them [2].

I began to ponder about a shortcut in libisofs which would trust the
checksum file (-checksum-list , -md5-list) enough to omit the reading of
all the package files' content.
Size, ownership, permissions, et.al would still be taken from the package
files on disk. No ISO image would emerge (because of no valid file content)
but .jigdo and .template would be created.
Probably libjte would need an API extension so that it knows that only
the count parameter of a libjte_show_data_chunk() call is valid.
Vice versa libisofs would have to ask libjte whether a particular file
is covered by the checksum list.

All tricky and probably not worth the risk of embarassing failure.


So back to my idea of merging ISOs:

> The debian-cd code in tools/make_disc_trees.pl is not documentation
> **as such**, but it's exactly how we create disc trees:

I am using it now for checking detail questions.


> It's baiscally just making a self-contained apt repository on each medium.

So
  https://wiki.debian.org/DebianRepository/Format
looks like the specs to follow.

Question (to everybody):

The description of Packages[.gz] files talks of "paragraphs" but does not
exactly define a paragraph's end delimiter. From Packages.gz in the ISO
i'd guess it is an empty line or the "Package" field of the next paragraph.

Is an empty line needed between paragraphs ?
Would more than one empty line between paragraphs damage the readability ?

Reason: I want to merge the Packages.gz files like

  (gunzip temp_file

but am not sure that the found Packages.gz will always end by an empty
line.  So i could simply insert an echo between the gunzips, or simply
trust that the empty line is not needed as separator, or begin to think ...


Have a nice day :)

Thomas

Bug#1011343: WISHLIST: Offical ALL-IN-ONE images?

[Thomas Schmitt]

Hi,

Zhang Boyang wrote:
> I guess there is no ready tool can merge two ISOs together.

Not directly. But xorriso can load the meta-data of an ISO, manipulate
that loaded model, and write the result as new ISO with the same boot
equipment as found in the loaded ISO.
As said, that would be no problem. Only enough disk space is needed, of
course, and a Debian version >= 9 or locally built GNU xorriso >= 1.4.2.


> After a quick look on dists/,
> I think these files are in same format with theses on online mirrors, so
> referring apt's documentation might be useful.

Searching "Debian Packages.gz" brought me to
  https://wiki.debian.org/DebianRepository/Format
which gives some hope for enlightenment.
I will try to merge DVD-1 and DVD-2 but then need a way to verify that
the result is fully functional.

It looks like i have to merge these files from DVD-1:

  ./dists/bullseye/contrib/binary-amd64/Packages.gz
  ./dists/bullseye/main/binary-amd64/Packages.gz
  ./dists/bullseye/main/debian-installer/binary-amd64/Packages.gz

  ./dists/bullseye/contrib/binary-amd64/Release
  ./dists/bullseye/main/binary-amd64/Release
  ./dists/bullseye/main/debian-installer/binary-amd64/Release
  ./dists/bullseye/Release

with these from DVD-2:

  ./bullseye/contrib/binary-amd64/Packages.gz
  ./bullseye/main/binary-amd64/Packages.gz

  ./bullseye/contrib/binary-amd64/Release
  ./bullseye/main/binary-amd64/Release
  ./bullseye/main/debian-installer/binary-amd64/Release
  ./bullseye/Release

Bystanders: Do i miss something yet ?

--

To demonstrate the rest of my sketch i tried this with DVD instead of DLBD
(xorriso is GNU xorriso-1.5.2, equivalent to xorriso in Debian 11):

  DLBD_1=debian-11.2.0-amd64-DVD-1.iso
  DLDB_1_MOUNT=/mnt/dlbd_1
  DLDB_2=debian-11.2.0-amd64-DVD-2.iso
  DLDB_2_MOUNT=/mnt/dlbd_2
  RESULT=ALL_IN_ONE.iso

  sudo mkdir "$DLDB_1_MOUNT" "$DLDB_2_MOUNT"
  sudo mount "$DLBD_1" "$DLDB_1_MOUNT"
  cp -a "$DLDB_1_MOUNT/dists" merged_dists
  sudo umount "$DLDB_1_MOUNT"
  chmod -R u+w merged_dists
  sudo mount "$DLBD_2" "$DLDB_2_MOUNT"

  # TODO:
  # Merge the files in merged_dists with those from "$DLDB_2_MOUNT"

  xorriso -indev "$DLBD_1" \
  -outdev "$RESULT" \
  -map "$DLDB_2_MOUNT"/pool /pool \
  -map merged_dists /dists \
  -chown_r 0 /dists -- \
  -chgrp_r 0 /dists -- \
  -chmod_r a-w /dists -- \
  -boot_image any replay \
  -blank as_needed \
  -stdio_sync off \
  -padding included \
  -compliance no_emul_toc

  sudo umount "$DLDB_2_MOUNT"

  xorriso -indev "$RESULT" -report_el_torito plain -report_system_area plain

yields the typical boot jackalope (still escapes more coyotes than any
other ISO partition layout):

  El Torito catalog  : 4301  1
  El Torito cat path : /isolinux/boot.cat
  El Torito images   :   N  Pltf  B   Emul  Ld_seg  Hdpt  Ldsiz LBA
  El Torito boot img :   1  BIOS  y   none  0x  0x00  45598
  El Torito boot img :   2  UEFI  y   none  0x  0x00   51844302
  El Torito img path :   1  /isolinux/isolinux.bin
  El Torito img opts :   1  boot-info-table isohybrid-suitable
  El Torito img path :   2  /boot/grub/efi.img
  System area options: 0x0202
  System area summary: MBR isohybrid cyl-align-off GPT APM
  ISO image size/512 : 16754652
  Partition offset   : 0
  MBR heads per cyl  : 0
  MBR secs per head  : 0
  MBR partition table:   N Status  TypeStart   Blocks
  MBR partition  :   1   0x80  0x000 16754652
  MBR partition  :   2   0x00  0xef17208 5184
  MBR partition path :   2  /boot/grub/efi.img
  GPT:   N  Info
  GPT disk GUID  :  657a86211710b54c8e25101781372d9f
  GPT entry array:  12  208  overlapping
  GPT lba range  :  64  16754598  16754651
  GPT partition name :   1  490053004f00480079006200720069006400
  GPT partname local :   1  ISOHybrid
  GPT partition GUID :   1  657a86211710b54c8e24101781372d9f
  GPT type GUID  :   1  a2a0d0ebe5b9334487c068b6b72699c7
  GPT partition flags:   1  0x1001
  GPT start and size :   1  0  16754596
  GPT partition name :   2  490053004f004800790062007200690064003100
  GPT partname local :   2  ISOHybrid1
  GPT partition GUID :   2  657a86211710b54c8e27101781372d9f
  GPT type GUID  :   2  a2a0d0ebe5b9334487c068b6b72699c7
  GPT partition flags:   2  0x1001
  GPT start and size :   2  17208  5184
  GPT partition path :   2  /boot/grub/efi.img
  APM:   N  Info
  APM block size :  2048
  APM gap fillers:  0
  APM partition name :   1  EFI
  APM partition type :   1  Apple_HFS
  APM start and size :   1  4302  1296
  APM partition path :   1  /boot/grub/efi.img

For details of the xorriso run, see man xorriso.
For the report format, see
  xorriso -report_el_torito help -report_system_area help | less


Have a nice 

Bug#1011343: WISHLIST: Offical ALL-IN-ONE images?

[Thomas Schmitt]

Hi,

i reply to 1011...@bugs.debian.org and zhangboyang...@gmail.com because
of my question and sketch for a procedure to merge DLBD-1 and -2 after
download into the desired ALL-IN-ONE ISO.


Andrew M.A. Cater wrote to debian-cd@lists.debian.org:
> If the original poster wants one huge .iso as one file to download from
> cdimage.debian.org - then 2 x double layer Blu-Ray (say) as one file
> would be 100GB or so.

11.3.0 is at (48245475328+35798159360)/1073741824 = 78.272 GiB.

  $ gunzip   Even on a good quality link, that's quite a time.

Yes. Especially since at least with jigdo-lite the bottleneck is the
latency of downloading the packages. A 78 GiB ISO might need a day to
become complete. jigdo-lite is graceful about being interrupted and resumed
the other day, though.

Running several jigdo-lite downloads simultaneously might mitigate the
latency problem so that bandwidth becomes the bottleneck.


> I don't actually know how long it take to create the .jigdo and .template
> files - I don't think it's as long as generating the full .iso files

They are created by libjte under control of libisofs under control of
xorriso as side effect of actual ISO production under control of debian-cd.
(I'm developer of libisofs and xorriso, and co-funder of libjte, which is
now back in the hands of Steve McIntyre from whose genisoimage code it
got large parts of its entrails.)

I was possibly wrong with guessing that the ISO is dumped into /dev/null,
although debian-cd seems to be smart enough to read the various checksums
from the .jigdo files, rather than calculating them from the .iso files.
The checksum code
  https://sources.debian.org/src/debian-cd/3.1.35/tools/imagesums/#L57
reminded me that there are .torrent files made, which obviously need
the .iso files at least at build time.

-

> it's always good to have otehr people to think round a problem with.

Is there documentation from which i could learn how the stuff in (i guess)
/dists of DLBD-1 and DLBD-2 could be merged so that it properly describes
a merged pool tree ?

My rough idea would be:
- mount both ISOs
- derive merged /dists files
- run xorriso to let it
  - load DLBD-1 with its boot equipment
  - merge-in the pool tree of mounted DLBD-2
  - overwrite the old /dists files by the newly derived ones
  - automatically replay the commands for the loaded boot equipment
  - store the result as new .iso file

The second step is where i would need info or advise.


Have a nice day :)

Thomas

Bug#1011343: WISHLIST: Offical ALL-IN-ONE images?

[Thomas Schmitt]

Hi,

some technical nitpicking.

Andrew M.A. Cater wrote:
> Also, keeping large files around on disk for a long
> time - there's some likelihood of data corruption.

The .jigdo and .template files of the DLBD ISOs are together smaller than
a netinst CD ISO. (Less than 90 MiB, see
https://cdimage.debian.org/debian-cd/current/amd64/jigdo-dlbd/ )

Building an ALL-IN-ONE would cost the computing time of another ISO set (*)
and 50 % percent more virtual memory than building DLBD1 (**).

I assume that the main workload with an additional set of jigdoized ISOs
is the need to once more shovel 75 GB of package files through libisofs
and libjte so that they can create .template and .jigdo files. The ISOs
themselves get directly piped into /dev/null, i guess.


> I'd hate a couple of
> bit flips three quarters of the way through a 6TB file, say, to mean that
> the whole thing isuseless.

The vast majority of data is stored as packages on the worldwide mirrors.
I'd expect some quality of filesystem and backup which keeps damage
confined to a few packages.

There are a lot of outdated mirrors around where one could dig for those
which really got lost from all active mirrors. I remember the hunt for
a package which once was overwritten by a newer version with the same
.deb file name. It was needed for building an old ISO from .jigdo which
was created when the new version did not yet exist.
But even without finding the older version, the emerging ISO would have
been usable for all purposes which don't touch that one rogue package.



Every time this wish pops up, i begin to ponder what is theoretically
needed to unite several pool trees from multiple ISOs into one, so that
it works like an official ALL-IN-ONE ISO.

Putting all files together and making the ISO bootable would be no
problem. But what does a neat pool have to offer as merged lists or other
meta-data so that it properly announces its content ?



Footnotes:
(*) Computing time:
Maybe it needs a bit more than DLBD[12] together, because the insertion
algorithm of libisofs has a quadratic aspect in its personality. But
the branchy pool tree helps a lot to keep this problem under the cover.
(**) Virtual memory consumption:
The memory consumption of libisofs mainly depends on the number of
files and the length of their names. *11.3.0-amd64-DLBD-[12].jigdo
together list ~ 59,000 files with ~ 2.2 MiB of basename length.
My daily BD backups have ~ 70,000 files with ~ 1.4 MiB of basename
length and do not demand gigabytes of RAM.


Have a nice day :)

Thomas

Re: problem with DVD installation

[Thomas Schmitt]

Hi,

J.A. Bezemer wrote:
> When the CD is scanned properly using this process, but the DVD fails, then
> your DVD is bad. Maybe the DVD has bad data on it (corrupted download), or
> the DVD burner is bad, or the DVD reader is bad.

I agree in principle. But the particular reason for the firmware's refusal
to consider booting the DVD should be found out.

So i would still be interested in seeing the output of

  xorriso -indev /dev/sr0 -toc -report_el_torito plain

while the DVD is in the drive, and of

  xorriso -outdev /dev/sr0 -list_profiles

regardless whether the medium is loaded or not.
(If there is more than one optical drive, the address might be /dev/sr1 or
/dev/sr2.)


Have a nice day :)

Thomas

Re: problem with DVD installation

[Thomas Schmitt]

Hi,

Gianmarco wrote:
> -> I have tried with: 10.3, 11.1, 11.2 and 11.3.
> [...]
> -> my results (with DVD-1 debian version 11.2):
> 1903144+0 record in
> 1903144+0 record out
> 3897638912 byte (3,9 GB) copied, 260,578 s, 15,0 MB/s
> d1fc0ddc81d980b9eddc9d110344bcf17a6cbd5750e147112ccc23bef4d61a8a  -

The DVD is readable at least as far it is needed for the BIOS to find
the EL Torito boot record, the catalog, and the boot images.
But the SHA256 does not match the one of
  
https://cdimage.debian.org/mirror/cdimage/archive/11.2.0/amd64/iso-dvd/debian-11.2.0-amd64-DVD-1.iso

Checking the DVD-1 of 10.3, 11.1, 11.3 would need other count= arguments
for dd. They would have to be derived from their image sizes or from
inspecting the DVDs.
See below.


> I just wanted to tell you that with a 2021-machine, of a friend of mine,
> we tried to download the ISO file (DVD-1) from this 2021-machine, always
> burn it in his 2021-machine (windows-OS) and we always tried to launch the
> installation from the same machine. the result was for me unexpectedly the
> same as all the other tests I performed.

I still believe that the DVDs are somehow bad.

Please inspect those which you still have by:

  xorriso -indev /dev/sr0 -toc -report_el_torito plain

and report the resulting information lines.

With the DVD+RW to which i burnt debian-11.2.0-amd64-DVD-1.iso i get:

  xorriso : NOTE : Loading ISO image tree from LBA 0
  xorriso : UPDATE :  80 nodes read in 1 seconds
  ...
  xorriso : UPDATE :9791 nodes read in 8 seconds
  xorriso : NOTE : Detected El-Torito boot information which currently is set 
to be discarded
  Drive current: -indev '/dev/sr0'
  Media current: DVD+RW
  Media status : is written , is appendable
  Boot record  : El Torito , MBR isohybrid cyl-align-on GPT APM
  Media summary: 1 session, 1903144 data blocks, 3717m data,  766m free
  Volume id: 'Debian 11.2.0 amd64 1'
  Drive current: -indev '/dev/sr0'
  Drive access : exclusive:unrestricted
  Drive type   : vendor 'HL-DT-ST' product 'BDDVDRW GGC-H20L' revision '1.03'
  Drive id : '... nobody needs to know ...'
  Media current: DVD+RW
  Media product: MBIPG101/W04/48 , Moser Baer India Limited
  Media status : is written , is appendable
  Media blocks : 1903168 readable , 391936 writable , 2295104 overall
  Boot record  : El Torito , MBR isohybrid cyl-align-on GPT APM
  Boot catalog : '/isolinux/boot.cat'
  Boot image   : '/isolinux/isolinux.bin' , boot_info_table=on
  Boot image   : '/boot/grub/efi.img' , platform_id=0xEF
  ISO offers   : Rock_Ridge Joliet
  ISO loaded   : Rock_Ridge
  TOC layout   : Idx ,  sbsector ,   Size , Volume Id
  ISO session  :   1 , 0 ,   1903144s , Debian 11.2.0 amd64 1
  Media summary: 1 session, 1903144 data blocks, 3717m data,  766m free
  Media nwa: 1903168s
  El Torito catalog  : 5946  1
  El Torito cat path : /isolinux/boot.cat
  El Torito images   :   N  Pltf  B   Emul  Ld_seg  Hdpt  Ldsiz LBA
  El Torito boot img :   1  BIOS  y   none  0x  0x00  47243
  El Torito boot img :   2  UEFI  y   none  0x  0x00   51845947
  El Torito img path :   1  /isolinux/isolinux.bin
  El Torito img opts :   1  boot-info-table isohybrid-suitable
  El Torito img path :   2  /boot/grub/efi.img

The size to be used with dd's count= is given in the lines

  TOC layout   : Idx ,  sbsector ,   Size , Volume Id
  ISO session  :   1 , 0 ,   1903144s , Debian 11.2.0 amd64 1
  Media summary: 1 session, 1903144 data blocks, 3717m data,  766m free

Other ISOs will show different numbers than 1903144.

Older xorriso versions might omit some of the shown lines. But the
assessment of node numbers (files and directories), of ISO session size,
of volume id, and of El Torito boot images is supposed to be the same as
in my above example.

You will probably get other info in these lines:

  Media current: DVD+RW
  Media product: MBIPG101/W04/48 , Moser Baer India Limited

It will be interesting to see what DVD type from which manufacturer you
used.

---

Sorry for my lack of more particular ideas about what goes wrong.
The fact that the other subscribers to this mailing list are not offering
own ideas might mean that they currently have none.

To the bystanders:
A test with an old BIOS machine and a Debian 11.3 DVD-1 would nevertheless
be helpful. I can currently only test the EFI boot path.

---

To get you going with Debian 11:

Would a USB stick of at least 4 GB capacity be available for being completely
overwritten ?

If so, i propose (with the usual warnings not to overwrite your hard disk
if it happens to be /dev/sdc):

  # Make a backup of the complete USB stick which i assume to be /dev/sdc
  usb_stick_dev=/dev/sdc
  usb_stick_backup="$HOME"/usb_stick_backup.gz
  dd if="$usb_stick_dev" bs=1M | gzip > "$usb_stick_backup"

  # Copy ISO image t

Re: problem with DVD installation

[Thomas Schmitt]

Hi,

i wrote:
> > What ISO exactly did you use ?

Gianmarco wrote:
> i downloaded ISO files from here (i have tried both format):
> https://cdimage.debian.org/debian-cd/current/source/iso-dvd/

That's 11.3.0 meanwhile. But as you report to have used 11.2, i assume that
it is the same ISO as i downloaded from the archives and tested a few hours
ago.


> I used brasero, for example.
> I selected: burn ISO image.

That's the correct choice.

If you still have the 11.2 DVD:
What do you get from

  dd if=/dev/sr0 bs=2048 count=1903144 | sha256sum

I get

  022370f066bc91b2cdac3837ff5fa9f3822c5afb2fc34f68084416079fe5a408  -

as with debian-11.2.0-amd64-DVD-1.iso from which i got the count of
1903144 blocks (dividing 3897638912 bytes by block size 2048).


> > So it did not actively offer you the DVD but rather you told it to prefer
> > DVD over hard disk ?

> yes: what you wrote is right.

All in all this looks like the DVD drive does not tell the BIOS that a
medium is loaded or that the BIOS does not recognize the El Torito boot
record and the boot catalog on the medium.
(But exactly this stuff has not changed between Debian 8 and 11.)

Insofar it would be very interesting to see what above dd | sha256sum
yields.


Have a nice day :)

Thomas

Re: problem with DVD installation

[Thomas Schmitt]

Hi,

i wrote:
> > So does the following small (251 MiB) ISO here work better ?
> >   
> > https://cdimage.debian.org/mirror/cdimage/archive/8.11.1/amd64/iso-cd/debian-8.11.1-amd64-netinst.iso

Gianmarco wrote:
> i have already installed the jessie version of linux some years ago in one
> of the pc where im try now to install 11.2.
> [...]
> but some days ago i tried to install debian jessie newly in the same pc and
> the installation was very good.

What ISO exactly did you use ?

By what program did you put it on the DVD ?
Please describe either command line options or the GUI menu actions which
you selected.


> the bios here is very simple.

So it did not actively offer you the DVD but rather you told it to prefer
DVD over hard disk ?

Since you say it's old BIOS, does this BIOS-only ISO boot:

  
https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-mac-11.3.0-amd64-netinst.iso

(The name "mac" is misleading and refers to a family of Apple computers
which boot by BIOS but refuse if the EFI equipment is advertised.)


The reason for my questions is to find out what works and what does not.
There were some changes between Debian 8 and 11. But the boot lures, which
lead the firmware to the boot images, have not changed.
So your report is quite a riddle, given that nobody else seems to have
experienced a similar refusal of multiple machines.


Have a nice day :)

Thomas

Re: problem with DVD installation

[Thomas Schmitt]

Hi,

Gianmarco wrote:
> maybe you are telling me that i have to use grub to start the DVD ?

No. GRUB is the bootloader in the ISO which gets started by EFI if it
is not in legacy emulation mode and thus would start the ISOLINUX bootloader
which is in the ISO too.


> - I set the bios with the boot from DVD.

I assume that the newer machine has EFI as firmware.
So it offers the DVD to you in its boot menue ?

In my ASUS machine's EFI i then double click the item and GRUB gets started
by the EFI firmware.


> - the system went straight into the pre-installed OS boot.

It shouldn't do this.
If it offers the DVD as boot opportunity then it should start one of the
bootloader images which are mentioned in the El Torito catalog of the ISO.

If the firmware believes to be a legacy BIOS (or emulates it) then it
should start the ISOLINUX boot program.
If it believes to be native EFI, then it should look into the boot image
for EFI, which is actually a FAT filesystem. In that filesystem it will
find /EFI/BOOT/BOOTX64.EFI which it then will run.
Both boot loaders will then present their boot menus which bring you
to the first Linux kernel and the installation software.

--

Are you sure older ISOs boot better on your machines ?
You mentioned Jessie.
So does the following small (251 MiB) ISO here work better ?

  
https://cdimage.debian.org/mirror/cdimage/archive/8.11.1/amd64/iso-cd/debian-8.11.1-amd64-netinst.iso

You can put it on DVD, although it's a CD sized ISO.


Have a nice day :)

Thomas

Re: problem with DVD installation

[Thomas Schmitt]

Hi,

> in the case of debian 11.2 no computer, not even the most recent one, can
> start with the installation directly from the 1st DVD.

Well, at least for me with a halfways modern EFI it boots to some Debian
software.

- What kind of firmware did you try ? What mainboards or laptop models ?
- Is always the same DVD drive used for the installation attempt ?
  (E.g. because is it attached via USB.)
- How far does the DVD boot for you ? Do you see indications that GRUB
  or ISOLINUX (in case of legacy BIOS) was started ?

---
What i tested:

I downloaded

  
https://cdimage.debian.org/mirror/cdimage/archive/11.2.0/amd64/iso-dvd/debian-11.2.0-amd64-DVD-1.iso

verified its SHA256
  022370f066bc91b2cdac3837ff5fa9f3822c5afb2fc34f68084416079fe5a408
and burnt it onto a DVD+RW

  xorriso -as cdrecord dev=/dev/sr5 -v -eject fs=32m 
debian-11.2.0-amd64-DVD-1.iso

Then i put the ISO into a Xeon machine with an ASUS board of 2020.
The EFI firmware offers me in its boot menu the model name of the DVD drive
for booting. I double click (it's that kind of EFI with icons and animated
CPU fans) and quite immediately see the word "GRUB" on the screen.
Then a GRUB menu appears which offers me to install. I rather choose
Advanced Options and Rescue Mode. All seems well until i want to go
back to choose another country.
It begins to talk about "Installation" which i don't want. So i press
the hardware Reset button and rather check whether my Debian on that
machine is still operational. It is. Still with fvwm. What a relief.


Have a nice dyay :)

Thomas

Re: non-free firmware unofficial BD jigdo files missing

[Thomas Schmitt]

Hi,

Steve McIntyre wrote:
> We don't provide firmware-included
> versions of every media type, just netinst and DVD.

sp...@caiway.net wrote:
> > Perhaps you can give me the link with info how to make my own files
> > *.jigdo & *.template

> For that, you'll need a full local mirror and use debian-cd.

How about adding the firmware files to an existing BD ISO by means of
xorriso ?

  xorriso -indev debian-11.2.0-amd64-DLBD-1.iso \
  -outdev my-firmware-11.2.0-amd64-DLBD-1.iso \
  ... add firmware files at suitable position in ISO tree ... \
  -boot_image any replay \
  ... jigdo production options ...

or like in
  https://wiki.debian.org/RepackBootableISO
(without removing the jigdo options, of course).

How difficult would it be to use such added firmware when the ISO is
booted ?
(I remember that, a few years ago, some Debian ISO had an appended
partition for storing firmware files, which were to be used when the ISO
was booted from a USB stick.)

I offer help with the xorriso part. But the Debian installer part would
need other experts.


Have a nice day :)

Thomas

Re: I Need Dvd Image Debian 10 Complete Edition, please, HELP-ME!

[Thomas Schmitt]

Hi,

Raphael Maria wrote:
> At the moment I need a DVD Image release of Debian 10, for a specific
> project that requires it, but I can't find it on the website.

Those ISOs are in the archive now:
  https://cdimage.debian.org/mirror/cdimage/archive/10.11.0/

The first three amd64 DVDs are available directly via https:
  https://cdimage.debian.org/mirror/cdimage/archive/10.11.0/amd64/iso-dvd/
Via jigdo you can get the full set of 16 DVDs:
  https://cdimage.debian.org/mirror/cdimage/archive/10.11.0/amd64/jigdo-dvd/
See:
  https://www.debian.org/CD/jigdo-cd/


Have a nice day :)

Thomas

Re: Debian 11.1.0 DVD 2,3,4,5 ?

[Thomas Schmitt]

Hi,

Christoph Prokop wrote:
> Is it on purpose only DVD1?

Yep. Stated by Andrew M.A. Cater in
  https://lists.debian.org/msgid-search/yvwb9cz42+gxr...@einval.com

> > [...] the number of DVDs was [...] reduced for Debian 11. Whereas
> > for Buster we made the first three .iso images, for Bullseye there is one:
> > the others can be built using jigdo.

That would be

  https://cdimage.debian.org/debian-cd/current/amd64/jigdo-dvd/

.jigdo and .template files are to be used as described in
  https://www.debian.org/CD/jigdo-cd/#how

I wrote a wiki article how to get to a Live Debian, install jigdo-lite
on it for the time it is running, and use it to download one or more
Jigdo ISOs:
  https://wiki.debian.org/JigdoOnLive
Maybe you find inspiration in there if riddles arise ...


> This Site lists 1-5 (but i am not sure who provides theses isos):
> Is this even an official mirror?
> http://debian-cd.inspire.net.nz/current/amd64/iso-dvd/

Hard to say. Maybe they are just nice and did the jigdo work for you.

If you can afford the bandwidth to download the ISOs and to discard them
if they are bad, then check the downloaded ISOs by the checksums in
  https://cdimage.debian.org/debian-cd/current/amd64/jigdo-dvd/SHA256SUMS
and/or
  https://cdimage.debian.org/debian-cd/current/amd64/jigdo-dvd/SHA512SUMS
I.e. _not_ by the checksums on debian-cd.inspire.net.nz .

A cautiously mistrusting downloader will want to verify the checksum
files by their .sig files and a run of gpg --verify.


Have a nice day :)

Thomas

Re: isorecorder

[Thomas Schmitt]

Hi,

Steve McIntyre wrote:
> [Dropping CC to Lou as we've wandered, re-adding Thomas for the GPT
> question.]

[Lou and i are subscribed to debian-cd, obviously. :))
It seems you dropped the original poster: The7up 
who might indeed not be interested in partitioning.]


> For the
> sake of this kind of use-case, is it sensible when driving xorriso to
> tweak the created partition table to list the ESP as partition 1? At
> the moment we're just using "-append_partition 2 0xef
> /path/to/ESP.img", would just changing that to be partition 1 instead
> work?

That would still work, but leave the ISO fileystem without a partition
entry to mark it.

The man page of xorrisofs says:

partition_number may be 1 to 4. Number 1 will put the whole  ISO
image  into  the unclaimed space before partition 1.

Minimalistic experiment:

  xorriso -as mkisofs -o test.iso /bin/ls -append_partition 1 0xef /bin/cp

yields (with /bin/ls as ISO payload having ~ 120 KB and /bin/cp as
appended payload having ~150 KB):

  $ sbin/fdisk -l test.iso
  ...
  DeviceBoot Start   End Sectors  Size Id Type
  test.iso1364   659 296  148K ef EFI (FAT-12/16/32)

--

I could tweak libisofs so that the ISO filesystem can appear as partition 2
but that would be unusual by partition 2 sitting before partition 1.

On the other hand, there is urgent need for an ISO filesystem partition
only if the reader cannot mount the base device but only its partitions.


Have a nice day :)

Thomas

Re: isorecorder

[Thomas Schmitt]

Hi,

Pete Batard wrote:
> Yes, as we've discussed elsewhere, that is pretty much my point.

That's why i summarized the results of our wiggling and fitting with
Ubuntu ISO development.


> symlinks [...] will of course not translate to FAT [...]

Duh. I forgot that aspect in my summary.
So its proposal part should be:

Both approaches are combinable:
- Have all EFI equipment as directory tree in the ISO filesystem. I.e.
  not only as image file or as appended partition.
- Make sure that the installation works without name conflicts from a
  filesystem with sloppy naming rules and without symbolic links.
  I.e. from FAT.


As Pete pointed out in the Ubuntu discussions, name length is not
restricted to 8.3. Nevertheless very long package names can cause truncation
which might become a problem if two names get truncated to the same text,
or if truncation implies mangling to avoid such collisions.


Have a nice day :)

Thomas

Re: isorecorder

[Thomas Schmitt]

Hi,

the advantages of putting an unmodified ISO image plainly onto a USB are:
- The boot and installation process is as near to DVD as possible.
- If anything goes wrong during that process, it is clear that debian-cd
  is in charge of diagnosing and hopefully managing the effort to provide
  a correction.

Nevertheless, the goals of Pete are valid too:
- A USB stick with large FAT partition is what many users expect.
- The (unspecified by UEFI) way of copying EFI boot equipment and data
  payload into a FAT filesystem seems to be indeed the way how experienced
  users of MS-Windows prepare a bootable USB stick.
- Copying from filesystem to filesystem is at least one order of magnitude
  less prone to fatal mishaps than is copying onto a device.

Both approaches are combinable:
- Have all EFI equipment as directory tree in the ISO filesystem. I.e.
  not only as image file or as appended partition.
- Make sure that the installation works without name conflicts from a
  filesystem with sloppy naming rules. I.e. from FAT.

And to promote my own song:

Debian ISOs should strive for a specs compliant partition table with as
few borderline hacks as possible.

It would be fully compliant if bearing a MBR partition table with disjoint
ISO and EFI partitions. But during the Ubuntu ISO reform there were
problems with new Lenovos which booted only with GPT
  https://bugs.launchpad.net/ubuntu-cdimage/+bug/1886148
and with old HPs which are well known to boot only if some MBR partition
bears the boot flag
  https://bugs.launchpad.net/ubuntu-cdimage/+bug/1899308

The Lenovo case is probably not fully explored, because it seems that
the same machines boot from our current abominable MBR-with-invalid-GPT
layout. The HP case needed a small hack by adding an MBR partition of
type 0x00 and size 1 which bears the boot flag.
I am not aware of failure reports with the Ubuntu 20.10 ISOs which could
be blamed on their partition layout.


Have a nice day :)

Thomas

Re: isorecorder

[Thomas Schmitt]

Hi,

The7up originally wrote:
> ... > It looks like isorecorder not available since a while ago. May I
> ... > remove it from the list in w.d.o/faq/CD index page?
> ... > At the same time, rufus (https://rufus.ie/ [rufus.ie]) should be
> ... > mentioned as a great app on Windows to make bootable usb stick from
> ... > iso (GNU licensing, source code available on GitLab)

Lou Poppler wrote:
> > > If you want to recommend rufus, please do so ONLY with a prominent, 
> > > un-ignoreable warning explaining that the user MUST select rufus'
> > > "DD mode" at the prompt immediately before writing the media.

i wrote:
> > I understand that manual selection of this mode has been removed in
> > favor of automatic detection of dd candidates:
> >   https://github.com/pbatard/rufus/issues/1148#issuecomment-395562137

Lou Poppler wrote:
> I corresponded with rufus's author Pete Batard, including some logs from
> #debian discussing my testing of rufus, and he kindly returned a lengthy
> response to my email.

On Thu, 1 Oct 2020, Pete Batard  wrote:
> > > > Now the one thing that might have thrown this user off is that Rufus 
> > > > always prompts to download these components BEFORE it asks to choose 
> > > > between ISO and DD mode,

Then i misunderstood that GitHub issue post from 2018.

So yes, the urge to use DD mode should be very visible within the advise
to copy a Debian amd64, i386, or arm64 ISO flatly to an USB-Stick.

I know that Pete Batard prefers other methods. But the flat copy is what
our ISOs strive for on the first hand.


Pete Batard wrote:
> > > > DD-mode image writing does often throw 
> > > > Windows people off, as they think that their drive has been reduced in 
> > > > size to just the ESP since Windows will not mount the main, larger 
> > > > installation partition.

I wonder whether that problem is resolved by the new layout of Ubuntu 20.10
which is a GPT variation of my proposals for debian-cd:
  https://lists.debian.org/debian-cd/2019/07/msg7.html

To my theory, the unwillingness of MS-Windows to show the ISO partition of
the current Debian ISO layout is in its partition type 0x00, which is
needed to keep the nested MBR partition layout legal for EFI.
My proposed layout without nested partitions makes it possible to have
a conventional partition type for the ISO partition or to use a GPT instead
of an MBR partition table.


Have a nice day :)

Thomas

Re: isorecorder

[Thomas Schmitt]

Hi,

Lou Poppler wrote:
> If you want to recommend rufus, please do so ONLY with a prominent, 
> un-ignoreable warning explaining that the user MUST select rufus'
> "DD mode" at the prompt immediately before writing the media.

I understand that manual selection of this mode has been removed in
favor of automatic detection of dd candidates:
  https://github.com/pbatard/rufus/issues/1148#issuecomment-395562137

I don't use Rufus myself but it is my answer to users of MS-Windows when
they ask what to do with an ISO that is bootable from USB-stick.


Have a nice day :)

Thomas

Re: Porting the standard image from live-wrapper to live-build

[Thomas Schmitt]

Hi,

Roland Clobus wrote:
> * xorriso complains about issues with Joliet (symlinks not supported,
> volid too long)

Joliet uses 16-bit characters which halves the field length in the
supplementary volume descriptor of Joliet. So no workaround is possible
other than shortening the volume id.

If there is knowledge how to represent symbolic links in Joliet so that
MS-Windows understands them, i'd be willing to work on an implementation
in libisofs.


> ** AP: Do we need support for Joliet? Untested: does Windows XP/7/10
> support RockRidge sufficiently well? And is it needed?

I am not aware that any MS-Windows system supports Rock Ridge natively.
There might be add-on software which can.

Having Joliet filenames of up to 64 characters length is better than
having ISO 9660 names of limited charset and max length 31.
It opens a better path to a copy method onto USB stick which is popular
among MS users. They simply copy all files from the ISO into a FAT
partition.
If a copy of /EFI/BOOT from the ISO's EFI partition is among them or gets
added, then the result is said to boot on all EFI firmwares.

See the complaint of the author of Rufus when Ubuntu hid the /EFI/BOOT
equipment in its new ISO layout:
  https://bugs.launchpad.net/ubuntu/+source/casper/+bug/1895131

When looking into mounted
  debian-live-10.2.0-amd64-xfce+nonfree.iso
  debian-10.6.0-amd64-netinst.iso
i see
  /EFI/boot
  /EFI/boot/bootx64.efi
  /EFI/boot/grubx64.efi
The files match those in the EFI partition image
  /boot/grub/efi.img
But the partition image contains an additional file
  /efi/debian/grub.cfg

Maybe one should put it into the /EFI/boot/ directory of the ISO ?


Have a nice day :)

Thomas

Re: posible wrong alert about malware at ISO but telling anyway

[Thomas Schmitt]

Hi,

Fernando Oliveira wrote:
> When downloading today debian 9.13 i386 netinstall, or debian 9.12 i386
> netinstall, my bitdefender anti-virus gave me an alert they have virus or
> malware. I do not believe it, but telling anyway :-)

The problem is known:
  https://lists.debian.org/debian-cd/2020/07/msg00056.html
  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=966538
  https://lists.debian.org/debian-cd/2020/10/msg00010.html
  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=973683

The offending byte patterns are most probably not in the ISO but in other
files on the server which offers the ISO:
  https://lists.debian.org/debian-cd/2020/11/msg5.html

Can you confirm that you get the ISO from umu.se ?
E.g. by a wget run to download the ISO which reports
  ...
  Location: https://saimei.ftp.acc.umu.se/...
  ...


Have a nice day :)

Thomas

Bug#973683: some images blacklisted by Google Chrome safe browsing

[Thomas Schmitt]

Hi,

> For all they will know, Debian has been pwned :-/

Yeah. I tried hard to keep my previous mail in a civilized tone towards
the intellectual entities who decided to deduce the purity of Debian ISOs
from .exe files on the same server.
(Quotation marks in the air are a warning sign towards myself that i am
about to start a rant.)

The only mitigation i can imagine is to put warning signs on pages like
  https://cdimage.debian.org/debian-cd/current-live/amd64/iso-hybrid/
which say something like:

- The browser warnings are known to be bogus. They appear because Google
  mistrusts especially our swedish mirrors.

- Please fulfill the (standard) request not to download by a web browser.
  See first paragraphs of https://www.debian.org/CD/http-ftp/ .

- If you have to use a web browser, you might be able to avoid the
  classification as malware by using a mirror server from the lower three
  quarters of
https://www.debian.org/CD/http-ftp/


Have a nice day :)

Thomas

Bug#973683: some images blacklisted by Google Chrome safe browsing

[Thomas Schmitt]

Hi,

the Google problem is getting worse. Firefox only warns because of their
botched malware "detection".

According to statements by Mattias Wadenstein and Steve McIntyre
in the thread around

  https://lists.debian.org/debian-cd/2020/11/msg5.html

it is probably about this naive reasoning from
  https://support.google.com/chrome/answer/6261569?p=ib_download_blocked

  "We've found that the website you tried to download the file from has
   been known to distribute malware."

So use some more brainful program for downloading. wget for example.


Have a nice day :)

Thomas

Re: Firefox claims i386 images contain virus or malware

[Thomas Schmitt]

Hi,

i wrote:
> > Do i understand right that it is not about the ISO and its content but
> > rather about other files which are offered by the same server ?

Mattias Wadenstein wrote:
> Well, as far as we know, yes.

"Sippenhaft" ... www.dict.cc ... "kin liability".
At least they don't resort to blood feuds.

I tested downloading
  
https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-10.6.0-amd64-netinst.iso
and wget reports

  Location: https://saimei.ftp.acc.umu.se/...

So indeed we germans get our ISOs from sweden. Wonders of globalization.


> it is non-trivial to make separate servers for 50+ different free
> software projects we mirror here,

I see.

Is there a public overview of the accusation process ?
If so, then i would write a wiki article which could then be linked from
places like
  https://www.debian.org/CD/http-ftp/


Have a nice day :)

Thomas

Re: Firefox claims i386 images contain virus or malware

[Thomas Schmitt]

Hi,

Andrew M.A. Cater wrote:
> If all else fails, find a Linux box and use jigdo-file and the jigdo-lite
> command to assemble a medium from individual Debian packages.

Or follow
  https://wiki.debian.org/JigdoOnLive

But the problem is with convenience users who download some netinst or
DVD-1 ISO and get warned by the Firefox of Debian and Ubuntu that these
ISOs contain malware.
We need to be able to explain to those people what's going on.
Just saying "don't worry" is not really convincing.


Have a nice day :)

Thomas

Re: Firefox claims i386 images contain virus or malware

[Thomas Schmitt]

Hi,

Steve McIntyre wrote:
> > Previously, this seemed to be caused by another (unrelated) download
> > from the same server in Sweden.

Do i understand right that it is not about the ISO and its content but
rather about other files which are offered by the same server ?

Back in july i downloaded an amd64 ISO by Debian 10 for testing and got
the warning once, but not on the second download. So reproducing the
effect is not easy.
I'm quite sure that the mirror i used was not in sweden. Do the offending
files get mirrored to german servers ?


Mattias Wadenstein wrote:
> Google has decided that a) annother ancient win95 antivirus binary is evil
> and b) [worse] a current free software projects' USB-SD-creator.exe is evil.

If my puzzled theory is right and Google stays as it is, then it might
become necessary to separate Debian ISOs from software from other worlds.

If my theory is wrong, please correct me, so i don't spread false rumors
among the ISO users.


Have a nice day :)

Thomas

Re: Firefox claims i386 images contain virus or malware

[Thomas Schmitt]

Hi,

Ruud Bos wrote:
> I tried to download two different ISO files via
> https://cdimage.debian.org/cdimage/release/current-live/i386/iso-hybrid/
> Everytime a download has completed, Firefox mentions that the ISO contains
> a virus. I tried downloading both the GNOME and MATE ISO's.

This accusation by Firefox is not new:
  https://lists.debian.org/debian-cd/2020/07/msg00056.html
  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=966538
Regrettably there is no hint what part of the ISOs exactly is accused of
being malware.
Do you see any information beyond the statement that there is a virus ?

It is unlikely that Debian inadvertently packages any virus.
To verify that the ISOs have not been tampered with, download into the
same directory as the ISOs:

  
https://cdimage.debian.org/cdimage/release/current-live/i386/iso-hybrid/SHA512SUMS
  
https://cdimage.debian.org/cdimage/release/current-live/i386/iso-hybrid/SHA512SUMS.sign

Verify the authenticity of SHA512SUMS by a run of

  gpg --keyserver keyring.debian.org --verify SHA512SUMS.sign SHA512SUMS

which must report

  gpg: Good signature from "Debian CD signing key "
  ...
  Primary key fingerprint: DF9B 9C49 EAA9 2984 3258  9D76 DA87 E80D 6294 BE9B

The reported "key fingerprint" must match one of those listed at
  https://www.debian.org/CD/verify .

When the SHA512SUMS was verified by SHA512SUMS.sign and the public Debian
key, let program "sha512sum" check for matches:

  sha512sum -c SHA512SUMS

must report

  debian-live-10.6.0-i386-gnome.iso : OK
  debian-live-10.6.0-i386-mate.iso : OK
  ...other.ISOs... : FAILED open or read
  sha512sum: WARNING: ... listed files could not be read

It must not report

  debian-live-10.6.0-i386-gnome.iso : FAILED
  ...
  sha512sum: WARNING: ... listed files could not be read
  sha512sum: WARNING: 1 computed checksum did NOT match

If the verification succeeds, consider to file a bug in Ubuntu against
the virus checker of Firefox.


Have a nice day :)

Thomas

Re: Debian Strech 9.13 ISO Download - Firefox says its containing a virus

[Thomas Schmitt]

Hi,

Steve McIntyre wrote:
> I hope you're keeping ok!

Sufficiently. I'm bored enough for digging out old kernel bugs and trying
to fix them. (I now got a half assembled new Xeon box with the best
Linux 4.19 drivers/scsi/sr.c and drivers/cdrom/cdrom.c ever ...)

But at this occasion i remember that i wanted to pester you again about

  "Copying a Debian hybrid ISO safely onto a USB flash drive"
  https://wiki.debian.org/XorrisoDdTarget

whether this shall go into the FAQ and whether you as my occasional sponsor
would want it as separate Debian package (else it would be installed as
part of xorriso).

--

Back to the bug:

Since the warning appeared only with the first download of
debian-9.13.0-amd64-netinst.iso but not with a subsequent one, i used now
the debian-10.5.0-amd64-DVD-1.iso which was reported by Triston Line.

No warning. All seems well again.


> a different Windows program
> hosted elsewhere on the acc.umu.se download service has been flagged
> as malware and that tainted everything hosted at their site.

I know that Google has high expectations towards the capabilities of their
programmers. So i wonder what kind of professional is allowed to define
the strategies of their virus guard.


Have a nice day :)

Thomas

Re: Debian Update BD no longer produced/avaliable

[Thomas Schmitt]

Hi,

ralfs wrote:
> > With buster, only the set of current BD images is available on
> > the servers but not an  " debian-update-10.x.x-amd64-DVD-x.iso "

I see
  debian-update-10.5.0-amd64-DVD-1.jigdo
  debian-update-10.5.0-amd64-DVD-1.template
  debian-update-10.5.0-amd64-DVD-2.jigdo
  debian-update-10.5.0-amd64-DVD-2.template
  debian-update-10.5.0-amd64-DVD-3.jigdo
  debian-update-10.5.0-amd64-DVD-3.template
on
  https://cdimage.debian.org/debian-cd/10.5.0/amd64/jigdo-dvd/


Steve McIntyre wrote:
> Sorry, do you mean BD (Blu-Ray disk) or DVD here?

Given that 10.5.0 already needs 3 update DVDs, it might be worth to produce
a single update BD image, too.
Not ony for BD but also for decently sized USB sticks. Maybe it helps
usability of the update ISO.


Have a nice day :)

Thomas

Re: Debian Strech 9.13 ISO Download - Firefox says its containing a virus

[Thomas Schmitt]

Hi,

i wrote:
> > Quite surely Debian does not put malware into its ISOs.

Steve McIntyre wrote:
> No, *definitely* not.

Well, as upstream programmer i could - intentionally or as victim of
malware myself - be the culprit who sneaks malware into a Debian ISO.
I try hard to keep my machines clean and my moral reputable, but in the end
i do not dare to be more affirmative than "Quite surely".


> I've raised a few tickets with Google

Please notify bug 966538 about any progress.


> We're wondering if it might just be a hash collision or something.

If it's not weaker than MD5 then this is extremely unlikely without a
systematic connection to malware classification. If some program binary
was registered as malware by mistake, that would be such a connection.


> It's difficult to tell with ~zero diagnostics. :-/

To my humble opinion, this obscure behavior should be reason enough to
disable the malware check by default.

My Debian 10 is only 6 days old and i did not install anything browser
related after the netinst installation was done. So the checker came
quite surely by default with the Firefox installation. (Currently the
desktop is LXDE, if this matters. Switching to fvwm is still on my todo
list ... after i learned how to keep the galaxy collision screensaver ...)


Have a nice day :)

Thomas

Re: Debian Strech 9.13 ISO Download - Firefox says its containing a virus

[Thomas Schmitt]

Hi,

i filed
  https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=966538
  "Firefox accuses debian-9.13.0-amd64-netinst.iso of containing malware"

If you are interested in getting possible replies by mail, send a mail to
  966538-subscr...@bugs.debian.org


Have a nice day :)

Thomas

Re: Debian Strech 9.13 ISO Download - Firefox says its containing a virus

[Thomas Schmitt]

Hi,

Deparade Max wrote:
> I tried three times to download the 9.13 version ISO but Firefox keep saying
> that it might conaint a virus or malware. I need a verified answer whats
> going on there.
> I used following site:
> https://cdimage.debian.org/mirror/cdimage/archive/9.13.0/amd64/iso-cd/
> And following ISO:
> debian-9.13.0-amd64-netinst.iso

My quite freshly installed Firefox on Debian 10 says the same.

But only on the first download. A second download does not cause that
warning of Firefox.
Neither do downloads of the other two ISOs on that page.

My old Iceweasel on Debian 8 does not complain.

--

Quite surely Debian does not put malware into its ISOs.

It would be interesting to see more details about how Firefox or other
virus scanners come to that idea. If they can tell a file name in the ISO
or a byte range in the ISO, then it would be possible to examine the
problem deeper.
Well, my firefox offers me to "open" the ISO ... if i understand it
correctly this "open" offers me to overwrite my hard disk by that ISO.
That's fewly helpful.

--

There is always the risk of man-in-the-middle malice or of corrupted
mirror servers.

Against that threat, Debian has two levels of verifying.
At
  https://cdimage.debian.org/mirror/cdimage/archive/9.13.0/amd64/iso-cd/
there are
  SHA512SUMS
  SHA512SUMS.sign
Download them and get the potential signing keys by
  gpg --keyserver keyring.debian.org --recv-keys 64E6EA7D
  gpg --keyserver keyring.debian.org --recv-keys 6294BE9B
Verify the validity of SHA512SUMS by
  gpg --verify SHA512SUMS.sign SHA512SUMS
which should say
  gpg: Good signature from "Debian CD signing key " 
[unknown]
  gpg: WARNING: This key is not certified with a trusted signature!
  gpg:  There is no indication that the signature belongs to the owner.
  Primary key fingerprint: DF9B 9C49 EAA9 2984 3258  9D76 DA87 E80D 6294 BE9B

The warning can be ignored. But the "Primary key fingerprint" must completely
match one of the fingerprints shown at
  https://www.debian.org/CD/verify

After you can trust the file SHA512SUMS, compute the SHA512 sum of the
downloaded ISO:
  sha512sum debian-9.13.0-amd64-netinst.iso
and compare it with the sum that is listed for this ISO in file SHA512SUMS.

If the SHA512 sums match, then you very surely have an original Debian ISO.
(And if virus scanner don't like it, they should tell more info now or
 forever hold their peace.)


My verification of the offending ISO was successful. I would trust it,
because i have few reason to trust Firefox and its ideas of malware
detection.


Have a nice day :)

Thomas

Re: Final CDs being written for Stretch - 9.13 release - prior to LTS

[Thomas Schmitt]

Hi,

Andrew Cater wrote:
> It does seem that
> we may be able to stop routine production of as many images. Netinst,
> something DVD-ish sized (so smaller than 8G) and some (larger file size ??)
> may do it.

How about being storage-neutral and drop amd64-DVD-2.iso and
amd64-DVD-3.iso in favor of a new amd64-STICK8GB-1.iso ?

That would mean the dropped DVDs 2 and 3 have to be downloaded by jigdo
like DVDs 4 to 16 (or like DVD 2 and 3 of arm64).
Question is what is on DVD 3 that will be missing for a "normal" Debian
installation from STICK8GB.

Really generous with more than all DVD.iso packages would be a readily
downloadable amd64-STICK16GB-1.iso (currently only available via jigdo).


> Jigdo does work well to build .iso images
> - it might need more documentation.

Especially the documentation needs to be where the .jigdo files are.

I meanwhile learned that SuSE has a package "jigdo" which provides
jigdo-lite. (It's not easy to get a file list of their packages online.)
Fedora too has jigdo-lite in package "jigdo".
(Well, when debian-cd switches to SHA256 for jigdo, those will need a
 nudge.)


Have a nice day :)

Thomas

Re: Final CDs being written for Stretch - 9.13 release - prior to LTS

[Thomas Schmitt]

Hi,

Stefan Monnier wrote:
> I was thinking of a scheme by which the ISO is constructed and streamed
> at the same time, so the complete ISO images aren't ever stored whole
> anywhere on the server.

Although there are no fundamental obstacles for stream production,
the current software uncompresses the .template file to a file on disk
and then fills the marked holes by the packages which it downloads from
the chosen mirror server.


I wrote:
> > The production of jigdo files is a linear effort only if the producer
> > knows where the filesystem stores file content data.

Stefan Monnier wrote:
> I'm not sure I understand what this means,

Indeed "linear" is not the right term. I apologize for being misleading.

Given the original description
  http://atterer.org/jigdo/debian-jigdo-mini-howto#PREPARINGTHEISOFORDOWNLOAD
of the production process it is more about "scientific" versus "magic":

  "Through magic, jigdo-file finds out which of the loose files are
   contained in the ISO image and their offsets within the ISO file.
   It outputs two files: a ".template" file and a ".jigdo" file."

All tools for creating .jigdo and .template get a list of MD5 sums and
package file paths (the "loose" files).

The original tool, jigdo-file, scans the readily prepared ISO image for
the content of the data files without knowing where their content starts.
This implies a large amount of try-and-error, even with a smart algorithm.
The mismatching efforts are wasted.

genisoimage and libjte (under libisofs and xorriso) get to see the content
data of each data file, compute the MD5 and look up this MD5 in the list.
They don't have to guess where the files are stored in the ISO.

On the other hand, jigdo-file can apply its magic to any filesystem which
stores file content unchanged in one byte string, whereas the non-guessing
approach needs help from the filesystem producer.


> it does sound like it
> implies that streaming production of ISOs is technically possible.

I actually meant production of .template and .jigdo, not re-assembly of
the .iso from them and Debian packages.

But yes, streaming re-assembly of .iso from .jigdo only needs structural
changes in the download software. (The problem is how to motivate the
change given that there hardly will be an official ISO-from-Jigdo server.)

-

Tixy wrote:
> > Debian's 'DVD' images are isohybrid and work for booting from USB flash
> > drives,

Stefan Monnier wrote:
> The discussion was about the creation of special ISOs for old Mac
> hardware whose firmware isn't able to boot from those hybrid ISOs.

The "mac" ISOs for i386 and amd64 are hybrid ISOs too. Just not for EFI.

They have legacy MBR boot code which hops on the boot image for CD booting
after it was started by legacy BIOS from USB stick.
But they don't have a second boot image for EFI from CD (which also
would serve as EFI System Partition on USB stick).


Have a nice day :)

Thomas

Re: Final CDs being written for Stretch - 9.13 release - prior to LTS

[Thomas Schmitt]

Hi,

(The mail headers indicate that the OP, Andrew Cater, is subscribed to
 debian-cd but not to debian-user. So replies to debian-user only might
 not get to his attention.)

Stefan Monnier wrote:
> An alternative is to have "virtual ISO images", i.e. images which are
> constructed on the fly (presumably by jigdo) on the web-server side.

Assuming that a complete set of ISOs for whatever medium occupies
at most 100 GB, it seems better to have the images ready rather than to
assemble them on demand, even if the mirror latency and bandwidth are
no problem.
This would spare the nightmare of managing the life cycle of temporary ISOs
on the server. I assume that all images would fit on a single modern HDD.

But the reason for letting the user perform jigdo download is that the network
load vanishes in the normal traffic of the package servers. If download gets
interrupted, one just has to start it again to get the remaining work
completed.
These adavantages for Debian can hardly be uphold by a jigdo assembling
server on Debian side.


> I'm not sure why the discussion is always around DVD images.

That's mainly about the image size factor of a 4+ GB DVD.

There is one size factor that is not defined by optical media sizes
  https://cdimage.debian.org/debian-cd/current/amd64/jigdo-16G/


> Would you be equally happy with an image only for a USB flash drive, or
> is it important for it to be an image for optical media?

The ISOs work from optical media and from hard-disk-like media.
The latter class includes USB sticks and memory cards.

The production of jigdo files is a linear effort only if the producer
knows where the filesystem stores file content data. This knowledge is in
the ISO 9660 producers genisoimage and xorriso. For other filesystems
the matching jigdo producer software is not available yet.

Further the filesystem must store the content of each Debian package as one
consequtive sequence of bytes. Fragmentation is not supported by jigdo.


For some reason even fancy container systems use ISOs for deployment.
  https://nanikjava.github.io/post/insideminikubeiso/
The example i found is BIOS-only and (virtual) CD-only
  https://storage.googleapis.com/minikube/iso/minikube-v1.0.0.iso
Maybe it's because the user would need to start an ISO 9660 manipulating
program in order to alter them. This hardly happens by mistake.


Have a nice day :)

Thomas

Re: Final CDs being written for Stretch - 9.13 release - prior to LTS

[Thomas Schmitt]

Hi,

Andrew Cater wrote:
> If anybody knows of any [Mac Mini] that they really must keep
> running, speak now or forever hold your peace.

It is possible to create an ISO without EFI boot lures from a normal
i386 or amd64 ISO.
Follow
  https://wiki.debian.org/RepackBootableISO
but leave out the options beginning with
  -eltorito-alt-boot

Others seem to have solved their mini mac problems similarly
  
https://wiki.archlinux.org/index.php/Unified_Extensible_Firmware_Interface#Remove_UEFI_boot_support_from_optical_media


> For the future, we should produce physical media for
> the netinst.iso, the first DVD image in any set and one larger image to be
> written to a USB stick if wanted - and corresponding source for each size.

A large image for USB stick or 25 GB Blu-ray would help those with
insufficient network access at installation time.


> All other .iso files to be distributed as .jigdo and .template files.

Most DVD images and images for all larger media are already only available
via .jigdo.

Jigdo's software status is healthy. But we need better documentation and more
advertising for it. How about a jigdo help file in the download directories
with .jigdo files, which tells how to get the download software and how to
use it.

Steve McIntyre offers binaries for MS-Windows and source at
  https://www.einval.com/~steve/software/jigdo/download/
I documented the endeavor of getting and booting a Debian Live for the
purpose of downloading jigdo ISOs in
  https://wiki.debian.org/JigdoOnLive
(Still lacking knowledge about how to mount foreign OS disks for writing.)
Then there is
  https://www.debian.org/CD/jigdo-cd/
We could need a list of distros and names of their packages which contain
jigdo-lite.

This all should be mashed up and condensed to a tangible description what
to do with the presented .jigdo and .template files.


There is also the fact that due to network and server latency the download
on a fast internet connection via jigdo is much slower than direct download
via HTTPS.
(Netinst installation might have the same drawbacks. How much faster is
 installation from a 32 GB USB stick in comparison to netinst ?)


Have a nice day :)

Thomas

Re: Final CDs being written for Stretch - 9.13 release - prior to LTS

[Thomas Schmitt]

Hi,

Andrew Cater wrote:
> [...] here are
> a couple of architectures that are built but not tested because the testers
> have no hardware.
> One set  is an old Mac on Intel media,

Do you mean these ?
  
https://cdimage.debian.org/debian-cd/current/i386/iso-cd/debian-mac-10.4.0-i386-netinst.iso
  
https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-mac-10.4.0-amd64-netinst.iso

They are related to
  https://wiki.debian.org/MacMiniIntel#Macmini_1.2C1
and following sections.

-

If it is not about above "mac" ISOs, which then ? I am curious.


Have a nice day :)

Thomas

Bug#963469: cdrom: /dev/sr0 scanned on and on for no visible reason

[Thomas Schmitt]

Hi,

I am not aware of a package that is in charge for drive behavior.
libburn or kernel are near the hardware. But i don't see this as libburn
problem and for now not as a kernel problem either.

It is unusual that an empty DVD drive makes noises, except when the tray
was freshly loaded and the drive has a look whether there is a medium.
So even if there is a process which periodically tries to read from the
drive, this should be a silent thing.

My first suspicion would be that the drive has problems to recognize media.
- Does the drive still read DVDs ?
- Do you see system log messages around the times when the drive is active ?
- Does the noise happen while the computer is under control of its firmware
  (BIOS or EFI) waiting for some user input (e.g. in a setup screen) ?

A bit far fetched:
What text output do you get from a xorriso run with SCSI log while no medium
is in the drive ?

  xorriso -scsi_log on -outdev /dev/sr0 >/tmp/xorriso_sr0.log 2>&1

The resulting file xorriso_sr0.log normally has 300 to 500 lines, which is
probably too long for a bug report. So send it to scdbac...@gmx.net in
private. I will then send my assessment to this bug report.


Have a nice day :)

Thomas

Upcomming trouble with hybrid ISOs and libblkid

[Thomas Schmitt]

Hi,

do the hybrid ISOs of debian-cd and debian-live depend on blkid properly
detecting the ISO 9660 filesystem on the base device (/dev/srN, /dev/sdX) ?
Does any other Debian software depend on this detection ?

If so, then trouble is at the horizon by
  
https://github.com/karelzak/util-linux/commit/7ef86a08914427d6486734614d7d3bbed1f108fe

This causes Fedora ISOs to fail booting from /dev/srN
  https://bugzilla.redhat.com/show_bug.cgi?id=1783066
because the /dev/disk/by-label/ link is not created by udev.

The commit shall obviously heal a shortcomming of the old questionable
udev rule
  # for partitions import parent information
  ENV{DEVTYPE}=="partition", IMPORT{parent}="ID_*"
when it meets a USB stick with an ISO on the base device and a partition
which has no own filesystem.

The effect on /dev/srN is obviously unintended.
To my theory, libblkid detects the partition table on the DVD and bets
that the Linux kernel will create partition devices.
But with /dev/srN there will no partition devices be created.

I also expect grub-mkrescue ISOs to show no ISO 9660 at all, because
they get a MBR partition starting at disk block 1 in the BIOS-only case,
and a GPT with no ISO 9660 partition in the EFI-case.
The only way to access their ISO filesystems is via the base device.


The change is not yet in
  
https://sources.debian.org/src/util-linux/2.34-0.1/libblkid/src/superblocks/iso9660.c/
Can we warn the maintainer in advance ?


Have a nice day :)

Thomas

Re: sha256 checksums

[Thomas Schmitt]

Hi,

wdstud...@protonmail.ch wrote:
> This may come as a shock to you but files without file extensions are not
> very useful. What am I supposed to open them with? Powerpoint? Daemon Tools?

The checksum files like
  https://cdimage.debian.org/debian-cd/current/amd64/iso-dvd/SHA256SUMS
are plain text files encoded in ASCII.
You may open them online with your web browser or offline by a text editor.

If you want to give them extensions, i'd propose .txt. But i am living in
MS-free environment since 30 years. So you might want to ask some experts
for MS-Windows for advice.


The signature files of the checksum files, like
  https://cdimage.debian.org/debian-cd/current/amd64/iso-dvd/SHA256SUMS.sign
are to be used by program "gpg".
  gpg --keyserver keyring.debian.org --verify SHA512SUMS.sign SHA512SUMS
(How to do this on MS-Windows is beyond my knowledge.)


Have a nice day :)

Thomas

Re: sha256 checksums

[Thomas Schmitt]

Hi,

wdstud...@protonmail.ch wrote:
> Can someone please tell me where [...] I can find the correct sha256
> checksums for the CD/DVD images?

In the same directories as the ISO images.


> The information at www.debian.org/CD/verify was NOT helpful.

Would this article be more helpful ?
  https://wiki.debian.org/JigdoOnLive#Verify_the_Debian_Live_download

It applies not only to Live ISOs but to the others too.


Have a nice day :)

Thomas

Re: Adding sha256 support into jigdo tools (jigdo and jigit)

[Thomas Schmitt]

Hi,

Steve McIntyre wrote:
> Let's go with "-jigdo-checksum-algorithm" to keep the separation?

The user is king.


Other mail:
I proposed to keep versioning and SONAME at 1
> > LIBJTE1 {

Steve McIntyre wrote:
> Surely that works for upwards compatibility, but won't stop somebody
> from building and depending on the newer libjte2 then trying to run a
> binary with libjte1?

The only user of libjte already has precautions against this:

> > Fine check for compatibility happens at compile time by the required
> > minimum version (e.g. iso_libjte_req_* versus LIBJTE_VERSION_*) and
> > at run time by calling libjte__is_compatible().

This kind of test works well between e.g. xorriso's binary and libisoburn,
and then between libisoburn and libisofs.
The test between libisoburn/libisofs and libjte has never been challenged
but i am optimistic that it works.

The compile time test is at
  https://sources.debian.org/src/libisofs/1.5.0-1/libisofs/messages.c/#L157
(This is how i let a passive piece of source kick an unknown compiler:
  https://sources.debian.org/src/libisofs/1.5.0-1/libisofs/messages.c/#L174
)
Runtime test
  https://sources.debian.org/src/libisofs/1.5.0-1/libisofs/messages.c/#L180
Here i let libjte judge whether it finds its own version not older than the
compile time version seen by libisoburn (LIBJTE_VERSION_* from libjte.h).

As long as you stay with a plausible sequence of version numbers,
libjte__is_compatible() should not need your attention.
(For the curious:
  https://sources.debian.org/src/jigit/1.21-1/libjte/libjte.c/#L68
)


Have a nice day :)

Thomas

Re: Adding sha256 support into jigdo tools (jigdo and jigit)

[Thomas Schmitt]

Hi,

SONAME comes from libjte/configure.ac. Dunno from where libburn got the
slightly braindamaged way to compose it. From there it spread over the
other libraries. libjte is its youngest victim. Shrug.

The upstream revision "2.0.0" is not necessarily the .so suffix.
The macro values from libjte.h have to be reflected in AC_INIT().
Thus:

-AC_INIT([libjte], [1.0.0], [http://www.libburnia-project.org])
+AC_INIT([libjte], [2.0.0], [... better URL ...])


The .so suffix should increase numerically from current 1.0.0.
But you are free to choose the SONAME.

If you want SONAME=1 (libjte.so.1.1.0, libjte1*.deb) then:

-LT_CURRENT=1
-LT_AGE=0
+LT_CURRENT=2
+LT_AGE=1

If you want SONAME=2 (libjte.so.2.0.0, libjte2*.deb) then:

-LT_CURRENT=1
+LT_CURRENT=2
 LT_AGE=0


Have a nice day :)

Thomas

Re: Adding sha256 support into jigdo tools (jigdo and jigit)

[Thomas Schmitt]

Hi,

while exploring the need for adaptions in the build system of libsofs,
i wonder whether it is really necessary to create LIBJTE2 in libjte.ver.
Adding new functions is upward compatible.
In my libraries i add them to the current (and only) SONAME version.

Fine check for compatibility happens at compile time by the required
minimum version (e.g. iso_libjte_req_* versus LIBJTE_VERSION_*) and
at run time by calling libjte__is_compatible().

If libisoburn gets a configure.ac test for existence of
libjte_set_checksum_algorithm(), a compiler macro to enable its use,
and a configure option to disable it despite being available,
then it can even be built with the older libjte.
(Fully autostylish.)

So how about this in libjte/libjte.ver ?

 LIBJTE1 {
   global
   ...
 libjte_get_next_message;
 libjte_clear_msg_list;
+libjte_set_checksum_algorithm;
+libjte_set_checksum_path;
+libjte_add_checksum_demand;
+libjte_decide_file_jigdo2;
+libjte_write_match_record2;
   local:
 *;
 };


(Good news is that even libjte2 would probably not cause changes in the
 build system of libisofs. Currently it depends only on 
 and on libjte.so. I hope that both can keep their names.
 debian/control depends on libjte1 indirectly via libjte-dev.
 Adding "(>= 2.0.0)" would not really be necessary.)


Have a nice day :)

Thomas