Bug#988120: jigdo-file: Fails to re-use its own cache file correctly

[mwonl]

Hello,

I just wanted to report that I have noticed the same problem.

I usually mount all old DVD images in one parent directory. On update 
jigdo just scanned this whole directory for the first new DVD and 
created a jigdo-file-cache.db file. For all other following new DVDs 
jigdo-file-cache.db was used and no additional scan of the directory was 
done.


After upgrading to bullseye (and using jigdo-file 0.8.0-1) I see the 
same behaviour as reported in this bug. A complete rescan of the whole 
directory is done for EVERY DVD and jigdo-file-cache.db seems to be 
ignored. This is very time consuming compared to the old behaviour.


I just downgraded to the buster version 0.7.3-5+deb10u1 and the 
behaviour is like before and there is only one scan for the first new 
DVD and the created jigdo-file-cache.db is then used for the following 
new ones.


(Testing was done with the 11.2.0 jigdo images)

So it seems a bug was introduced which corrupts (?) jigdo-file-cache.db 
for directories with a (very) large number of files.


If needed I can do some debugging with my setup.

Best regards,

Mark

Bug#988120: jigdo-file: Fails to re-use its own cache file correctly

[Pascal]

Package: jigdo-file
Version: 0.8.0-1
Followup-For: Bug #988120
X-Debbugs-Cc: pascal.mart...@gmx.fr


Dear Maintainer,

I have been worrying about those bugs and sensing the trouble would probably be
with "jigdo-file-cache.db", I made new tests beginning with the last of the
five first DVDs.

The version used for jigdo-file is always 0.8.0-1 with debian bullseye.

So I mounted "debian-10.7.0-amd64-BD-1.iso" under "/media/user/" and then I
typed :

$ jigdo-lite debian-10.7.0-amd64-DVD-5.jigdo

I chose "Files to scan : /media/user/" when jigdo asked me. The scanning for
the whole BD takes 35mn on my computer. This slowness is not surprising as I
understood that with jigdo 0.8.0-1 checksums for each *.deb file were made with
SHA256. With jigdo 0.7.3-5 it was with MD5, much quicker.

The surprising thing is that all worked fine. Jigdo created "jigdo-file-
cache.db" for the first time and it could find in
"debian-10.7.0-amd64-BD-1.iso" everything it needed to make up
"debian-10.7.0-amd64-DVD-5.iso".

Following my intuition I removed "jigdo-file-cache.db", or rather displaced it
into another directory. Then I remade a try with
"debian-10.7.0-amd64-DVD-4.jigdo" and then "debian-10.7.0-amd64-DVD-3.jigdo",
each time removing the preceding "jigdo-file-cache.db" before lauching jigdo-
lite.

And again it worked fine. Jigdo could find all it needed from
"debian-10.7.0-amd64-BD-1.iso" to make up DVD-4 and DVD-3 ISO images.

So I would say that the trouble seems to come from the overwriting of "jigdo-
file-cache.db". Comparing each "jigdo-file-cache.db" from the preceding one
with the "cmp -l" command line, I remarked that they were very different from
one another. The one for DVD-3 is even a little bigger :

DVD 5 : Found 1814 files  "jigdo-file-cache.db" : 16 568 320 bytes
DVD 4 : Found 2759 files  "jigdo-file-cache.db" : 16 568 320 bytes
DVD 3 : Found 2480 files  "jigdo-file-cache.db" : 16 576 512 bytes

After overwriting "jigdo-file-cache.db" jigdo seems to be induced into errors.
As if something went wrong in the process of overwriting versus writing for the
first time. Since this cache file is obviously not re-used correctly, jigdo
would work well if "jigdo-file-cache.db" was automatically erased at each jigdo
launching.
But "jigdo-file-cache.db" should be re-used in all cases (and particularly in
our case) to save time. Or else jigdo has to re-read the BD-1 image (or any
other ISO) all over again, and again.

With Jigdo 0.7.3-5, I have a backup of "jigdo-file-cache.db" (only to be used
with that version of jigdo because of different *.deb checksums). With that
cache file, Jigdo 0.7.3-5 does not even check the BD-1 ISO content. It produces
DVD-1 to 5 ISOs directly, and at the end it says the DVD-1 to 5 images (MD5
check) are correct.

Cordially,
Pascal.


-- System Information:
Debian Release: bullseye/sid
  APT prefers testing
  APT policy: (500, 'testing')
Architecture: amd64 (x86_64)

Kernel: Linux 5.10.0-6-amd64 (SMP w/2 CPU threads)
Kernel taint flags: TAINT_PROPRIETARY_MODULE, TAINT_OOT_MODULE,
TAINT_UNSIGNED_MODULE
Locale: LANG=fr_FR.UTF-8, LC_CTYPE=fr_FR.UTF-8 (charmap=UTF-8), LANGUAGE not
set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages jigdo-file depends on:
ii  libbz2-1.0   1.0.8-4
ii  libc62.31-11
ii  libdb5.3 5.3.28+dfsg1-0.8
ii  libgcc-s1 [libgcc1]  10.2.1-6
ii  libstdc++6   10.2.1-6
ii  wget 1.21-1+b1
ii  zlib1g   1:1.2.11.dfsg-2

jigdo-file recommends no packages.

jigdo-file suggests no packages.