https://bugs.kde.org/show_bug.cgi?id=526185
Bug ID: 526185
Summary: Large-library ("XXL") mode: re-evaluate the single-CPU
limit on Sync Metadata (bug 329091, 2016) and batch
face-change work
Classification: Applications
Product: digikam
Version First 9.1.0
Reported In:
Platform: Microsoft Windows
OS: Microsoft Windows
Status: REPORTED
Severity: wishlist
Priority: NOR
Component: Maintenance-Metadata
Assignee: [email protected]
Reporter: [email protected]
Target Milestone: ---
CONTEXT
hi guys - I run digiKam 9.1.0 on Windows with a local MariaDB 12.2 database and
about 130,000 images: photos hosted on one NVMe data disk, the maria database
on a second NVMe drive, 20 logical cores and 64 GB of RAM. After a large
face-tagging session, a Maintenance > Sync Metadata run (database to files)
took about 20 hours.
MEASUREMENTS DURING THE SYNC
- digiKam used 99 % of exactly one core, 5 % of the machine; 1 of 34 threads
was running.
- digiKam I/O was 26 MB/s read (about 6,000 small reads per second) and 13 MB/s
write. The photo drive was 30 % busy with an empty queue. The database was not
the bottleneck.
- Progress was about 2.8 images per second, with an average rewritten file of
about 9 MB.
- For videos (MOV/MP4) digiKam's ExifTool process rewrote each whole file to
add tags: 212 GB read and 210 GB written during this run, at 80-140 MB/s during
video-heavy albums.
- and to confirm The "Work on all processor cores" option was enabled.
WHY IT IS SINGLE-THREADED (from the 9.1.0 source)
- utilities/maintenance/manager/maintenancemngr.cpp, stage9(), forces
setUseMultiCoreCPU(false) on the MetadataSynchronizer with the comment "See Bug
#329091: Multicore CPU support with Exiv2 is problematic, even with 0.25
release". The same override applies to tag and person rename rewrites
(albummanager_talbum.cpp).
- Bug 329091 is really about MySQL max_allowed_packet. Comment 31 (2016)
limited the synchronizer to one CPU "for the moment" because of Exiv2 0.25.
Comment 40 (2018) closed the bug after a thread-cleanup fix. The limit itself
has not been revisited since.
- All Exiv2 access also goes through one process-wide s_metaEngineMutex, held
across the whole file load and the whole open/read/write.
- After every file write, ScanController::finishFileMetadataWrite() rescans the
file, re-reading its metadata and recomputing the unique hash. That is about
three metadata parses per image.
EXIV2 TODAY
digiKam 9.1.0 bundles Exiv2 0.28.8. Exiv2 documents Exif and IPTC as
re-entrant, and separate Image objects as safe to use concurrently. The
remaining known XMP race (Exiv2 issue #3449: encode() iterating the namespace
registry while another thread registers a namespace) is fixed on Exiv2 main (PR
#3448, February 2026), but it is not in 0.28.8 or 0.28.9. therefore I have
asked Exiv2 to backport it to 0.28.x: see
https://github.com/Exiv2/exiv2/issues/9514
Parallelism therefore looks safe for Exif/IPTC work and file I/O, as long as
XMP encoding and decoding and runtime namespace registration stay serialised.
digiKam currently calls XmpParser::initialize() without a lock function.
my wish : AN OPT-IN "LARGE LIBRARY" OPTION GROUP, OFF BY DEFAULT, that will
enable -
1. Lets Sync Metadata and tag/person rewrites use N worker threads (a spin
box), with a reader/writer lock: Exif/IPTC and file I/O run in parallel, XMP
and namespace registration stay exclusive, and it is version-gated so Exiv2
0.27.x keeps today's behaviour.
2. Replaces the full rescan after digiKam's own metadata write with an update
of the stored size, date and hash, or at least makes that rescan cheaper.
3. Writes video metadata to .xmp sidecars instead of rewriting the whole video,
and skips files whose stored metadata already matches.
4. Queues face-classifier retraining and background recognition after face
changes, with visible "train now" and "recognise now" actions, instead of
retraining on every confirm and rescanning all albums afterwards.
5. Makes a Sync Metadata run resumable, so an interrupted 20-hour run does not
restart from the beginning.
RELATED REPORTS: bug 433405 (rename ignores lazy sync), bug 509161 (sync
reloads all image data), bug 513515 (accepting a face scans all pictures), bug
507441 (resume interrupted metadata sync), bug 511761.
I am happy to test builds on this collection and provide measurements.
--
You are receiving this mail because:
You are watching all bug changes.
