[digikam] [Bug 362023] Extremely slow metadata writing via maintenance

[Maik Qualmann]

https://bugs.kde.org/show_bug.cgi?id=362023

Maik Qualmann  changed:

   What|Removed |Added

   Version Fixed In||7.7.0
 Status|REPORTED|RESOLVED
 Resolution|--- |FIXED

--- Comment #21 from Maik Qualmann  ---
Hmm, it looks like closing the bug didn't work. I close it.

Maik

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 362023] Extremely slow metadata writing via maintenance

[soredake]

https://bugs.kde.org/show_bug.cgi?id=362023

soredake  changed:

   What|Removed |Added

 CC||ndrzj1...@relay.firefox.com

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 362023] Extremely slow metadata writing via maintenance

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=362023

--- Comment #20 from caulier.gil...@gmail.com ---
Maik,

Why this file still open even the comment from comment #18 ?

Gilles

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 362023] Extremely slow metadata writing via maintenance

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=362023

--- Comment #19 from caulier.gil...@gmail.com ---
digiKam 7.0.0 stable release is now published:

https://www.digikam.org/news/2020-07-19-7.0.0_release_announcement/

We need a fresh feedback on this file using this version.

Best Regards

Gilles Caulier

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 362023] Extremely slow metadata writing via maintenance

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=362023

caulier.gil...@gmail.com changed:

   What|Removed |Added

  Component|Metadata|Maintenance-Metadata

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 362023] Extremely slow metadata writing via maintenance

[Mario Frank]

https://bugs.kde.org/show_bug.cgi?id=362023

--- Comment #18 from Mario Frank  ---
Git commit 2f8ddd42ef62d7aea9e490cdb05ffcc644810c81 by Mario Frank.
Committed on 22/02/2017 at 15:05.
Pushed by mfrank into branch 'master'.

Merged the current state of the garbage collection branch which improves the
database cleanup stage of the maintenance
and improves the reactiveness of the maintenance overall. We ported the way
items are processed to a queue based method
that can use the CPUs more effectively and does not create thousands of
threads.
Related: bug 283062, bug 216895, bug 374225, bug 351658, bug 329353
FIXED-IN: 5.5.0

M  +17   -12   NEWS

https://commits.kde.org/digikam/2f8ddd42ef62d7aea9e490cdb05ffcc644810c81

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 362023] Extremely slow metadata writing via maintenance

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=362023

caulier.gil...@gmail.com changed:

   What|Removed |Added

Version|5.0.0   |5.4.0

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 362023] Extremely slow metadata writing via maintenance

[Simon]

https://bugs.kde.org/show_bug.cgi?id=362023

--- Comment #17 from Simon  ---
Hi Gilles,

This problem is still the same. I reduced it for me by using internal
mysql database and preloading most of the database to memory.
When testing with appimage, sqlite on system hd and data on separate hd
(no ssl in my laptop :) ), now even the UI gets unresponsive during
writing. Maybe it would be more efficient to remember which files were
written and issue a rescan after writing of metadata is done?
Generally the only issue I have with syncthing is its constant scanning
of stuff on the HD. Whenever I start it causes tons of read access (to
images, not database) producing command line output like
"digikam.dimg***: JPEG file identified" and "digikam.metaengine:
Orientation => Exif.Image.Orientation => 1", as if these files were new
or modified (they are not).

Cheers,
Simon

On 25/11/16 15:38, bugzilla_nore...@kde.org wrote:
> https://bugs.kde.org/show_bug.cgi?id=362023
>
> caulier.gil...@gmail.com changed:
>
>What|Removed |Added
> 
>  CC||caulier.gil...@gmail.com
>
> --- Comment #16 from caulier.gil...@gmail.com ---
> What's about this file using digiKam AppImage bundle 5.4.0 pre release given 
> at
> this url :
>
> https://drive.google.com/drive/folders/0BzeiVr-byqt5Y0tIRWVWelRJenM
>
> Gilles Caulier
>

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 362023] Extremely slow metadata writing via maintenance

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=362023

caulier.gil...@gmail.com changed:

   What|Removed |Added

 CC||caulier.gil...@gmail.com

--- Comment #16 from caulier.gil...@gmail.com ---
What's about this file using digiKam AppImage bundle 5.4.0 pre release given at
this url :

https://drive.google.com/drive/folders/0BzeiVr-byqt5Y0tIRWVWelRJenM

Gilles Caulier

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 362023] Extremely slow metadata writing via maintenance

[Bizy via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=362023

Bizy  changed:

   What|Removed |Added

 CC||koen.b...@gmail.com

--- Comment #15 from Bizy  ---
Same here (Ubuntu 16.4).

Already more than 4 hours to update (via 'Maintenance', tags database -->
images) a folder with some 4000 images.  Memory use more than 4 Gb...
Progression window still indicating 0%...
Guess that's not how it's supposed to be...

Workaround:  selecting all images and same command via 'Edit', takes 15
minutes...

PS:  If you want me to do something, please be very specific... (most of the
conversation above is beyond my comprehension...)

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 362023] Extremely slow metadata writing via maintenance

[Simon via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=362023

--- Comment #14 from Simon  ---
Indeed, my setup is far from optimal for disk io. I have both the 
database and the images on a ntfs pratition of a hard disk on my laptop 
(at least not system hd). I thought that the database would be 
automatically cached in ram. I will look at it again some time.
Thanks again for your help.

On 02/05/16 21:14, Maik Qualmann via KDE Bugzilla wrote:
> https://bugs.kde.org/show_bug.cgi?id=362023
>
> --- Comment #13 from Maik Qualmann  ---
> This is are long waiting times, up to 5 seconds until a scan is completed for
> one image. Disabling the scanning does not help, he would be rescheduled in 
> any
> case. Modification date or file size have changed and need to be updated in 
> the
> DB. Writing to the SQLite DB is the time problem. The SQLite DB to put on an
> SSD drive is strongly recommended. Here are a few measured values, writing of
> one image information in the DB this include read new information from image
> (images on HDD - EXT4):
>
> HDD:
> SQLite: 180-270ms
> internal MySQL: 40-70ms
>
> SSD:
> SQLite: 30-60ms
>
> Are the images on an NTFS partition? Is also here the SQLite DB?
>
> Maik
>

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 362023] Extremely slow metadata writing via maintenance

[Maik Qualmann via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=362023

--- Comment #13 from Maik Qualmann  ---
This is are long waiting times, up to 5 seconds until a scan is completed for
one image. Disabling the scanning does not help, he would be rescheduled in any
case. Modification date or file size have changed and need to be updated in the
DB. Writing to the SQLite DB is the time problem. The SQLite DB to put on an
SSD drive is strongly recommended. Here are a few measured values, writing of
one image information in the DB this include read new information from image
(images on HDD - EXT4):

HDD:
SQLite: 180-270ms
internal MySQL: 40-70ms

SSD:
SQLite: 30-60ms

Are the images on an NTFS partition? Is also here the SQLite DB?

Maik

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 362023] Extremely slow metadata writing via maintenance

[Simon via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=362023

--- Comment #12 from Simon  ---
Created attachment 98665
  --> https://bugs.kde.org/attachment.cgi?id=98665=edit
Command line output in the middle of writing metadat to files with
scancontroller2.patch

I applied the patch and added the command line output in the same fashion as
before.

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 362023] Extremely slow metadata writing via maintenance

[Simon via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=362023

--- Comment #11 from Simon  ---
Created attachment 98664
  --> https://bugs.kde.org/attachment.cgi?id=98664=edit
Startup of digikam and start of writing metadata with scancontroller2.patch

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 362023] Extremely slow metadata writing via maintenance

[Simon via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=362023

--- Comment #10 from Simon  ---
Do I apply this patch on top of the current HEAD or on top of your 
previous patch? I guess the first, but just to be sure.

On 27/04/16 21:14, Maik Qualmann via KDE Bugzilla wrote:
> https://bugs.kde.org/show_bug.cgi?id=362023
>
> --- Comment #9 from Maik Qualmann  ---
> Created attachment 98651
>--> https://bugs.kde.org/attachment.cgi?id=98651=edit
> scancontroller2.patch
>
> That it is now slowly working because now images are processed with symbolic
> links.
> Please try this patch. He also adds a time measurement.
>
> Maik
>

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 362023] Extremely slow metadata writing via maintenance

[Maik Qualmann via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=362023

--- Comment #9 from Maik Qualmann  ---
Created attachment 98651
  --> https://bugs.kde.org/attachment.cgi?id=98651=edit
scancontroller2.patch

That it is now slowly working because now images are processed with symbolic
links.
Please try this patch. He also adds a time measurement.

Maik

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 362023] Extremely slow metadata writing via maintenance

[Maik Qualmann via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=362023

Maik Qualmann  changed:

   What|Removed |Added

  Attachment #98521|0   |1
is obsolete||

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 362023] Extremely slow metadata writing via maintenance

[Simon via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=362023

--- Comment #8 from Simon  ---
And the scan is now going clearly slower than before the patch. I am now 
running it almost two days and its at 3% only.

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 362023] Extremely slow metadata writing via maintenance

[Simon via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=362023

--- Comment #7 from Simon  ---
Created attachment 98528
  --> https://bugs.kde.org/attachment.cgi?id=98528=edit
Command line output in the middle of writing metadat to files with
scancontroller.patch

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 362023] Extremely slow metadata writing via maintenance

[Simon via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=362023

--- Comment #6 from Simon  ---
Created attachment 98527
  --> https://bugs.kde.org/attachment.cgi?id=98527=edit
Startup of digikam and start of writing metadata with  scancontroller.patch

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 362023] Extremely slow metadata writing via maintenance

[Simon via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=362023

--- Comment #5 from Simon  ---
Thanks for looking into this. I applied your patch. The results seems to be the
same (maybe somewhat less frequent rescans). I attached the initial part of the
log after startup, where redundant stuff is excluded (marked by [...]). The
actual scanning starts at line 400. A second command line output is from later
on during the scan.

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 362023] Extremely slow metadata writing via maintenance

[Maik Qualmann via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=362023

--- Comment #4 from Maik Qualmann  ---
Created attachment 98521
  --> https://bugs.kde.org/attachment.cgi?id=98521=edit
scancontroller.patch

Can you try this test patch? And report how digiKam now behaves.

Maik

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 362023] Extremely slow metadata writing via maintenance

[Maik Qualmann via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=362023

Maik Qualmann  changed:

   What|Removed |Added

 CC||metzping...@gmail.com

--- Comment #3 from Maik Qualmann  ---
This commit fixes only the writing of metadata for images which are linked via
a symbolic link.

Maik

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 362023] Extremely slow metadata writing via maintenance

[Maik Qualmann via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=362023

--- Comment #2 from Maik Qualmann  ---
Git commit 773f9361e5df5904c938ad9ee4cbb19acd1aa1f6 by Maik Qualmann.
Committed on 21/04/2016 at 19:03.
Pushed by mqualmann into branch 'master'.

fix absolute file path without symbolic links

M  +2-2libs/dmetadata/metaengine.cpp

http://commits.kde.org/digikam/773f9361e5df5904c938ad9ee4cbb19acd1aa1f6

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 362023] Extremely slow metadata writing via maintenance

[Simon via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=362023

--- Comment #1 from Simon  ---
Created attachment 98490
  --> https://bugs.kde.org/attachment.cgi?id=98490=edit
Command line output of writing metadata to files.

-- 
You are receiving this mail because:
You are watching all bug changes.