[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=365202

caulier.gil...@gmail.com changed:

   What|Removed |Added

   Severity|normal  |crash

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[bugzilla_noreply]

https://bugs.kde.org/show_bug.cgi?id=365202

caulier.gil...@gmail.com changed:

   What|Removed |Added

  Component|Faces-Engine|Database-Faces

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[Evert Vorster via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #67 from Evert Vorster  ---
Thanks for the updated patch. It's building now. 

My latest test was with all the patches installed, and it scanned all the way
through my collection.  There are a few smaller bugs where the program does not
work as expected that could very well be due to the ABI change, so I'll test
all of them again.

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[Maik Qualmann via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #66 from Maik Qualmann  ---
Created attachment 100038
  --> https://bugs.kde.org/attachment.cgi?id=100038=edit
currentFace31.patch

New patch, face modul updated to the latest version of OpenCV3.1.
Please test it with  this patch.

Maik

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[Maik Qualmann via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

Maik Qualmann  changed:

   What|Removed |Added

  Attachment #2|0   |1
is obsolete||

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #65 from caulier.gil...@gmail.com ---
As i said previously, i processed on my VM with my collection all faces scan
with Mysql server + multicore CPU support. No crash.

About TBB support in OpenCV 3.1 that i compiled, i don't set anything to
configure OpenCV, excepted the fact to pass path to OpenCV-Contrib modules
source code to include Face module as well. That all.

As i can see TBB is disabled by default, so i suspect that no TBB here.

Gilles Caulier

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #64 from caulier.gil...@gmail.com ---
Evert,

My patch use OpenCV 3.1 tarball. OpenCV-contrib come from github as well
(current implementation. As i know there is no OpenCV-contrib tarball released.
This is very problematic as API/ABI version can be the hell to follow. This is
why i included face module code in digiKam core as well instead to use
OpenCV-contrib instead. Sound like my firts choice is not too bad after all.

As i already said somewhere and sometime, openCV is a another big puzzle...

I can update code in digiKam core about face module, as i know now what i need
to change to compile fine with OpenCV 3.1.

Gilles Caulier

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[Evert Vorster via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #63 from Evert Vorster  ---
Update on the status. 
I am now compiling digikam with all the patches except for Gilles' patch to use
face module form opencv_contrib instead from digiKam core. That patch causes a
compile error. I think it's due to the abi version of opencv that I have is
slightly different. 

With just Maik's face31.patch installed I was able to scan through my
collection with about 4 tries. I ended up with about 4000 faces identified out
of 32000 photos. 
I am now going to try again with all the patches installed and see if they
address different issues.

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[Evert Vorster via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #62 from Evert Vorster  ---
Just tested with the vanilla code, and digikam does indeed crash earlier than
with the patch. I am now installing just Maik's patch, and re-running it.

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[Evert Vorster via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #61 from Evert Vorster  ---
Small correction, it seems my source tree was not cleaned as thoroughly as I
thought it was, and so I appear to have built digikam with all the patches
applied when I thought it was not. 
My apologies. I am now rebuilding vanilla digikam, to see if it still fails
with TBB disabled in OpenCV.

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[Evert Vorster via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #60 from Evert Vorster  ---
Hi there.. 

OK, so, I recompiled digikam with no patch, and ran the face detection against
my picture library. 
The face detect would usually fail anywhere from 1 to 4% through the library. 

Ever since I have recompiled opencv to not use TBB, digikam scans much further
into the library, regardsless of whether there is a patch applied or not. So I
think we are at the root of this issue.

I will open a bug report with opencv, as the bug is in their handling of TBB.

I will open another bug for the failures I am seeing now, just to keep the
troubleshooting that was done seperate and not confuse the issue. Thank you
very much for your help!

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #59 from caulier.gil...@gmail.com ---
Note : with my patch, multi-coree enabled, and mysql server, no crash. All work
fine.

I didn't see the tbb opencv dependency. I will look if mine as this module
enabled.

Gilles Caulier

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[Evert Vorster via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #58 from Evert Vorster  ---
OK, so now DigiKam crashes with both internal and external SQL, at
approximately 30% through my collection. I'll start a new bug for that. 
I get a compile error with Gilles' patch, and Maik's seems to head off the
error with TBB if it is disabled in opencv.
I'll compile digikam without any patch, and run it against my collection with
both databases to confirm that Maik's patch is needed in conjunction with a TBB
free opencv.

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[Evert Vorster via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #57 from Evert Vorster  ---
Created attachment 12
  --> https://bugs.kde.org/attachment.cgi?id=12=edit
Compile error Gilles patch to use external face module

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[Evert Vorster via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #56 from Evert Vorster  ---
Now we are getting somewhere. 

I finally managed to compile a version of opencv that has tbb disabled.

With Maik's patch installed I was able to scan through 25% of my library before
I hit the bug of "MySQL server has gone away", so definitely have one issue
nailed down to opencv's use of tbb. 
I can try again with internal SQlite, and see if I can scan through my
collection completely. 

Since this version of opencv I have now is the contrib version, I will try
again to build with Gilles' patch. 

Looking back over the patches, which, if any, are recommended for me?
Do I file a bug report at OpenCV for the crashes in TBB? (I did a bit of
reading about it, and appears that the memory errors that valgrind reports is
because TBB uses their own scheduler and that can confuse valgrind)

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[Maik Qualmann via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #55 from Maik Qualmann  ---
The cause is compiled libtbb2 support in OpenCV. Gilles, look in the crash logs
from Evert, our distributions have libtbb2 disabled in OpenCV. See my Comment
52

Maik

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[Evert Vorster via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #54 from Evert Vorster  ---
Gilles,
 I found the contrib version of OpenCV, but the network link I am behind is
terribly slow, and I have been unable to download and compile it. I tried your
patch anyways, but digikam absolutely requires the contrib package of opencv. 

Right now I am trying Maik's patch, as that allows me to compile against the
version of OpenCV that is included in standard Arch. 

In a couple of weeks I will be home, with faster and more reliable internet,
and then I will try your patch again.

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #53 from caulier.gil...@gmail.com ---
Ever,

Faces scan with Mysql dataabse server is complete without a crash. I used
single core to compute face detection.

I will not try to use whole cpu core.

Gilles Caulier

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[Maik Qualmann via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #52 from Maik Qualmann  ---
My packages for OpenCV are not using a TBB (libtbb2). Your Yes.
It could be related herewith:

http://code.opencv.org/issues/4489

Maik

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #51 from caulier.gil...@gmail.com ---
Screenshot of face scan with remote mysql in action :

https://www.flickr.com/photos/digikam/28121515122/in/dateposted-public/

Gilles Caulier

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #50 from caulier.gil...@gmail.com ---
Evert,

With my patch, and Mysql remote server, face scan is started over the same
collection and no crash appear until now Wait and see...

Gilles Caulier

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #49 from caulier.gil...@gmail.com ---
Evert,

The face scan with my patch is finished with 244 faces found and no crash

Now i will test with a Mysql internal database.

Gilles Caulier

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[Maik Qualmann via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #48 from Maik Qualmann  ---
Created attachment 2
  --> https://bugs.kde.org/attachment.cgi?id=2=edit
face31.patch

New face modul from OpenCV-3.1. But not the latest version.

Maik

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[Maik Qualmann via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #47 from Maik Qualmann  ---
Gilles,

I have a patch created for the new face recognition already a few months ago.
But I see a slight change in the ABI to the current code from github. OpenCV
minimum would be version 3.1. I will upload the patch for testing here.

Maik

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #46 from caulier.gil...@gmail.com ---
With my patch, detection process my whole collection with 2271 images using
sqlite database.

Yes, it's not too huge but it's a test collection on my virtual machine.

Until now, all work fine. No crash, faces are detected (15%)

Gilles Caulier

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #45 from caulier.gil...@gmail.com ---
Created attachment 1
  --> https://bugs.kde.org/attachment.cgi?id=1=edit
patch to use face module from opencv_contrib instead from digiKam core

Evert,

New patch to drop face module in digiKam core and use last one from
opencv_contrib.
Opencv3 need to be compiled with OpenCV_Contrib module of course...

Maik,

I have a big doubt on the right way to  implement this new pure virtual method
:

void predict(InputArray src, Ptr collector) const = 0;

... defined in face.hpp and to code in facerec_borrowed.cpp

Gilles Caulier

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[Evert Vorster via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #44 from Evert Vorster  ---
Created attachment 0
  --> https://bugs.kde.org/attachment.cgi?id=0=edit
The list of files from OpenCV

I am on Arch linux, we don't have contrib. The closest thing is self compile
from git. :)
I have attached the file list of the OpenCV I have installed on my system. It's
version 3.1.0, and looks like it includes face detection. 

What is weird about this bug is that it does do quite a few face detections
before it fails, but not always at the same point.

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #43 from caulier.gil...@gmail.com ---
Do you have opencv3-contrib installed. This one include face detection
algorithm.

Typically for 5.0.0 i included this opencv-face directly in digiKam core, as
opencv 2 to 3 still a transition switch.

If code backported few mont ago has a bug, i tried to update it as well without
success. 

So the idea is to require opencv contrib at configuration time though digiKam
cmake script and to remove opencv-face module from digiKam core and use system
based module instead.

Gilles Caulier

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #42 from caulier.gil...@gmail.com ---
Here with sqlite, the crash is not reproducible. I compiled OpenCV3 myself and
uninstalled openCV2 before, to prevent binaries mix of the library.

Using valgrind reduce execution speed as you have seen. The program is executed
in a "VM" like which check all memory allocation and use. If the crash is due
to a race condition as i suspect, you will not able to reproduce it. This is
why i would to add a mutex is the critical section of the OpenCV call code to
be sure that method is not called more than one time by separated threads.

But as you say that crash is also reproducible just when tagging face as well.
perhaps the problem is in OpenCV as well... I don't know

This is why a check with OpenCV2 can be interesting to do.

Gilles Caulier

Gilles Caulier

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[Evert Vorster via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #41 from Evert Vorster  ---
Created attachment 99988
  --> https://bugs.kde.org/attachment.cgi?id=99988=edit
Segfault with facedetect patch applied, running on SQlite db

Looks like it's not the database backend, then, as it segfaults in exactly the
same way when running internal sqlite instead of MySQL. 

I suppose the only thing left to do now would be to get it to segfault while
running valgrind, right?

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[Evert Vorster via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #40 from Evert Vorster  ---
I was just updating some of the face tags that the recognizer got wrong,
without scanning for new faces, and this rather quickly segfaults digikam. 

I am now starting up digikam with the sqlite db, and of course it has to scan
through all my photos first, which takes a while.

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #39 from caulier.gil...@gmail.com ---
>I found that just changing the tag associated to a picture will segfault 
>digikam

Do you mean a simple tag to image or a face tag ? Both are different in the way
to process data in background.

Gilles Caulier

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[Evert Vorster via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #38 from Evert Vorster  ---
Created attachment 99987
  --> https://bugs.kde.org/attachment.cgi?id=99987=edit
gdb crash log with facedetect patch loaded.

Unfortunately, it seems that the facedetect patch did not stop the segfaults. 

I ran it through gdb, as that is quite a bit faster than valgrind. This crash
was reproduced by just scanning for faces, no recognition, and not surfing
around in the detected faces tags. I found that just changing the tag
associated to a picture will segfault digikam, in fact, that happens quite a
lot. 

I will now go disable the external mysql, and see if I can make it crash.

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #37 from caulier.gil...@gmail.com ---
Maik,

Your facedetector.patch must be applied to git/master in all cases...

Gilles Caulier

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #36 from caulier.gil...@gmail.com ---
>I will notify the maintainer of digikam-git that libexiv2 is no longer a 
>requirement for >digikam. 

Not libexiv2, but libkexiv2.

For dependencies details, look here :

https://quickgit.kde.org/?p=digikam-software-compilation.git=blob=DEPENDENCIES

Gilles Caulier

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[Evert Vorster via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #35 from Evert Vorster  ---
I am currently running digikam through valgrind. Once that crashes, I will
install Maik's patch only, and see if I can reproduce the issue. 

I will notify the maintainer of digikam-git that libexiv2 is no longer a
requirement for digikam. 

I can turn off the external database for testing purposes, but having the
database external is very much a desired feature for me. 

Unfortunately, downgrading to opencv 2 is not an option on this machine, there
is too much of my other software that depend on it.

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[Maik Qualmann via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #34 from Maik Qualmann  ---
Created attachment 99986
  --> https://bugs.kde.org/attachment.cgi?id=99986=edit
facedetector.patch

Please test also this patch.

Maik

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #33 from caulier.gil...@gmail.com ---
There is some test to do , if possible.

1/ using OpenCV2 instead OpenCV3. There is a flags to turn off in digiKam cmake
configuration script before to compile.
2/ using sqlite database instead Mysql, to see if crash is reproducible.
3/ in all case single core and multicore must be tested to validate.

Gilles Caulier

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #32 from caulier.gil...@gmail.com ---
Exiv2 is fine.

We don't use libkexiv2 anymore. Implementation is not in digiKam core, to
reduce the puzzle.

Gilles Caulier

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #31 from caulier.gil...@gmail.com ---
My patch must be the right direction but certainly not at the right level in
source code.

Look well as we touch data not initialized while OpenCV API call  It sound like
a non re-entrancy somewhere.

Remember that face fingerprints while detection is computed in a separated
thread. It's more complex when multi-cores are used. Perhaps the patch must be
applied in some top level call in facedetector class.

Gilles Caulier

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[Evert Vorster via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #30 from Evert Vorster  ---
Gilles:
extra/exiv2 0.25-3 
and libkexiv2 from git, r782.6c196e4-1

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[Evert Vorster via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #29 from Evert Vorster  ---
Maik, thanks again for looking into this. 

I am currently doing a valgrind log hoping to catch another segfault. It's just
so terribly slow!
On every try I get about 3 - 6% through the database before digikam quits. Like
I mentioned before there seems to be two distict issues, one where the database
goes away, and one where digikam segfaults. 

I have applied the small patch from Gilles,  but it does not seem to have made
a difference to either error. 

I have to re-iterate that this is scanning through the database doing faces
detect only. If I do faces detect and recognize, the process completes very
quickly, and no faces are detected. 

The four pictures listed above open fine with Gwenview.

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #28 from caulier.gil...@gmail.com ---
Maik,

even if images are broken, or if video/raw files crash Exiv2, all Exiv2 API are
wrapped around exception catch which must be handle by high level 
implementation in digiKam. 

Typically if a file break Exiv2, we must able to take the right direction to
prevent an broken workflow in face detection threads.

It's know that Exiv2 < 0.25 are very sensible of video files for ex, but it's
better now with last 0.25 stable release.

Evert;

Just to be sure, which Exiv2 version do you use exactly ?

Gilles Caulier

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[Maik Qualmann via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #27 from Maik Qualmann  ---
Can you check if these images are broken?

/data/DigiKam/Photos/Home/2015/02-28 Oval Track/Evert Pictures/IMG_7377.JPG
/data/DigiKam/Photos/Home/2015/02-28 Oval Track/Evert Pictures/IMG_7378.JPG
/data/DigiKam/Photos/Home/2015/02-28 Oval Track/Evert Pictures/IMG_7381.JPG
/data/DigiKam/Photos/Home/2015/02-28 Oval Track/Evert Pictures/IMG_7383.JPG

Maik

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[Evert Vorster via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #26 from Evert Vorster  ---
Created attachment 99984
  --> https://bugs.kde.org/attachment.cgi?id=99984=edit
Console output of crash after removing movie files.

So, I have removed all the movie files from the directory, and let the face
scanner do it's thing. 
Now I'll try again through gdb, and also remove anything that is not a .jpg,
just to be sure.

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[Evert Vorster via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #25 from Evert Vorster  ---
Created attachment 99981
  --> https://bugs.kde.org/attachment.cgi?id=99981=edit
valgrind log

OK, Thanks for looking into this. 
I have moved all the .mov and .mp4 from that directory. 

There are still some cr2, png, and tiff mixed in, but exiv should not have a
problem with those. 
Right now digikam is running through my collection, and let's see if it
crashes.

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[Maik Qualmann via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #24 from Maik Qualmann  ---
I see in the valgrind log that QuickTime video is included. I think it's
possible that Exiv2 crashes here. Can you these video files for test removed in
face recognition?

Maik

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[Evert Vorster via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #23 from Evert Vorster  ---
Ah, Maik, the same error is repeated in the valgrind log that I just attached,
and since I did not type that one, it definitely does not contain typos. 

Thanks for looking into this!
-Evert-

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[Evert Vorster via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #22 from Evert Vorster  ---
Re: Maik Qualmann, 
I did indeed make a typo. 

Thanks for pointing that out. 
-Evert-

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[Maik Qualmann via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

Maik Qualmann  changed:

   What|Removed |Added

 CC||metzping...@gmail.com

--- Comment #21 from Maik Qualmann  ---
> digikam.dbengine: Failure executing query:
>  "SELECT orientation FROM ImagaInformation WHERE imageid=?;"

Have you copied this string or it may contain a typo?

ImagaInformation => ImageInformation
The query string is correct in digKam source code.

Maik

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[Evert Vorster via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #20 from Evert Vorster  ---
Created attachment 99978
  --> https://bugs.kde.org/attachment.cgi?id=99978=edit
valgrind with mysql gone away error

There seems to be two distict modes of failure here. One is the "mysql gone
away" error, and the other is a segfault. 
I just attached a valgrind for the mysql gone away error, I'll re-run and see
if I am "lucky" enough to catch the program segfaulting in valgrind.

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[Evert Vorster via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #19 from Evert Vorster  ---
Created attachment 99977
  --> https://bugs.kde.org/attachment.cgi?id=99977=edit
Segfault, with the patch applied on detect faces only

Once I rebuilt digikam with debugging enabled, I ran it through gdb with
detecting faces only. 
It failed fairly quickly with a segfault, and I have attached the log here. 

Right now I am running the same through Valgrind, but it might be a while
before it crashes. 

-Evert-

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[Evert Vorster via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #18 from Evert Vorster  ---
Hey there... 
Unfortunately, that patch did not stop the crashing. 
With just scanning for faces, it got about 6% through my collection of 30,000
photos ( 160Gb ) before crashing again. This time it's not a segfault, but the
program halts with this weird error:
digikam.dbengine: Prepare failed!
digikam.dbengine: Failure executing query:
 "SELECT orientation FROM ImagaInformation WHERE imageid=?;"
Error messages: "QMYSQL3: Unable to prepare statement" "MySQL server has gone
away" 2006 2
Bound values: ()

One thing worth noticing though, is that I also tried to scan and recognize
faces, and it zipped through that entire collection in seconds, and did not
find a single face. 

Just recognizing faces on their own seems to work fine, however. 

I'll start building digikam with debugging enabled, and see if I can track this
one down.,

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault [patch]

[via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

caulier.gil...@gmail.com changed:

   What|Removed |Added

Summary|External MySQL database |External MySQL database
   |faces update segfault   |faces update segfault
   ||[patch]

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault

[via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #17 from caulier.gil...@gmail.com ---
Created attachment 99970
  --> https://bugs.kde.org/attachment.cgi?id=99970=edit
add mutex to prevent non re-entrency in OpenCV API

Try with my little patch if it fix the problem.
Gilles Caulier

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault

[Evert Vorster via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #16 from Evert Vorster  ---
I'm sorry, I can't read that log as clearly as you do. I know how to produce
them, and then hopefully someone smarter than me can pick at it and hopefully
update the git with a fix. 

Can you put it in layman's terms, please? What happens next?

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault

[via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #15 from caulier.gil...@gmail.com ---
Sound like the source of the problem :

==16753== Conditional jump or move depends on uninitialised value(s)
==16753==at 0x973DFAA: ??? (in /usr/lib/libopencv_objdetect.so.3.1.0)
==16753==by 0x9744026: ??? (in /usr/lib/libopencv_objdetect.so.3.1.0)
==16753==by 0xC22C56C: cv::parallel_for_(cv::Range const&,
cv::ParallelLoopBody const&, double) (in /usr/lib/libopencv_core.so.3.1.0)
==16753==by 0x9745AE0: ??? (in /usr/lib/libopencv_objdetect.so.3.1.0)
==16753==by 0x9750104: ??? (in /usr/lib/libopencv_objdetect.so.3.1.0)
==16753==by 0x973AE2D: ??? (in /usr/lib/libopencv_objdetect.so.3.1.0)
==16753==by 0x974EF74:
cv::CascadeClassifier::detectMultiScale(cv::_InputArray const&,
std::vector >&, double, int,
int, cv::Size_, cv::Size_) (in /usr/lib/libopencv_objdetect.so.3.1.0)
==16753==by 0x5000F2D:
FacesEngine::OpenCVFaceDetector::cascadeResult(cv::Mat const&,
FacesEngine::Cascade&, FacesEngine::DetectObjectParameters const&) const
(opencvfacedetector.cpp:469)
==16753==by 0x5001F5A: FacesEngine::OpenCVFaceDetector::verifyFace(cv::Mat
const&, QRect const&) const (opencvfacedetector.cpp:536)
==16753==by 0x500360C: FacesEngine::OpenCVFaceDetector::detectFaces(cv::Mat
const&, cv::Size_ const&) (opencvfacedetector.cpp:767)
==16753==by 0x5019A31: FacesEngine::FaceDetector::detectFaces(QImage
const&, QSize const&) (facedetector.cpp:160)
==16753==by 0x5324A5A:
Digikam::DetectionWorker::process(QExplicitlySharedDataPointer)
(facepipeline.cpp:483)
==16753==  Uninitialised value was created by a heap allocation
==16753==at 0x4C2ABD0: malloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==16753==by 0xC079B21: cv::fastMalloc(unsigned long) (in
/usr/lib/libopencv_core.so.3.1.0)
==16753==by 0xC1CD3D0: cv::Mat::create(int, int const*, int) (in
/usr/lib/libopencv_core.so.3.1.0)
==16753==by 0x974BB2E: ??? (in /usr/lib/libopencv_objdetect.so.3.1.0)
==16753==by 0x9745609: ??? (in /usr/lib/libopencv_objdetect.so.3.1.0)
==16753==by 0x9750104: ??? (in /usr/lib/libopencv_objdetect.so.3.1.0)
==16753==by 0x973AE2D: ??? (in /usr/lib/libopencv_objdetect.so.3.1.0)
==16753==by 0x974EF74:
cv::CascadeClassifier::detectMultiScale(cv::_InputArray const&,
std::vector >&, double, int,
int, cv::Size_, cv::Size_) (in /usr/lib/libopencv_objdetect.so.3.1.0)
==16753==by 0x5000F2D:
FacesEngine::OpenCVFaceDetector::cascadeResult(cv::Mat const&,
FacesEngine::Cascade&, FacesEngine::DetectObjectParameters const&) const
(opencvfacedetector.cpp:469)
==16753==by 0x5001F5A: FacesEngine::OpenCVFaceDetector::verifyFace(cv::Mat
const&, QRect const&) const (opencvfacedetector.cpp:536)
==16753==by 0x500360C: FacesEngine::OpenCVFaceDetector::detectFaces(cv::Mat
const&, cv::Size_ const&) (opencvfacedetector.cpp:767)
==16753==by 0x5019A31: FacesEngine::FaceDetector::detectFaces(QImage
const&, QSize const&) (facedetector.cpp:160)

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault

[Evert Vorster via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #14 from Evert Vorster  ---
Created attachment 99947
  --> https://bugs.kde.org/attachment.cgi?id=99947=edit
Konsole output around the error.

Noticed this error about halfway down the attached file. For some reason MySQL
was "not available"

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault

[Evert Vorster via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #13 from Evert Vorster  ---
Created attachment 99946
  --> https://bugs.kde.org/attachment.cgi?id=99946=edit
Valgrind log of a segfault

Finally, a Valgrind log of the error

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault

[via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

caulier.gil...@gmail.com changed:

   What|Removed |Added

 CC||swatilodh...@gmail.com

--- Comment #12 from caulier.gil...@gmail.com ---
If the crash is a race condition between the FaceManagement code and the
database interface, perhaps valgrind can help to identify when memory is
corrupted. Typically it's in face histogram computation thread.

Note : Face detection do not crash using Sqlite database here. I process 20K
images in 10 minutes on my main linux computer.

Gilles Caulier

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault

[Evert Vorster via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #11 from Evert Vorster  ---
The database is MariaDB, which is the same as MySQL, I believe. 

I use OpenCV 3.x

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault

[via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

--- Comment #10 from caulier.gil...@gmail.com ---
So you use Mysql as database. Right ?

and which Opencv library version you use ? 2.x or 3.x ?

Gilles Caulier

-- 
You are receiving this mail because:
You are watching all bug changes.

[digikam] [Bug 365202] External MySQL database faces update segfault

[via KDE Bugzilla]

https://bugs.kde.org/show_bug.cgi?id=365202

caulier.gil...@gmail.com changed:

   What|Removed |Added

  Component|Faces-Engine|Database-Mysql
Summary|External MySQL database |External MySQL database
   |update segfault |faces update segfault

-- 
You are receiving this mail because:
You are watching all bug changes.