Re: [Shotwell] My post on face recognition in shotwell
I pushed a change to add ifdef protection around the dnn code.https://gitlab.gnome.org/nma83/shotwell/commit/41da9cc11c4344a3ac317013fd4450d1df1673fa On Wednesday, 22 August, 2018, 12:42:31 AM IST, Jens Georg wrote: > > Any idea why debian does not include the dnn module in the opencv > > package? > > Not really. I am downloading the build infra for the Debian opencv > and > see what I can do. It seems that the opencv-contrib package only > contains the shared libs, but there is no development package for the > respective header files. > > I will see what is possible. I wanted to contact the debian maintainer/uloader of the OpenCV package. it was disabled in debian git without any reason given. You can use the flatpak, it has dnn enabled, but it currently doesn't find the helper, I'm still looking into that. ___ shotwell-list mailing list shotwell-list@gnome.org https://mail.gnome.org/mailman/listinfo/shotwell-list ___ shotwell-list mailing list shotwell-list@gnome.org https://mail.gnome.org/mailman/listinfo/shotwell-list
Re: [Shotwell] My post on face recognition in shotwell
Thanks for the feedback Jens. I will push the change in a day or two. On Wednesday, 22 August, 2018, 2:14:34 AM IST, Jens Georg wrote: On Tue, 2018-08-21 at 21:12 +0200, Jens Georg wrote: > > > Any idea why debian does not include the dnn module in the opencv > > > package? > > > > Not really. I am downloading the build infra for the Debian opencv > > and > > see what I can do. It seems that the opencv-contrib package only > > contains the shared libs, but there is no development package for > > the > > respective header files. > > > > I will see what is possible. > > I wanted to contact the debian maintainer/uloader of the OpenCV > package. it was disabled in debian git without any reason given. > > You can use the flatpak, it has dnn enabled, but it currently doesn't > find the helper, I'm still looking into that. Narendra, the binary is owning the "wrong" name pattern for the flatpak, it should own "org.gnome.Shotwell.Faces" - or even better "org.gnome.Shotwell.Faces1" otherwise the sandbox will not allow it without further configuration ___ shotwell-list mailing list shotwell-list@gnome.org https://mail.gnome.org/mailman/listinfo/shotwell-list ___ shotwell-list mailing list shotwell-list@gnome.org https://mail.gnome.org/mailman/listinfo/shotwell-list
Re: [Shotwell] My post on face recognition in shotwell
Looks like the dnn module is shipped in OpenCV3 but current dependency is defined as >= 2.3.0. We could update the version to be > 3.0. Any idea why debian does not include the dnn module in the opencv package?My archlinux install does have it as part of the standard opencv install.https://www.archlinux.org/packages/extra/x86_64/opencv/On Tuesday, 21 August, 2018, 6:38:48 PM IST, Norbert Preining wrote: Hi Narendra, > OpenCV uses the folder opencv2 to include the object oriented > definitions even in the version 3.0 . Apparently the dnn.hpp should be > in the package libopencv-contrib > (https://packages.debian.org/buster/libopencv-contrib3.2) , but I'm > using right now Ubuntu 16.04 and it doesn't ship the dnn.hpp file. Indeed, also Debian does not ship it. One can check here for existence of this file in Debian/unstable: https://packages.debian.org/search?searchon=contents=dnn.hpp=exactfilename=unstable=any > The only advice I could give is, if you are going to compile OpenCV > from https://github.com/opencv/opencv after clone the repository Umpf, that is not something I am going to do. It seems the dnn module is not compiled for Debian or Ubuntu. This is a pain. Thanks for checking. Norbert -- PREINING Norbert http://www.preining.info Accelia Inc. + JAIST + TeX Live + Debian Developer GPG: 0x860CDC13 fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13 ___ shotwell-list mailing list shotwell-list@gnome.org https://mail.gnome.org/mailman/listinfo/shotwell-list
[Shotwell] Basic face recognition working!
Hi, I managed to get a basic version of face recognition working in my branch.https://gitlab.gnome.org/nma83/shotwell/commit/f7b2c2203d2d7b5a722f5133f79ea8baf14d7fcd It uses the OpenFace DNN model I had described earlier.There are some optimizations to be done for storing the intermediate face representation vector.The UI may also need improvement. I added a 'Train Face from Photo' menu item in the FacePage menu for now.A face from a single photo is used as reference to compare with unknown faces in other photos when running detection.I also modified the DBus interface to do both face boundary detection and vector generation in the same call.More feedback is welcome. Thanks,Narendra___ shotwell-list mailing list shotwell-list@gnome.org https://mail.gnome.org/mailman/listinfo/shotwell-list
Re: [Shotwell] Adding face recognition to shotwell
Hi, I modified the DBus code to use activation for launching the facedetect process. The build and install process has not been updated to install the DBus service file at the right place though. I presume there needs to be a meson directive to do that.As for the DNN model, should we create a post install hook in the various packages to prompt for downloading the model from the URL into shotwell's data directory? There could be a license prompt before downloading for the Apache 2.0 license. Thanks,Narendra On Wednesday, 18 July, 2018, 8:45:09 PM IST, Narendra Acharya wrote: Hi Jens, I agree that we should not be forcing the DNN model download for everyone. It could be an optional 'data' package which is required only for face recognition and not detection. This data package could probably be under a different license like you said.So I will assume the basic facedetect binary would be packaged with shotwell, with a dbus service hook to launch it. On Wednesday, 18 July, 2018, 8:40:16 PM IST, Jens Georg wrote: > So do we now create 2 packages out of the shotwell tree? How would the > facedetect package be published to various distributions? > And should the facedetect binary become a service which can be > launched by shotwell, or maybe by the OS? You should not care about this at all. Either me (for flatpak) or whoever is responsible for the distribution packages. Being D-Bus (and running on the public busses) automatic launching comes for free with D-Bus activation, given the D-Bus service files are installed properly. But we can do that in a second step. The more problematic thing I see here is that I would not like to force everyone to download 31Mb of data for a feature they are probably not even using The other issue is that I have a vague memory of Apache + GPL being a bit of a problem. So probably the face detect binary might have to go into a separate, differently licensed package that uses D-Bus for "license isolation" anyway. Let me have a look at the code first, though. ___ shotwell-list mailing list shotwell-list@gnome.org https://mail.gnome.org/mailman/listinfo/shotwell-list
Re: [Shotwell] Adding face recognition to shotwell
Hi Jens, I agree that we should not be forcing the DNN model download for everyone. It could be an optional 'data' package which is required only for face recognition and not detection. This data package could probably be under a different license like you said.So I will assume the basic facedetect binary would be packaged with shotwell, with a dbus service hook to launch it. On Wednesday, 18 July, 2018, 8:40:16 PM IST, Jens Georg wrote: > So do we now create 2 packages out of the shotwell tree? How would the > facedetect package be published to various distributions? > And should the facedetect binary become a service which can be > launched by shotwell, or maybe by the OS? You should not care about this at all. Either me (for flatpak) or whoever is responsible for the distribution packages. Being D-Bus (and running on the public busses) automatic launching comes for free with D-Bus activation, given the D-Bus service files are installed properly. But we can do that in a second step. The more problematic thing I see here is that I would not like to force everyone to download 31Mb of data for a feature they are probably not even using The other issue is that I have a vague memory of Apache + GPL being a bit of a problem. So probably the face detect binary might have to go into a separate, differently licensed package that uses D-Bus for "license isolation" anyway. Let me have a look at the code first, though. ___ shotwell-list mailing list shotwell-list@gnome.org https://mail.gnome.org/mailman/listinfo/shotwell-list
Re: [Shotwell] Adding face recognition to shotwell
Hi Andreas, I am not familiar with flatpak, so will read up and see how I can deploy this service using it.So the preferred method seems to be separate package for the facedetect binary which comes with the DNN model. Thanks,Narendra On Tuesday, 17 July, 2018, 12:44:49 PM IST, Andreas Brauchli wrote: Hi Narendra, On Mon, 2018-07-16 at 15:28 +, Narendra Acharya via shotwell-list wrote: > Hi, > > As mentioned in my recent posts, I am working on better integration > of face detection (via DBus) and add face recognition as the next > step. The DBus connection cleanup is mostly done in my > fork: https://gitlab.gnome.org/nma83/shotwell/commits/wip/faces. Thanks, looking forward to it. > In order to get to face recognition, I would like to use the OpenFace > method described here: https://cmusatyalab.github.io/openface/ > It involves running the extracted face image (already available from > detection) through a DNN to get a 128 element vector which supposedly > represents the face. This vector needs to be stored (say in > FaceTable) per face. Face recognition can be performed by > transforming the face to be recognized to a similar vector and > correlating via dot-product with each stored face's vector. The one > with the highest dot product is the most likely match. > To do this, a DNN model is needed which is open-source under Apache > 2.0. > https://storage.cmusatyalab.org/openface-models/nn4.small2.v1.t7t7 - > 31MB size > How could we add this model as part of the shotwell package? Or > create another package which can be an optional dependency. Since you're using DBus, the application coupling is very low already, so it sounds like the better approach is indeed to provide a standalone face-recognition service. This could then also be used by other apps wishing to connect to it (say, gnome photos). As such, you'd bundle it separately. Shotwell should then only need the code to interface with the DBus service: detect if it's available, maybe start it if necessary and send photo-handles to run the recognition on. Maybe you can even Flatpak your bundle and have it export the DBus service. That would make it quite easy to deploy. > > Thanks, > Narendra Cheers, Andreas > ___ > shotwell-list mailing list > shotwell-list@gnome.org > https://mail.gnome.org/mailman/listinfo/shotwell-list ___ shotwell-list mailing list shotwell-list@gnome.org https://mail.gnome.org/mailman/listinfo/shotwell-list
[Shotwell] Adding face recognition to shotwell
Hi, As mentioned in my recent posts, I am working on better integration of face detection (via DBus) and add face recognition as the next step. The DBus connection cleanup is mostly done in my fork: https://gitlab.gnome.org/nma83/shotwell/commits/wip/faces.In order to get to face recognition, I would like to use the OpenFace method described here: https://cmusatyalab.github.io/openface/It involves running the extracted face image (already available from detection) through a DNN to get a 128 element vector which supposedly represents the face. This vector needs to be stored (say in FaceTable) per face. Face recognition can be performed by transforming the face to be recognized to a similar vector and correlating via dot-product with each stored face's vector. The one with the highest dot product is the most likely match.To do this, a DNN model is needed which is open-source under Apache 2.0. https://storage.cmusatyalab.org/openface-models/nn4.small2.v1.t7 - 31MB size How could we add this model as part of the shotwell package? Or create another package which can be an optional dependency. Thanks,Narendra___ shotwell-list mailing list shotwell-list@gnome.org https://mail.gnome.org/mailman/listinfo/shotwell-list
Re: [Shotwell] Enhance face detection
Thanks for the response Jens. I will try this out. On Friday, 6 July, 2018, 2:07:15 PM IST, Jens Georg wrote: Hi, regarding your question on IRC: set_crop is probably not a good idea because it will set the transformation in the database. Rather get the photo, use pb = photo.get_pixbuf_with_options(Scaling.for_original()) with all transformations (scaling, rotation etc.) applied. You can then use new Pixbuf.subpixbuf from that and the coordinates in the face description. This is under the assumption that the coordinates in the Face are applicable to the final transformed image > I have made the first level changes to migrate from process execution > for every detect faces call, to one time execution of facedetect > process and then communicate over DBus. > https://gitlab.gnome.org/nma83/shotwell/commit/4c84248b80ebf3a1f1eb05aa60625d9d8c5d0134 > > The DBus interface XML is at: > https://gitlab.gnome.org/nma83/shotwell/blob/wip/faces/facedetect/org.gnome.ShotwellFaces1.xml > > I have implemented DetectFaces and Terminate methods. The next steps > are: > > 1. Extract the faces detected in photos and cache them (downsizing if > needed) - maybe in $HOME/.cache/shotwell > 2. Run face training on the labelled faces, with a minimum number of > faces per label (maybe 10) > 3. Attempt to automatically recognise and label faces in photos after > training > > On Saturday, 23 June, 2018, 8:54:43 PM IST, Jens Georg > wrote: > > On Fri, 2018-06-22 at 08:08 +, Narendra Acharya via shotwell-list > wrote: > >> Hi, >> >> I was looking at the current face detection code in shotwell 0.29.2 >> and feel that it can be enhanced. >> So I pulled the code from >> https://gitlab.gnome.org/GNOME/shotwell.git and started making some >> changes. >> 1. Do not run another process for face detection >> (shotwell-facedetect) but integrate the OpenCV calls into shotwell >> and link to OpenCV libraries > > This is potentially problematic as OpenCV can be compiled against Gtk2 > - which will break Shotwell then and is the reason why the detection > was done in a separate process in the first place. > >> 2. Ability to use OpenCL when available on the platform using T-API >> in OpenCV 3.x >> 3. Eventually add face recognition on photos using a trained model >> based on faces labelled by the user - this would need a few faces to >> be tagged by the user before the OpenCV face recognition calls can >> be used >> > (https://docs.opencv.org/2.4/modules/contrib/doc/facerec/facerec_tutorial.html). > > There is also some prior work on this by the original author of > facedetect: https://github.com/Sanva/facedetect > >> Is there anyone else working on this area? > >> Thanks, >> Narendra >> >> ___ >> >> shotwell-list mailing list >> >> shotwell-list@gnome.org >> >> https://mail.gnome.org/mailman/listinfo/shotwell-list ___ shotwell-list mailing list shotwell-list@gnome.org https://mail.gnome.org/mailman/listinfo/shotwell-list
Re: [Shotwell] Enhance face detection
I have made the first level changes to migrate from process execution for every detect faces call, to one time execution of facedetect process and then communicate over DBus.https://gitlab.gnome.org/nma83/shotwell/commit/4c84248b80ebf3a1f1eb05aa60625d9d8c5d0134 The DBus interface XML is at:https://gitlab.gnome.org/nma83/shotwell/blob/wip/faces/facedetect/org.gnome.ShotwellFaces1.xml I have implemented DetectFaces and Terminate methods. The next steps are: 1. Extract the faces detected in photos and cache them (downsizing if needed) - maybe in $HOME/.cache/shotwell2. Run face training on the labelled faces, with a minimum number of faces per label (maybe 10)3. Attempt to automatically recognise and label faces in photos after training On Saturday, 23 June, 2018, 8:54:43 PM IST, Jens Georg wrote: On Fri, 2018-06-22 at 08:08 +, Narendra Acharya via shotwell-list wrote: Hi, I was looking at the current face detection code in shotwell 0.29.2 and feel that it can be enhanced.So I pulled the code from https://gitlab.gnome.org/GNOME/shotwell.git and started making some changes.1. Do not run another process for face detection (shotwell-facedetect) but integrate the OpenCV calls into shotwell and link to OpenCV libraries This is potentially problematic as OpenCV can be compiled against Gtk2 - which will break Shotwell then and is the reason why the detection was done in a separate process in the first place. 2. Ability to use OpenCL when available on the platform using T-API in OpenCV 3.x3. Eventually add face recognition on photos using a trained model based on faces labelled by the user - this would need a few faces to be tagged by the user before the OpenCV face recognition calls can be used (https://docs.opencv.org/2.4/modules/contrib/doc/facerec/facerec_tutorial.html). There is also some prior work on this by the original author of facedetect: https://github.com/Sanva/facedetect Is there anyone else working on this area? Thanks,Narendra ___shotwell-list mailing listshotwell-list@gnome.orghttps://mail.gnome.org/mailman/listinfo/shotwell-list ___ shotwell-list mailing list shotwell-list@gnome.org https://mail.gnome.org/mailman/listinfo/shotwell-list
Re: [Shotwell] Enhance face detection
OK great. I will post an update here once I have the current functionality working over D-bus and a facedetect background process. Hope I can use the IRC channel for any other minor questions I may have. On Sunday, 24 June, 2018, 3:59:20 PM IST, Jens Georg wrote: > I understand the need for keeping OpenCV code in a separate process. > Even so, I think we should consider starting this process as a > 'daemon' when starting shotwell (controlled by an option) and > communicate with it over RPC from the shotwell process instead of > executing it for every 'Detect faces' button press. I will start > looking at adding an RPC layer from today. Excellent! Please use D-Bus, that should be the easiest and is nicely supported in Vala. I would suggest a private D-Bus, not going on system/user/session Bus. (A small example is at https://gitlab.gnome.or g/GNOME/glib/blob/master/gio/tests/gdbus-example-peer.c ) but session Bus should work as well. > That said, do we expect OpenCV to link to a different GTK version in > the long term? The OpenCV 3.4 I use in the flatpak is still looking for GTK2, so there's always potential that someone will enable that combination :-/ > > Once the RPC is done, will try to integrate the latest from the > Sanva/facedetect repo and hope to get some basic face recognition > capability. ___ shotwell-list mailing list shotwell-list@gnome.org https://mail.gnome.org/mailman/listinfo/shotwell-list
Re: [Shotwell] Enhance face detection
Hi Jens, I understand the need for keeping OpenCV code in a separate process. Even so, I think we should consider starting this process as a 'daemon' when starting shotwell (controlled by an option) and communicate with it over RPC from the shotwell process instead of executing it for every 'Detect faces' button press. I will start looking at adding an RPC layer from today.That said, do we expect OpenCV to link to a different GTK version in the long term? Once the RPC is done, will try to integrate the latest from the Sanva/facedetect repo and hope to get some basic face recognition capability. Thanks,Narendra On Saturday, 23 June, 2018, 8:57:20 PM IST, Jens Georg wrote: On Fri, 2018-06-22 at 10:48 -0300, Ricardo via shotwell-list wrote: > Hello. > In https://valadoc.org/opencv/index.htm there is a vapi to link > direct > from vala to opencv without c/c++ in the middle. Some considerations: > First, it use OpenCV 2 which cannot detect automatically the presence > of a graphic card capable to use OpenCL. > Second, in the last stable Debian the OpenCV3 is not available yet. > So > it would not be possible to easily compile it in Debian. > In the vapi, the functions about matrix eigenvalues to recognize > specific person faces are available too. The nightly flatpak comes with OpenCV 3, so using Gnome builder + flatpack might make that possible. > > I'm working in a new user experience - > https://gitlab.gnome.org/GNOME/shotwell/issues/13 - for faces. If you > work in bring the OpenCV3 to valadoc and use that vapi in shotwell > probably we will not work in the same lines of code and we will not > have much work in a merge. > > Phako, some time ago in a pull-request you mentioned that the use of > c++ is not a big issue. Is ok to Narendra work in the use of the > OpenCV vapi in shotwell now? See my other mail, there is a potential problem of introducing an incompatibility when linking against OpenCV directly. ___ shotwell-list mailing list shotwell-list@gnome.org https://mail.gnome.org/mailman/listinfo/shotwell-list
Re: [Shotwell] Enhance face detection
Hi Ricardo, There are problems with the OpenCV VAPI that I discovered while trying to integrate OpenCV directly.OpenCV is not C compatible anymore in the recent 3.x versions (https://github.com/opencv/opencv/issues/10963). So I had to create another VAPI bridge that took in plain C structures and then called the OpenCV functions. This bridge is wrapped in extern "C" so that vala can talk to it. OpenCV libraries are linked to the shotwell executable at the link stage.I think we can defer OpenCL usage until OpenCV3 gets into debian (it is available on my archlinux machine). I have forked the shotwell repo and created wip/faces branch with my current WIP changes:https://gitlab.gnome.org/nma83/shotwell/tree/wip/faces The face detection works as it used to in this branch, but using the VAPI bridge to OpenCV instead of another process.I am exploring other options for better accuracy - OpenCV with DNN or dlib. The UI changes should not conflict with my change like you said, so merge should be seamless once we get there. Thanks,Narendra From: Ricardo To: Narendra Acharya Cc: "shotwell-list gnome org" Subject: Re: [Shotwell] Enhance face detectionDate: Fri, 22 Jun 2018 10:48:21 -0300Hello.In https://valadoc.org/opencv/index.htm there is a vapi to link directfrom vala to opencv without c/c++ in the middle. Some considerations:First, it use OpenCV 2 which cannot detect automatically the presenceof a graphic card capable to use OpenCL.Second, in the last stable Debian the OpenCV3 is not available yet. Soit would not be possible to easily compile it in Debian.In the vapi, the functions about matrix eigenvalues to recognizespecific person faces are available too.I'm working in a new user experience -https://gitlab.gnome.org/GNOME/shotwell/issues/13 - for faces. If youwork in bring the OpenCV3 to valadoc and use that vapi in shotwellprobably we will not work in the same lines of code and we will nothave much work in a merge.Phako, some time ago in a pull-request you mentioned that the use ofc++ is not a big issue. Is ok to Narendra work in the use of theOpenCV vapi in shotwell now?Bye,Ricardo.2018-06-22 5:08 GMT-03:00, Narendra Acharya via shotwell-list: Hi, I was looking at the current face detection code in shotwell 0.29.2 and feel that it can be enhanced.So I pulled the code from https://gitlab.gnome.org/GNOME/shotwell.git and started making some changes.1. Do not run another process for face detection (shotwell-facedetect) but integrate the OpenCV calls into shotwell and link to OpenCV libraries2. Ability to use OpenCL when available on the platform using T-API in OpenCV 3.x3. Eventually add face recognition on photos using a trained model based on faces labelled by the user - this would need a few faces to be tagged by the user before the OpenCV face recognition calls can be used (https://docs.opencv.org/2.4/modules/contrib/doc/facerec/facerec_tutorial.html). Is there anyone else working on this area? Thanks,Narendra ___ shotwell-list mailing list shotwell-list@gnome.org https://mail.gnome.org/mailman/listinfo/shotwell-list
[Shotwell] Enhance face detection
Hi, I was looking at the current face detection code in shotwell 0.29.2 and feel that it can be enhanced.So I pulled the code from https://gitlab.gnome.org/GNOME/shotwell.git and started making some changes.1. Do not run another process for face detection (shotwell-facedetect) but integrate the OpenCV calls into shotwell and link to OpenCV libraries2. Ability to use OpenCL when available on the platform using T-API in OpenCV 3.x3. Eventually add face recognition on photos using a trained model based on faces labelled by the user - this would need a few faces to be tagged by the user before the OpenCV face recognition calls can be used (https://docs.opencv.org/2.4/modules/contrib/doc/facerec/facerec_tutorial.html). Is there anyone else working on this area? Thanks,Narendra ___ shotwell-list mailing list shotwell-list@gnome.org https://mail.gnome.org/mailman/listinfo/shotwell-list
