Re: [Shotwell] Enhance face detection
Thanks for the response Jens. I will try this out. On Friday, 6 July, 2018, 2:07:15 PM IST, Jens Georg wrote: Hi, regarding your question on IRC: set_crop is probably not a good idea because it will set the transformation in the database. Rather get the photo, use pb = photo.get_pixbuf_with_options(Scaling.for_original()) with all transformations (scaling, rotation etc.) applied. You can then use new Pixbuf.subpixbuf from that and the coordinates in the face description. This is under the assumption that the coordinates in the Face are applicable to the final transformed image > I have made the first level changes to migrate from process execution > for every detect faces call, to one time execution of facedetect > process and then communicate over DBus. > https://gitlab.gnome.org/nma83/shotwell/commit/4c84248b80ebf3a1f1eb05aa60625d9d8c5d0134 > > The DBus interface XML is at: > https://gitlab.gnome.org/nma83/shotwell/blob/wip/faces/facedetect/org.gnome.ShotwellFaces1.xml > > I have implemented DetectFaces and Terminate methods. The next steps > are: > > 1. Extract the faces detected in photos and cache them (downsizing if > needed) - maybe in $HOME/.cache/shotwell > 2. Run face training on the labelled faces, with a minimum number of > faces per label (maybe 10) > 3. Attempt to automatically recognise and label faces in photos after > training > > On Saturday, 23 June, 2018, 8:54:43 PM IST, Jens Georg > wrote: > > On Fri, 2018-06-22 at 08:08 +, Narendra Acharya via shotwell-list > wrote: > >> Hi, >> >> I was looking at the current face detection code in shotwell 0.29.2 >> and feel that it can be enhanced. >> So I pulled the code from >> https://gitlab.gnome.org/GNOME/shotwell.git and started making some >> changes. >> 1. Do not run another process for face detection >> (shotwell-facedetect) but integrate the OpenCV calls into shotwell >> and link to OpenCV libraries > > This is potentially problematic as OpenCV can be compiled against Gtk2 > - which will break Shotwell then and is the reason why the detection > was done in a separate process in the first place. > >> 2. Ability to use OpenCL when available on the platform using T-API >> in OpenCV 3.x >> 3. Eventually add face recognition on photos using a trained model >> based on faces labelled by the user - this would need a few faces to >> be tagged by the user before the OpenCV face recognition calls can >> be used >> > (https://docs.opencv.org/2.4/modules/contrib/doc/facerec/facerec_tutorial.html). > > There is also some prior work on this by the original author of > facedetect: https://github.com/Sanva/facedetect > >> Is there anyone else working on this area? > >> Thanks, >> Narendra >> >> ___ >> >> shotwell-list mailing list >> >> shotwell-list@gnome.org >> >> https://mail.gnome.org/mailman/listinfo/shotwell-list ___ shotwell-list mailing list shotwell-list@gnome.org https://mail.gnome.org/mailman/listinfo/shotwell-list
Re: [Shotwell] Enhance face detection
Hi, regarding your question on IRC: set_crop is probably not a good idea because it will set the transformation in the database. Rather get the photo, use pb = photo.get_pixbuf_with_options(Scaling.for_original()) with all transformations (scaling, rotation etc.) applied. You can then use new Pixbuf.subpixbuf from that and the coordinates in the face description. This is under the assumption that the coordinates in the Face are applicable to the final transformed image I have made the first level changes to migrate from process execution for every detect faces call, to one time execution of facedetect process and then communicate over DBus. https://gitlab.gnome.org/nma83/shotwell/commit/4c84248b80ebf3a1f1eb05aa60625d9d8c5d0134 The DBus interface XML is at: https://gitlab.gnome.org/nma83/shotwell/blob/wip/faces/facedetect/org.gnome.ShotwellFaces1.xml I have implemented DetectFaces and Terminate methods. The next steps are: 1. Extract the faces detected in photos and cache them (downsizing if needed) - maybe in $HOME/.cache/shotwell 2. Run face training on the labelled faces, with a minimum number of faces per label (maybe 10) 3. Attempt to automatically recognise and label faces in photos after training On Saturday, 23 June, 2018, 8:54:43 PM IST, Jens Georg wrote: On Fri, 2018-06-22 at 08:08 +, Narendra Acharya via shotwell-list wrote: Hi, I was looking at the current face detection code in shotwell 0.29.2 and feel that it can be enhanced. So I pulled the code from https://gitlab.gnome.org/GNOME/shotwell.git and started making some changes. 1. Do not run another process for face detection (shotwell-facedetect) but integrate the OpenCV calls into shotwell and link to OpenCV libraries This is potentially problematic as OpenCV can be compiled against Gtk2 - which will break Shotwell then and is the reason why the detection was done in a separate process in the first place. 2. Ability to use OpenCL when available on the platform using T-API in OpenCV 3.x 3. Eventually add face recognition on photos using a trained model based on faces labelled by the user - this would need a few faces to be tagged by the user before the OpenCV face recognition calls can be used (https://docs.opencv.org/2.4/modules/contrib/doc/facerec/facerec_tutorial.html). There is also some prior work on this by the original author of facedetect: https://github.com/Sanva/facedetect Is there anyone else working on this area? Thanks, Narendra ___ shotwell-list mailing list shotwell-list@gnome.org https://mail.gnome.org/mailman/listinfo/shotwell-list ___ shotwell-list mailing list shotwell-list@gnome.org https://mail.gnome.org/mailman/listinfo/shotwell-list
Re: [Shotwell] Enhance face detection
I have made the first level changes to migrate from process execution for every detect faces call, to one time execution of facedetect process and then communicate over DBus.https://gitlab.gnome.org/nma83/shotwell/commit/4c84248b80ebf3a1f1eb05aa60625d9d8c5d0134 The DBus interface XML is at:https://gitlab.gnome.org/nma83/shotwell/blob/wip/faces/facedetect/org.gnome.ShotwellFaces1.xml I have implemented DetectFaces and Terminate methods. The next steps are: 1. Extract the faces detected in photos and cache them (downsizing if needed) - maybe in $HOME/.cache/shotwell2. Run face training on the labelled faces, with a minimum number of faces per label (maybe 10)3. Attempt to automatically recognise and label faces in photos after training On Saturday, 23 June, 2018, 8:54:43 PM IST, Jens Georg wrote: On Fri, 2018-06-22 at 08:08 +, Narendra Acharya via shotwell-list wrote: Hi, I was looking at the current face detection code in shotwell 0.29.2 and feel that it can be enhanced.So I pulled the code from https://gitlab.gnome.org/GNOME/shotwell.git and started making some changes.1. Do not run another process for face detection (shotwell-facedetect) but integrate the OpenCV calls into shotwell and link to OpenCV libraries This is potentially problematic as OpenCV can be compiled against Gtk2 - which will break Shotwell then and is the reason why the detection was done in a separate process in the first place. 2. Ability to use OpenCL when available on the platform using T-API in OpenCV 3.x3. Eventually add face recognition on photos using a trained model based on faces labelled by the user - this would need a few faces to be tagged by the user before the OpenCV face recognition calls can be used (https://docs.opencv.org/2.4/modules/contrib/doc/facerec/facerec_tutorial.html). There is also some prior work on this by the original author of facedetect: https://github.com/Sanva/facedetect Is there anyone else working on this area? Thanks,Narendra ___shotwell-list mailing listshotwell-list@gnome.orghttps://mail.gnome.org/mailman/listinfo/shotwell-list ___ shotwell-list mailing list shotwell-list@gnome.org https://mail.gnome.org/mailman/listinfo/shotwell-list
Re: [Shotwell] Enhance face detection
Hi, OK great. I will post an update here once I have the current functionality working over D-bus and a facedetect background process. Hope I can use the IRC channel for any other minor questions I may have. On IRC I am best reachable during European business hours ~(9am - 4pm CEST). I was also pointed to a private d-bus example in gnome-photos: https://gitlab.gnome.org/GNOME/gnome-photos/blob/master/src/photos-thumbnail-factory.c https://gitlab.gnome.org/GNOME/gnome-photos/blob/master/src/photos-thumbnailer.c Sorry, no vala :-/ ___ shotwell-list mailing list shotwell-list@gnome.org https://mail.gnome.org/mailman/listinfo/shotwell-list
Re: [Shotwell] Enhance face detection
OK great. I will post an update here once I have the current functionality working over D-bus and a facedetect background process. Hope I can use the IRC channel for any other minor questions I may have. On Sunday, 24 June, 2018, 3:59:20 PM IST, Jens Georg wrote: > I understand the need for keeping OpenCV code in a separate process. > Even so, I think we should consider starting this process as a > 'daemon' when starting shotwell (controlled by an option) and > communicate with it over RPC from the shotwell process instead of > executing it for every 'Detect faces' button press. I will start > looking at adding an RPC layer from today. Excellent! Please use D-Bus, that should be the easiest and is nicely supported in Vala. I would suggest a private D-Bus, not going on system/user/session Bus. (A small example is at https://gitlab.gnome.or g/GNOME/glib/blob/master/gio/tests/gdbus-example-peer.c ) but session Bus should work as well. > That said, do we expect OpenCV to link to a different GTK version in > the long term? The OpenCV 3.4 I use in the flatpak is still looking for GTK2, so there's always potential that someone will enable that combination :-/ > > Once the RPC is done, will try to integrate the latest from the > Sanva/facedetect repo and hope to get some basic face recognition > capability. ___ shotwell-list mailing list shotwell-list@gnome.org https://mail.gnome.org/mailman/listinfo/shotwell-list
Re: [Shotwell] Enhance face detection
> I understand the need for keeping OpenCV code in a separate process. > Even so, I think we should consider starting this process as a > 'daemon' when starting shotwell (controlled by an option) and > communicate with it over RPC from the shotwell process instead of > executing it for every 'Detect faces' button press. I will start > looking at adding an RPC layer from today. Excellent! Please use D-Bus, that should be the easiest and is nicely supported in Vala. I would suggest a private D-Bus, not going on system/user/session Bus. (A small example is at https://gitlab.gnome.or g/GNOME/glib/blob/master/gio/tests/gdbus-example-peer.c ) but session Bus should work as well. > That said, do we expect OpenCV to link to a different GTK version in > the long term? The OpenCV 3.4 I use in the flatpak is still looking for GTK2, so there's always potential that someone will enable that combination :-/ > > Once the RPC is done, will try to integrate the latest from the > Sanva/facedetect repo and hope to get some basic face recognition > capability. ___ shotwell-list mailing list shotwell-list@gnome.org https://mail.gnome.org/mailman/listinfo/shotwell-list
Re: [Shotwell] Enhance face detection
Hi Jens, I understand the need for keeping OpenCV code in a separate process. Even so, I think we should consider starting this process as a 'daemon' when starting shotwell (controlled by an option) and communicate with it over RPC from the shotwell process instead of executing it for every 'Detect faces' button press. I will start looking at adding an RPC layer from today.That said, do we expect OpenCV to link to a different GTK version in the long term? Once the RPC is done, will try to integrate the latest from the Sanva/facedetect repo and hope to get some basic face recognition capability. Thanks,Narendra On Saturday, 23 June, 2018, 8:57:20 PM IST, Jens Georg wrote: On Fri, 2018-06-22 at 10:48 -0300, Ricardo via shotwell-list wrote: > Hello. > In https://valadoc.org/opencv/index.htm there is a vapi to link > direct > from vala to opencv without c/c++ in the middle. Some considerations: > First, it use OpenCV 2 which cannot detect automatically the presence > of a graphic card capable to use OpenCL. > Second, in the last stable Debian the OpenCV3 is not available yet. > So > it would not be possible to easily compile it in Debian. > In the vapi, the functions about matrix eigenvalues to recognize > specific person faces are available too. The nightly flatpak comes with OpenCV 3, so using Gnome builder + flatpack might make that possible. > > I'm working in a new user experience - > https://gitlab.gnome.org/GNOME/shotwell/issues/13 - for faces. If you > work in bring the OpenCV3 to valadoc and use that vapi in shotwell > probably we will not work in the same lines of code and we will not > have much work in a merge. > > Phako, some time ago in a pull-request you mentioned that the use of > c++ is not a big issue. Is ok to Narendra work in the use of the > OpenCV vapi in shotwell now? See my other mail, there is a potential problem of introducing an incompatibility when linking against OpenCV directly. ___ shotwell-list mailing list shotwell-list@gnome.org https://mail.gnome.org/mailman/listinfo/shotwell-list
Re: [Shotwell] Enhance face detection
On Fri, 2018-06-22 at 10:48 -0300, Ricardo via shotwell-list wrote: > Hello. > In https://valadoc.org/opencv/index.htm there is a vapi to link > direct > from vala to opencv without c/c++ in the middle. Some considerations: > First, it use OpenCV 2 which cannot detect automatically the presence > of a graphic card capable to use OpenCL. > Second, in the last stable Debian the OpenCV3 is not available yet. > So > it would not be possible to easily compile it in Debian. > In the vapi, the functions about matrix eigenvalues to recognize > specific person faces are available too. The nightly flatpak comes with OpenCV 3, so using Gnome builder + flatpack might make that possible. > > I'm working in a new user experience - > https://gitlab.gnome.org/GNOME/shotwell/issues/13 - for faces. If you > work in bring the OpenCV3 to valadoc and use that vapi in shotwell > probably we will not work in the same lines of code and we will not > have much work in a merge. > > Phako, some time ago in a pull-request you mentioned that the use of > c++ is not a big issue. Is ok to Narendra work in the use of the > OpenCV vapi in shotwell now? See my other mail, there is a potential problem of introducing an incompatibility when linking against OpenCV directly. ___ shotwell-list mailing list shotwell-list@gnome.org https://mail.gnome.org/mailman/listinfo/shotwell-list
Re: [Shotwell] Enhance face detection
On Fri, 2018-06-22 at 08:08 +, Narendra Acharya via shotwell-list wrote: > Hi, > > I was looking at the current face detection code in shotwell 0.29.2 > and feel that it can be enhanced. > So I pulled the code from https://gitlab.gnome.org/GNOME/shotwell.git > and started making some changes. > 1. Do not run another process for face detection (shotwell- > facedetect) but integrate the OpenCV calls into shotwell and link to > OpenCV libraries This is potentially problematic as OpenCV can be compiled against Gtk2 - which will break Shotwell then and is the reason why the detection was done in a separate process in the first place. > 2. Ability to use OpenCL when available on the platform using T-API > in OpenCV 3.x > 3. Eventually add face recognition on photos using a trained model > based on faces labelled by the user - this would need a few faces to > be tagged by the user before the OpenCV face recognition calls can be > used (https://docs.opencv.org/2.4/modules/contrib/doc/facerec/facerec > _tutorial.html). There is also some prior work on this by the original author of facedetect: https://github.com/Sanva/facedetect > Is there anyone else working on this area? > Thanks,Narendra > > > > > > ___shotwell-list mailing > listshotwell-list@gnome.orghttps://mail.gnome.org/mailman/listinfo/sh > otwell-list___ shotwell-list mailing list shotwell-list@gnome.org https://mail.gnome.org/mailman/listinfo/shotwell-list
Re: [Shotwell] Enhance face detection
Hi Ricardo, There are problems with the OpenCV VAPI that I discovered while trying to integrate OpenCV directly.OpenCV is not C compatible anymore in the recent 3.x versions (https://github.com/opencv/opencv/issues/10963). So I had to create another VAPI bridge that took in plain C structures and then called the OpenCV functions. This bridge is wrapped in extern "C" so that vala can talk to it. OpenCV libraries are linked to the shotwell executable at the link stage.I think we can defer OpenCL usage until OpenCV3 gets into debian (it is available on my archlinux machine). I have forked the shotwell repo and created wip/faces branch with my current WIP changes:https://gitlab.gnome.org/nma83/shotwell/tree/wip/faces The face detection works as it used to in this branch, but using the VAPI bridge to OpenCV instead of another process.I am exploring other options for better accuracy - OpenCV with DNN or dlib. The UI changes should not conflict with my change like you said, so merge should be seamless once we get there. Thanks,Narendra From: Ricardo To: Narendra Acharya Cc: "shotwell-list gnome org" Subject: Re: [Shotwell] Enhance face detectionDate: Fri, 22 Jun 2018 10:48:21 -0300Hello.In https://valadoc.org/opencv/index.htm there is a vapi to link directfrom vala to opencv without c/c++ in the middle. Some considerations:First, it use OpenCV 2 which cannot detect automatically the presenceof a graphic card capable to use OpenCL.Second, in the last stable Debian the OpenCV3 is not available yet. Soit would not be possible to easily compile it in Debian.In the vapi, the functions about matrix eigenvalues to recognizespecific person faces are available too.I'm working in a new user experience -https://gitlab.gnome.org/GNOME/shotwell/issues/13 - for faces. If youwork in bring the OpenCV3 to valadoc and use that vapi in shotwellprobably we will not work in the same lines of code and we will nothave much work in a merge.Phako, some time ago in a pull-request you mentioned that the use ofc++ is not a big issue. Is ok to Narendra work in the use of theOpenCV vapi in shotwell now?Bye,Ricardo.2018-06-22 5:08 GMT-03:00, Narendra Acharya via shotwell-list: Hi, I was looking at the current face detection code in shotwell 0.29.2 and feel that it can be enhanced.So I pulled the code from https://gitlab.gnome.org/GNOME/shotwell.git and started making some changes.1. Do not run another process for face detection (shotwell-facedetect) but integrate the OpenCV calls into shotwell and link to OpenCV libraries2. Ability to use OpenCL when available on the platform using T-API in OpenCV 3.x3. Eventually add face recognition on photos using a trained model based on faces labelled by the user - this would need a few faces to be tagged by the user before the OpenCV face recognition calls can be used (https://docs.opencv.org/2.4/modules/contrib/doc/facerec/facerec_tutorial.html). Is there anyone else working on this area? Thanks,Narendra ___ shotwell-list mailing list shotwell-list@gnome.org https://mail.gnome.org/mailman/listinfo/shotwell-list
