Hi, This is absolutely great news. Thanks a lot. Please do open a WIP PR. We (at INRIA) were planning to allocate time from someone this summer to work on this. So you'll have someone reviewing / advicing.
With regards to releasing the gil, you need to use the 'with nogil' statement in cython. Have a look for instance at page 203 of the cython book: https://books.google.fr/books?id=H3JmBgAAQBAJ&pg=PA203&lpg=PA203&dq=with+nogil+statement&source=bl&ots=QH3Nh0uhsT&sig=-F-q8NDOHkJx_lyXlJSnektZjus&hl=en&sa=X&ved=0CEYQ6AEwBWoVChMI1Iigj-OCxwIVAlkUCh0MFQOQ#v=onepage&q=with%20nogil%20statement&f=false The parallel computing is then done with joblib parallel, using the threadhing backend. G On Thu, Jul 30, 2015 at 03:13:30PM +0530, Maheshakya Wijewardena wrote: > Hi, > I've started to look into the matter of improving performance of LSHForest. As > we have discussed sometime before(in fact, quite a long time), main concern is > to Cythonize distance calculations. Currently, this done by iteratively moving > over all the query vectors when `kneighbors` method is called for a set of > query vectors. It has been identified that iterating over each query with > Python loops is a huge overhead. I have implemented a few Cython hacks to > demonstrate the distance calculation in LSHForest and I was able to get an > approximate speedup 10x compared to current distance calculation with a Python > loop. However, I came across some blockers while trying to do this and need > some clarifications. > What I need to know is, do we use a mechanism to release GIL when we want to > parallelize. One of my observations is `pairwise_distance` uses all the cores > even when I don't specify `n_jobs` parameter which is 1 in default. Is this an > expected behavior? > If I want to release GIL, can I use OpenMP module in Cython? Or is that a task > of Joblib? > Any input on this is highly appreciated. > Best regards, -- Gael Varoquaux Researcher, INRIA Parietal NeuroSpin/CEA Saclay , Bat 145, 91191 Gif-sur-Yvette France Phone: ++ 33-1-69-08-79-68 http://gael-varoquaux.info http://twitter.com/GaelVaroquaux ------------------------------------------------------------------------------ _______________________________________________ Scikit-learn-general mailing list Scikit-learn-general@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
