Re: [Scikit-learn-general] Joblib and IPython

Howdy, On Fri, Jan 27, 2012 at 6:44 AM, Andreas wrote: > At the moment all parallelism is handled by joblib. On the other hand it > seems > IPython can talk to the SGE scheduling. > So I would love to have a way for joblib to talk to IPython. just to say that I'm sorry not to jump in the discuss

Re: [Scikit-learn-general] Announce: scikit-learn 0.10

My pleasure, I'm sorry for the delay! Sent from my iPod On 27.01.2012, at 20:49, Vincent Dubourg wrote: > Thank you Vlad! After a slight upgrade of both numpy and scipy I managed > to get a brand new working 0.10 sklearn! > > On 27/01/2012 17:32, Gael Varoquaux wrote: >> On Fri, Jan 27, 2012 at

Re: [Scikit-learn-general] [Scikit-learn-commits] [scikit-learn/scikit-learn] bc6255: ENH merge sparse and dense SVMs, part 1

You should add an entry to the what's new file. -- Olivier -- Try before you buy = See our experts in action! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePo

Re: [Scikit-learn-general] Announce: scikit-learn 0.10

Thank you Vlad! After a slight upgrade of both numpy and scipy I managed to get a brand new working 0.10 sklearn! On 27/01/2012 17:32, Gael Varoquaux wrote: > On Fri, Jan 27, 2012 at 03:25:45PM +0100, Fabian Pedregosa wrote: >> On Fri, Jan 27, 2012 at 3:20 PM, Vlad Niculae wrote: >>> The binarie

Re: [Scikit-learn-general] Joblib and IPython

On Fri, Jan 27, 2012 at 11:42:15AM -0500, Satrajit Ghosh wrote: > i understand, but you have to have a glue somewhere unless each of these > distribution libraries expose the same api. Agreed > the problem is that you have to agree on the data model for job dispatch > and that might take a while

Re: [Scikit-learn-general] Joblib and IPython

hi gael, > I am not too entousiastic about that: joblib is meant to be a light > library. Such a solution would pretty much force the joblib release > manager to have an SGE cluster in order to run the tests and debug. > > Just like I think that IPython-specific stuff should live in IPython, I >

Re: [Scikit-learn-general] Announce: scikit-learn 0.10

On Fri, Jan 27, 2012 at 03:25:45PM +0100, Fabian Pedregosa wrote: > On Fri, Jan 27, 2012 at 3:20 PM, Vlad Niculae wrote: > > The binaries are up! Awesome. Thanks heaps, Gael -- Try before you buy = See our experts in ac

Re: [Scikit-learn-general] Fused types in Cython

On Sat, Jan 28, 2012 at 01:07:17AM +0900, Mathieu Blondel wrote: > It seems that the next release of Cython will have support for some > kind of templates - they are called fused types. Awesome! These guys are extremely proactive. Gael

Re: [Scikit-learn-general] Joblib and IPython

On Fri, Jan 27, 2012 at 11:15:43AM -0500, Satrajit Ghosh wrote: >one option we could consider is to take the job distribution capability in >nipype and make it general purpose and add it to joblib (will require some >effort - it won't be quick). it would be nice as i have stated before

Re: [Scikit-learn-general] Joblib and IPython

>> What do you think would be the hard part / why >> do you think this is much work? >> > I think that their is a bit of learning to be done. In particular, the > code would have to be well tested, and I have no idea of what the right > way to test it would be. The second problem would be to

Re: [Scikit-learn-general] Fused types in Cython

2012/1/27 Mathieu Blondel : > Hello, > > It seems that the next release of Cython will have support for some > kind of templates - they are called fused types. \o/ Thanks for the heads up. -- Olivier http://twitter.com/ogrisel - http://github.com/ogrisel

Re: [Scikit-learn-general] Joblib and IPython

On Fri, Jan 27, 2012 at 05:10:11PM +0100, Olivier Grisel wrote: > The problem is that for multiprocessing, a n_jobs argument is enough > (to tell the number of cores). But for cluster computing you will have > to pass some kind of active cluster session (e.g. a > IPython.parallel.Client instance th

Re: [Scikit-learn-general] Joblib and IPython

On Fri, Jan 27, 2012 at 05:01:12PM +0100, Andreas wrote: > I was not sure whether you want other backends in joblib. The parallel part of joblib is meant to be a convenience wrapper, not a parallel execution model of its own. As such, enriching it with other backends is definitely in the scope as

Re: [Scikit-learn-general] Joblib and IPython

hi gael, I would really like to avoid having any direct import to IPython in > scikit-learn: I would like our set of dependencies to stick to scipy, > numpy, and optionally matplotlib for the examples (note that this also > means that I would like to get rid of the pyamg dependency, that has > pro

Re: [Scikit-learn-general] Joblib and IPython

2012/1/27 Gael Varoquaux : > On Fri, Jan 27, 2012 at 04:58:30PM +0100, Olivier Grisel wrote: >> I would advise you to start by experimenting with your own version of >> GridSearchCV (by deriving from the version of sklearn) and passing a >> LoadBalancedView instance as argument to the constructor a

Re: [Scikit-learn-general] Joblib and IPython

Hi Olivier. Thanks for your suggestions. It certainly seems easier to directly use IPython but I agree with Gael about not wanting to add additional dependencies. I'll try doing it in joblib and if that is to hard, I'll try doing it directly in sklearn. Let's see how this goes! Cheers, Andy On

Re: [Scikit-learn-general] Joblib and IPython

hi andreas, when you launch ipcluster on SGE for example, it queues up a set of python engines as jobs. these jobs will get distributed to the SGE execution pool depending on it's current job distribution. The key to note here is that no real job execution (in your case forests) have taken place y

Re: [Scikit-learn-general] Joblib and IPython

2012/1/27 Andreas : > On 01/27/2012 04:55 PM, Gael Varoquaux wrote: >> On Fri, Jan 27, 2012 at 03:44:31PM +0100, Andreas wrote: >> >>> as it could be. So I was wondering whether there would be a >>> non-intrusive way to make sklearn parallelize over the cluster. >>> >> This is a very legitimate que

Re: [Scikit-learn-general] Joblib and IPython

On Fri, Jan 27, 2012 at 04:58:30PM +0100, Olivier Grisel wrote: > I would advise you to start by experimenting with your own version of > GridSearchCV (by deriving from the version of sklearn) and passing a > LoadBalancedView instance as argument to the constructor and use it in > the fit method in

[Scikit-learn-general] Fused types in Cython

Hello, It seems that the next release of Cython will have support for some kind of templates - they are called fused types. Discussion: http://groups.google.com/group/cython-users/browse_thread/thread/fbb707afa233d346?pli=1 Documentation: https://sage.math.washington.edu:8091/hudson/job/cython-d

Re: [Scikit-learn-general] Joblib and IPython

On 01/27/2012 04:55 PM, Gael Varoquaux wrote: > On Fri, Jan 27, 2012 at 03:44:31PM +0100, Andreas wrote: > >> as it could be. So I was wondering whether there would be a >> non-intrusive way to make sklearn parallelize over the cluster. >> > This is a very legitimate question. Basically,

Re: [Scikit-learn-general] Joblib and IPython

2012/1/27 Andreas : I would advise you to start by experimenting with your own version of GridSearchCV (by deriving from the version of sklearn) and passing a LoadBalancedView instance as argument to the constructor and use it in the fit method instead of calling joblib. The same could be followe

Re: [Scikit-learn-general] Joblib and IPython

On Fri, Jan 27, 2012 at 03:44:31PM +0100, Andreas wrote: > as it could be. So I was wondering whether there would be a > non-intrusive way to make sklearn parallelize over the cluster. This is a very legitimate question. Basically, it boils down to: how can we extend the parallelism model in sciki

Re: [Scikit-learn-general] Joblib and IPython

Hi Satra. Thanks for your comments. Can you explain what the "grap an engine" strategy means? Is it that you distribute the jobs to the engines before starting any jobs and not having them in a queue? This should be ok if my jobs and my engines are pretty homogeneous, right? The main question f

Re: [Scikit-learn-general] Joblib and IPython

hi andreas, a few notes: - a sprint planned for pycon will be looking at parallel computing with scikit-learn and ipython (http://wiki.ipython.org/PyCon12Sprint) - ipython currently uses a grab an engine and not release strategy in the context of distributed systems like SGE/PBS/LSF. this implie

[Scikit-learn-general] Joblib and IPython

Hi everybody. This question basically goes out to Gael, but might also be interesting for others. I am using sklearn on an SGE cluster at the moment and it is not as nice as it could be. So I was wondering whether there would be a non-intrusive way to make sklearn parallelize over the cluster. A

Re: [Scikit-learn-general] Announce: scikit-learn 0.10

On Fri, Jan 27, 2012 at 3:20 PM, Vlad Niculae wrote: > The binaries are up! Thanks Vlad, Fabian -- Try before you buy = See our experts in action! The most comprehensive online learning library for Microsoft developers

Re: [Scikit-learn-general] cache_size for SVM

Ok, thanks a lot! Cheers, Mathias On Fri, Jan 27, 2012 at 2:45 PM, Andreas wrote: > ** > On 01/27/2012 02:34 PM, Mathias Verbeke wrote: > > Dear Andy, > > I'm currently using version 0.9. > > Then you should upgrade ;) > It is available in Version .10 that has been released recently. > Cheer

Re: [Scikit-learn-general] cache_size for SVM

On 01/27/2012 02:34 PM, Mathias Verbeke wrote: Dear Andy, I'm currently using version 0.9. Then you should upgrade ;) It is available in Version .10 that has been released recently. Cheers, Andy Best, Mathias On Fri, Jan 27, 2012 at 2:30 PM, Andreas > wrot

Re: [Scikit-learn-general] cache_size for SVM

Dear Andy, I'm currently using version 0.9. Best, Mathias On Fri, Jan 27, 2012 at 2:30 PM, Andreas wrote: > ** > Dear Mathias. > Which version of sklearn to you have? > The "cache_size" argument was removed at some point and reintroduced later. > > Cheers, > Andy > > > On 01/27/2012 02:29 PM

Re: [Scikit-learn-general] cache_size for SVM

Dear Mathias. Which version of sklearn to you have? The "cache_size" argument was removed at some point and reintroduced later. Cheers, Andy On 01/27/2012 02:29 PM, Mathias Verbeke wrote: Dear all, When I want to set the cache_size of an SVM, using clf = svm.SVC(cache_size=200.0) I get: Type

[Scikit-learn-general] cache_size for SVM

Dear all, When I want to set the cache_size of an SVM, using clf = svm.SVC(cache_size=200.0) I get: TypeError: __init__() got an unexpected keyword argument 'cache_size', althought this should be possible according to the documentation. What am I doing wrong here? Or isn't this possible anymore

Re: [Scikit-learn-general] Announce: scikit-learn 0.10

sorry, I don't have a Windows system at the moment, if you have a VM could you do it? If you're not set up either, I'll do it in a day or two. Best, Vlad Sent from my iPod On 27.01.2012, at 12:29, Fabian Pedregosa wrote: > @vene: do you have time to make the windows binaries or should I do tha

Re: [Scikit-learn-general] Announce: scikit-learn 0.10

@vene: do you have time to make the windows binaries or should I do that ? Best, fabian On Wed, Jan 18, 2012 at 11:57 AM, Vincent Dubourg wrote: > Hi list, > > Is there any reason the Windows binary for the 0.10 release does not appear > on the sourceforge repos? > http://sourceforge.net/projec