2011/9/20 xinfan meng <[email protected]>: > Hi: > I ran the sklearn sentiment classification codes > (https://github.com/scikit-learn/scikit-learn-tutorial/blob/master/solutions/exercise_02_sentiment.py) > and found it keep creating new python.exe instance, and then my computer > crashed because ran out of memory. > > My OS system is Windows 7, and the scikits.learn comes from the > Enthought Python Distribution bundle. Since this scikits.learn is version > 0.8, I made several neccesary modification to the codes. The codes behaved > normally in MacOS. So I wonder if this is a problem with my OS. May be > someone can try on their system to see if it is reproducible? Thanks.
The main problem is using windows itself :) More seriously, yes the multiprocessing behavior is different on windows than on POSIX compliant systems such as MaxOSX or Linux. However the exercise you mention, the grid search has only two points in the grid to explore so the joblib.Parallel tool that is used when n_jobs != 1 should only fork 2 jobs even if it encounters some kind of multiprocessing.Pool misbehavior that prevents the finished tasks to release their process. How many python processes do you have running concurrently and what is the amount of memory allocated for each? Can you reproduce this problem using the development version of scikit-learn? -- Olivier http://twitter.com/ogrisel - http://github.com/ogrisel ------------------------------------------------------------------------------ All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense. http://p.sf.net/sfu/splunk-d2dcopy1 _______________________________________________ Scikit-learn-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
