On 01/23/2012 02:20 PM, Lars Buitinck wrote: > 2012/1/23 Dimitrios Pritsos<[email protected]>: >> On 01/23/2012 12:24 PM, Olivier Grisel wrote: >>> BTW: what is the structure of you data in PyTables? Is is mapped to a >>> scipy.sparse Compressed Sparse Row datastructure? How many features do >>> you have in your dataset? >> The training data are in a EArray (Compressed per row due to lots of >> zeros). >> I have 34000 Samples and the length of my Dictionary depending on the >> Training Set is about 1,500,000. >> However, using about 30,000 features seems satisfactory for a >> proof-of-concept case. However the samples needs to be approximately >> about 30-50k. > That would be doable. 30k features × 50k samples in a CSR matrix with > dtype=float32, assuming it's 90% zeros (a pessimistic guess for topic > spotting) would take just over 2GB. > I will give it a try however in some of my tests had a memory management problem. As I can recall it was mostly because of numpy function that might ask from pyTable to load every thing in main men. I guess some loops and some slicing might solve the problem.
However I fist try to figure out how to use linear_model.SGDClassifier which it suppose to be capable to be trained in stages. Plus since I am using Linear Kernel it won't effect my results. Still I will give a try to the Sparse structure. Thank you for the tip! ------------------------------------------------------------------------------ Try before you buy = See our experts in action! The most comprehensive online learning library for Microsoft developers is just $99.99! Visual Studio, SharePoint, SQL - plus HTML5, CSS3, MVC3, Metro Style Apps, more. Free future releases when you subscribe now! http://p.sf.net/sfu/learndevnow-dev2 _______________________________________________ Scikit-learn-general mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/scikit-learn-general
