Thanks for the responses Manuel and Mark. Manuel, I am swarming over 3 to 4 fields. That may be part of the cause. Also, are you using multiple cores?
Mark, CLA takes up a lot of RAM too, but never more than 1 or 2 gigs, so I've never had a problem there. The problem happens specifically during the swarming process. So I was wondering if others were seeing the same thing and if it's hinting at something wrong. Cheers, Ritchie ******* Message: 3 Date: Thu, 10 Apr 2014 10:31:05 +0000 From: "Scheele, Manuel" <[email protected]> To: NuPIC general mailing list. <[email protected]> Subject: Re: [nupic-discuss] Swarming process using a large amount of memory Message-ID: <[email protected]> Content-Type: text/plain; charset="us-ascii" Hi Ritchie, Unfortunately, I can't help you as to why your swarm uses up so much memory, but I have swarmed over data files with sizes of 8MB without problems (it only takes some time, as you would expect). I have 6GB of RAM. According to the resources monitor the swarming process uses about 320MB initially and grows to a total of 460MB. However, I don't think file size matters too much when swarming. What is more relevant is the number of fields in your file and the two things are not necessarily connected (a field can have any byte size, so a large file may not indicate a large number of fields). But I am not too confident about this. Let's see what the rest of the community has to say about it ;). A workaround would for now would be to limit the number of lines you swarm over (in the search_def.json), but that is equivalent to swarming over a smaller file. Manuel ________________________________________ From: nupic [[email protected]] on behalf of Ritchie Lee [ [email protected]] Sent: 09 April 2014 22:07 To: [email protected] Subject: [nupic-discuss] Swarming process using a large amount of memory Hi friends of NuPIC! I have been running swarms on csv data files that are around 3 megabytes in size, and I have found that it is using about >6 gigabytes of RAM during the process. If I run swarms on data files that are larger than that, my computer runs out of RAM and hangs (I have 8 GB of RAM). In particular I've tried swarming on a 13 MB data file and it froze very quickly. Memory usage seems to climb monotonically during the swarming process, and released all at the end on completion. I am wondering if anyone has had experience swarming large (>10 MB) csv files and your experiences with the memory consumption. Ideally I'd like to be able to swarm over much larger datasets (on the order of a hundred megs). Thanks, Ritchie Lee Research Engineer Carnegie Mellon University-Silicon Valley NASA Ames Research Center Bldg 23, Rm 115 Moffett Field, CA 94035 (650) 335-2847 ------------------------------ Message: 4 Date: Thu, 10 Apr 2014 13:21:03 +0200 From: Marek Otahal <[email protected]> To: "NuPIC general mailing list." <[email protected]> Subject: Re: [nupic-discuss] Swarming process using a large amount of memory Message-ID: <cach1_rq8kvp0gwwcprdexr+jb7pqv_m_tgylolf1y9npiwe...@mail.gmail.com> Content-Type: text/plain; charset="utf-8" Hi Ritchie, I did some experiments (not swarming) with large scale CLAs, it turns out python objects take a lot of resources. Your memory demands would depend on: encoders you use, the size and type of input data, and implementation of spatial pooler and temporal pooler (currently we have 2 - python and cpp). Could you share these details or best the "OPF settings file" where you describe the structure of data to Nupic (description.py or something)? Cheers, Mark
_______________________________________________ nupic mailing list [email protected] http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org
