On Thu, 2009-04-23 at 14:20 -0700, dnadavewa wrote: > I'm working on a large data problem where I'm reading in data from text files > with almost 2 million columns. In doing this, I can read in about 25 rows > before Mono bombs out with an out of memory error.
How are you reading in these lines? > What I found was the mono executable was indeed 64 bit, but gmcs.exe and > mcs.exe were 32 bit. As Chris Howie mentioned, these are actually in platform-neutral IL, and will be run using a 64-bit address space when using `mono`. > One other point, memory usage is horrible. I admit that I'm new to C# and > mono, so my coding skills are not as good as others, but a 300MB file should > not use 2GB RAM to read in 1/8 of the file. That depends ~entirely on how you're reading in the file. Also keep in mind that .NET strings are UTF-16, so if your input text is ASCII, you will require twice as much RAM as the size of the file, e.g. 600MB of RAM to store the entire file as a string. (Then there is various object overhead considerations, but these are likely tiny compared to the 300MB you're looking at.) > I stopped using classes to > store the data and went with List<string> and List<string[]> to read in this > much data. Any comments on how I might improve this performance would be > appreciated. To provide any comments we'd need to know more about what you're trying to do. For example, reading a 300MB XML file using XmlDocument will require *lots* of RAM, as in addition to the UTF-16 string issue, each element, attribute, etc. will be represented as separate objects, with varying amounts of memory required. DOM would be something to avoid here, while XmlReader would be much better. The easiest question, though, is this: do you really need to keep the entire file contents in memory all at once? Or can you instead process each line independently (or while caching minimal data from one line to the next, so that the contents of previous lines don't need to be maintained). This would allow you to remove your List<string>, and save a ton of memory. - Jon _______________________________________________ Mono-list maillist - [email protected] http://lists.ximian.com/mailman/listinfo/mono-list
