G'day, Tomas, I've looked at my BaselineCorrectionTask and the only spot where SimpleScans are created and not immediately added to a ReadDataFileWriter is in copyScans(). I've rewritten copyScans to add each SimpleScan to a ReadDataFileWriter as it's created (change committed in Rev. 3232).
Unfortunately, I don't think this is going to fix Kazuya's problem because the really memory-hungry part of BaselineCorrectionTask is buildTICChromatograms (similarly buildBasePeakChromatograms), which creates a double[num_bins][num_scans]. Kazuya is using a m/z bin size of 0.01, i.e. num_bins is very large. Kazuya, are you able to use a larger bin size, say 0.1 or 0.05? Longer term, it might be possible to rewrite these methods to process the data set one bin at a time which would do away with the need for the double[][]. However, it would require that a full copy of the scans be held in memory at once. Alternatively, I wonder whether baseline correction might be better applied after Chromatogram Builder, i.e. it becomes a peak list method rather than a raw data file method? Regards, Chris. On 30/07/12 09:08, Tomas Pluskal wrote: > Hi Chris, > > I checked the code of the baseline correction module, and I think the > problem mentioned below comes from the way how the module deals with > the scans. > The module creates a copy of the whole raw data in memory (by creating > lots of SimpleScan instances) and then writes them all to > RawDataFileWriter. > If the amount of the data is large, that will cause heap overload. > IMHO scans should be processed one by one and immediately added to the > RawDataFileWriter. > Adding the new scan to the RawDataFileWriter will turn the scan into a > StorableScan and therefore the memory can be reclaimed. > > What do you think? > > Cheers, > > Tomas > > > > On Jul 29, 2012, at 2:34 PM, 石田 和也 wrote: > >> Dear developpers >> >> My name is Kazuya, and I am researching metabolomics by >> OrbitrapXL(Thermo) at CERI. >> I am triing to introduce MZmine2 for our processing scheme. >> >> This reason why I am writting this e-mail to you is I would like to >> ask questions about Java heap size of MZmine2. >> >> I imported to the software our .raw files(MS scan data about >> 150Mb/file), then I performed "Baseline correction"(conditions are >> following). >> But, this processing was aborted at once and diplayed a message >> "Java heap space". >> I adjusted heap size to 2048 kB(maximum memmory size in our PC is >> 4MB) , but this process aborted. >> >> So, could you advise to me how many need a heap size at least? >> >> If you needs other informations to solve this problem, please do not >> hesitate to ask. >> >> Processing conditions >> Chromatogram type; "TIC" >> MS level; "1" >> Smoothing; "100,000(=10^5)" >> Asymmetry; "0.01" >> m/z bin width; "0.01" >> >> Sincerely, >> Kazuya >> >> ++++++++++++++++++++++++++++++++++++++++++ >> Kazuya Ishida >> Chemicals Evaluation and Research Institute, Japan >> Email: [email protected] <mailto:[email protected]> >> Phone: +81-480-37-2601 Fax: +81-480-37-2521 >> URL: http://www.cerij.or.jp/ceri_en/index_e4.shtml >> ++++++++++++++++++++++++++++++++++++++++++ >> ------------------------------------------------------------------------------ >> Live Security Virtual Conference >> Exclusive live event will cover all the ways today's security and >> threat landscape has changed and how IT managers can respond. >> Discussions >> will include endpoint security, mobile security and the latest in >> malware >> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ >> _______________________________________________ >> Mzmine-devel mailing list >> [email protected] >> <mailto:[email protected]> >> https://lists.sourceforge.net/lists/listinfo/mzmine-devel > > =============================================== > Tom?・ Pluskal > G0 Cell Unit, Okinawa Institute of Science and Technology Graduate > University > 1919-1 Tancha, Onna-son, Okinawa 904-0495, Japan > TEL: +81-98-966-8684 > Fax: +81-98-966-2890 > This message may contain confidential information. If you are not the designated recipient, please notify the sender immediately, and delete the original and any copies. Any use of the message by you is prohibited. ------------------------------------------------------------------------------ Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ _______________________________________________ Mzmine-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/mzmine-devel
