Hi Moe, Thanks for pinpointing the cause. Please just change the macro of #if 1 to #if 0 and you will get the old behaviour exactly like what you have program.
I have probably committed this code with the wrong macro by accident. It is based on a undocumented "API". The motivation is that mzWiff can reduce it execution time by 30-60%. But my testing has revealed that it is very unstable depending on the version of Analyst / AnalystQS. As I wish to return to it some days and thus have not removed the chunk of code from the check in. Sorry for causing you the problem. Note: AnalystQS 1.1, Analyst 1.4 and Analyst 1.5 are not affected by this "bug". CH On Tue, Mar 17, 2009 at 2:38 PM, Moe <[email protected]> wrote: > > The memory leak appears to be isolated in class AnalystQS20Interface, > method processMSSpecData, in the call to "m_ipGetRawDataSpec- > >GetRawScanXYPoints()" when requesting the product spectrum's entire > set of data points (origical code snip below). > > /*** ORIGINAL CODE FROM RELEASE 4.2.0 ***/ > > inline void AnalystQS20Interface::processMSSpecData(Scan *pScan) { > #if 1 > ... > long lSpecDataPoints; > ... > VARIANT dXMass; VariantInit(&dXMass); > VARIANT dYInts; VariantInit(&dYInts); > ... > HRESULT hr = m_ipGetRawDataSpec->GetRawScanXYPoints > (&lSpecDataPoints, &dXMass, &dYInts); > ... > long lPeaksCount=0; > ... > for (long dpi=1; dpi<=lSpecDataPoints; dpi++, pXMass++, pYInts++) { > if (*pYInts>0.0) { > pScan->mzArray_[lPeaksCount] = *pXMass; > pScan->totalIonCurrent_ += > (pScan->intensityArray_[lPeaksCount] > =*pYInts); > lPeaksCount++; > > if (pScan->basePeakIntensity_<*pYInts) { > pScan->basePeakMZ_ = *pXMass; > pScan->basePeakIntensity_ = *pYInts; > } > } > } > ... > > It appears that the memory occupied at &dXMass and &xYInts VARIANTs > does not get freed even after descoping, probably because it is being > written externally to mzWiff by the AnalystService process. It does > not seem possible to free this memory internally, but I have not > explored all options here. > > Perhaps an ultimate call to "m_ipGetRawDataSpec->Release" may induce > the freeing, but I suppose this would render the IGetRawData structure > prematurely unusable. > > As a workaround, I avoid the call to GetRawScanXYPoints() altogether, > instead calling m_ipFMANSpecData->GetDataPointXValue() and - > >GetDataPointYValue() to request an individual datapoint, as per code > below. As this call must be made for each data point, there is a > performance hit. I estimated it to be about 30% longer execution time > for a 500 MB input, which is still reasonable for me at this time. > > /*** WORKAROUND CODE ***/ > ... > long lSpecDataPoints = m_ipFMANSpecData->GetNumberOfDataPoints(); > ... > double dblX, dblY; > long lPeaksCount=0; > for (long dpi=1; dpi<=lSpecDataPoints; dpi++) { > dblX = m_ipFMANSpecData->GetDataPointXValue(dpi); > dblY = m_ipFMANSpecData->GetDataPointYValue(dpi); > if (dblY>0) { > pScan->mzArray_[lPeaksCount] = dblX; > pScan->intensityArray_[lPeaksCount] = dblY; > pScan->totalIonCurrent_ += dblY; > lPeaksCount++; > > if (pScan->basePeakIntensity_ < dblY) { > pScan->basePeakMZ_ = dblX; > pScan->basePeakIntensity_ = dblY; > } > } > } > > > > On Mar 16, 6:09 pm, Moe <[email protected]> wrote: > > I have uploaded the files bsa-20fmol-2.wiff/.scan. Although the > > converter doesn't crash on this file, it did reach > 300 MB memory > > usage in my attempt. > > > > Let me know if you need anything else for your debugging. > > > > Thanks, > > Moyez > > > > On Mar 16, 5:46 pm, Natalie Tasman <[email protected]> wrote: > > > > > > > > >http://groups.google.com/group/spctools-discuss/web/how-to-upload-fil. > .. > > > > > On Mon, Mar 16, 2009 at 2:43 PM, Moe <[email protected]> wrote: > > > > > > Thanks for the VC tips. Where can I upload the BSA file? > > > > > > Moyez > > > > > > On Mar 16, 5:25 pm, Natalie Tasman <[email protected]> > wrote: > > > >> Hi Moyez, > > > > > >> Of course we have our own files to test, but yes, please upload your > > > >> file, as we've never observed the behavior you're reporting. > > > > > >> And also please rerun your conversions with the "-v" (verbose) > option > > > >> and copy and paste the output up to the "sha-1" calculation. > > > > > >> You're welcome to rebuild and debug the converter on your own. Just > > > >> open the TPP.sln file and build the mzWiff project from there. Note > > > >> you'll also need the "win_lib" project to be checked out at the same > > > >> level as the TPP code for boost, zlib, etc. > > > > > >> -Natalie > > > > > >> On Mon, Mar 16, 2009 at 2:07 PM, Moe <[email protected]> wrote: > > > > > >> > Hi Natalie, > > > > > >> > The leak is apparent in QStar WIFF files of any size. For a 30 > minute > > > >> > BSA QC run (25 MB file size), peak memory usage reached 314 MB (to > > > >> > produce ~195 MB mzXML output), which doesn't seem reasonable. Do > you > > > >> > have your own data files that you can debug with? If not, I can > upload > > > >> > the BSA run to your FTP site (need the details). > > > > > >> > The data I'm trying to analyze consists of 2 hour runs at about > 500 MB > > > >> > each. mzWiff crashes about halfway through a data file, leaving a > 1.3 > > > >> > GB (incomplete) mzXML file which ends at an elution time of about > 60 > > > >> > minutes (when memory reaches slightly > 2 GB). > > > > > >> > I have tried older releases of the executable up to 4.0.0 and all > > > >> > behave the same. I am considering recompiling a 64-bit executable, > or > > > >> > trying to locate the leak. I'll check out the code from > SourceForge, > > > >> > but if you have additional instructions for setting up the VC > project, > > > >> > please let me know. > > > > > >> > Thanks, > > > >> > Moyez > > > > > >> > On Mar 16, 1:41 pm, Natalie Tasman <[email protected]> > wrote: > > > >> >> Hello Moyez, > > > > > >> >> Thank you for the detailed bug report. I haven't heard of issues > with > > > >> >> mzWiff. Thanks for the MSVC hint-- I'm pretty sure this is not > turned > > > >> >> on yet, but actually probably shouldn't be necessary. I agree > with > > > >> >> you that that much memory usage sounds like a memory leak. The > > > >> >> converters are designed to process data scan-by-scan with a > relatively > > > >> >> low memory overhead. > > > > > >> >> If you can upload a sample file we can take a look and see if we > can > > > >> >> replicate behavior and then fix it. > > > > > >> >> Thanks, > > > > > >> >> Natalie > > > > > >> >> On Mon, Mar 16, 2009 at 9:11 AM, Moe <[email protected]> wrote: > > > > > >> >> > I'm trying to convert Analyst QStar Elite data acquired with > Analyst > > > >> >> > QS 2.0, with mzWiff version 4.2.0(build Feb 19 2009 09:48:08), > on > > > >> >> > Windows XP Pro SP3 and Windows Server 2003 R2 x64 Standard SP2. > > > > > >> >> > There appears to be a memory leak causing process memory to > accumulate > > > >> >> > to > 2GB, shortly after which the program crashes with error: > > > > > >> >> > ERROR: COM error 80004005 while processing sample#2 > > > >> >> > INFO: error message: Unspecified error > > > > > >> >> > This happens both on 32-bit XP and 64-bit Server 2003, both > systems > > > >> >> > have > 4GB physical memory. The per-process addressable limit > for a 32- > > > >> >> > bit process is 2GB unless compiled with the /LARGEADDRESSAWARE > Visual C > > > >> >> > ++ linker flag. > > > > > >> >> > Is anyone aware of this problem? Is this in fact a leak that > can be > > > >> >> > avoided? > > > > > >> >> > Thanks, > > > >> >> > Moyez- Hide quoted text - > > > > > >> >> - Show quoted text -- Hide quoted text - > > > > > >> - Show quoted text -- Hide quoted text - > > > > > - Show quoted text -- Hide quoted text - > > > > - Show quoted text - > > > > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "spctools-discuss" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/spctools-discuss?hl=en -~----------~----~----~----~------~----~------~--~---
