Thank you, Moyez. We appreciate your work! By the way, I still haven't received your file for testing.
Thanks, Natalie On Tue, Mar 17, 2009 at 2:38 PM, Moe <[email protected]> wrote: > > The memory leak appears to be isolated in class AnalystQS20Interface, > method processMSSpecData, in the call to "m_ipGetRawDataSpec- >>GetRawScanXYPoints()" when requesting the product spectrum's entire > set of data points (origical code snip below). > > /*** ORIGINAL CODE FROM RELEASE 4.2.0 ***/ > > inline void AnalystQS20Interface::processMSSpecData(Scan *pScan) { > #if 1 > ... > long lSpecDataPoints; > ... > VARIANT dXMass; VariantInit(&dXMass); > VARIANT dYInts; VariantInit(&dYInts); > ... > HRESULT hr = m_ipGetRawDataSpec->GetRawScanXYPoints > (&lSpecDataPoints, &dXMass, &dYInts); > ... > long lPeaksCount=0; > ... > for (long dpi=1; dpi<=lSpecDataPoints; dpi++, pXMass++, pYInts++) { > if (*pYInts>0.0) { > pScan->mzArray_[lPeaksCount] = *pXMass; > pScan->totalIonCurrent_ += > (pScan->intensityArray_[lPeaksCount] > =*pYInts); > lPeaksCount++; > > if (pScan->basePeakIntensity_<*pYInts) { > pScan->basePeakMZ_ = *pXMass; > pScan->basePeakIntensity_ = *pYInts; > } > } > } > ... > > It appears that the memory occupied at &dXMass and &xYInts VARIANTs > does not get freed even after descoping, probably because it is being > written externally to mzWiff by the AnalystService process. It does > not seem possible to free this memory internally, but I have not > explored all options here. > > Perhaps an ultimate call to "m_ipGetRawDataSpec->Release" may induce > the freeing, but I suppose this would render the IGetRawData structure > prematurely unusable. > > As a workaround, I avoid the call to GetRawScanXYPoints() altogether, > instead calling m_ipFMANSpecData->GetDataPointXValue() and - >>GetDataPointYValue() to request an individual datapoint, as per code > below. As this call must be made for each data point, there is a > performance hit. I estimated it to be about 30% longer execution time > for a 500 MB input, which is still reasonable for me at this time. > > /*** WORKAROUND CODE ***/ > ... > long lSpecDataPoints = m_ipFMANSpecData->GetNumberOfDataPoints(); > ... > double dblX, dblY; > long lPeaksCount=0; > for (long dpi=1; dpi<=lSpecDataPoints; dpi++) { > dblX = m_ipFMANSpecData->GetDataPointXValue(dpi); > dblY = m_ipFMANSpecData->GetDataPointYValue(dpi); > if (dblY>0) { > pScan->mzArray_[lPeaksCount] = dblX; > pScan->intensityArray_[lPeaksCount] = dblY; > pScan->totalIonCurrent_ += dblY; > lPeaksCount++; > > if (pScan->basePeakIntensity_ < dblY) { > pScan->basePeakMZ_ = dblX; > pScan->basePeakIntensity_ = dblY; > } > } > } > > > > On Mar 16, 6:09 pm, Moe <[email protected]> wrote: >> I have uploaded the files bsa-20fmol-2.wiff/.scan. Although the >> converter doesn't crash on this file, it did reach > 300 MB memory >> usage in my attempt. >> >> Let me know if you need anything else for your debugging. >> >> Thanks, >> Moyez >> >> On Mar 16, 5:46 pm, Natalie Tasman <[email protected]> wrote: >> >> >> >> >http://groups.google.com/group/spctools-discuss/web/how-to-upload-fil... >> >> > On Mon, Mar 16, 2009 at 2:43 PM, Moe <[email protected]> wrote: >> >> > > Thanks for the VC tips. Where can I upload the BSA file? >> >> > > Moyez >> >> > > On Mar 16, 5:25 pm, Natalie Tasman <[email protected]> wrote: >> > >> Hi Moyez, >> >> > >> Of course we have our own files to test, but yes, please upload your >> > >> file, as we've never observed the behavior you're reporting. >> >> > >> And also please rerun your conversions with the "-v" (verbose) option >> > >> and copy and paste the output up to the "sha-1" calculation. >> >> > >> You're welcome to rebuild and debug the converter on your own. Just >> > >> open the TPP.sln file and build the mzWiff project from there. Note >> > >> you'll also need the "win_lib" project to be checked out at the same >> > >> level as the TPP code for boost, zlib, etc. >> >> > >> -Natalie >> >> > >> On Mon, Mar 16, 2009 at 2:07 PM, Moe <[email protected]> wrote: >> >> > >> > Hi Natalie, >> >> > >> > The leak is apparent in QStar WIFF files of any size. For a 30 minute >> > >> > BSA QC run (25 MB file size), peak memory usage reached 314 MB (to >> > >> > produce ~195 MB mzXML output), which doesn't seem reasonable. Do you >> > >> > have your own data files that you can debug with? If not, I can upload >> > >> > the BSA run to your FTP site (need the details). >> >> > >> > The data I'm trying to analyze consists of 2 hour runs at about 500 MB >> > >> > each. mzWiff crashes about halfway through a data file, leaving a 1.3 >> > >> > GB (incomplete) mzXML file which ends at an elution time of about 60 >> > >> > minutes (when memory reaches slightly > 2 GB). >> >> > >> > I have tried older releases of the executable up to 4.0.0 and all >> > >> > behave the same. I am considering recompiling a 64-bit executable, or >> > >> > trying to locate the leak. I'll check out the code from SourceForge, >> > >> > but if you have additional instructions for setting up the VC project, >> > >> > please let me know. >> >> > >> > Thanks, >> > >> > Moyez >> >> > >> > On Mar 16, 1:41 pm, Natalie Tasman <[email protected]> wrote: >> > >> >> Hello Moyez, >> >> > >> >> Thank you for the detailed bug report. I haven't heard of issues >> > >> >> with >> > >> >> mzWiff. Thanks for the MSVC hint-- I'm pretty sure this is not >> > >> >> turned >> > >> >> on yet, but actually probably shouldn't be necessary. I agree with >> > >> >> you that that much memory usage sounds like a memory leak. The >> > >> >> converters are designed to process data scan-by-scan with a >> > >> >> relatively >> > >> >> low memory overhead. >> >> > >> >> If you can upload a sample file we can take a look and see if we can >> > >> >> replicate behavior and then fix it. >> >> > >> >> Thanks, >> >> > >> >> Natalie >> >> > >> >> On Mon, Mar 16, 2009 at 9:11 AM, Moe <[email protected]> wrote: >> >> > >> >> > I'm trying to convert Analyst QStar Elite data acquired with >> > >> >> > Analyst >> > >> >> > QS 2.0, with mzWiff version 4.2.0(build Feb 19 2009 09:48:08), on >> > >> >> > Windows XP Pro SP3 and Windows Server 2003 R2 x64 Standard SP2. >> >> > >> >> > There appears to be a memory leak causing process memory to >> > >> >> > accumulate >> > >> >> > to > 2GB, shortly after which the program crashes with error: >> >> > >> >> > ERROR: COM error 80004005 while processing sample#2 >> > >> >> > INFO: error message: Unspecified error >> >> > >> >> > This happens both on 32-bit XP and 64-bit Server 2003, both systems >> > >> >> > have > 4GB physical memory. The per-process addressable limit for >> > >> >> > a 32- >> > >> >> > bit process is 2GB unless compiled with the /LARGEADDRESSAWARE >> > >> >> > Visual C >> > >> >> > ++ linker flag. >> >> > >> >> > Is anyone aware of this problem? Is this in fact a leak that can be >> > >> >> > avoided? >> >> > >> >> > Thanks, >> > >> >> > Moyez- Hide quoted text - >> >> > >> >> - Show quoted text -- Hide quoted text - >> >> > >> - Show quoted text -- Hide quoted text - >> >> > - Show quoted text -- Hide quoted text - >> >> - Show quoted text - > > > > --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "spctools-discuss" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/spctools-discuss?hl=en -~----------~----~----~----~------~----~------~--~---
