I see the #else code now, which is pretty much the same as my fix. Too bad I didn't notice it earlier.
However, the latest TPP release still contains this "unstable" version. Moyez On Mar 17, 7:17 pm, Chee-Hong Wong <[email protected]> wrote: > Hi Moe, > Thanks for pinpointing the cause. > > Please just change the macro of #if 1 to #if 0 and you will get the old > behaviour exactly like what you have program. > > I have probably committed this code with the wrong macro by accident. It is > based on a undocumented "API". The motivation is that mzWiff can reduce it > execution time by 30-60%. But my testing has revealed that it is very > unstable depending on the version of Analyst / AnalystQS. As I wish to > return to it some days and thus have not removed the chunk of code from the > check in. > > Sorry for causing you the problem. > > Note: AnalystQS 1.1, Analyst 1.4 and Analyst 1.5 are not affected by this > "bug". > > CH > > > > On Tue, Mar 17, 2009 at 2:38 PM, Moe <[email protected]> wrote: > > > The memory leak appears to be isolated in class AnalystQS20Interface, > > method processMSSpecData, in the call to "m_ipGetRawDataSpec- > > >GetRawScanXYPoints()" when requesting the product spectrum's entire > > set of data points (origical code snip below). > > > /*** ORIGINAL CODE FROM RELEASE 4.2.0 ***/ > > > inline void AnalystQS20Interface::processMSSpecData(Scan *pScan) { > > #if 1 > > ... > > long lSpecDataPoints; > > ... > > VARIANT dXMass; VariantInit(&dXMass); > > VARIANT dYInts; VariantInit(&dYInts); > > ... > > HRESULT hr = m_ipGetRawDataSpec->GetRawScanXYPoints > > (&lSpecDataPoints, &dXMass, &dYInts); > > ... > > long lPeaksCount=0; > > ... > > for (long dpi=1; dpi<=lSpecDataPoints; dpi++, pXMass++, pYInts++) { > > if (*pYInts>0.0) { > > pScan->mzArray_[lPeaksCount] = *pXMass; > > pScan->totalIonCurrent_ += > > (pScan->intensityArray_[lPeaksCount] > > =*pYInts); > > lPeaksCount++; > > > if (pScan->basePeakIntensity_<*pYInts) { > > pScan->basePeakMZ_ = *pXMass; > > pScan->basePeakIntensity_ = *pYInts; > > } > > } > > } > > ... > > > It appears that the memory occupied at &dXMass and &xYInts VARIANTs > > does not get freed even after descoping, probably because it is being > > written externally to mzWiff by the AnalystService process. It does > > not seem possible to free this memory internally, but I have not > > explored all options here. > > > Perhaps an ultimate call to "m_ipGetRawDataSpec->Release" may induce > > the freeing, but I suppose this would render the IGetRawData structure > > prematurely unusable. > > > As a workaround, I avoid the call to GetRawScanXYPoints() altogether, > > instead calling m_ipFMANSpecData->GetDataPointXValue() and - > > >GetDataPointYValue() to request an individual datapoint, as per code > > below. As this call must be made for each data point, there is a > > performance hit. I estimated it to be about 30% longer execution time > > for a 500 MB input, which is still reasonable for me at this time. > > > /*** WORKAROUND CODE ***/ > > ... > > long lSpecDataPoints = m_ipFMANSpecData->GetNumberOfDataPoints(); > > ... > > double dblX, dblY; > > long lPeaksCount=0; > > for (long dpi=1; dpi<=lSpecDataPoints; dpi++) { > > dblX = m_ipFMANSpecData->GetDataPointXValue(dpi); > > dblY = m_ipFMANSpecData->GetDataPointYValue(dpi); > > if (dblY>0) { > > pScan->mzArray_[lPeaksCount] = dblX; > > pScan->intensityArray_[lPeaksCount] = dblY; > > pScan->totalIonCurrent_ += dblY; > > lPeaksCount++; > > > if (pScan->basePeakIntensity_ < dblY) { > > pScan->basePeakMZ_ = dblX; > > pScan->basePeakIntensity_ = dblY; > > } > > } > > } > > > On Mar 16, 6:09 pm, Moe <[email protected]> wrote: > > > I have uploaded the files bsa-20fmol-2.wiff/.scan. Although the > > > converter doesn't crash on this file, it did reach > 300 MB memory > > > usage in my attempt. > > > > Let me know if you need anything else for your debugging. > > > > Thanks, > > > Moyez > > > > On Mar 16, 5:46 pm, Natalie Tasman <[email protected]> wrote: > > > > >http://groups.google.com/group/spctools-discuss/web/how-to-upload-fil. > > .. > > > > > On Mon, Mar 16, 2009 at 2:43 PM, Moe <[email protected]> wrote: > > > > > > Thanks for the VC tips. Where can I upload the BSA file? > > > > > > Moyez > > > > > > On Mar 16, 5:25 pm, Natalie Tasman <[email protected]> > > wrote: > > > > >> Hi Moyez, > > > > > >> Of course we have our own files to test, but yes, please upload your > > > > >> file, as we've never observed the behavior you're reporting. > > > > > >> And also please rerun your conversions with the "-v" (verbose) > > option > > > > >> and copy and paste the output up to the "sha-1" calculation. > > > > > >> You're welcome to rebuild and debug the converter on your own. Just > > > > >> open the TPP.sln file and build the mzWiff project from there. Note > > > > >> you'll also need the "win_lib" project to be checked out at the same > > > > >> level as the TPP code for boost, zlib, etc. > > > > > >> -Natalie > > > > > >> On Mon, Mar 16, 2009 at 2:07 PM, Moe <[email protected]> wrote: > > > > > >> > Hi Natalie, > > > > > >> > The leak is apparent in QStar WIFF files of any size. For a 30 > > minute > > > > >> > BSA QC run (25 MB file size), peak memory usage reached 314 MB (to > > > > >> > produce ~195 MB mzXML output), which doesn't seem reasonable. Do > > you > > > > >> > have your own data files that you can debug with? If not, I can > > upload > > > > >> > the BSA run to your FTP site (need the details). > > > > > >> > The data I'm trying to analyze consists of 2 hour runs at about > > 500 MB > > > > >> > each. mzWiff crashes about halfway through a data file, leaving a > > 1.3 > > > > >> > GB (incomplete) mzXML file which ends at an elution time of about > > 60 > > > > >> > minutes (when memory reaches slightly > 2 GB). > > > > > >> > I have tried older releases of the executable up to 4.0.0 and all > > > > >> > behave the same. I am considering recompiling a 64-bit executable, > > or > > > > >> > trying to locate the leak. I'll check out the code from > > SourceForge, > > > > >> > but if you have additional instructions for setting up the VC > > project, > > > > >> > please let me know. > > > > > >> > Thanks, > > > > >> > Moyez > > > > > >> > On Mar 16, 1:41 pm, Natalie Tasman <[email protected]> > > wrote: > > > > >> >> Hello Moyez, > > > > > >> >> Thank you for the detailed bug report. I haven't heard of issues > > with > > > > >> >> mzWiff. Thanks for the MSVC hint-- I'm pretty sure this is not > > turned > > > > >> >> on yet, but actually probably shouldn't be necessary. I agree > > with > > > > >> >> you that that much memory usage sounds like a memory leak. The > > > > >> >> converters are designed to process data scan-by-scan with a > > relatively > > > > >> >> low memory overhead. > > > > > >> >> If you can upload a sample file we can take a look and see if we > > can > > > > >> >> replicate behavior and then fix it. > > > > > >> >> Thanks, > > > > > >> >> Natalie > > > > > >> >> On Mon, Mar 16, 2009 at 9:11 AM, Moe <[email protected]> wrote: > > > > > >> >> > I'm trying to convert Analyst QStar Elite data acquired with > > Analyst > > > > >> >> > QS 2.0, with mzWiff version 4.2.0(build Feb 19 2009 09:48:08), > > on > > > > >> >> > Windows XP Pro SP3 and Windows Server 2003 R2 x64 Standard SP2. > > > > > >> >> > There appears to be a memory leak causing process memory to > > accumulate > > > > >> >> > to > 2GB, shortly after which the program crashes with error: > > > > > >> >> > ERROR: COM error 80004005 while processing sample#2 > > > > >> >> > INFO: error message: Unspecified error > > > > > >> >> > This happens both on 32-bit XP and 64-bit Server 2003, both > > systems > > > > >> >> > have > 4GB physical memory. The per-process addressable limit > > for a 32- > > > > >> >> > bit process is 2GB unless compiled with the /LARGEADDRESSAWARE > > Visual C > > > > >> >> > ++ linker flag. > > > > > >> >> > Is anyone aware of this problem? Is this in fact a leak that > > can be > > > > >> >> > avoided? > > > > > >> >> > Thanks, > > > > >> >> > Moyez- Hide quoted text - > > > > > >> >> - Show quoted text -- Hide quoted text - > > > > > >> - Show quoted text -- Hide quoted text - > > > > > - Show quoted text -- Hide quoted text - > > > > - Show quoted text -- Hide quoted text - > > - Show quoted text - --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "spctools-discuss" group. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/spctools-discuss?hl=en -~----------~----~----~----~------~----~------~--~---
