I think it's objects creating new objects whose memory is than never released even if the first object is unset.(most likely the php problem as described in the link you sent me in your original reply). I tried to keep track where which objects are created but I became quickly rather confused. I could release a few extra K bytes but not enough to make any real difference. However, after I found in a moment of enlightnment a way to retrieve the data I need without using any objects at all I didn't try any further.
Cheers, Gu Quoting Markus Krötzsch <[email protected]>: > Ok, thanks for this first analysis. The built-in MW profiling actually > reports > memory usage as well. This might further increase our understanding of the > issue. But ultimately, some detailed report on memory usage for allocated > data > structures in PHP would be most valuable to find out what really occupies > those additional bytes. I have some guesses but it will take me some time > until I can look into the details. > > -- Markus > > On Donnerstag, 8. Januar 2009, [email protected] wrote: > > Hi, > > > > I added some memory_get_usage() calls to the code > > > > 'before' is the return value before the call to > > SMWQueryProcessor::getResultFromQueryString (with different offset values) > > '#1' value in getInstanceQueryResult() after > > $result = new SMWQueryResult() > > '#2' value in getInstanceQueryResult() after the foreach loops > > assigning $row[] = new SMWResultArray > > > > Offset:0 before:15,295,804 > > #1: 16,775,940 > > #2: 17,468,932 > > Offset:10 before:17,557,768 > > #1: 17,581,712 > > #2: 18,729,208 > > Offset:20 before:18,701,124 > > #1: 18,733,052 > > #2: 20,044,556 > > > > Offset:480 before:128,938,860 > > #1: 128,981,036 > > #2: 131,807,872 > > Offset:490 before:131,779,456 > > #1: 131,821,492 > > <b>Fatal error</b>: Allowed memory size of 134217728 bytes > > > > Basically there are app. 240K memory used up for my query after calling > > getResultFromQueryString.and at present that limits me to less than 500 > > returned results. > > > > It would of course be nice if there would be a way to release some of that > > memory again after the result has been processed (but I guess first one > > would have to know where it went, to the SMWQueryResult object or > somewhere > > else?) > > > > Another question is, if (as I only just need the page titles from the > query > > result) there is a simpler way to do that than calling > > getResultFromQueryString? > > > > Cheers, > > Gu > > > > Quoting Markus Krötzsch <[email protected]>: > > > Hi, > > > > > > I am not aware of a good way of debugging memory issues in PHP yet. Can > > > anyone > > > recommend a free profiler or the like? As for your problem, there is a > > > general > > > tendency of PHP to hoard memory during (very) long runs. This is a known > > > bug > > > > > > that apparently can only be avoided by re-engineering the object > > > dependencies > > > > > > in affected code [1]. However, without a proper memory profiling, I have > > > no feasible approach for finding out where this should be done. > > > > > > Of course, it is also possible that some other problem causes memory to > > > be eaten without PHP being to blame. Usually this is caused by look-up > > > arrays that are used as caches in PHP. For example, MediaWiki has a > > > LinkCache that can grow without control; if you do long runs that may > > > (indirectly) use this > > > > > > cache, then it is required to purge it manually from time to time. > Again, > > > there might be further such problems, though I have tried to limit all > > > caches' > > > size in SMW. Ideas for better debugging are welcome. > > > > > > Regards, > > > > > > Markus > > > > > > [1] http://paul-m-jones.com/?p=262 > > > > > > On Mittwoch, 7. Januar 2009, [email protected] wrote: > > > > Hi, > > > > > > > > I wonder if anyone can tell me if the functions used when I call > > > > SMWQueryProcessor::getResultFromQueryString should release all the > > > > memory after they finished? > > > > > > > > I thought I could avoid a memory problem by calling > > > > getResultFromQueryString with different 'offset' values to retrieve > > > > > > smaller > > > > > > > chunks of articles. However, at some point I still run out of memory > > > > > > during > > > > > > > the loop, even if I just call the function and ignore the result. > > > > > > > > $maxresults = 500; > > > > $limit = $params['limit']; # is 10 > > > > for ($ofs = 0; $ofs < $maxresults ; $ofs += $limit) { > > > > $params['offset'] = $ofs; > > > > $dummy = > > > > SMWQueryProcessor::getResultFromQueryString($querystring, $params, > > > > $printouts, SMW_OUTPUT_WIKI); > > > > } > > > > > > > > When $ofs reaches 490 I run out of memory (Fatal error: Allowed memory > > > > > > size > > > > > > > of 134217728 bytes exhausted). > > > > > > > > The test query is '[[MyProp::~*;?;?]][[Category:MainArticle]]' with > > > > about 19000 articles matching (as using [[MyProp::?;?;?]] gives the > > > > error 'The value of property "MyProp" was not understood', I have to > > > > replace one '?' with '~*'). > > > > > > > > MediaWiki 1.13.1 > > > > PHP 5.2.1 > > > > MySQL 6.0.45 > > > > SMW 1.5a SVN > > > > > > > > Thanks, > > > > Gu > > > > > > > ------------------------------------------------------------------------- > > >-- > > > > > > >--- Check out the new SourceForge.net Marketplace. > > > > It is the best place to buy or sell services for > > > > just about anything Open Source. > > > > http://p.sf.net/sfu/Xq1LFB > > > > _______________________________________________ > > > > Semediawiki-devel mailing list > > > > [email protected] > > > > https://lists.sourceforge.net/lists/listinfo/semediawiki-devel > > > > > > -- > > > Markus Krötzsch > > > Semantic MediaWiki http://semantic-mediawiki.org > > > http://korrekt.org [email protected] > > > -- > Markus Krötzsch > Semantic MediaWiki http://semantic-mediawiki.org > http://korrekt.org [email protected] > > ------------------------------------------------------------------------------ This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ Semediawiki-devel mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/semediawiki-devel
