Bugs item #2518276, was opened at 2009-01-19 00:39
Message generated for change (Comment added) made by lafanasi
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2518276&group_id=56967
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: PFtijah
Group: Pathfinder "stable"
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Loredana Afanasiev (lafanasi)
Assigned to: Henning Rode (hrode)
Summary: memory leak
Initial Comment:
Dear developers,
When running the simple keyword query below ~100 times, the MServer's memory
footprint grows to 100% and the server crashes with no error messages. My
collection has ~1GB.
let $opt := <TijahOptions ft-index="polietiekedata" ir-model="NLLR"/>
let $c := collection("HAN")
let $qid := tijah:query-id($c, "//spreker[about(.,%KEYWORD%)]", $opt)
for $res in tijah:nodes($qid)
return <pair>{( string($res/@naam), tijah:score($qid, $res))}</pair>
thank you!
l.
OS: Linux, 2.6.24.3-50.fc8
Mem: 19GB
Compiler: gcc 4.1.2
Parser tools: ?
autoconf (GNU Autoconf) 2.61
automake (GNU automake) 1.10
ltmain.sh (GNU libtool) 1.5.24
----------------------------------------------------------------------
>Comment By: Loredana Afanasiev (lafanasi)
Date: 2009-02-23 16:36
Message:
Hi all,
the problem is not fixed yet, but Henning seems to have found the
problem..
"i was just reformulating your query to this one:
let $opt := <TijahOptions ft-index="polietiekedata" ir-model="NLLR" />
(:let $query := tijah:tokenize("knettergek") :)
let $query := "knettergek"
(: We need the "else" because empty queries are not allowed :)
let $nexi := concat("//spreker[about(.,",$query,")]")
for $res in tijah:queryall($nexi, $opt)
return
<pair>{string($res/@naam)}</pair>
and suddenly there was no memory leak anymore.. "
So it seems that the problem sits in the implementation of query-id()
function.
Henning the floor is yours :)
l.
----------------------------------------------------------------------
Comment By: Stefan Manegold (stmane)
Date: 2009-02-15 21:31
Message:
Loredana, JanF,
could you please
(1) check whether the problem is indeed fixed
(2) provide/add a test
(3) close the bug report once the fix is confirmed and a test has been
added.
Thanks!
Stefan
----------------------------------------------------------------------
Comment By: Jan Flokstra (jflokstra)
Date: 2009-02-03 11:40
Message:
I had a more systematic look at the problem here. I think I solved the
problem by:
1) removing 2 unnecessary BBPfixes()
2) assigning the nexi_result variable in the generated pftijah script to
nil;
This was a very strange error. The bat is removed from the buffer pool
when the script is run throught "mclient -lmil" and we thought our
generated script was memory leak free. But when we called it from "C" by
"monet_exec(script_str);" the bat was not cleared and we have to assign nil
manually. To prevent further problems in the futrue we plan to put the
entire scripts between { }. This way the MIL interpreter should kill all
used variables also when called from "C"
We reduced memory consumption for 'big' queries' (eg. 50.000 results) from
850K to 16K which is about avarage for Pathfinder,
I wait for conformation from Loredana before I close the bug.
JanF.
----------------------------------------------------------------------
Comment By: Henning Rode (hrode)
Date: 2009-01-19 16:55
Message:
i just checked it together with jan f..
result: we have indeed a memory leak here.
5K per simple pathfinder query.
1.8M per simple pf/tijah query.
26K per pf/tijah query with new query processor.
the 1.8M are of course quite bad, but since it only occurs in the old
version of the query processor, we won't fix that anymore.
the 26K in the new version are now on my todo list. i still have to find
out, whether it occurs in the C code of the query processor, or whether we
simply do not give enough hints to MonetDB to free MIL vars.
loredana, you can maybe use as an additional option to your queries:
<TijahOptions newversion="true"/>
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2518276&group_id=56967
------------------------------------------------------------------------------
Open Source Business Conference (OSBC), March 24-25, 2009, San Francisco, CA
-OSBC tackles the biggest issue in open source: Open Sourcing the Enterprise
-Strategies to boost innovation and cut costs with open source participation
-Receive a $600 discount off the registration fee with the source code: SFAD
http://p.sf.net/sfu/XcvMzF8H
_______________________________________________
Monetdb-bugs mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/monetdb-bugs
