Bugs item #1854224, was opened at 2007-12-19 17:14
Message generated for change (Comment added) made by vzzzbx
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=1854224&group_id=56967
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: PF/runtime
Group: Pathfinder 0.20
>Status: Closed
>Resolution: Works For Me
Priority: 5
Private: No
Submitted By: Wouter Alink (vzzzbx)
Assigned to: Wouter Alink (vzzzbx)
Summary: XQ: large memory usage during multiple inserts
Initial Comment:
when doing many small updates in a single query on a single document, a (very)
large amount of memory is used.
this can be reproduced by rewriting the many subsequenced-queries in the script
from bug 1854215 to a single query. MonetDB will then use at least 11GB (with
oid64) of memory (that's when i killed mserver, i don't know if it will use
more). the source document in which the inserts are being made is about 1M in
size (plain text). peter and lefteris have seen this happen and might have more
info on this.
i guess it is a separate bug (or feature).
note: the work-around "chopping the query into statements with less inserts per
query" works for me.
----------------------------------------------------------------------
>Comment By: Wouter Alink (vzzzbx)
Date: 2007-12-21 16:23
Message:
Logged In: YES
user_id=621590
Originator: YES
hello sjoerd, thanks for looking at this. you are right. i guess i added
it during debugging. i will close this bug.
----------------------------------------------------------------------
Comment By: Sjoerd Mullender (sjoerd)
Date: 2007-12-21 15:08
Message:
Logged In: YES
user_id=43607
Originator: NO
The insert queries are all of the form:
let $todo := subsequence(doc("prepared5123.data")/XIRAF/item[*], 0,128)
let $dest := for $i in doc("test9.xml")//[EMAIL PROTECTED]
where some $k in $todo/@xid satisfies $k = $i/@xid
return $i
for $item in $todo
return
do insert $item/*
into exactly-one([EMAIL PROTECTED] = $item/@xid]])
If you want to do them all at once, it becomes:
let $todo := doc("prepared5123.data")/XIRAF/item[*]
let $dest := for $i in doc("test9.xml")//[EMAIL PROTECTED]
where some $k in $todo/@xid satisfies $k = $i/@xid
return $i
for $item in $todo
return
do insert $item/*
into exactly-one([EMAIL PROTECTED] = $item/@xid]])
However, why do you use this complex query. Isn't this query equivalent
to:
let $todo := doc("prepared5123.data")/XIRAF/item[*]
for $item in $todo
return
do insert $item/*
into exactly-one(doc("test9.xml")//[EMAIL PROTECTED] = $item/@xid])
The advantage of this last query is that it runs in finite time using a
finite amount of memory (it completes on my 2GB system within 4 minutes.
The problem with the original complete query is in the actual querying
part, i.e. before it gets to updating the document.
Wouter, please comment and then reassign.
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=1854224&group_id=56967
-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft
Defy all challenges. Microsoft(R) Visual Studio 2005.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Monetdb-bugs mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/monetdb-bugs
