Bugs item #1976341, was opened at 2008-05-28 11:43
Message generated for change (Comment added) made by stmane
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=1976341&group_id=56967
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: PF/runtime
Group: MonetDB4 4.24
Status: Open
Resolution: None
Priority: 7
Private: No
Submitted By: Wouter Alink (vzzzbx)
Assigned to: Stefan Manegold (stmane)
Summary: XQ: leftovers after deleting document
Initial Comment:
Using todays stable build (preJune, May28, 2008) but it happens with earlier
builds as well), the dbfarm isn't cleaned up as expected after deleting a
document with 'pf:del-doc()'.
When repeatedly adding and deleting documents, the leftovers start piling up,
and eventually the system runs out of disk-space.
See example below.
bash-3.2$ wc -c /tmp/aap.xml
292890833 /tmp/aap.xml
bash-3.2$ echo 'pf:add-doc("/tmp/aap.xml","aap.xml")' | mclient -lxq
bash-3.2$ du -sh /tmp/dbfolder/
367M /tmp/dbfolder/
bash-3.2$ echo 'pf:add-doc("/tmp/aap.xml","aap_1.xml")' | mclient -lxq
bash-3.2$ du -sh /tmp/dbfolder/
733M /tmp/dbfolder/
bash-3.2$ echo 'pf:add-doc("/tmp/aap.xml","aap_2.xml")' | mclient -lxq
bash-3.2$ du -sh /tmp/dbfolder/
1.1G /tmp/dbfolder/
bash-3.2$
bash-3.2$ echo 'pf:del-doc("aap_2.xml")' | mclient -lxq
bash-3.2$ du -sh /tmp/dbfolder/
997M /tmp/dbfolder/
bash-3.2$ echo 'pf:del-doc("aap_1.xml")' | mclient -lxq
bash-3.2$ du -sh /tmp/dbfolder/
895M /tmp/dbfolder/
bash-3.2$ du -sh /tmp/dbfolder/
895M /tmp/dbfolder/
bash-3.2$ du -sh /tmp/dbfolder/
895M /tmp/dbfolder/
bash-3.2$ echo 'pf:del-doc("aap.xml")' | mclient -lxq
bash-3.2$ du -sh /tmp/dbfolder/
794M /tmp/dbfolder/
bash-3.2$ du -sh /tmp/dbfolder/
794M /tmp/dbfolder/
bash-3.2$
----------------------------------------------------------------------
>Comment By: Stefan Manegold (stmane)
Date: 2008-06-04 02:16
Message:
Logged In: YES
user_id=572415
Originator: NO
See also
http://sourceforge.net/mailarchive/forum.php?thread_name=20080603233242.GA31340%40cwi.nl&forum_name=monetdb-developers
----------------------------------------------------------------------
Comment By: Stefan Manegold (stmane)
Date: 2008-05-29 18:15
Message:
Logged In: YES
user_id=572415
Originator: NO
The previous release (Feb2008: MonetDB 4.22.0 + Pathfinder 0.22.0) shows
the same behavior.
----------------------------------------------------------------------
Comment By: Stefan Manegold (stmane)
Date: 2008-05-29 18:09
Message:
Logged In: YES
user_id=572415
Originator: NO
first analysis seems to show that after a "pf:del-doc()", the BATs holding
the document are (correctly) turned from "persistent" into "transient", and
their "lrefcnt" is reduced from "2" to "1", but (most probably due to the
"lrefcnt" of "1"), they stay "around" until this one reference to them is
released, e.g., by stopping (and restarting) the server --- need to check,
why/where this one lref comes from ...
----------------------------------------------------------------------
Comment By: Lefteris Sidirourgos (lsidir)
Date: 2008-05-29 15:28
Message:
Logged In: YES
user_id=1856546
Originator: NO
Before adding the doc:
232K MonetDB4
12K MonetDB4/xquery_logs
8.0K MonetDB4/xquery_logs/demo
216K MonetDB4/dbfarm
212K MonetDB4/dbfarm/demo
204K MonetDB4/dbfarm/demo/bat
4.0K MonetDB4/dbfarm/demo/bat/LEFTOVERS
8.0K MonetDB4/dbfarm/demo/bat/BACKUP
4.0K MonetDB4/dbfarm/demo/bat/03
44K MonetDB4/dbfarm/demo/bat/01
4.0K MonetDB4/dbfarm/demo/bat/05
4.0K MonetDB4/dbfarm/demo/bat/04
After adding the doc:
366M MonetDB4
12K MonetDB4/xquery_logs
8.0K MonetDB4/xquery_logs/demo
366M MonetDB4/dbfarm
366M MonetDB4/dbfarm/demo
366M MonetDB4/dbfarm/demo/bat
4.0K MonetDB4/dbfarm/demo/bat/LEFTOVERS
8.0K MonetDB4/dbfarm/demo/bat/BACKUP
4.0K MonetDB4/dbfarm/demo/bat/03
44K MonetDB4/dbfarm/demo/bat/01
179M MonetDB4/dbfarm/demo/bat/05
188M MonetDB4/dbfarm/demo/bat/04
After deleting the doc:
265M MonetDB4
12K MonetDB4/xquery_logs
8.0K MonetDB4/xquery_logs/demo
265M MonetDB4/dbfarm
265M MonetDB4/dbfarm/demo
265M MonetDB4/dbfarm/demo/bat
4.0K MonetDB4/dbfarm/demo/bat/LEFTOVERS
8.0K MonetDB4/dbfarm/demo/bat/BACKUP
4.0K MonetDB4/dbfarm/demo/bat/03
44K MonetDB4/dbfarm/demo/bat/01
77M MonetDB4/dbfarm/demo/bat/05
188M MonetDB4/dbfarm/demo/bat/04
----------------------------------------------------------------------
Comment By: Stefan Manegold (stmane)
Date: 2008-05-29 12:12
Message:
Logged In: YES
user_id=572415
Originator: NO
could you please also report the results of
`find .../MonetDB4 -type d | xargs du -sh`
?
Thanks!
----------------------------------------------------------------------
Comment By: Lefteris Sidirourgos (lsidir)
Date: 2008-05-29 12:01
Message:
Logged In: YES
user_id=1856546
Originator: NO
Hi, I repeated this bug, and I am attaching the results. This is not a
query compile time problem but a runtime, but because the mps and algebra
use a slightly different "play docmgm tape" I did the experiments with both
back-ends. The problem is there for both back-ends, and the disk is cleaned
only when the mserver *starts* again (and not when it is killed). The
document I used is around 280MB.
I am re-assigning this bug to Stefan:) (or Peter?)
File Added: leftovers_after_del.report
----------------------------------------------------------------------
Comment By: Stefan Manegold (stmane)
Date: 2008-05-28 12:40
Message:
Logged In: YES
user_id=572415
Originator: NO
Moreover, does /tmp/dbfolder/ only contain the dbfarm or also the log
directory?
How does the size of the log directory evolve/change in above example?
----------------------------------------------------------------------
Comment By: Stefan Manegold (stmane)
Date: 2008-05-28 12:38
Message:
Logged In: YES
user_id=572415
Originator: NO
Wouter,
could you please check
(1) which of the sub directories of /tmp/dbfolder/ contains the
left-overs, and
(2) whether stopping and re-starting Mserver does change (reduce) the size
of /tmp/dbfolder/ (and/or its subdirectories)
?
Thanks!
----------------------------------------------------------------------
Comment By: Wouter Alink (vzzzbx)
Date: 2008-05-28 11:50
Message:
Logged In: YES
user_id=621590
Originator: YES
(increased priority a little, as it seems a showstopper for us)
additional information:
- restarting mserver does clean up the data as expected
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=1976341&group_id=56967
-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://sourceforge.net/services/buy/index.php
_______________________________________________
Monetdb-bugs mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/monetdb-bugs
