Bugs item #2528563, was opened at 2009-01-22 14:32
Message generated for change (Comment added) made by stmane
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2528563&group_id=56967
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: PF general
Group: Pathfinder "stable"
Status: Closed
Resolution: None
Priority: 5
Private: No
Submitted By: Andreas Meinl (andreasmeinl)
Assigned to: Peter Boncz (boncz)
Summary: Server crashes
Initial Comment:
During the import of some xml files, the server crashed with the error message
"MonetDB>!FATAL: GDKmallocmax: failed for 252933552 bytes".
After that the server crashes repeatedly for example executing the very simple
xquery "count(collection("test"))".
The output of mclient, where I execute the xquery, is everytime more or less
the same:
xquery>count(collection("test"))
more><>
#GDKmalloc(210756272) fail => BBPtrim(enter) usage[mem=639556716,vm=1117388800]
#GDKmalloc(210756272) fail => BBPtrim(ready) usage[mem=413709780,vm=1096351744]
#GDKmallocmax: recovery ok. Continuing..
15831
MAPI = mone...@localhost:50000
QUERY = count(collection("test"))
ERROR = !ERROR: BBPdecref: 1000000163_rid_level does not have pointer fixes.
xquery>count(collection("test"))
more><>
15831
MAPI = mone...@localhost:50000
QUERY = count(collection("test"))
ERROR = !ERROR: BBPdecref: 1000000163_rid_level does not have pointer fixes.
!ERROR: BBPdecref: 1000000163_attr_qn does not have pointer fixes.
!ERROR: BBPdecref: 1000000163_attr_prop does not have pointer fixes.
xquery>count(collection("test"))
more><>
MAPI = mone...@localhost:50000
ACTION= read_line
QUERY = count(collection("test"))
ERROR = Connection terminated
The environment I use is Windows XP Professional x86 and MonetDB/XQuery server
and client, 32 bit, windows installer, version 20081216.
----------------------------------------------------------------------
>Comment By: Stefan Manegold (stmane)
Date: 2009-04-07 13:03
Message:
Andreas,
In the MonetDB kernel, memory management in particular for 32-bit Windows
has recently been improved and made more robust (as far as possible)
against address space fragmentation; hence, you might want to try, again,
with the latest Feb2009-SP1 release ...
... otherwise, you'd need to resort to a 64-bit system & 64-bit MonetDB
...
Stefan
----------------------------------------------------------------------
Comment By: Peter Boncz (boncz)
Date: 2009-04-07 12:05
Message:
(1) coherent system state after failed add-doc() is tested already
(2) scalability on 32-bits (windows) is unfixable
closing
----------------------------------------------------------------------
Comment By: Peter Boncz (boncz)
Date: 2009-02-12 18:04
Message:
Hi Jens,
I am sorry that your bug report remained unattended for some time.
There are two problems here:
(1) MXQ is not able apparently to import your dataset (crash)
(2) after crashing in the add-doc, it leaves the collection in a corrupted
state
We will be releasing in a few weeks, and the good news about (2) is that
we fixed some bugs regarding error handing in a shred.
About (1), the problem is that 32-bits windows only has 2GB of address
space, and given your collection size (0.5GB) but taking into account
overhead and memory fragmentation, the crash is most likely related to too
little address space. This is very hard to fix, because MonetDB is using
virtual memory for managing databases.
For large collections like these, we advise using 64-bits windows, and use
the 64-bits builds we provide.
You might try to workaround by splitting up your import job in batches,
e.g. adding 100 documents at a time to the collection.
----------------------------------------------------------------------
Comment By: Andreas Meinl (andreasmeinl)
Date: 2009-01-26 08:18
Message:
Eventually I did the same import again on a newly installed instance of
MonetDB/XQuery on another computer and got the same crash of MonetDB again.
The only difference is the first time I had been able to import 15831 xml
files and the second time only 12161 files before the crash happened.
Is there any server log I could attach?
----------------------------------------------------------------------
Comment By: Andreas Meinl (andreasmeinl)
Date: 2009-01-24 18:37
Message:
I wanted to rescue my imported files (or to be more precisely, the time I
had to spent for the import) and to backup the database to restore it on
another instance of MonetDB/XQuery on another computer.
But unfortunately the backup also failed with the error message I've
attached as a PNG file.
Is there any way to repair my database and not having to import all files
again?
File Added: monetdb.png
----------------------------------------------------------------------
Comment By: Andreas Meinl (andreasmeinl)
Date: 2009-01-22 15:25
Message:
- My notebook is equipped with two GB of RAM.
After I have started the server, 1164 MB are still available (and the
system cache has 620 MB) - referring to Windows Task-Manager.
- I wanted to import 492086 files (into one collection), but the server
crashed after having imported 15831 files (with the message I mentioned
before). The size of one file is about 20 to 70 KB.
- Sorry, but these documents represent confidential business documents;
I'm not allowed to distribute them.
- For the import I just use a quick & simple Java loop of the form:
statement = connection.createStatement();
statement.executeUpdate("pf:add-doc(\"C:\\[...]\\test.xml\", \"test.xml\",
\"test\")");
statement.close();
- The crash seems to happen always in the same way, that is after
executing the xquery three times. After the crash there are 1205 MB
available and 794 MB are assigned to the system cache.
----------------------------------------------------------------------
Comment By: Stefan Manegold (stmane)
Date: 2009-01-22 15:00
Message:
Hm, this look as if your Mserver fails to allocate an extra ~200 MB,
probably running out of available address spcae on your 32-bit system (or
at least not finding a consecutive free 200 MB address range due to
excessive memory/address space fragmentation on Windows).
The first question is, why these 200 MB are required, or differently:
How many documents did you (sucessfully) load into how many collections,
and how large are the individual documents and the collections you're
using?
How much physical and virtual memory does your machine have?
Stefan
----------------------------------------------------------------------
Comment By: Ying Zhang (yingying)
Date: 2009-01-22 14:49
Message:
Hello Andreas,
Would you please provide us some more information:
- How large is your memory? How many memory is still available when you
start monet?
- How many files are you trying to add? How large is each document, or
how large are they in total?
- Is it possible to provide us your complete set of documents, in a
compressed form, as attachment or via a URL?
- How do you exactly add the documents? which command do you use?
- How often does the crash happen (always or sometimes)?
Regards,
Jennie
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2528563&group_id=56967
------------------------------------------------------------------------------
This SF.net email is sponsored by:
High Quality Requirements in a Collaborative Environment.
Download a free trial of Rational Requirements Composer Now!
http://p.sf.net/sfu/www-ibm-com
_______________________________________________
Monetdb-bugs mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/monetdb-bugs
