Bugs item #2785756, was opened at 2009-05-02 22:04
Message generated for change (Comment added) made by stmane
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2785756&group_id=56967
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: XML
Group: MonetDB4 "stable"
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: winston (wwinston)
Assigned to: Nobody/Anonymous (nobody)
Summary: crashes on shredding of large XML 1.2 GB XMark document
Initial Comment:
xquery>pf:add-doc("D:\Projects_Data_2009\data\XMarkStandard\fact8.xml","fact8.xm
l","fact8")
more>
more><>
MAPI = mone...@localhost:50000
QUERY = pf:add-doc("D:\Projects_Data_2009\data\XMarkStandard\fact8.xml","fact8.x
ml","fact8")
ERROR = !↕#GDKmmap(193462272) fails, try to free up space [memory in use=1433139
34,virtual memory in use=1171783680]
#GDKmmap(193462272) result [mem=143313934,vm=1171783680]
MAPI = mone...@localhost:50000
QUERY = pf:add-doc("D:\Projects_Data_2009\data\XMarkStandard\fact8.xml","fact8.x
ml","fact8")
ERROR = !ERROR: [shred_url]: 1 times inserted nil due to errors at tuples
0...@0.
!ERROR: [shred_url]: first error was:
!ERROR: HEAPextend: failed to extend to 193148746 for 10\1001theap
!ERROR: shredBAT_append_str: APPEND-STR[_prop_text](
!ERROR: tale cordelia sick terra wor crutch dates fie slee
p boarded avoid
!ERROR: arbour slippery yet sampson prove within dejected
graze clock
!ERROR: gentleman friendly brass best answering lord front
pitifully
!ERROR: forever expect cool halfpenny very gar post hinge
telling absence
!ERROR: affairs footed murderer author call key bent stran
ger
!ERROR: guildenstern yoke kneeling virtue diadem great cou
nterfeit
!ERROR: bootless cousin honestly councils troyans examinat
ion cruel
!ERROR: fearful chambermaids people prone conspirators lie
until oph
!ERROR: letting abortive appears purposed jour nod souther
ly dearest
!ERROR: vexation affairs theft preventions sparrow opinion
hue two
!ERROR: suddenly year mothers moon majesty tongue mortalit
y thereof
!ERROR: gratify march preventions harmless others intents
awhile alas
!ERROR: tweaks greek weeping renews penny masks shines bat
he pump choke
!ERROR: bawd gerard even beside load fall peeping ), BUNap
pend fails
!ERROR: CMDshred_url: operation failed.
xquery>
----------------------------------------------------------------------
>Comment By: Stefan Manegold (stmane)
Date: 2009-05-05 08:48
Message:
Winston,
please note that MonetDB will (try to) use virtual memory (swap) and/or
memory-mapped files once it runs out of available free physical memory. The
problem on 32-bit systems, in particular on Windows, is the 32-bit address
space limitation, i.e., the system and/or any process cannot address more
than (in theory) 4 GB (2^32) --- in practice it's often as "little" as 3 GB
or even 2 GB (2^31). You are most probably hitting this limit, combined
with address space fragmentation as explained earlier.
That said, neither during loading nor during querying, MonetDB needs to
have the whole document physically in memory, but it needs to have it
accessible entirely in the address space --- though having it in physical
memory will speed-up processing.
----------------------------------------------------------------------
Comment By: winston (wwinston)
Date: 2009-05-05 02:46
Message:
Stefan,
Thank you for your assistance.
The 1.2 GB document load crashed the server (i.e. the server process
exited and ended) and corrupted the dbfarm so that I can't run queries
against the other collections. I will need to reload the dbfarm.
I am running the MonetDB4 20090324 release, the March 2009 release.
I have over 12 GB of disk space left.
I am running WinXP 32-bit on a Dell laptop. The laptop has 2GB ram and
that is the maximum that can be installed, so as far as ram is concerned, I
am stuck.
When loading the 1.2GB document, I am sure that there was less than 1 GB
of ram available, as other Windows processes take up 800KB just after boot
and I had a few other programs running too.
I will try to shut down as much as I can and load the document right after
a cold boot.
However, I have a few questions:
(1) Does MonetDB always load the full document into RAM when loading?
(2) If I run xquery against the 1.2GB document in the database will
MonetDB load the whole document into memory or will MonetDB load a small
portion of the document into buffers even if the whole document needs to be
scanned?
(3) Can I configure MonetDB to grab 1.2GB + 200KB = 1.5GB on startup via
configuration so that the memory is grabbed in advance?
----------------------------------------------------------------------
Comment By: Stefan Manegold (stmane)
Date: 2009-05-03 08:27
Message:
and: does your server indeed *crash*, or does it just fail to load that
document, giving the reported error, but continue running?
----------------------------------------------------------------------
Comment By: Stefan Manegold (stmane)
Date: 2009-05-03 08:25
Message:
ps: it could of course also be that your disk has less than 200 MB free
...
----------------------------------------------------------------------
Comment By: Stefan Manegold (stmane)
Date: 2009-05-03 08:24
Message:
I guess from your previous bug report that you are running on 32-bit
Windows using the Feb2009 release of MonetDB.
I this case (32-bit Windows), you most probably simply hit the 2 GB
(31-bit) address space limitation in combination with address space
fragmentation on Win32: While loading that 1.2 GB document, Mserver grows
to almost 1.2 GB (vm=1171783680) and then needs to allocate (memory map) an
other almost 200 MB (GDKmmap(193462272)) apparently this fails, most
probably because there is no single consecutive free range of 200 MB left
in remaining <= 800 MB of the 2 GB address space, most probably due to
address space fragmentation.
You might want to upgrade to the latest Feb2009-SP2 bug fix release --- we
implemented some fixes in Feb2009-SP1 (and hence Feb2009-SP2) to better
cope with the memory/address space limitations and fragmentation problems
on Win32 --- and start your Mserver freshly only to load that 1.2 GB
document.
If that does not help, either, I see only two solution:
1) Migrate to 32-bit Unix (e.g., Linux)
2) (better) migrate to a 64-bit OS using a 64-bit build of MonetDB
Stefan
----------------------------------------------------------------------
Comment By: Stefan Manegold (stmane)
Date: 2009-05-02 22:19
Message:
The usual questions:
- Which version of MonetDB? (preferably the output of `Mserver
--version`)
- Which OS? (The paths indicate Windows, but which version, and 32-bit or
64-bit?)
- What kind of hardware? (mainly how much main memory?)
- How much free disk space on the partition (or "drive") that your dbfarm
is located on?
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2785756&group_id=56967
------------------------------------------------------------------------------
The NEW KODAK i700 Series Scanners deliver under ANY circumstances! Your
production scanning environment may not be a perfect world - but thanks to
Kodak, there's a perfect scanner to get the job done! With the NEW KODAK i700
Series Scanner you'll get full speed at 300 dpi even with all image
processing features enabled. http://p.sf.net/sfu/kodak-com
_______________________________________________
Monetdb-bugs mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/monetdb-bugs
