Bugs item #2785756, was opened at 2009-05-02 22:04
Message generated for change (Comment added) made by boncz
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2785756&group_id=56967
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: XML
Group: Pathfinder "stable"
>Status: Closed
>Resolution: Invalid
Priority: 5
Private: No
Submitted By: winston (wwinston)
Assigned to: Nobody/Anonymous (nobody)
Summary: XQ: crashes on shredding of large XML 1.2 GB XMark document
Initial Comment:
xquery>pf:add-doc("D:\Projects_Data_2009\data\XMarkStandard\fact8.xml","fact8.xm
l","fact8")
more>
more><>
MAPI = mone...@localhost:50000
QUERY = pf:add-doc("D:\Projects_Data_2009\data\XMarkStandard\fact8.xml","fact8.x
ml","fact8")
ERROR = !↕#GDKmmap(193462272) fails, try to free up space [memory in use=1433139
34,virtual memory in use=1171783680]
#GDKmmap(193462272) result [mem=143313934,vm=1171783680]
MAPI = mone...@localhost:50000
QUERY = pf:add-doc("D:\Projects_Data_2009\data\XMarkStandard\fact8.xml","fact8.x
ml","fact8")
ERROR = !ERROR: [shred_url]: 1 times inserted nil due to errors at tuples
0...@0.
!ERROR: [shred_url]: first error was:
!ERROR: HEAPextend: failed to extend to 193148746 for 10\1001theap
!ERROR: shredBAT_append_str: APPEND-STR[_prop_text](
!ERROR: tale cordelia sick terra wor crutch dates fie slee
p boarded avoid
!ERROR: arbour slippery yet sampson prove within dejected
graze clock
!ERROR: gentleman friendly brass best answering lord front
pitifully
!ERROR: forever expect cool halfpenny very gar post hinge
telling absence
!ERROR: affairs footed murderer author call key bent stran
ger
!ERROR: guildenstern yoke kneeling virtue diadem great cou
nterfeit
!ERROR: bootless cousin honestly councils troyans examinat
ion cruel
!ERROR: fearful chambermaids people prone conspirators lie
until oph
!ERROR: letting abortive appears purposed jour nod souther
ly dearest
!ERROR: vexation affairs theft preventions sparrow opinion
hue two
!ERROR: suddenly year mothers moon majesty tongue mortalit
y thereof
!ERROR: gratify march preventions harmless others intents
awhile alas
!ERROR: tweaks greek weeping renews penny masks shines bat
he pump choke
!ERROR: bawd gerard even beside load fall peeping ), BUNap
pend fails
!ERROR: CMDshred_url: operation failed.
xquery>
----------------------------------------------------------------------
>Comment By: Peter Boncz (boncz)
Date: 2009-11-12 10:52
Message:
This is MXQ running out of memory on 32-bits windows. Nothing unusual, MXQ
has scalability limits, certainly on 32-bits windows. Not a bug at all.
----------------------------------------------------------------------
Comment By: Stefan Manegold (stmane)
Date: 2009-08-14 14:10
Message:
tagged subject
----------------------------------------------------------------------
Comment By: Sjoerd Mullender (sjoerd)
Date: 2009-08-05 14:54
Message:
I just tried shredding the scalefactor-10 XMark document on my 32 bit
WIndows machine (2GB RAM, plenty of free disk).
As expected, this failed, but it seems to have failed in a pretty
unspectacular way. The client reported an error, and the server just
stayed alive. Memory use of the server went up during shredding until it
tried to get more than was possible, and then went back to a pretty
reasonable level.
The size of the document was 1172322349 bytes (on Linux, i.e. with just LF
line endings--on Windows, it would need another 20393169 bytes for the
CRs).
I tried this with the newly created stable branch for the upcoming August
2009 release.
I'd say, we can close this bug as Rejected since it involves going beyond
the boundaries of what the platform can do.
However, the error message that I got is extremely unclear, so we may want
to do something about that:
#GDKmmap(278921216) fails, try to free up space [memory in
use=146786972,virtual memory in use=1135542272]
#GDKmmap(278921216) result [mem=134294220,vm=1121058816]
MAPI = mone...@localhost:50000
QUERY = pf:add-doc('c:\Documents and Settings\sjoerd\My
Documents\XMark-f10.xml','XMark-f10.xml')
ERROR = !ERROR: [shred_url]: 1 times inserted nil due to errors at tuples
0...@0.
!ERROR: [shred_url]: first error was:
!ERROR: HEAPextend: failed to extend to 278712451 for 05\502theap
!ERROR: shredBAT_append_str: APPEND-STR[_prop_text](
!ERROR: ), BUNappend fails
!ERROR: CMDshred_url: operation failed.
----------------------------------------------------------------------
Comment By: Peter Boncz (boncz)
Date: 2009-06-05 13:21
Message:
sorry, I missed the previous comments in the sf interface, so stefan had
already commented about this. anyway, the documentation link describes how
you could try to use 3GB on 32-bits windoze. You might just manage, maybe.
----------------------------------------------------------------------
Comment By: Peter Boncz (boncz)
Date: 2009-06-05 13:19
Message:
http://monetdb.cwi.nl/projects/monetdb/XQuery/Documentation/Scalability.html#Scalability
----------------------------------------------------------------------
Comment By: Peter Boncz (boncz)
Date: 2009-06-05 13:18
Message:
This appears to be a scalability problem. Please provide information on
your OS, machine and the version of MonetDB you use.
Windoze 32-bits is limited by the 2GB virtual memory limit (no matter how
much RAM you have), and there always is fragmentation, so a 1.2GB file
won't shred. Use 64-bits windoze, and a 64-bits build.
----------------------------------------------------------------------
Comment By: Stefan Manegold (stmane)
Date: 2009-05-05 08:48
Message:
Winston,
please note that MonetDB will (try to) use virtual memory (swap) and/or
memory-mapped files once it runs out of available free physical memory. The
problem on 32-bit systems, in particular on Windows, is the 32-bit address
space limitation, i.e., the system and/or any process cannot address more
than (in theory) 4 GB (2^32) --- in practice it's often as "little" as 3 GB
or even 2 GB (2^31). You are most probably hitting this limit, combined
with address space fragmentation as explained earlier.
That said, neither during loading nor during querying, MonetDB needs to
have the whole document physically in memory, but it needs to have it
accessible entirely in the address space --- though having it in physical
memory will speed-up processing.
----------------------------------------------------------------------
Comment By: winston (wwinston)
Date: 2009-05-05 02:46
Message:
Stefan,
Thank you for your assistance.
The 1.2 GB document load crashed the server (i.e. the server process
exited and ended) and corrupted the dbfarm so that I can't run queries
against the other collections. I will need to reload the dbfarm.
I am running the MonetDB4 20090324 release, the March 2009 release.
I have over 12 GB of disk space left.
I am running WinXP 32-bit on a Dell laptop. The laptop has 2GB ram and
that is the maximum that can be installed, so as far as ram is concerned, I
am stuck.
When loading the 1.2GB document, I am sure that there was less than 1 GB
of ram available, as other Windows processes take up 800KB just after boot
and I had a few other programs running too.
I will try to shut down as much as I can and load the document right after
a cold boot.
However, I have a few questions:
(1) Does MonetDB always load the full document into RAM when loading?
(2) If I run xquery against the 1.2GB document in the database will
MonetDB load the whole document into memory or will MonetDB load a small
portion of the document into buffers even if the whole document needs to be
scanned?
(3) Can I configure MonetDB to grab 1.2GB + 200KB = 1.5GB on startup via
configuration so that the memory is grabbed in advance?
----------------------------------------------------------------------
Comment By: Stefan Manegold (stmane)
Date: 2009-05-03 08:27
Message:
and: does your server indeed *crash*, or does it just fail to load that
document, giving the reported error, but continue running?
----------------------------------------------------------------------
Comment By: Stefan Manegold (stmane)
Date: 2009-05-03 08:25
Message:
ps: it could of course also be that your disk has less than 200 MB free
...
----------------------------------------------------------------------
Comment By: Stefan Manegold (stmane)
Date: 2009-05-03 08:24
Message:
I guess from your previous bug report that you are running on 32-bit
Windows using the Feb2009 release of MonetDB.
I this case (32-bit Windows), you most probably simply hit the 2 GB
(31-bit) address space limitation in combination with address space
fragmentation on Win32: While loading that 1.2 GB document, Mserver grows
to almost 1.2 GB (vm=1171783680) and then needs to allocate (memory map) an
other almost 200 MB (GDKmmap(193462272)) apparently this fails, most
probably because there is no single consecutive free range of 200 MB left
in remaining <= 800 MB of the 2 GB address space, most probably due to
address space fragmentation.
You might want to upgrade to the latest Feb2009-SP2 bug fix release --- we
implemented some fixes in Feb2009-SP1 (and hence Feb2009-SP2) to better
cope with the memory/address space limitations and fragmentation problems
on Win32 --- and start your Mserver freshly only to load that 1.2 GB
document.
If that does not help, either, I see only two solution:
1) Migrate to 32-bit Unix (e.g., Linux)
2) (better) migrate to a 64-bit OS using a 64-bit build of MonetDB
Stefan
----------------------------------------------------------------------
Comment By: Stefan Manegold (stmane)
Date: 2009-05-02 22:19
Message:
The usual questions:
- Which version of MonetDB? (preferably the output of `Mserver
--version`)
- Which OS? (The paths indicate Windows, but which version, and 32-bit or
64-bit?)
- What kind of hardware? (mainly how much main memory?)
- How much free disk space on the partition (or "drive") that your dbfarm
is located on?
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2785756&group_id=56967
------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Monetdb-bugs mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/monetdb-bugs
