On Wed, Jan 21, 2009 at 04:16:00AM -0800, sylver_b wrote: > By the way ... what will happen if i only have 4GB of memory on a 64-bit > system and table size over 10GB (for example) ?
MonetDB will use virtual memory, mostly memory mapped files, partly swap --- until the available disk capacity or address space is exceeded --- whatever limit comes first (holds for both 32- & 64-bit systems). Stefan > > > > > ________________________________ > De : Stefan Manegold <[email protected]> > À : sylver_b <[email protected]> > Cc : [email protected] > Envoyé le : Mercredi, 21 Janvier 2009, 11h48mn 42s > Objet : Re : [Monetdb-developers] Re : mclient running out of memory, > crashing mserver5 > > On Wed, Jan 21, 2009 at 03:12:20AM -0800, sylver_b wrote: > > Hi Stephan, > > > > I will reply below : > > > > > > > > ________________________________ > > > > De : Stefan Manegold <[email protected]> > > À : sylver_b <[email protected]> > > Cc : [email protected] > > Envoyé le : Mercredi, 21 Janvier 2009, 10h28mn 41s > > Objet : Re: [Monetdb-developers] Re : mclient running out of memory, > > crashing mserver5 > > > > > > >> Yes > > > > How big is/was your mserver5 when running or just before crashing? > > > > >> missed to check that, but this is what i have at the moment : > > > > USER PID %CPU %MEM VSZ RSS TTY STAT START > > TIME COMMAND > > root 31679 0.0 3.2 1950816 131056 pts/3 Sl+ > > 10:05 0:00 | \_ mserver5 --dbinit=include sql; > > This while being idle or while loading data? > > In any case, with almost 2 GB vitual size, your mserver5 has obviously > reached the addressable limit on a 32-bit system (well, theoretically, > 32-bit systems can address 4 GB, but in practice processes are often only > allowed to address 2 GB). > > > Your machine and OS seem to be 64-bit, right? > > >> 32-bit > > > What about your MonetDB installation? > > Could you please provide us with the output of > > `mserver5 --version` > > ? > > >> # mserver5 --version > > MonetDB server v5.8.0 (32-bit), based on kernel v1.26.0 (32-bit oids) > > Copyright (c) 1993-July 2008 CWI > > Copyright (c) August 2008- MonetDB B.V., all rights reserved > > Visit http://monetdb.cwi.nl/for further information > > Configured for prefix: /root/MonetDB > > Libraries: > > openssl: OpenSSL 0.9.8b 04 May 2006 (compiled with OpenSSL 0.9.8b 04 May > > 2006) > > Compiled by: r...@ulys > > Compilation: gcc -O2 -std=c99 > > Linking : /usr/bin/ld > > Ok, on a 32-bit system, 32-bit MonetDB can only handle data sizes up to 2 GB > at a time. Th efollwoing two limitations apply: each column of your tables > (called BAT in MonetDB lingo) cannot exceed 2 GB AND the total size of all > columns (BATs) active at a time while evaluating queries cannot exceed 2 GB, > i.e., the addressable data size on 32-bit systems. When bulkloading data, > all columns (BATs) of a table are active concurrently, hence, the total > table size cannot exceed 2 GB. > > The GDKmmap failure in you previous posting suggetst that MonetDB cannot > allocate more memory (or to be more precise: address space), most probably > as the available 2 GB address space is already exhausted. > > As Stefan (de Konink) suggested, you can test whether loading your data > succeeds without constraints (primary keys, foreign keys, etc.) defined on > your table(s). If not, your data size might just be too large for a 32-bit > MonetDB, and you'd need to resort to a 64-bit machine + 64-bit MonetDB or > consider fragmenting your data. > > Hope this helps ... > > Stefan > > > How much free space is on your disk partition where your dbfarm is located? > > >> # df -h > > Filesystem Size Used Avail Use% Mounted on > > /dev/sda1 212G 180G 22G 90% / > > varrun 2.0G 56K 2.0G 1% /var/run > > varlock 2.0G 0 2.0G 0% /var/lock > > procbususb 10M 52K 10M 1% /proc/bus/usb > > udev 10M 52K 10M 1% /dev > > devshm 2.0G 0 2.0G 0% /dev/shm > > > > > > > Is there anything to do to prevent this ? should the DB swap at all ? > > > > MonetDB does in-memory processing, i.e., all columns that are active at a > > time > > need to be in the processes address space, either being loaded or being > > memory mapped; so yes, with huge amount of data MonetDB will also use > > virtual memory, either as swap or as memory-mapped files. > > > > Stefan > > > > > > > > > > > > > > > > > > > > ________________________________ > > > De : Stefan de Konink <[email protected]> > > > À : sylver_b <[email protected]> > > > Cc : [email protected] > > > Envoyé le : Mercredi, 21 Janvier 2009, 1h09mn 13s > > > Objet : Re: [Monetdb-developers] mclient running out of memory, crashing > > > mserver5 > > > > > > sylver_b wrote: > > > > I don't think i'm running out of memory. I had to stop and restart > > > > mserver5, flush the table, then re-run the command - all the records > > > > were inserted. Also I noticed this type of messages on the mserver5 > > > > console when piping records through the mclient (over 1million records) > > > > . It seems that the mclient is quiet instable (cf my post about "COPY, > > > > terminating connection" few months ago). > > > > So, is there a way to increase the memory limit or make sure mserver5 > > > >won't crash when running bash inserts via a cron job ? > > > > > > The only thing I was able to do against that; more swap for the job. And > > > of course the standard stuff as loading CSV files, preferably no indices > > > on the table you are loading huge amounts of data to. > > > > > > > > > Stefan > > > > > > > > > > > > > > > ------------------------------------------------------------------------------ > > > This SF.net email is sponsored by: > > > SourcForge Community > > > SourceForge wants to tell your story. > > > http://p.sf.net/sfu/sf-spreadtheword > > > _______________________________________________ > > > Monetdb-developers mailing list > > > [email protected] > > > https://lists.sourceforge.net/lists/listinfo/monetdb-developers > > > > > > -- > > | Dr. Stefan Manegold | mailto:[email protected] | > > | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | > > | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | > > | The Netherlands | Fax : +31 (20) 592-4312 | > > > > On Wed, Jan 21, 2009 at 02:07:04AM -0800, sylver_b wrote: > > > Hi > > > > > > Interesting.. > > > > > > Even worst, this morning i found out that all the inserts of last night > > > failed !! :( > > > > > > > > > #BBPTRIM_ENTER: memsize=1144385432,vmsize=1599613808 > > > #BBPTRIM: memtarget=819717481 vmtarget=0 > > > #BBPTRIM_EXIT: memsize=366265384,vmsize=1599605544 > > > #BBPTRIM_ENTER: memsize=1173710640,vmsize=1628939016 > > > #BBPTRIM: memtarget=849042689 vmtarget=0 > > > #BBPTRIM_EXIT: memsize=366265384,vmsize=1628939016 > > > #BBPTRIM_ENTER: memsize=1143635024,vmsize=1629665320 > > > #BBPTRIM: memtarget=0 vmtarget=19052584 > > > #BBPTRIM_EXIT: memsize=366265384,vmsize=1598862320 > > > #BBPTRIM_ENTER: memsize=1143633968,vmsize=1645786120 > > > #BBPTRIM: memtarget=0 vmtarget=35173384 > > > #BBPTRIM_EXIT: memsize=366265384,vmsize=1645786120 > > > #BBPTRIM_ENTER: memsize=1143633992,vmsize=1676588064 > > > #BBPTRIM: memtarget=0 vmtarget=65975328 > > > #BBPTRIM_EXIT: memsize=366265384,vmsize=1598862320 > > > #BBPTRIM_ENTER: memsize=1143633944,vmsize=1613149168 > > > #BBPTRIM: memtarget=0 vmtarget=2536432 > > > #BBPTRIM_EXIT: memsize=366265384,vmsize=1582347224 > > > #BBPTRIM_ENTER: memsize=1143633920,vmsize=1620095960 > > > #BBPTRIM: memtarget=0 vmtarget=9483224 > > > #BBPTRIM_EXIT: memsize=366265384,vmsize=1620095960 > > > #GDKmmap(148111360) fail => BBPtrim(enter) > > > usage[mem=1143633920,vm=1496757208] > > > #BBPTRIM_ENTER: memsize=1143633920,vmsize=1496757208 > > > #BBPTRIM: memtarget=0 vmtarget=1073741824 > > > #BBPTRIM_EXIT: memsize=366265384,vmsize=1496757208 > > > #GDKmmap(148111360) fail => BBPtrim(ready) > > > usage[mem=1143633920,vm=1496757208] > > > !ERROR: GDKload: cannot mmap(): name=32/3267, ext=theap > > > !OS: Cannot allocate memory > > > !ERROR: GDKload: failed name=32/3267, ext=theap > > > > > > top - 10:00:34 up 249 days, 20:31, 4 users, load average: 0.28, 0.12, > > > 0.04 > > > Tasks: 86 total, 1 running, 85 sleeping, 0 stopped, 0 zombie > > > Cpu(s): 0.0%us, 0.0%sy, 0.0%ni,100.0%id, 0.0%wa, 0.0%hi, 0.0%si, > > > 0.0%st > > > Mem: 4025600k total, 3633228k used, 392372k free, 107544k buffers > > > Swap: 9502408k total, 92k used, 9502316k free, 1520104k cached > > > > Is that after mserver5 had crashed? > > > > > > > -- > | Dr. Stefan Manegold | mailto:[email protected] | > | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | > | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | > | The Netherlands | Fax : +31 (20) 592-4312 | > > > > -- | Dr. Stefan Manegold | mailto:[email protected] | | CWI, P.O.Box 94079 | http://www.cwi.nl/~manegold/ | | 1090 GB Amsterdam | Tel.: +31 (20) 592-4212 | | The Netherlands | Fax : +31 (20) 592-4312 | ------------------------------------------------------------------------------ This SF.net email is sponsored by: SourcForge Community SourceForge wants to tell your story. http://p.sf.net/sfu/sf-spreadtheword _______________________________________________ Monetdb-developers mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/monetdb-developers
