Bugs item #2970780, was opened at 2010-03-15 19:35 Message generated for change (Comment added) made by sjoerd You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2970780&group_id=56967
Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: SQL/Core Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Hering Cheng (heringcheng) Assigned to: Sjoerd Mullender (sjoerd) Summary: Memory Usage Initial Comment: Can MonetDB handle terabytes of data? I dimly understand that MonetDB maps data in each column into memory, which is partly responsible for its performance. However, what if the table is so large that all the data in a column do not fit in RAM + swap on the server? I am currently loading such large data set into MonetDB and I can see that the mserver5 process is nearing the physical limit on my server. Here is from "top" command on my SPARC/Solaris: PID USERNAME LWP PRI NICE SIZE RES STATE TIME CPU COMMAND 26004 chenher 39 60 0 219G 28G sleep 648:22 0.32% mserver5 Here, total size is 219 GB and resident size is 28 GB. Will it run out of memory? Is there a way to limit how much memory mserver5 consumes? ---------------------------------------------------------------------- >Comment By: Sjoerd Mullender (sjoerd) Date: 2010-03-17 14:38 Message: MonetDB can definitely handle terabytes of data. However, since all data that a query uses is memory mapped, you need a coorespondingly large address space. In other words, a 64 bit architecture and a 64 bit build of MonetDB are essential. MonetDB memory maps large files. This means that under certain conditions MonetDB does not use your swap space. However, as soon as you start querying, intermediate results are created, and they do occupy real memory and real swap space. Because MonetDB does everything in bulk, those intermediate results may be large as well. In other words, to work effectively with such large databases, you need enough memory and swap space (the memory is needed to avoid thrashing). ---------------------------------------------------------------------- Comment By: Fabian (mr-meltdown) Date: 2010-03-16 17:56 Message: > Worse yet, I now cannot even restart it. How do I delete a database? monetdb destroy <database> ---------------------------------------------------------------------- Comment By: Hering Cheng (heringcheng) Date: 2010-03-16 15:14 Message: So mserver5 eventually core dumped. Unfortunately, I did not build my SPARC binary with debugging info: t...@17 (l...@17) terminated by signal SEGV (no mapping at the fault address) 0xffffffff7ec5401c: BAThashjoin+0x19c4: stx %o0, [%l0] (dbx) where current thread: t...@17 =>[1] BAThashjoin(0xffffffff698f9dc0, 0x123099bc8, 0xffffffff7ef0a570, 0x123099a10, 0x112bff9c8, 0xffffffff7ede1f78), at 0xffffffff7ec5401c [2] BATjoin(0x112bff9c8, 0x123099a10, 0x1000000, 0x0, 0x2636bc, 0xffffffff76014c68), at 0xffffffff7ec7e8fc [3] ALGjoinestimate(0x1194f48e0, 0xffffffff7582caf8, 0x1194f48d0, 0x8000000000000000, 0x10e924, 0x112bff9c8), at 0xffffffff7581e97c [4] ALGjoin(0x8000000000000000, 0xffffffff7ede1f70, 0x40, 0x0, 0x10e7ac, 0xffffffff7592d1f8), at 0xffffffff7581ea80 [5] DFLOWstep(0x1194f48e0, 0xffffffff698fa0d0, 0xffffffff698fa130, 0xffffffff7f372ac8, 0x115, 0x1001069c8), at 0xffffffff7f230344 [6] runDFLOWworker(0x1012ceb98, 0xffffffff32af7198, 0x10123fae8, 0x0, 0x1, 0xffffffff7f26d6a8), at 0xffffffff7f2338a4 Worse yet, I now cannot even restart it. How do I delete a database? 2010-03-16 07:02:33 ERR control[1316]: rdcuxsrv220:58360: failed to fork mserver: database 'dent' has inconsistent state (running but dead), review merovingian''s logfile for any peculiarities 2010-03-16 07:02:40 MSG merovingian[1316]: database 'dent' (16384) has crashed (dumped core) t...@1 (l...@1) terminated by signal SEGV (no mapping at the fault address) 0xffffffff6c2c2220: load_key+0x02a0: ld [%o0], %o1 (dbx) where current thread: t...@1 =>[1] load_key(0x10024fc28, 0x100d58158, 0x100258328, 0x100c30aa8, 0x100d5e208, 0xffffffff6c4096d0), at 0xffffffff6c2c2220 [2] load_table(0x10024fc28, 0x100d0af78, 0x100d581c8, 0xffffffff7ede1f78, 0x100d58164, 0x1002582c8), at 0xffffffff6c2c32dc [3] load_schema(0x10024fc28, 0x100d0af84, 0x100d0af98, 0xffffffff6c455610, 0x100d0af78, 0x100250f68), at 0xffffffff6c2c4100 [4] load_trans(0x10024fc28, 0x4ba, 0x10024fc50, 0x100274598, 0xffffffff6c2f78d8, 0x8000000000000000), at 0xffffffff6c2c443c [5] store_init(0x0, 0x0, 0xffffffff6c4121c0, 0x4ba, 0x400, 0x100105560), at 0xffffffff6c2c6708 [6] mvc_init(0x0, 0x0, 0x0, 0x10016a360, 0x17d4ec, 0xffffffff6c2f4668), at 0xffffffff6c28c26c [7] SQLinit(0x0, 0xffffffff6c455658, 0xffffffff6c4120a8, 0xffffffff6c4096d0, 0xffffffff7ef0a4d8, 0x0), at 0xffffffff6c252254 [8] SQLprelude(0x0, 0xffffffff6c2ec4a0, 0x290, 0xffffffff6c4120b0, 0xffffffff7f376b80, 0xffffffff6c2ec4b0), at 0xffffffff6c251f24 [9] runMALsequence(0x1001062c0, 0x100190dc8, 0x6974, 0xffffffff7f26d5b0, 0xffffffff7fffd4a0, 0x10024f758), at 0xffffffff7f223ae0 [10] runMAL(0x1001062c0, 0x2, 0x10, 0xffffffff7fffd518, 0x0, 0x1), at 0xffffffff7f21c4b8 [11] MALengine(0x0, 0x1001062c0, 0x100190dc8, 0x0, 0xffffffff7f372ac8, 0x100117178), at 0xffffffff7f21b74c [12] callString(0x1001062c0, 0xffffffff7f26cf88, 0xffffffff7f21b4b4, 0x0, 0x1008b14e8, 0xffffffff7f372ac8), at 0xffffffff7f219fa0 [13] main(0x100002000, 0x0, 0x10010fb7e, 0x100105660, 0x10010f000, 0x100105e70), at 0x100003090 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2970780&group_id=56967 ------------------------------------------------------------------------------ Download Intel® Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev _______________________________________________ Monetdb-bugs mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/monetdb-bugs
