Agustin Schapira wrote: > Dear Monet developers, > > Once again, here I am, writing for help ;-) > > We are working with a 'large' table, with 250 million rows, 11 > columns. This takes up a lot of memory space, as you can imagine: a > single column of 250 million 'ints' requires almost a gigabyte, and I would say 1GB is a minimum. What if you are on a 64bit machine? And also, if this table requires a hash index, then add another 1GB/table.
> there's several of those; in fact, I think that the entire table > takes up a bit over 6GB in memory (some columns are strings, and can > be compressed). The problem is that we have a machine with 8GB, and > every time we try to do something with that table, Monet (4.16.2) > crashes. For example, I've been trying to delete rows from the table > where two columns have the same value: > > var var_0 := [=](col1,col2).select(true).mirror(); Hard to deduce what's happening. > col1.delete(var_0); > commit(); > col2.delete(var_0); > commit(); > col3.delete(var_0); > commit(); I would first try to make sure if the commit is not sitting in the way. Do you need to commit here ? > ... > > After deleting from a few of those columns, Monet dies: > > !ERROR: BATSIGcrash: Mserver internal error (Segmentation fault), > please restart. > !ERROR: (One potential cause could be that your disk might be full...) > > > Do you know if there a fix for this? In principle, Monet should be > able to unload the BATs that it has already deleted from, to release > memory. Also, would doing a semijoin() instead of a delete() help? (I > though that a semijoin would need to create a new BAT, thereby using > even more memory!). And, in general: is Monet 5 capable of accessing > BATs that don't entirely fit in memory? Would the new version fix > this problem? Maybe it's time for us to adapt it... > > Thanks again for all your help, and regards from Amherst, > > -- Agustin > > ------------------------------------------------------------------------- > This SF.net email is sponsored by DB2 Express > Download DB2 Express C - the FREE version of DB2 express and take > control of your XML. No limits. Just data. Click to get it now. > http://sourceforge.net/powerbar/db2/ > _______________________________________________ > Monetdb-developers mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/monetdb-developers ------------------------------------------------------------------------- This SF.net email is sponsored by DB2 Express Download DB2 Express C - the FREE version of DB2 express and take control of your XML. No limits. Just data. Click to get it now. http://sourceforge.net/powerbar/db2/ _______________________________________________ Monetdb-developers mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/monetdb-developers
