Bugs item #1856034, was opened at 2007-12-21 21:10 Message generated for change (Comment added) made by mlkersten You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=482468&aid=1856034&group_id=56967
Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core Group: MonetDB5 5.2 >Status: Closed >Resolution: Fixed Priority: 5 Private: No Submitted By: mharrison (matt_harrison) Assigned to: Niels Nes (nielsnes) Summary: Memory leaks when bulk loading Initial Comment: While attempting to load 1Mil rows into a table memory use goes 700+Megs. Loading the same tables in My/postgre/sql doesn't ever take more than 20Megs! After bulk load is done memory comes down a bit (100M or so) but still stays high. Am attaching valgrind log. ---------------------------------------------------------------------- >Comment By: Martin Kersten (mlkersten) Date: 2008-01-27 10:55 Message: Logged In: YES user_id=490798 Originator: NO On behalf of the Niels/Harrison, this bug can be closed. Many leakage problems have been resolved and no new complaint has been received. The testweb does not need additional tests. ---------------------------------------------------------------------- Comment By: Niels Nes (nielsnes) Date: 2007-12-30 20:42 Message: Logged In: YES user_id=43556 Originator: NO we now only need 400MB. Not sure yet why we need way more then the on disk size (which is about 180MB). ---------------------------------------------------------------------- Comment By: mharrison (matt_harrison) Date: 2007-12-22 17:47 Message: Logged In: YES user_id=1164236 Originator: YES Am running 32bit, but can run 64bit. How much mem was required to load 100G snapshot? Did you have to load it in steps? Am still confused as I thought bulk load would only put the data in the filesystem. Again my|postgres/sql use no more than 20M when doing the same bulk load.... Yes I care about performance and if you are going to give it to me during queries, excellent. But using up all the memory for bulk loading seems like an architectural flaw to me (a simple minded guy, who perhaps doesn't understand the benefits of it). ---------------------------------------------------------------------- Comment By: Martin Kersten (mlkersten) Date: 2007-12-22 14:56 Message: Logged In: YES user_id=490798 Originator: NO If you are running on a 32-bit you might indeed run into problems :( And may have to perform an incremental load, such that the individual columns fit your machine limits. We successfully loaded a 100 Gbyte snapshot of the SkyServer database and are awaiting the hardware for the 3TB load. ---------------------------------------------------------------------- Comment By: mharrison (matt_harrison) Date: 2007-12-22 14:42 Message: Logged In: YES user_id=1164236 Originator: YES Memory consumed by ram/top. Memory get's mmapped during bulk load? Then there's no way I'm every going to get 100+Mil rows loaded. ---------------------------------------------------------------------- Comment By: Martin Kersten (mlkersten) Date: 2007-12-22 09:21 Message: Logged In: YES user_id=490798 Originator: NO Are you referring to the on-disk size? or the memory consumption as seen by 'top' Note that the memory consumption of the ^sql are based on the memory buffer pool set aside for processing. Not the table sizes. In MonetDB you see the table size as a large memory mapped file. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=482468&aid=1856034&group_id=56967 ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ Monetdb-bugs mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/monetdb-bugs
