Bugs item #1892560, was opened at 2008-02-13 10:57 Message generated for change (Comment added) made by stmane You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=482468&aid=1892560&group_id=56967
Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: PF/compiler Group: Pathfinder 0.22 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Stefan Manegold (stmane) Assigned to: Stefan Manegold (stmane) Summary: PF: "insufficient memory" while compiling XMark module Initial Comment: When compiling larger queries, such as e.g, the XMark module, pf crashes with fatal error: insufficient memory (allocating 2048 bytes failed) in .\..\..\..\compiler\mem\array.c (PFarray), line 96 on Windows and Solaris/x86 (on machines that have 1 GB RAM ...) See e.g., http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/Int.64.32.d-Windows5.2/benchmarks_XMark_mod_XQ/q01.err.00.html http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.32.32.d-SunOS5.10/benchmarks_XMark_mod_XQ/q01.err.00.html http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.32.32.d-SunOS5.10/benchmarks_XMark_opt_XQ/q01.err.00.html http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/Int.64.32.d-Windows5.2/benchmarks_XMark_opt_XQ/q01.err.00.html http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/Int.64.32.d-Windows5.2/benchmarks_XMark_mod_MIL/xq2mil.err.00.html http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.32.32.d-SunOS5.10/benchmarks_XMark_mod_MIL/xq2mil.err.00.html http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/Int.64.32.d-Windows5.2/benchmarks_XMark_opt_MIL/xq2mil.err.00.html http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.32.32.d-SunOS5.10/benchmarks_XMark_opt_MIL/xq2mil.err.00.html http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/Int.64.32.d-Windows5.2/benchmarks_XMark_XRpc/q01.err.00.html http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.32.32.d-SunOS5.10/benchmarks_XMark_XRpc/q01.err.00.html http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.32.32.d-SunOS5.10/benchmarks_XMach-1/q1.err.00.html http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/Int.64.32.d-Windows5.2/benchmarks_XMach-1/q1.err.00.html ---------------------------------------------------------------------- >Comment By: Stefan Manegold (stmane) Date: 2008-02-13 22:00 Message: Logged In: YES user_id=572415 Originator: YES In the XQuery_0-22 branch, only (propagation prevented): as suggested by Jan: reduce the default (inital) size of (dynamic) arrays from 2048 byte to 256 byte [post release-tag check-in (approved by Stefan ;-))] ---------------------------------------------------------------------- Comment By: Stefan Manegold (stmane) Date: 2008-02-13 13:54 Message: Logged In: YES user_id=572415 Originator: YES Indeed, lowering the default allocation size is just a HACK to cure symptoms, not to solve the problem. We might consider doing it, anyway, as a post-release fix just to keep/make testing work reasonable (again). It will not be in the initial release (which should be base on last night code base) --- unless we want to postpone the release even more ... ---------------------------------------------------------------------- Comment By: Jan Rittinger (tsheyar) Date: 2008-02-13 13:04 Message: Logged In: YES user_id=993208 Originator: NO I can only agree with Stefan -- In the testing I get a time out on my machine and a very high memory usage. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 9635 rittinge 15 0 1005m 662m 90m S 2.3 65.4 0:03.62 Mserver If I use stand-alone Pathfinder everything works fine and consumes only 50MB of memory (thanks to garbage collection). ---------- After making the point about the garbage collection here comes the proposal to fix this. We use dynamic arrays for almost all generic structures. And because we do not want to resize (which maps to a new malloc+copy in the embedded variant) very often we set the initial size to 2KB. Of course a lot of these generic structures are used which leads to the reported memory problem. I think for the release the initial size can be decreased as in many cases we only store very few data in an array. This change however only postpones the problem (of allocating memory only without freeing it) and is not necessary for the stand-alone variant. In the future the embedded variant thus needs a better and cleaner solution. Reassigning it to the release people. Index: mem/array.c =================================================================== RCS file: /cvsroot/monetdb/pathfinder/compiler/mem/array.c,v retrieving revision 1.24 diff -r1.24 array.c 76c76 < #define ACHUNK 2048U --- > #define ACHUNK 256U ---------------------------------------------------------------------- Comment By: Stefan Manegold (stmane) Date: 2008-02-13 11:53 Message: Logged In: YES user_id=572415 Originator: YES Just for info: On Solaris/x86, compiling the XMark module worked fine until October 20 2007. The "normal" (non-"optimized") version has been failing since October 21 2007: http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.32.32.d-SunOS5.10/benchmarks_XMark_mod_MIL/xq2mil.err.00.html The "optimized" version has been failing since November 16 2007: http://monetdb.cwi.nl/testing/projects/monetdb/Stable/pathfinder/.mTests103/GNU.32.32.d-SunOS5.10/benchmarks_XMark_opt_MIL/xq2mil.err.00.html On Windows, both versions have been failing since nightly Windows testing has been started mid December 2007. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=482468&aid=1892560&group_id=56967 ------------------------------------------------------------------------- This SF.net email is sponsored by: Microsoft Defy all challenges. Microsoft(R) Visual Studio 2008. http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/ _______________________________________________ Monetdb-bugs mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/monetdb-bugs
