Bugs item #2687705, was opened at 2009-03-15 15:28
Message generated for change (Comment added) made by stmane
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2687705&group_id=56967
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Core
Group: MonetDB Common CVS Head
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Jan Rittinger (tsheyar)
Assigned to: Peter Boncz (boncz)
Summary: Performance degradation due to memory management
Initial Comment:
I open this bug as a reminder for memory problems in the current STABLE and
HEAD versions.
(All observations where made on Mac OS X 10.5.6)
Running the example queries on dblp.xml (441 MB):
Q0: count(/dblp//author[text()="Michael Stonebraker"])
Q1: count(/dblp/*/author[text()="Michael Stonebraker"])
leads to various observations:
1.) Release 0.24 is faster than Feb2009
2.) Release Feb2009 is faster than Nov2008
3.) Release Nov2008 is faster than HEAD (03/10/2009)
4.) Document loading is getting slower if query is run multiple times
5.) Queries are getting slower if run multiple times
For more details please look into the MonetDB developers mailing list topic:
[MonetDB-users] too deep recursion.
A related observation is that running XMark Q11 and Q12 (sf=1) needs twice the
time due to the checkin to MonetDB/src/gdk gdk_heap.mx,,1.107,1.108 on
February, 5th. This problem is even more dramatic in the current HEAD version
on XIRAF queries (see pathfinder wiki).
----------------------------------------------------------------------
>Comment By: Stefan Manegold (stmane)
Date: 2009-03-15 16:06
Message:
a) from 1.) - 3.) I derive
0.24 < Feb2009 < Nov2008 < HEAD
b) the various releases diff by more than only memory management
c) for analysis of their respective performance impact,
here are the details of Peter's recent memory managment changes
(changes that involve more than 2 files have been wrapped in CVS tags
as indicated below):
===================================================================
2009/02/10 - sjoerd: MonetDB/src/gdk/gdk_heap.mx,1.110
propagated changes of Tuesday Feb 10 2009
from the Feb2009 branch to the development trunk
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2009/02/10 - stmane: src/gdk/gdk_heap.mx,1.107.2.1
[on behalf of Peter] re-doing
2009/02/05 - boncz: MonetDB/src/gdk/gdk_heap.mx,1.108
- make large allocs fall back again on mmap
now only the actually intended changes
and as bug fix, i.e., in the Feb2009 release branch
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
===================================================================
===================================================================
2009/03/02 - boncz: MonetDB/src/common/monet_options.py.in,1.40
MonetDB/src/gdk/gdk.mx,1.275 MonetDB/src/gdk/gdk_bat.mx,1.209
MonetDB/src/gdk/gdk_heap.mx,1.111
MonetDB/src/gdk/gdk_utils.mx,1.228
MonetDB4/conf/MonetDB.conf.in,1.17
MonetDB4/src/gdk/Tests/directsemijoin.milS,1.3
MonetDB4/src/gdk/Tests/directsemijoin.stable.out,1.3
MonetDB4/src/gdk/Tests/malloc_mmap_priv.milS,1.2
MonetDB4/src/gdk/Tests/malloc_mmap_priv.stable.out,1.2
MonetDB4/src/modules/plain/sys.mx,1.18
MonetDB4/src/tools/Tests/client.stable.out,1.2
MonetDB4/src/tools/Tests/client.stable.out.64bit,1.2
MonetDB4/src/tools/Tests/multiClientSeq.stable.out,1.2
MonetDB4/src/tools/Tests/multiClientSeq.stable.out.64bit,1.2
MonetDB4/src/tools/Tests/server.stable.out,1.2
MonetDB4/tests/Availability/Tests/01_Modules_static.stable.out,1.18
MonetDB4/tests/BugTracker/Tests/mmap_bats.SF-1428389.prelude,1.2
MonetDB4/tests/BugTracker/Tests/project_on_BAT_with_batFirst.SF-2020806.milS,1.3
MonetDB4/tests/BugTracker/Tests/project_on_BAT_with_batFirst.SF-2020806.stable.out,1.5
MonetDB4/tests/by_Peter/Tests/tst_append_readonly.milS,1.3
MonetDB4/tests/by_Peter/Tests/tst_append_readonly.stable.out,1.5
MonetDB4/tests/by_Peter/Tests/tst_append_readonly_commit.milS,1.3
MonetDB4/tests/by_Peter/Tests/tst_append_readonly_commit.stable.out,1.6
MonetDB4/tests/by_Peter/Tests/tst_mmap.milS,1.4
MonetDB4/tests/by_Peter/Tests/tst_mmap.stable.out,1.4
MonetDB4/tests/by_Peter/Tests/tst_updatestr.milS,1.4
MonetDB4/tests/by_Peter/Tests/tst_updatestr.stable.out,1.5
MonetDB4/tests/by_Peter/Tests/tst_writestr.milS,1.4
MonetDB4/tests/by_Peter/Tests/tst_writestr.stable.out,1.4
MonetDB5/conf/monetdb5.conf.in,1.41
MonetDB5/src/mal/Tests/performanceTests/perf.conf,1.16
MonetDB5/src/modules/kernel/status.mx,1.58
OK, back from holiday I decided to fix this mmap issue.
There are two distinct vm thresholds of importance:
(1) the miminum heap size after which a heap may become memory mapped
(2) the allocation size after which mmap is obligatory
I have decided for (2) to simply use GDK_mem_maxsize (ie allocations
larger than the RAM). Creating files for heap allocation is to
be avoided unless in extreme situations. I also checked that
HEAPalloc/HEAPextend fall back onto file-based mmap if allocation
fails.
Regarding (1) there was already inconsistency, because apart from
GDK_vm_minsize, some other code pieces were already using
REMAP_PAGE_MAXSIZE. I have decided to convert all such checks to the
latter.
This means GDK_vm_minsize is removed as a variable in GDK, and also
as a configurable environment variable. Less knobs is better.
===================================================================
is wrapped in CVS tags "with_GDK_vm_minsize" & "without_GDK_vm_minsize"
===================================================================
2009/03/05 - boncz: MonetDB/src/common/monet_options.mx,1.31
MonetDB/src/common/monet_options.py.in,1.41
MonetDB/src/gdk/gdk.mx,1.276 MonetDB/src/gdk/gdk_heap.mx,1.112
MonetDB/src/gdk/gdk_utils.mx,1.229
re-introducing a knob to set the threshold above which to use mmap-
on-tempfile now named: gdk_mmap_minsize
on large jobs, setting this one to e.g. 200MB should be of interest
===================================================================
is wrapped in CVS tags "without_GDK_mmap_minsize" &
"with_GDK_mmap_minsize"
===================================================================
2009/03/06 - boncz: MonetDB/src/gdk/gdk_utils.mx,1.230
- adaptively swap GDK_mmap_minsize back to small (128MB) if vmtrim detects
memory crunch.
===================================================================
===================================================================
2009/03/07 - boncz: MonetDB/src/gdk/gdk_posix.mx,1.164
MonetDB/src/gdk/gdk_utils.mx,1.231
- start vmtrim actions when rss > .5mem_maxsize (was: .75mem_maxsize)
- make the mmap_minsize adaptive between 256MB and 16MB depending on
the severity of the memory shortage
===================================================================
===================================================================
2009/03/08 - boncz: MonetDB/src/gdk/gdk_posix.mx,1.165
MonetDB/src/gdk/gdk_utils.mx,1.232
- made the mmap-on-tempfile a bit more adaptive
===================================================================
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2687705&group_id=56967
------------------------------------------------------------------------------
Apps built with the Adobe(R) Flex(R) framework and Flex Builder(TM) are
powering Web 2.0 with engaging, cross-platform capabilities. Quickly and
easily build your RIAs with Flex Builder, the Eclipse(TM)based development
software that enables intelligent coding and step-through debugging.
Download the free 60 day trial. http://p.sf.net/sfu/www-adobe-com
_______________________________________________
Monetdb-bugs mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/monetdb-bugs
