Bugs item #2687705, was opened at 2009-03-15 15:28
Message generated for change (Settings changed) made by boncz
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2687705&group_id=56967
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Core
Group: MonetDB Common CVS Head
>Status: Closed
>Resolution: Fixed
Priority: 5
Private: No
Submitted By: Jan Rittinger (tsheyar)
Assigned to: Peter Boncz (boncz)
Summary: Performance degradation due to memory management
Initial Comment:
I open this bug as a reminder for memory problems in the current STABLE and
HEAD versions.
(All observations where made on Mac OS X 10.5.6)
Running the example queries on dblp.xml (441 MB):
Q0: count(/dblp//author[text()="Michael Stonebraker"])
Q1: count(/dblp/*/author[text()="Michael Stonebraker"])
leads to various observations:
1.) Release 0.24 is faster than Feb2009
2.) Release Feb2009 is faster than Nov2008
3.) Release Nov2008 is faster than HEAD (03/10/2009)
4.) Document loading is getting slower if query is run multiple times
5.) Queries are getting slower if run multiple times
For more details please look into the MonetDB developers mailing list topic:
[MonetDB-users] too deep recursion.
A related observation is that running XMark Q11 and Q12 (sf=1) needs twice the
time due to the checkin to MonetDB/src/gdk gdk_heap.mx,,1.107,1.108 on
February, 5th. This problem is even more dramatic in the current HEAD version
on XIRAF queries (see pathfinder wiki).
----------------------------------------------------------------------
Comment By: Peter Boncz (boncz)
Date: 2009-04-17 15:57
Message:
The performance problems on Jan's machine and especially those related to
multiple queries were related to swapping (BBPtrim) triggering on the near
full use of the VM address space. On 32-bits systems, MonetDB tries to
limit VM usage below 2GB, though Jan in his machine has 4GB RAM.
In february there have been a number of tweaks regarding TPCH 100GB
experiments, some iterations of that process indeed brought performance
degradation. However, I consider those problems as solved.
btw, I ran xmark 100MB on the same platform as the sigmod paper
sigmod now
q01 0.12 0.10
q02 0.19 0.12
q03 1.20 0.37
q04 0.42 0.29
q05 0.08 0.08
q06 0.00 0.02
q07 0.01 0.02
q08 0.47 0.52
q09 0.52 0.62
q10 5.18 2.99
q11 3.62 7.16
q12 2.11 4.37
q13 0.10 0.07
q14 0.93 1.10
q15 0.07 0.10
q16 0.08 0.13
q17 0.15 0.19
q18 0.05 0.09
q19 0.38 0.44
q20 0.62 0.23
This is mostly a mps/alg comparison, not so much memory consumption. One
can say performance is at least on par, generally better. Only q11/q12 are
slower. I remember we did some special things in mps to avoid materializing
columns that were not needed in the aggregation, but you would expect that
algebra DCE wouls automatically do that as well.
anyway, I am closing this one for now
----------------------------------------------------------------------
Comment By: Stefan Manegold (stmane)
Date: 2009-03-15 16:06
Message:
a) from 1.) - 3.) I derive
0.24 < Feb2009 < Nov2008 < HEAD
b) the various releases diff by more than only memory management
c) for analysis of their respective performance impact,
here are the details of Peter's recent memory managment changes
(changes that involve more than 2 files have been wrapped in CVS tags
as indicated below):
===================================================================
2009/02/10 - sjoerd: MonetDB/src/gdk/gdk_heap.mx,1.110
propagated changes of Tuesday Feb 10 2009
from the Feb2009 branch to the development trunk
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2009/02/10 - stmane: src/gdk/gdk_heap.mx,1.107.2.1
[on behalf of Peter] re-doing
2009/02/05 - boncz: MonetDB/src/gdk/gdk_heap.mx,1.108
- make large allocs fall back again on mmap
now only the actually intended changes
and as bug fix, i.e., in the Feb2009 release branch
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
===================================================================
===================================================================
2009/03/02 - boncz: MonetDB/src/common/monet_options.py.in,1.40
MonetDB/src/gdk/gdk.mx,1.275 MonetDB/src/gdk/gdk_bat.mx,1.209
MonetDB/src/gdk/gdk_heap.mx,1.111
MonetDB/src/gdk/gdk_utils.mx,1.228
MonetDB4/conf/MonetDB.conf.in,1.17
MonetDB4/src/gdk/Tests/directsemijoin.milS,1.3
MonetDB4/src/gdk/Tests/directsemijoin.stable.out,1.3
MonetDB4/src/gdk/Tests/malloc_mmap_priv.milS,1.2
MonetDB4/src/gdk/Tests/malloc_mmap_priv.stable.out,1.2
MonetDB4/src/modules/plain/sys.mx,1.18
MonetDB4/src/tools/Tests/client.stable.out,1.2
MonetDB4/src/tools/Tests/client.stable.out.64bit,1.2
MonetDB4/src/tools/Tests/multiClientSeq.stable.out,1.2
MonetDB4/src/tools/Tests/multiClientSeq.stable.out.64bit,1.2
MonetDB4/src/tools/Tests/server.stable.out,1.2
MonetDB4/tests/Availability/Tests/01_Modules_static.stable.out,1.18
MonetDB4/tests/BugTracker/Tests/mmap_bats.SF-1428389.prelude,1.2
MonetDB4/tests/BugTracker/Tests/project_on_BAT_with_batFirst.SF-2020806.milS,1.3
MonetDB4/tests/BugTracker/Tests/project_on_BAT_with_batFirst.SF-2020806.stable.out,1.5
MonetDB4/tests/by_Peter/Tests/tst_append_readonly.milS,1.3
MonetDB4/tests/by_Peter/Tests/tst_append_readonly.stable.out,1.5
MonetDB4/tests/by_Peter/Tests/tst_append_readonly_commit.milS,1.3
MonetDB4/tests/by_Peter/Tests/tst_append_readonly_commit.stable.out,1.6
MonetDB4/tests/by_Peter/Tests/tst_mmap.milS,1.4
MonetDB4/tests/by_Peter/Tests/tst_mmap.stable.out,1.4
MonetDB4/tests/by_Peter/Tests/tst_updatestr.milS,1.4
MonetDB4/tests/by_Peter/Tests/tst_updatestr.stable.out,1.5
MonetDB4/tests/by_Peter/Tests/tst_writestr.milS,1.4
MonetDB4/tests/by_Peter/Tests/tst_writestr.stable.out,1.4
MonetDB5/conf/monetdb5.conf.in,1.41
MonetDB5/src/mal/Tests/performanceTests/perf.conf,1.16
MonetDB5/src/modules/kernel/status.mx,1.58
OK, back from holiday I decided to fix this mmap issue.
There are two distinct vm thresholds of importance:
(1) the miminum heap size after which a heap may become memory mapped
(2) the allocation size after which mmap is obligatory
I have decided for (2) to simply use GDK_mem_maxsize (ie allocations
larger than the RAM). Creating files for heap allocation is to
be avoided unless in extreme situations. I also checked that
HEAPalloc/HEAPextend fall back onto file-based mmap if allocation
fails.
Regarding (1) there was already inconsistency, because apart from
GDK_vm_minsize, some other code pieces were already using
REMAP_PAGE_MAXSIZE. I have decided to convert all such checks to the
latter.
This means GDK_vm_minsize is removed as a variable in GDK, and also
as a configurable environment variable. Less knobs is better.
===================================================================
is wrapped in CVS tags "with_GDK_vm_minsize" & "without_GDK_vm_minsize"
===================================================================
2009/03/05 - boncz: MonetDB/src/common/monet_options.mx,1.31
MonetDB/src/common/monet_options.py.in,1.41
MonetDB/src/gdk/gdk.mx,1.276 MonetDB/src/gdk/gdk_heap.mx,1.112
MonetDB/src/gdk/gdk_utils.mx,1.229
re-introducing a knob to set the threshold above which to use mmap-
on-tempfile now named: gdk_mmap_minsize
on large jobs, setting this one to e.g. 200MB should be of interest
===================================================================
is wrapped in CVS tags "without_GDK_mmap_minsize" &
"with_GDK_mmap_minsize"
===================================================================
2009/03/06 - boncz: MonetDB/src/gdk/gdk_utils.mx,1.230
- adaptively swap GDK_mmap_minsize back to small (128MB) if vmtrim detects
memory crunch.
===================================================================
===================================================================
2009/03/07 - boncz: MonetDB/src/gdk/gdk_posix.mx,1.164
MonetDB/src/gdk/gdk_utils.mx,1.231
- start vmtrim actions when rss > .5mem_maxsize (was: .75mem_maxsize)
- make the mmap_minsize adaptive between 256MB and 16MB depending on
the severity of the memory shortage
===================================================================
===================================================================
2009/03/08 - boncz: MonetDB/src/gdk/gdk_posix.mx,1.165
MonetDB/src/gdk/gdk_utils.mx,1.232
- made the mmap-on-tempfile a bit more adaptive
===================================================================
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=482468&aid=2687705&group_id=56967
------------------------------------------------------------------------------
Stay on top of everything new and different, both inside and
around Java (TM) technology - register by April 22, and save
$200 on the JavaOne (SM) conference, June 2-5, 2009, San Francisco.
300 plus technical and hands-on sessions. Register today.
Use priority code J9JMT32. http://p.sf.net/sfu/p
_______________________________________________
Monetdb-bugs mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/monetdb-bugs
