Changeset: 94ae82635677 for MonetDB
URL: http://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=94ae82635677
Modified Files:
Branch: default
Log Message:
merged
diffs (truncated from 481 to 300 lines):
diff -r 27d9c8547be0 -r 94ae82635677 MonetDB5/src/modules/kernel/Tests/All
--- a/MonetDB5/src/modules/kernel/Tests/All Sat Dec 18 12:57:47 2010 +0100
+++ b/MonetDB5/src/modules/kernel/Tests/All Sat Dec 18 12:58:06 2010 +0100
@@ -1,2 +1,3 @@
time01
TriBool
+vacuum
diff -r 27d9c8547be0 -r 94ae82635677
MonetDB5/src/modules/kernel/Tests/vacuum.mal
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/MonetDB5/src/modules/kernel/Tests/vacuum.mal Sat Dec 18 12:58:06
2010 +0100
@@ -0,0 +1,39 @@
+#building blocks for SQL vacuum actions
+
+b:= bat.new(:oid,:int);
+c:= bat.new(:oid,:oid);
+
+barrier (go,i):= language.newRange(0:lng);
+ k:= mmath.rand();
+ barrier t:= k < 1000000;
+ o:= calc.oid(i);
+ bat.append(c,o);
+ exit t;
+ bat.append(b,k);
+ redo (go,i):= language.nextElement(1:lng,1000000:lng);
+exit (go,i);
+
+d:= algebra.copy(b);
+
+cnt:= aggr.count(b);
+io.print(cnt);
+cnt:= aggr.count(c);
+io.print(cnt);
+#s:= algebra.sample(c,20);
+#io.print(s);
+
+t0:= alarm.usec();
+x:= bat.shrink(b,c);
+t1:= alarm.usec();
+d0:= t1-t0;
+io.printf("#bat.shrink() %d\n",d0);
+cnt:= aggr.count(x);
+io.print(cnt);
+
+t0:= alarm.usec();
+y:= bat.reuse(d,c);
+t1:= alarm.usec();
+d0:= t1-t0;
+io.printf("#bat.reuse() %d\n",d0);
+cnt:= aggr.count(y);
+io.print(cnt);
diff -r 27d9c8547be0 -r 94ae82635677
MonetDB5/src/modules/kernel/Tests/vacuum.stable.err
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/MonetDB5/src/modules/kernel/Tests/vacuum.stable.err Sat Dec 18
12:58:06 2010 +0100
@@ -0,0 +1,75 @@
+stderr of test 'vacuum` in directory 'src/modules/kernel` itself:
+
+
+# 12:11:50 >
+# 12:11:50 > mserver5 "--config=/ufs/mk/current//Linux/etc/monetdb5.conf"
--debug=10 --set gdk_nr_threads=0 --set
"monet_mod_path=/ufs/mk/current//Linux/lib/MonetDB5:/ufs/mk/current//Linux/lib/MonetDB5/lib:/ufs/mk/current//Linux/lib/MonetDB5/bin"
--set "gdk_dbfarm=/ufs/mk/current//Linux/var/MonetDB5/dbfarm" --set
mapi_open=true --set xrpc_open=true --set mapi_port=36446 --set xrpc_port=43827
--set monet_prompt= --trace --forcemito --set mal_listing=2
--dbname=mTests_src_modules_kernel vacuum.mal
+# 12:11:50 >
+
+# builtin opt gdk_arch = 64bitx86_64-unknown-linux-gnu
+# builtin opt gdk_version = 1.41.0
+# builtin opt prefix = /ufs/mk/current//Linux
+# builtin opt exec_prefix = ${prefix}
+# builtin opt gdk_dbname = demo
+# builtin opt gdk_dbfarm = ${prefix}/var/MonetDB/dbfarm
+# builtin opt gdk_debug = 0
+# builtin opt gdk_alloc_map = no
+# builtin opt gdk_vmtrim = yes
+# builtin opt monet_admin = adm
+# builtin opt monet_prompt = >
+# builtin opt monet_welcome = yes
+# builtin opt monet_mod_path = ${exec_prefix}/lib/MonetDB
+# builtin opt monet_daemon = no
+# builtin opt host = localhost
+# builtin opt mapi_port = 50000
+# builtin opt mapi_clients = 2
+# builtin opt mapi_open = false
+# builtin opt mapi_autosense = false
+# builtin opt default_pipe =
inline,remap,evaluate,costModel,coercions,emptySet,aliases,mitosis,mergetable,deadcode,commonTerms,joinPath,reorder,deadcode,reduce,dataflow,history,multiplex,garbageCollector
+# builtin opt minimal_pipe = inline,remap,deadcode,multiplex,garbageCollector
+# builtin opt sql_optimizer = default_pipe
+# builtin opt sql_debug = 0
+# builtin opt standoff_ns =
+# builtin opt standoff_start = start
+# builtin opt standoff_end = end
+# config opt prefix = /ufs/mk/current//Linux
+# config opt config = ${prefix}/etc/monetdb5.conf
+# config opt prefix = /ufs/mk/current//Linux
+# config opt exec_prefix = ${prefix}
+# config opt gdk_dbfarm = ${prefix}/var/MonetDB5/dbfarm
+# config opt monet_mod_path =
${exec_prefix}/lib/MonetDB5:${exec_prefix}/lib/MonetDB5/lib:${exec_prefix}/lib/MonetDB5/bin
+# config opt mero_pidfile = ${prefix}/var/run/MonetDB/merovingian.pid
+# config opt mero_controlport = 50001
+# config opt no_mitosis_pipe =
inline,remap,evaluate,costModel,coercions,emptySet,aliases,mergetable,deadcode,commonTerms,joinPath,reorder,deadcode,reduce,dataflow,history,multiplex,garbageCollector
+# config opt sequential_pipe =
inline,remap,evaluate,costModel,coercions,emptySet,aliases,mergetable,deadcode,commonTerms,joinPath,reorder,deadcode,reduce,history,multiplex,garbageCollector
+# config opt nov2009_pipe =
inline,remap,evaluate,costModel,coercions,emptySet,aliases,mergetable,deadcode,constants,commonTerms,joinPath,deadcode,reduce,dataflow,history,multiplex,garbageCollector
+# config opt replication_pipe =
inline,remap,evaluate,costModel,coercions,emptySet,aliases,mergetable,deadcode,constants,commonTerms,joinPath,deadcode,reduce,dataflow,history,replication,multiplex,garbageCollector
+# config opt accumulator_pipe =
inline,remap,evaluate,costModel,coercions,emptySet,aliases,mergetable,deadcode,constants,commonTerms,joinPath,deadcode,reduce,accumulators,dataflow,history,multiplex,garbageCollector
+# config opt recycler_pipe =
inline,remap,evaluate,costModel,coercions,emptySet,aliases,deadcode,constants,commonTerms,joinPath,deadcode,recycle,reduce,dataflow,history,multiplex,garbageCollector
+# config opt cracker_pipe =
inline,remap,evaluate,costModel,coercions,emptySet,aliases,selcrack,deadcode,constants,commonTerms,joinPath,deadcode,reduce,dataflow,history,multiplex,garbageCollector
+# config opt sidcrack_pipe =
inline,remap,evaluate,costModel,coercions,emptySet,aliases,sidcrack,deadcode,constants,commonTerms,joinPath,deadcode,reduce,dataflow,history,multiplex,garbageCollector
+# config opt datacell_pipe =
inline,remap,evaluate,costModel,coercions,emptySet,aliases,deadcode,constants,commonTerms,joinPath,datacell,deadcode,reduce,dataflow,history,multiplex,garbageCollector
+# config opt octopus_pipe =
inline,remap,evaluate,costModel,coercions,emptySet,aliases,mitosis,mergetable,deadcode,constants,commonTerms,joinPath,deadcode,octopus,reduce,dataflow,history,multiplex,garbageCollector
+# config opt tarantula_pipe =
inline,remap,evaluate,costModel,coercions,emptySet,aliases,mitosis,mergetable,deadcode,constants,commonTerms,joinPath,tarantula,deadcode,reduce,dataflow,history,multiplex,garbageCollector
+# config opt mapreduce_pipe =
inline,remap,evaluate,costModel,coercions,emptySet,aliases,mapreduce,mergetable,deadcode,commonTerms,joinPath,reorder,deadcode,reduce,dataflow,history,multiplex,garbageCollector
+# config opt datacyclotron_pipe =
inline,remap,evaluate,costModel,coercions,emptySet,aliases,datacyclotron,mergetable,deadcode,commonTerms,joinPath,reorder,deadcode,reduce,dataflow,history,replication,multiplex,garbageCollector
+# config opt derive_pipe =
inline,remap,evaluate,costModel,coercions,emptySet,aliases,mitosis,mergetable,deadcode,commonTerms,derivePath,joinPath,reorder,deadcode,reduce,dataflow,history,multiplex,garbageCollector
+# config opt dictionary_pipe =
inline,remap,dictionary,evaluate,costModel,coercions,emptySet,aliases,mergetable,deadcode,constants,commonTerms,joinPath,deadcode,reduce,dataflow,history,multiplex,garbageCollector
+# config opt compression_pipe =
inline,remap,evaluate,costModel,coercions,emptySet,aliases,mergetable,deadcode,constants,commonTerms,joinPath,deadcode,reduce,dataflow,compression,dataflow,history,multiplex,garbageCollector
+# cmdline opt config = /ufs/mk/current//Linux/etc/monetdb5.conf
+# cmdline opt gdk_nr_threads = 0
+# cmdline opt monet_mod_path =
/ufs/mk/current//Linux/lib/MonetDB5:/ufs/mk/current//Linux/lib/MonetDB5/lib:/ufs/mk/current//Linux/lib/MonetDB5/bin
+# cmdline opt gdk_dbfarm = /ufs/mk/current//Linux/var/MonetDB5/dbfarm
+# cmdline opt mapi_open = true
+# cmdline opt xrpc_open = true
+# cmdline opt mapi_port = 36446
+# cmdline opt xrpc_port = 43827
+# cmdline opt monet_prompt =
+# cmdline opt mal_listing = 2
+# cmdline opt gdk_dbname = mTests_src_modules_kernel
+#warning: please don't forget to set your vault key!
+#(see /ufs/mk/current//Linux/etc/monetdb5.conf)
+
+# 12:11:52 >
+# 12:11:52 > Done.
+# 12:11:52 >
+
diff -r 27d9c8547be0 -r 94ae82635677
MonetDB5/src/modules/kernel/Tests/vacuum.stable.out
--- /dev/null Thu Jan 01 00:00:00 1970 +0000
+++ b/MonetDB5/src/modules/kernel/Tests/vacuum.stable.out Sat Dec 18
12:58:06 2010 +0100
@@ -0,0 +1,64 @@
+stdout of test 'vacuum` in directory 'src/modules/kernel` itself:
+
+
+# 12:11:50 >
+# 12:11:50 > mserver5 "--config=/ufs/mk/current//Linux/etc/monetdb5.conf"
--debug=10 --set gdk_nr_threads=0 --set
"monet_mod_path=/ufs/mk/current//Linux/lib/MonetDB5:/ufs/mk/current//Linux/lib/MonetDB5/lib:/ufs/mk/current//Linux/lib/MonetDB5/bin"
--set "gdk_dbfarm=/ufs/mk/current//Linux/var/MonetDB5/dbfarm" --set
mapi_open=true --set xrpc_open=true --set mapi_port=36446 --set xrpc_port=43827
--set monet_prompt= --trace --forcemito --set mal_listing=2
--dbname=mTests_src_modules_kernel vacuum.mal
+# 12:11:50 >
+
+# MonetDB server v5.23.0, based on kernel v1.41.0
+# Not released
+# Serving database 'mTests_src_modules_kernel', using 4 threads
+# Compiled for x86_64-unknown-linux-gnu/64bit with 64bit OIDs dynamically
linked
+# Found 7.750 GiB available main-memory.
+# Copyright (c) 1993-July 2008 CWI.
+# Copyright (c) August 2008-2010 MonetDB B.V., all rights reserved
+# Visit http://monetdb.cwi.nl/ for further information
+# Listening for connection requests on mapi:monetdb://eir.ins.cwi.nl:36446/
+# MonetDB/SQL module v2.41.0 loaded
+# MonetDB/GIS module v0.21.0 loaded
+function user.main():void;
+#building blocks for SQL vacuum actions
+ b := bat.new(:oid,:int);
+ c := bat.new(:oid,:oid);
+barrier (go,i) := language.newRange(0:lng);
+ k := mmath.rand();
+barrier t := calc.<(k,1000000);
+ o := calc.oid(i);
+ bat.append(c,o);
+exit t;
+ bat.append(b,k);
+ redo (go,i) := language.nextElement(1:lng,1000000:lng);
+exit (go,i);
+ d := algebra.copy(b);
+ cnt := aggr.count(b);
+ io.print(cnt);
+ cnt := aggr.count(c);
+ io.print(cnt);
+#s:= algebra.sample(c,20);
+#io.print(s);
+ t0 := alarm.usec();
+ x := bat.shrink(b,c);
+ t1 := alarm.usec();
+ d0 := calc.-(t1,t0);
+ io.printf("#bat.shrink() %d\n",d0);
+ cnt := aggr.count(x);
+ io.print(cnt);
+ t0 := alarm.usec();
+ y := bat.reuse(d,c);
+ t1 := alarm.usec();
+ d0 := calc.-(t1,t0);
+ io.printf("#bat.reuse() %d\n",d0);
+ cnt := aggr.count(y);
+ io.print(cnt);
+end main;
+[ 1000000 ]
+[ 471 ]
+#bat.shrink() 9138
+[ 999529 ]
+#bat.reuse() 7587
+[ 999529 ]
+
+# 12:11:52 >
+# 12:11:52 > Done.
+# 12:11:52 >
+
diff -r 27d9c8547be0 -r 94ae82635677 MonetDB5/src/modules/kernel/bat5.mx
--- a/MonetDB5/src/modules/kernel/bat5.mx Sat Dec 18 12:57:47 2010 +0100
+++ b/MonetDB5/src/modules/kernel/bat5.mx Sat Dec 18 12:58:06 2010 +0100
@@ -579,7 +579,7 @@
@mal
command shrink(b:bat[:oid,:any_1],del:bat[:oid,:oid]):bat[:oid,:any_1]
address BKCshrinkBAT
-comment "shrinkt the BAT based on a list of entries identified as
to-be-deleted";
+comment "Shrink the BAT based on a list of entries identified as
to-be-deleted";
command shrinkMap(b:bat[:oid,:any_1],del:bat[:oid,:oid]):bat[:oid,:oid]
address BKCshrinkBATmap
@@ -587,7 +587,7 @@
command reuse(b:bat[:oid,:any_1],del:bat[:oid,:oid]):bat[:oid,:any_1]
address BKCreuseBAT
-comment "shuffle the values around to restore a dense representation of buns.";
+comment "Shuffle the values around to restore a dense representation of buns.";
command reuseMap(b:bat[:oid,:any_1],del:bat[:oid,:oid]):bat[:oid,:oid]
address BKCreuseBATmap
diff -r 27d9c8547be0 -r 94ae82635677 MonetDB5/src/optimizer/opt_mapreduce.mx
--- a/MonetDB5/src/optimizer/opt_mapreduce.mx Sat Dec 18 12:57:47 2010 +0100
+++ b/MonetDB5/src/optimizer/opt_mapreduce.mx Sat Dec 18 12:58:06 2010 +0100
@@ -54,6 +54,8 @@
@:exportOptimizer(mapreduce)@
+#define OPTDEBUGmapreduce if ( optDebug & ((lng)1 <<DEBUG_OPT_MAPREDUCE) )
+
#endif
@c
#include "mal_config.h"
diff -r 27d9c8547be0 -r 94ae82635677 sql/src/backends/monet5/vaults/mseed.mx
--- a/sql/src/backends/monet5/vaults/mseed.mx Sat Dec 18 12:57:47 2010 +0100
+++ b/sql/src/backends/monet5/vaults/mseed.mx Sat Dec 18 12:58:06 2010 +0100
@@ -55,6 +55,8 @@
-- errors are returned for off-line analysis.
CREATE PROCEDURE mseedImport(vid int, source string, target string)
EXTERNAL NAME mseed.import;
+CREATE PROCEDURE mseedLoad(vid int, source string, target string)
+EXTERNAL NAME mseed.load;
-- The records are collected in SQL tables of the following structure
-- The are ordered on timestamp
@@ -87,29 +89,49 @@
module mseed;
pattern import{unsafe}(vid:int, source:str, target:str)
address MseedImport
-comment "Dump the record content of an mseed file from the vault into the
mseed catalog";
+comment "Fetch the record descriptor of a mseed file from the vault into the
mseed catalog";
+pattern load{unsafe}(vid:int, source:str, target:str)
+address MseedLoad
+comment "Load the content of a mseed file from the vault into a mseed table";
@{
@-
Performance experimentation on eir on 17 dec 2010.
The code base uses the SQLstatementIntern to populate both the catalog and the
mseed files.
-Experiment 0 concerns loading the mseedN file using a sequence of inserts.
This involves
+Experiment INSERTLOAD concerns loading the mseedN file using a sequence of
inserts. This involves
the complete code path from generating SQL statement, parsing, optimization
and execution.
No query caching is enabled. The first record from
ftp://www.orfeus-eu.org/pub/data/POND/1990/002/MN_BNI__BHN.1990.002.20.21.32.mseed
with 3768 events was loaded in 10.770 msec
+(Total would be around 260 sec, but there are leaks)
-The second experiment produced a copy file, separately loaded into MonetDB
using
-copy 400000 records into mseed2 from '/tmp/input/' delimiters ',','\n';
+Experiment COPYLOAD produced a copy file, separately loaded into MonetDB using
+copy 400000 records into mseed2 from '/tmp/data/' delimiters ',','\n';
this track leads to a load time of 326ms for the first record.
The first file with 26 records and 97592 tuples was loaded this way within
8710 msec.
This approach still uses the SQL logger. The expected speedup for direct
update would
be something like a factor 3.
-The third experiment produced the single binary BAT images followed by
+Experiment ATTACHLOAD produced the single binary BAT images followed by
copy into mseed2 from ('/tmp/time','/tmp/data');
-Attach procedure for complete file ran in about 232 ms.
+Attach procedure for complete file ran in about 232 ms, which excludes
+a forced saving of the state on disk.
+Experiment MANUALLOAD generates a file with insert statements, which is
executed manually
+against mclient. The total time for this first batch is 34 sec.
+This shows the gains from cached plans. (compared to experiment 0)
+The file grows from 0.1GB to 1GB
+-rw-r--r-- 1 mk ins 721464 2010-12-18 10:01 4156.tail
+-rw-r--r-- 1 mk ins 360732 2010-12-18 10:01 4272.tail
+
+
+The current ORFEUS repository is ca 3.2M files (ca 350GB). Excluding ftp
transfer cost,
+the following optimistic load times are foreseen:
+Experiment Loadtime
+INSERTLOAD 28 yrs
+COPYLOAD 322 days
+ATTACHLOAD 10 days (37days at 1sec)
+MANUALLOAD 3.2 yrs
@h
#ifndef _MSEED_H
@@ -134,6 +156,7 @@
#endif
_______________________________________________
Checkin-list mailing list
[email protected]
http://mail.monetdb.org/mailman/listinfo/checkin-list
