Update of /cvsroot/monetdb/pathfinder/modules/pftijah/Tests In directory sc8-pr-cvs16.sourceforge.net:/tmp/cvs-serv23721/Tests
Modified Files:
procs.stable.out sigs.stable.out
Log Message:
detailed log message of last commit
(the last check-in initially had a detailed log-message which got lost during
commit):
Changes to the PF/tijah index:
- introducing a new BAT to store the sizes of all element nodes. it does not
contain
any more the zero sizes of all terms, and thus remains by far smaller, causing
a big
performance gain.
- the old tijah-intern pre-size BAT is kept so far, though it is completely
redundant,
since it speeds up index updates.
- the index structure is used in all SRA operators, which caused changes to the
standard representation of a scored node set.
Changes to the representation of a scored node set:
- tijah nodes are not any longer identified by PRE values, instead newly
introduced NIDs
are used, which correspond to the OID given in the void-headed new size BAT.
- late evaluation: whenever nodes of certain tagname are selected, we do not
return a
scored node set containing a tuple for all thos nodes, but instead return a
special BAT
containing the start and end offset only, together with a score set to NIL. if
such a
construct is found in another SRA operator, it will get inflated when
neccessary.
Changes to the containment join evaluation:
- whenever a containment join is evaluated, we check in the query environment
params,
whether ancestor candidates are possibly nested.
- if the ancestor candidates are not nested, the containment join algorithm
will be simplified
and allows more skipping. these techniques showed as well a high performance
impact.
General Code Clean-Up:
- I have removed several unused functions.
- rewritten code of all score functions to show and make use of their
similiarity
- fixed bug in OKAPI scoring model
Remarks:
- I still get failures for two test cases that make use of our fragmentation
option,
which have to be solved soon. Jan does not get these failures
(!? maybe 64bit problem, only difference between the systems.)
Index: sigs.stable.out
===================================================================
RCS file: /cvsroot/monetdb/pathfinder/modules/pftijah/Tests/sigs.stable.out,v
retrieving revision 1.23
retrieving revision 1.24
diff -u -d -r1.23 -r1.24
--- sigs.stable.out 3 Aug 2007 07:32:22 -0000 1.23
+++ sigs.stable.out 14 Jan 2008 14:49:54 -0000 1.24
@@ -37,7 +37,6 @@
[ "mergeindex(BAT[oid,oid], BAT[void,oid], BAT[void,oid], int) :
BAT[void,BAT]"
]
[ "offsetindex(BAT[void,oid], int) : BAT[void,oid]"
]
[ "pf2tijah_node(BAT[oid,str], BAT[oid,oid], BAT[oid,oid], BAT[oid,oid],
BAT[oid,int], BAT[oid,str]) : BAT[void,oid]"
]
-[ "score_NLLR(BAT[oid,oid], BAT[void,int], BAT[oid,int], BAT[oid,int],
BAT[oid,any], dbl, int) : BAT[oid,dbl]"
]
[ "serialize_tijah_opt(BAT[void,BAT], int, BAT[void,oid], BAT[void,oid],
BAT[void,oid], BAT[void,int], BAT[void,lng], BAT[void,dbl], BAT[void,str]) :
BAT[str,str]" ]
[ "tijah_tokenize(str) : str"
]
[ "tj_add2collection(str, BAT[str,str], bit) : void"
]
@@ -53,8 +52,8 @@
[ "tj_normalizeTerm(str, str) : str"
]
[ "tj_ping() : void"
]
[ "tj_setlog(str) : void"
]
-[ "treemergejoin_sort(BAT[void,oid], BAT[void,int], BAT[void,oid]) :
BAT[oid,oid]"
]
-[ "treemergejoin_sort_unnested(BAT[void,oid], BAT[void,int], BAT[void,oid]) :
BAT[oid,oid]"
]
+[ "treemergejoin_sort(BAT[oid,oid], BAT[void,int], BAT[oid,oid]) :
BAT[oid,oid]"
]
+[ "treemergejoin_sort_unnested(BAT[oid,oid], BAT[void,int], BAT[oid,oid]) :
BAT[oid,oid]"
]
[ "union_add(BAT[oid,dbl], BAT[oid,dbl]) : BAT[oid,dbl]"
]
[ "union_div(BAT[oid,dbl], BAT[oid,dbl]) : BAT[oid,dbl]"
]
[ "union_mul(BAT[oid,dbl], BAT[oid,dbl]) : BAT[oid,dbl]"
]
Index: procs.stable.out
===================================================================
RCS file: /cvsroot/monetdb/pathfinder/modules/pftijah/Tests/procs.stable.out,v
retrieving revision 1.33
retrieving revision 1.34
diff -u -d -r1.33 -r1.34
--- procs.stable.out 18 Dec 2007 10:50:16 -0000 1.33
+++ procs.stable.out 14 Jan 2008 14:49:53 -0000 1.34
@@ -28,7 +28,6 @@
[ "CTrefine_rev(oid, BAT[oid,any]) : BAT[oid,oid]"
]
[ "CTrefine_rev(oid, any) : oid"
]
[ "CTsubgroup(BAT[oid,oid], BAT[oid,any], BAT[oid,any]) : BAT[oid,oid]"
]
-[ "NLLR_body(BAT[oid,dbl], BAT[oid,oid], str, BAT[oid,int], BAT[oid,int], dbl,
int, BAT[oid,str]) : BAT[oid,dbl]" ]
[ "TIMEZONE(str) : tzone"
]
[ "[:rotate_xor_hash=](BAT[any::1,int], int, BAT[any::1,any]) :
BAT[any::1,int]"
]
[ "[date](BAT[any::1,timestamp]) : BAT[any::1,date]"
]
@@ -58,20 +57,19 @@
[ "__ws_postcommit(BAT[void,BAT], oid, BAT[int,oid], BAT[int,oid],
BAT[int,oid], BAT[int,oid], BAT[oid,oid], BAT[oid,oid], BAT[void,oid],
BAT[void,oid]) : void"
]
[ "_buildIRindex(str, BAT[str,BAT]) : void"
]
[ "_collection_cleanup() : BAT[str,str]"
]
-[ "_containing_desc_tmj(BAT[oid,void], BAT[oid,oid], BAT[void,int]) :
BAT[oid,oid]"
]
[ "_del_doc(bit, BAT[any,str], BAT[oid,str], BAT[oid,str]) : bit"
]
[ "_del_doc_replace(BAT[oid,str], BAT[oid,str]) : void"
]
-[ "_getTermCollLH(oid, int, str) : dbl"
]
-[ "_getTermDocLHs0(oid, BAT[void,oid], str, bit) : BAT[oid,dbl]"
]
-[ "_getTermDocLHs1(oid, BAT[void,oid], str, bit) : BAT[oid,dbl]"
]
-[ "_getTermPositions(oid, str) : BAT[void,oid]"
]
+[ "_getTagPositions(oid, str) : BAT[void,oid]"
]
+[ "_getTermColCnt(oid, BAT[oid,str]) : int"
]
+[ "_getTermDocCnt(oid, BAT[oid,oid], BAT[void,int], BAT[oid,str]) :
BAT[oid,int]"
]
+[ "_getTermPositions(oid, BAT[oid,str]) : BAT[void,oid]"
]
[ "_runtime_create(str, lock, lock) : BAT[lock,BAT]"
]
[ "_runtime_flush() : void"
]
[ "_runtime_get(str) : BAT[lock,BAT]"
]
-[ "_score_LM(int, int, dbl, BAT[oid,dbl], dbl) : BAT[oid,dbl]"
]
-[ "_score_LMs(int, int, dbl, BAT[oid,dbl], dbl) : BAT[oid,dbl]"
]
-[ "_score_NLLR(int, int, dbl, BAT[oid,dbl], dbl) : BAT[oid,dbl]"
]
-[ "_score_OKAPI(oid, BAT[void,oid], str, dbl, dbl, int, dbl) : BAT[oid,dbl]"
]
+[ "_score_LM(int, int, int, int, BAT[oid,int], BAT[void,int], dbl) :
BAT[oid,dbl]"
]
+[ "_score_LMs(int, int, int, int, BAT[oid,int], BAT[void,int], dbl) :
BAT[oid,dbl]"
]
+[ "_score_NLLR(int, int, int, int, BAT[oid,int], BAT[void,int], dbl) :
BAT[oid,dbl]"
]
+[ "_score_OKAPI(int, BAT[oid,int], BAT[void,int], int, dbl, dbl, dbl) :
BAT[oid,dbl]"
]
[ "_shred_doc_base(BAT[oid,str], BAT[void,str], BAT[void,str], lng) :
BAT[oid,BAT]"
]
[ "_shred_doc_cleanup(lng, bit) : BAT[any,any]"
]
[ "_shred_into_docBAT(BAT[str,BAT], BAT[void,str], BAT[void,str],
BAT[void,timestamp], str, oid, lng) : oid"
]
@@ -110,8 +108,7 @@
[ "add_xrpc_trusted(BAT[void,str]) : void"
]
[ "add_xrpc_trusted(str) : void"
]
[ "addmonths(timestamp, int) : timestamp"
]
-[ "anc_desc(BAT[oid,any], BAT[oid,any], BAT[void,int]) : BAT[oid,oid]"
]
-[ "anc_desc_llscj(BAT[oid,any], BAT[oid,any], BAT[void,int]) : BAT[oid,oid]"
]
+[ "anc_desc(BAT[oid,any], BAT[oid,any], BAT[oid,str]) : BAT[oid,oid]"
]
[ "ancestor(BAT[void,oid], BAT[void,oid], oid, BAT[void,BAT], int) :
BAT[void,BAT]"
]
[ "ancestor_or_self(BAT[void,oid], BAT[void,oid], oid, BAT[void,BAT], int) :
BAT[void,BAT]" ]
[ "and_exp(BAT[any,any], BAT[any,any], int) : BAT[any,any]"
]
@@ -144,14 +141,12 @@
[ "clear(BAT[any::1,any::2]) : BAT[any::1,any::2]"
]
[ "clients() : void"
]
[ "cold(BAT[any::1,any::2]) : void"
]
-[ "collTermCount(str, BAT[oid,int]) : BAT[oid,int]"
]
[ "coll_lock_set(BAT[void,BAT], oid, int, str) : void"
]
[ "coll_lock_unset(BAT[void,BAT], oid, int, str, lng) : void"
]
[ "collection_cleanup(BAT[str,str]) : void"
]
[ "combine_node_info(any, BAT[void,oid], any, any, BAT[void,oid], any) :
BAT[void,BAT]"
]
-[ "contained_by(BAT[oid,dbl], BAT[oid,dbl], str, BAT[oid,str]) : BAT[oid,dbl]"
]
+[ "contained_by(BAT[oid,dbl], BAT[oid,dbl], BAT[oid,str]) : BAT[oid,dbl]"
]
[ "contained_by(BAT[oid,dbl], BAT[oid,str]) : BAT[oid,dbl]"
]
-[ "contained_by_frag(BAT[oid,BAT], BAT[oid,BAT], BAT[oid,str]) : BAT[oid,BAT]"
]
[ "containing(BAT[oid,dbl], BAT[oid,dbl], BAT[oid,str]) : BAT[oid,dbl]"
]
[ "correct_sizes(oid, BAT[void,oid], BAT[void,int]) : BAT[void,int]"
]
[ "cos(flt) : flt"
]
@@ -241,7 +236,6 @@
[ "get_xrpc_open() : bit"
]
[ "get_xrpc_options() : str"
]
[ "get_xrpc_port() : int"
]
-[ "gt(BAT[any,any], int) : BAT[any,any]"
]
[ "help() : void"
]
[ "help(str) : void"
]
[ "help_sig(oid) : void"
]
@@ -254,7 +248,6 @@
[ "htordered_unique_thetajoin(int, BAT[oid,any::1], BAT[oid,any::1], any, any)
: BAT[any,any]" ]
[ "impl_attr_own(BAT[oid,BAT], any, any) : BAT[oid,oid]"
]
[ "insert(str, int, int, str) : str"
]
-[ "intersect(BAT[any,any], BAT[any,any]) : BAT[any,any]"
]
[ "intersect(BAT[any::1,any::2], BAT[any::1,any::2]) : BAT[any::1,any::2]"
]
[ "io() : BAT[str,int]"
]
[ "is_type(int, int) : bit"
]
@@ -477,15 +470,16 @@
[ "order_regions(BAT[any,any]) : BAT[any,any]"
]
[ "orderby_table(str, str, ..BAT[any::1,any]..) : void"
]
[ "outerjoin(BAT[any::1,any::2], BAT[any::2,any::3]) : BAT[any::1,any::3]"
]
+[ "p_contained_by_max(BAT[oid,dbl], BAT[oid,dbl], BAT[oid,str]) :
BAT[oid,dbl]"
]
+[ "p_contained_by_max(BAT[oid,dbl], BAT[oid,str]) : BAT[oid,dbl]"
]
[ "p_contained_by_sum(BAT[oid,dbl], BAT[oid,dbl], BAT[oid,str]) :
BAT[oid,dbl]"
]
[ "p_contained_by_sum(BAT[oid,dbl], BAT[oid,str]) : BAT[oid,dbl]"
]
-[ "p_containing_q_LM(BAT[oid,dbl], BAT[void,str], str, BAT[oid,str]) :
BAT[oid,dbl]"
]
-[ "p_containing_q_LMs(BAT[oid,dbl], BAT[void,str], str, BAT[oid,str]) :
BAT[oid,dbl]"
]
-[ "p_containing_q_NLLR(BAT[oid,dbl], BAT[void,str], str, BAT[oid,str]) :
BAT[oid,dbl]"
]
-[ "p_containing_q_NLLR2(BAT[oid,dbl], BAT[void,str], str, BAT[oid,str]) :
BAT[oid,dbl]"
]
-[ "p_containing_q_NLLR_frag(BAT[oid,BAT], BAT[void,str], flt, BAT[oid,str]) :
BAT[oid,BAT]" ]
-[ "p_containing_q_OKAPI(BAT[oid,dbl], BAT[void,str], str, BAT[oid,str]) :
BAT[oid,dbl]"
]
-[ "p_containing_wsumd(BAT[oid,dbl], BAT[oid,dbl], int, BAT[oid,str]) :
BAT[oid,dbl]"
]
+[ "p_containing_max(BAT[oid,dbl], BAT[oid,dbl], BAT[oid,str]) : BAT[oid,dbl]"
]
+[ "p_containing_q_LM(BAT[oid,dbl], BAT[void,str], BAT[oid,str]) :
BAT[oid,dbl]"
]
+[ "p_containing_q_LMs(BAT[oid,dbl], BAT[void,str], BAT[oid,str]) :
BAT[oid,dbl]"
]
+[ "p_containing_q_NLLR(BAT[oid,dbl], BAT[void,str], BAT[oid,str]) :
BAT[oid,dbl]"
]
+[ "p_containing_q_OKAPI(BAT[oid,dbl], BAT[void,str], BAT[oid,str]) :
BAT[oid,dbl]"
]
+[ "p_containing_sum(BAT[oid,dbl], BAT[oid,dbl], BAT[oid,str]) : BAT[oid,dbl]"
]
[ "pagesample(BAT[oid,any]) : BAT[oid,void]"
]
[ "parent(BAT[void,oid], BAT[void,oid], oid, BAT[void,BAT], int) :
BAT[void,BAT]"
]
[ "patindex(str, str) : int"
]
@@ -500,8 +494,7 @@
[ "preceding(BAT[void,oid], BAT[void,oid], oid, BAT[void,BAT], int, chr) :
BAT[void,BAT]"
]
[ "preceding_sibling(BAT[void,oid], BAT[void,oid], oid, BAT[void,BAT], int,
chr) : BAT[void,BAT]"
]
[ "print(str, BAT[any::1,any], ..BAT[any::1,any]..) : void"
]
-[ "prior_logls(BAT[any,any], int, BAT[oid,str]) : BAT[any,any]"
]
-[ "prior_ls(BAT[any,any], int, BAT[oid,str]) : BAT[any,any]"
]
+[ "prior_ls(BAT[oid,dbl], BAT[oid,str]) : BAT[any,any]"
]
[ "procs() : void"
]
[ "prod(BAT[any,bte]) : bte"
]
[ "prod(BAT[any,dbl]) : dbl"
]
@@ -520,7 +513,7 @@
[ "replace(str, str, str) : str"
]
[ "replace_root(BAT[void,oid], BAT[oid,oid], BAT[oid,oid]) : void"
]
[ "rev_select(BAT[any::1,any::2], str, any::2) : BAT[any::1,any::2]"
]
-[ "rm_overlap(BAT[any,any], str) : BAT[any,any]"
]
+[ "rm_overlap(BAT[any,any], BAT[oid,str]) : BAT[any,any]"
]
[ "rotate_xor_hash(int, int, any) : int"
]
[ "rpc_client(str, str, str, str, str, str, int, int, lng, BAT[void,BAT],
BAT[void,oid], BAT[void,oid], BAT[void,oid], BAT[void,int], BAT[void,lng],
BAT[void,dbl], BAT[void,dbl], BAT[void,str]) : oid" ]
[ "rpcd_start() : void"
]
@@ -537,21 +530,18 @@
[ "select_node(BAT[oid,str]) : BAT[oid,dbl]"
]
[ "select_node(str, BAT[oid,str]) : BAT[oid,dbl]"
]
[ "select_node(str, bit, BAT[oid,str]) : BAT[oid,dbl]"
]
-[ "select_node_frag(str, bit, BAT[oid,str]) : BAT[oid,BAT]"
]
[ "select_phrase(BAT[void,oid], BAT[oid,str]) : BAT[oid,dbl]"
]
[ "select_root(BAT[oid,str]) : BAT[oid,dbl]"
]
[ "select_root(BAT[void,oid], BAT[oid,str]) : BAT[oid,dbl]"
]
-[ "select_root_frag(BAT[oid,str]) : BAT[oid,BAT]"
]
-[ "select_root_frag(BAT[void,oid], BAT[oid,str]) : BAT[oid,BAT]"
]
[ "select_startnodes(BAT[void,oid], BAT[oid,str]) : BAT[oid,dbl]"
]
-[ "select_startnodes_frag(BAT[void,oid], BAT[oid,str]) : BAT[oid,BAT]"
]
[ "select_term(str, BAT[oid,str]) : BAT[oid,dbl]"
]
[ "select_term(str, int, BAT[oid,str]) : BAT[oid,dbl]"
]
-[ "select_term_frag(str, BAT[oid,str]) : BAT[oid,BAT]"
]
[ "select_wide(BAT[oid,oid], BAT[oid,oid], oid, BAT[void,BAT], int,
BAT[void,oid], chr) : BAT[void,BAT]" ]
[ "semijoin(BAT[any::1,any::2], BAT[any::1,any]) : BAT[any::1,any::2]"
]
+[ "set_intersect(BAT[any,any], BAT[any,any], BAT[oid,str]) : BAT[any,any]"
]
[ "set_kind(BAT[void,oid], int) : BAT[void,int]"
]
[ "set_kind(oid, int) : int"
]
+[ "set_union(BAT[any,any], BAT[any,any], BAT[oid,str]) : BAT[any,any]"
]
[ "shred_doc(BAT[void,str], BAT[void,str], BAT[void,str], BAT[void,lng]) :
void" ]
[ "shred_doc(str, str) : void"
]
[ "shred_doc(str, str, str, lng) : void"
]
@@ -562,8 +552,6 @@
[ "sin(flt) : flt"
]
[ "sinh(flt) : flt"
]
[ "size(BAT[any,bit]) : wrd"
]
-[ "size_entity(BAT[oid,any], BAT[oid,str]) : BAT[oid,dbl]"
]
-[ "size_term(BAT[oid,any], BAT[oid,str]) : BAT[oid,dbl]"
]
[ "sort_ht(BAT[any::1,any::2]) : BAT[any::1,any::2]"
]
[ "space(int) : str"
]
[ "split2frag(BAT[oid,any], BAT[void,oid]) : BAT[oid,BAT]"
]
@@ -611,19 +599,13 @@
[ "tj_add2collection(str, BAT[str,str], bit) : void"
]
[ "tj_add2collection(str, str, str, bit) : void"
]
[ "tj_addEquivalenceClass(BAT[any,any]) : void"
]
-[ "tj_coll_init(str, str) : void"
]
-[ "tj_coll_remove(str) : void"
]
-[ "tj_collection(str) : BAT[void,BAT]"
]
[ "tj_delete_collection(str) : void"
]
[ "tj_delete_collection(str, bit) : void"
]
[ "tj_delete_global() : void"
]
[ "tj_dump_collection(str) : void"
]
[ "tj_extend_collection(str, BAT[void,str]) : void"
]
-[ "tj_finalize(str) : void"
]
[ "tj_get_collection_lock(str) : lock"
]
[ "tj_get_ft_index(BAT[str,str], bit) : str"
]
-[ "tj_global_init() : void"
]
-[ "tj_global_remove() : void"
]
[ "tj_initEquivalences() : void"
]
[ "tj_init_collection(str, BAT[str,str]) : void"
]
[ "tj_init_collection(str, BAT[str,str], BAT[void,str]) : void"
]
@@ -632,14 +614,15 @@
[ "tj_init_global(BAT[str,str], bit) : void"
]
[ "tj_is_indexed(str) : bit"
]
[ "tj_is_indexed(str) : bit"
]
+[ "tj_nid2pre(BAT[oid,any], BAT[oid,str]) : BAT[oid,any]"
]
[ "tj_ping() : void"
]
[ "tj_play_doc_tape(BAT[void,BAT], BAT[void,oid], BAT[void,int],
BAT[void,lng], BAT[void,str]) : void"
]
[ "tj_rebuild_collection(str) : void"
]
[ "tj_setBackgroundCollName(str, BAT[oid,str]) : BAT[void,str]"
]
[ "tj_set_parameter(str, str, str) : void"
]
+[ "tj_size_collection(str) : void"
]
+[ "tj_size_collection2(str) : void"
]
[ "tj_trace(str) : void"
]
-[ "tj_url2Collection(str, str, str, str) : void"
]
-[ "tj_url2Collection_index(str, str, str) : void"
]
[ "tmark_grp_unique(BAT[any,any], BAT[oid,any]) : BAT[oid,oid]"
]
[ "tmark_grp_unique(any, BAT[oid,any]) : BAT[oid,oid]"
]
[ "tmark_unique(BAT[oid,any], BAT[oid,any]) : BAT[oid,oid]"
]
@@ -649,7 +632,6 @@
[ "trace_ops(BAT[void,any], BAT[oid,BAT], BAT[oid,BAT], BAT[oid,BAT],
BAT[oid,int], BAT[oid,oid], int, oid, oid) : void"
]
[ "trace_qname_uri(oid, oid) : str"
]
[ "trace_scopes(BAT[void,any], BAT[oid,BAT], BAT[oid,BAT], BAT[oid,BAT],
BAT[oid,BAT], BAT[oid,BAT], BAT[oid,int], BAT[oid,oid], int, oid, oid) : void"
]
-[ "union(BAT[any,any], BAT[any,any]) : BAT[any,any]"
]
[ "union(BAT[any::1,any::2], BAT[any::1,any::2]) : BAT[any::1,any::2]"
]
[ "unique(BAT[any::1,any::2]) : BAT[any::1,any::2]"
]
[ "vars() : void"
]
-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
_______________________________________________
Monetdb-pf-checkins mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/monetdb-pf-checkins
