Changeset: 52e0d62f9040 for MonetDB
URL: https://dev.monetdb.org/hg/MonetDB?cmd=changeset;node=52e0d62f9040
Removed Files:
sql/test/in-value-list/debugpath.txt
sql/test/in-value-list/python-scripts/create-copy-into-query.py
sql/test/in-value-list/python-scripts/create-select-query.py
sql/test/in-value-list/python-scripts/generate-query.py
sql/test/in-value-list/python-scripts/init-db.py
Branch: in-value-list
Log Message:
Remove debug and experimentation files.
diffs (truncated from 702 to 300 lines):
diff --git a/sql/test/in-value-list/debugpath.txt
b/sql/test/in-value-list/debugpath.txt
deleted file mode 100644
--- a/sql/test/in-value-list/debugpath.txt
+++ /dev/null
@@ -1,593 +0,0 @@
-NOT IN
-
-(gdb) p nr_of_bats(mb,p)
-$95 = 3
-(gdb) p mb->var[p->argv[0]].type
-$96 = 65542
-(gdb) p mb->var[p->argv[1]].type
-$97 = 65542
-(gdb) p mb->var[p->argv[2]].type
-$98 = 65541
-(gdb) p mb->var[p->argv[3]].type
-$99 = 65541
-(gdb) p mb->var[p->argv[4]].type
-$100 = 65542
-(gdb) info br
-Num Type Disp Enb Address What
-15 hw watchpoint keep y (malType *) *0x7fffd404ec90
- breakpoint already hit 3 times
-(gdb) del 15
-(gdb) p->argv[0]
-Undefined command: "p->argv". Try "help".
-(gdb) p p->argv[0]
-$101 = 40
-(gdb) p p->argv[1]
-$102 = 41
-(gdb) p p->argv[2]
-$103 = 9
-(gdb) p p->argv[3]
-$104 = 39
-(gdb) p p->argv[4]
-$105 = 29
-(gdb) p *p
-$106 = {token = 50 '2', barrier = 0 '\000', typechk = 2 '\002', gc = 3 '\003',
polymorphic = 0 '\000', varargs = 0 '\000', jump = 0, pc = 0, fcn = 0x0, blk =
0x7fffd4041c70, mitosis = -1, clock = 0, ticks = 0, calls = 0,
- totticks = 0, wbytes = 0, modname = 0x7ffff4dfe5b8 "algebra", fcnname =
0x7ffff4dfeba8 "antijoin", argc = 8, retc = 2, maxarg = 16, argv =
0x7fffd404581c}
-(gdb) p p
-$107 = (InstrPtr) 0x7fffd40457b0
-(gdb) p mb
-$108 = (MalBlkPtr) 0x7fffd40027e0
-
-IN
-
-mb = (MalBlkPtr) 0x7fffd4029240
-p = (InstrPtr) 0x7fffd4145680
-
-(gdb) p mb->var[p->argv[0]].type
-$120 = 65542
-(gdb) p mb->var[p->argv[1]].type
-$121 = 65542
-(gdb) p mb->var[p->argv[2]].type
-$122 = 65541
-(gdb) p mb->var[p->argv[3]].type
-$123 = 65541
-(gdb) p mb->var[p->argv[4]].type
-$124 = 4
-
-(gdb) p p->argv[0]
-$127 = 36
-(gdb) p p->argv[1]
-$128 = 37
-(gdb) p p->argv[2]
-$129 = 18
-(gdb) p p->argv[3]
-$130 = 35
-(gdb) p p->argv[4]
-$131 = 38
-
-
-NOT IN
-NOTE: watching malType of VarRecord at 0x7fffd40588b0 (p
&(mb->var[p->argv[4]]))
-(gdb) p &(mb->var[p->argv[4]].type)
-$162 = (malType *) 0x7fffd40588f0
-this value is set in mat_apply2 which has only one call side in the codebase
in OPTmergetableImplementation.
-
-(IN) doesn't call mat_apply2
-
-(NOT IN) goes in because there is a 'select' instruction in the mal plan at
the point that it is handled in OPTmergetableImplementation.
-
-(IN) doesn't have this select instruction??!!?? Why??
-
-Inside OPTmergetableImplementation the mal plans look like these is the plan
according to (NOT IN)
-IN NOT in
-define define
-mvc mvc
-tid tid
-tid tid
-tid tid
-tid tid
-tid tid
-new new
-bind bind
-bind bind
-bind bind
-bind bind
-bind bind
-new new
-bind bind
-bind bind
-bind bind
-bind bind
-bind bind
-bind bind
-new new
-new new
-projection
-bind bind
-bind bind
-bind bind
-bind bind
-bind bind
-new new
-bind bind
-bind bind
-bind bind
-bind bind
-bind bind
-bind bind
-new new
-new new
-projection select
-new new
-append append
-groupdone groupdone
-projection projection
-join antijoin
-projection projection
-projection projection
-new new
-new new
-new new
-new new
-new new
-append append
-append append
-append append
-append append
-append append
-append append
-append append
-append append
-append append
-append append
-resultSet resultSet
-deadcode deadcode
-aliases aliases
-constants constants
-commonTerms commonTerms
-projectionpath projectionpath
-deadcode deadcode
-reorder reorder
-matpack matpack
-dataflow dataflow
-querylog querylog
-multiplex multiplex
-generator generator
-profiler profiler
-candidates candidates
-postfix postfix
-deadcode deadcode
-wlc wlc
-garbageCollector garbageCollector
-
-So in the (IN) situation there are two projections that aren't there in (NOT
IN). However in (NOT IN) we have a select.
-I am trying to figure out when the projection instruction is set. Seems that
the assiocated memory addresses change per run...
-(gdb) p &(p -> fcnname)
-$27 = (str *) 0x7fffd4045a68
-the first time that I catch it using a conditional breakpoint in
OPTmergetableImplementation it has the above value
-
-Ok I think I found the origin of this projection instruction. However it is
copied from some other malblock:
-(gdb) p &(old->stmt[i]->fcnname)
-$35 = (str *) 0x7fffd4041c08
-(gdb) watch * (str *) 0x7fffd4041c08
-Hardware watchpoint 11: * (str *) 0x7fffd4041c08
-Ok somewhere in the parsing a stmt_project_join occurs. This is where the
projection comes from. the stacktrace is
-#0 stmt_project_join (be=0x7fffd4002b60, op1=0x7fffd4136ec0,
op2=0x7fffd4137040, ins=0x0) at
../../../../sql/backends/monet5/sql_statement.c:1970
-#1 0x00007ffff49d4df2 in stmt_project (be=0x7fffd4002b60, op1=0x7fffd4136ec0,
op2=0x7fffd4137040) at ../../../../sql/backends/monet5/sql_statement.c:1990
-#2 0x00007ffff49ad31d in stmt_col (be=0x7fffd4002b60, c=0x5555573dadb0,
del=0x7fffd4136ec0) at ../../../../sql/backends/monet5/rel_bin.c:878
-#3 0x00007ffff49aeb16 in rel2bin_basetable (be=0x7fffd4002b60,
rel=0x7fffd4136020) at ../../../../sql/backends/monet5/rel_bin.c:1307
-#4 0x00007ffff49c18f0 in subrel_bin (be=0x7fffd4002b60, rel=0x7fffd4136020,
refs=0x7fffd4136de0) at ../../../../sql/backends/monet5/rel_bin.c:5287
-#5 0x00007ffff49b598d in rel2bin_select (be=0x7fffd4002b60,
rel=0x7fffd4136340, refs=0x7fffd4136de0) at
../../../../sql/backends/monet5/rel_bin.c:2707
-#6 0x00007ffff49c1a81 in subrel_bin (be=0x7fffd4002b60, rel=0x7fffd4136340,
refs=0x7fffd4136de0) at ../../../../sql/backends/monet5/rel_bin.c:5325
-#7 0x00007ffff49b4ec2 in rel2bin_project (be=0x7fffd4002b60,
rel=0x7fffd4136800, refs=0x7fffd4136de0, topn=0x0) at
../../../../sql/backends/monet5/rel_bin.c:2571
-#8 0x00007ffff49c1a53 in subrel_bin (be=0x7fffd4002b60, rel=0x7fffd4136800,
refs=0x7fffd4136de0) at ../../../../sql/backends/monet5/rel_bin.c:5321
-#9 0x00007ffff49c1d46 in _subrel_bin (be=0x7fffd4002b60, rel=0x7fffd4136800,
refs=0x7fffd4136de0) at ../../../../sql/backends/monet5/rel_bin.c:5384
-#10 0x00007ffff49c1e59 in output_rel_bin (be=0x7fffd4002b60,
rel=0x7fffd4136800) at ../../../../sql/backends/monet5/rel_bin.c:5407
-#11 0x00007ffff49e158e in sql_relation2stmt (be=0x7fffd4002b60,
r=0x7fffd4136800) at ../../../../sql/backends/monet5/sql_gencode.c:606
-#12 0x00007ffff49e1797 in backend_dumpstmt (be=0x7fffd4002b60,
mb=0x7fffd4132720, r=0x7fffd4136800, top=1, add_end=1, query=0x7fffd4001db0
"explain select * from foo where i in (1);")
- at ../../../../sql/backends/monet5/sql_gencode.c:646
-#13 0x00007ffff49e205b in backend_dumpproc (be=0x7fffd4002b60,
c=0x7ffff5054368, cq=0x7fffd41331e0, r=0x7fffd4136800) at
../../../../sql/backends/monet5/sql_gencode.c:776
-#14 0x00007ffff499cfd3 in SQLparser (c=0x7ffff5054368) at
../../../../sql/backends/monet5/sql_scenario.c:1262
-#15 0x00007ffff7a09d0e in runPhase (c=0x7ffff5054368, phase=1) at
../../../monetdb5/mal/mal_scenario.c:510
-#16 0x00007ffff7a09e17 in runScenarioBody (c=0x7ffff5054368, once=0) at
../../../monetdb5/mal/mal_scenario.c:532
-#17 0x00007ffff7a0a0d3 in runScenario (c=0x7ffff5054368, once=0) at
../../../monetdb5/mal/mal_scenario.c:569
-#18 0x00007ffff7a0be65 in MSserveClient (dummy=0x7ffff5054368) at
../../../monetdb5/mal/mal_session.c:525
-#19 0x00007ffff7a0b927 in MSscheduleClient (command=0x7fffd4000b30 "\020\v",
challenge=0x7fffef1f7eab "DIGTjxl4Q", fin=0x7fffe8000c80, fout=0x7fffe8002d60,
protocol=PROTOCOL_9, blocksize=8190)
----Type <return> to continue, or q <return> to quit---c
- at ../../../monetdb5/mal/mal_session.c:403
-#20 0x00007ffff7aa6d67 in doChallenge (data=0x7fffe8000c90) at
../../../../monetdb5/modules/mal/mal_mapi.c:270
-#21 0x00007ffff75df237 in thread_starter (arg=0x555555a35af0) at
../../gdk/gdk_system.c:492
-#22 0x00007ffff59106db in start_thread (arg=0x7fffef1f8700) at
pthread_create.c:463
-#23 0x00007ffff563988f in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95
-
-I THINK it is the first projection join that hits the following line as
breakpoint: sql_statement.c:1978
-
-
-
-4223222 CRASH
-
-Ok so I have found some code in rel_select that when deactivated, no longer
crashes the server. However it does creates resource leeks.
-
-aris@aris-MonetDB:~/Sources/monetdb$ time mclient -f rowcount
select_10K_10K.sql
-633678 tuples
-
-real 2m4.657s
-user 0m0.084s
-sys 0m0.039s
-aris@aris-MonetDB:~/Sources/monetdb$ time mclient -f rowcount
select_10K_10K.sql
-9366322 tuples
-
-real 5m24.929s
-user 0m0.950s
-sys 0m0.584s
-aris@aris-MonetDB:~/Sources/monetdb$ echo "first one is for default branch
with 'IN' operator."
-first one is for default branch with 'IN' operator.
-aris@aris-MonetDB:~/Sources/monetdb$ echo "second one is for default branch
with 'NOT IN' operator."
-second one is for default branch with 'NOT IN' operator.
-aris@aris-MonetDB:~/Sources/monetdb$ time mclient -f rowcount
select_10K_10K.sql
-9366322 tuples
-
-real 5m2.891s
-user 0m1.049s
-sys 0m0.709s
-aris@aris-MonetDB:~/Sources/monetdb$ echo "Previous one is "NOT IN" query with
default branch with weird "NOT IN clause" removed"
-Previous one is NOT IN query with default branch with weird NOT IN clause
removed
-aris@aris-MonetDB:~/Sources/monetdb$ time mclient -f rowcount
select_10K_10K.sql
-^C
-real 17m51.920s
-user 0m0.008s
-sys 0m0.000s
-
-aris@aris-MonetDB:~/Sources/monetdb$ echo "The previous query is a "NOT IN" on
the special branch with the "NOT IN clause" removed. the query never finishes
and mserver complains about running out of virtual memory????"
-The previous query is a NOT IN on the special branch with the NOT IN clause
removed. the query never finishes and mserver complains about running out of
virtual memory????
-aris@aris-MonetDB:~/Sources/monetdb$ echo "#GDKmremap(87777607680) fails, try
to free up space [memory in use=71173840,virtual memory in use=673214695120]"
-#GDKmremap(87777607680) fails, try to free up space [memory in
use=71173840,virtual memory in use=673214695120]
-aris@aris-MonetDB:~/Sources/monetdb$ kill -9 $(pgrep mserver5)
-
-Ok I have studied the dubious code in rel_select. What it basically does in
the "NOT IN" situation is that it adds an additional predicate expression
"isnull(c) = False". That's when the assertions fail.
-
-Now if I perform the following query:
-client -f rowcount -s "select * from foo as bar where bar.i = 3 or bar.i in
(1, 2);"
-
- it crashes in exactly the same way as the "NOT IN" version would crash
without the initial predicate expression:
- client -f rowcount -s "select * from foo as bar bar.i in (1, 2);"
-
-I think this is what Niels means with that the join statement doesn't merge
the candidate lists correctly.
-
-The following query runs fine:
-mclient -f rowcount -s "select foo.c from foo where i in (1) or i = 2;"
-
-it cals batjoin with multiple worker threads and an empty candidate list.
Which is not weird because the COMPARE expression comes after the IN expression.
-
-However if I change the order of the predicates it breaks again. Furthermore
handle_in_exps is called twice!
-
-mclient -f rowcount -s "select foo.c from foo where i = 2 or i in (1);"
-
-the associated stracktrace of the first and second column
-
-#0 handle_in_exps (be=0x7fffd4002b60, ce=0x7fffd4031a60, nl=0x7fffd4031c90,
left=0x7fffd4033300, right=0x0, grp=0x0, ext=0x0, cnt=0x0, sel=0x0, in=1,
use_r=0) at ../../../../sql/backends/monet5/rel_bin.c:327 #0
handle_in_exps (be=0x7fffd4002b60, ce=0x7fffd4031610, nl=0x7fffd40328b0,
left=0x7fffd4033300, right=0x0, grp=0x0, ext=0x0, cnt=0x0, sel=0x7fffd4033780,
in=1, use_r=0) at ../../../../sql/backends/monet5/rel_bin.c:327
-#1 0x00007ffff49abb94 in exp_bin (be=0x7fffd4002b60, e=0x7fffd4031e80,
left=0x7fffd4033300, right=0x0, grp=0x0, ext=0x0, cnt=0x0, sel=0x0) at
../../../../sql/backends/monet5/rel_bin.c:676 #1 0x00007ffff49abb94 in
exp_bin (be=0x7fffd4002b60, e=0x7fffd4032980, left=0x7fffd4033300, right=0x0,
grp=0x0, ext=0x0, cnt=0x0, sel=0x7fffd4033780) at
../../../../sql/backends/monet5/rel_bin.c:676
-#2 0x00007ffff49ac02a in exp_bin (be=0x7fffd4002b60, e=0x7fffd4032600,
left=0x7fffd4033300, right=0x0, grp=0x0, ext=0x0, cnt=0x0, sel=0x0) at
../../../../sql/backends/monet5/rel_bin.c:721
-#3 0x00007ffff49b5bd0 in rel2bin_select (be=0x7fffd4002b60,
rel=0x7fffd4031920, refs=0x7fffd4032da0) at
../../../../sql/backends/monet5/rel_bin.c:2745 #2 0x00007ffff49b5bd0 in
rel2bin_select (be=0x7fffd4002b60, rel=0x7fffd4031920, refs=0x7fffd4032da0) at
../../../../sql/backends/monet5/rel_bin.c:2745
-#4 0x00007ffff49c1a7d in subrel_bin (be=0x7fffd4002b60, rel=0x7fffd4031920,
refs=0x7fffd4032da0) at ../../../../sql/backends/monet5/rel_bin.c:5325 #3
0x00007ffff49c1a7d in subrel_bin (be=0x7fffd4002b60, rel=0x7fffd4031920,
refs=0x7fffd4032da0) at ../../../../sql/backends/monet5/rel_bin.c:5325
-#5 0x00007ffff49b4ebe in rel2bin_project (be=0x7fffd4002b60,
rel=0x7fffd4032080, refs=0x7fffd4032da0, topn=0x0) at
../../../../sql/backends/monet5/rel_bin.c:2571 #4 0x00007ffff49b4ebe in
rel2bin_project (be=0x7fffd4002b60, rel=0x7fffd4032080, refs=0x7fffd4032da0,
topn=0x0) at ../../../../sql/backends/monet5/rel_bin.c:2571
-#6 0x00007ffff49c1a4f in subrel_bin (be=0x7fffd4002b60, rel=0x7fffd4032080,
refs=0x7fffd4032da0) at ../../../../sql/backends/monet5/rel_bin.c:5321 #5
0x00007ffff49c1a4f in subrel_bin (be=0x7fffd4002b60, rel=0x7fffd4032080,
refs=0x7fffd4032da0) at ../../../../sql/backends/monet5/rel_bin.c:5321
-#7 0x00007ffff49c1d42 in _subrel_bin (be=0x7fffd4002b60, rel=0x7fffd4032080,
refs=0x7fffd4032da0) at ../../../../sql/backends/monet5/rel_bin.c:5384 #6
0x00007ffff49c1d42 in _subrel_bin (be=0x7fffd4002b60, rel=0x7fffd4032080,
refs=0x7fffd4032da0) at ../../../../sql/backends/monet5/rel_bin.c:5384
-#8 0x00007ffff49c1e55 in output_rel_bin (be=0x7fffd4002b60,
rel=0x7fffd4032080) at ../../../../sql/backends/monet5/rel_bin.c:5407 #7
0x00007ffff49c1e55 in output_rel_bin (be=0x7fffd4002b60, rel=0x7fffd4032080) at
../../../../sql/backends/monet5/rel_bin.c:5407
-#9 0x00007ffff49e158a in sql_relation2stmt (be=0x7fffd4002b60,
r=0x7fffd4032080) at ../../../../sql/backends/monet5/sql_gencode.c:606 #8
0x00007ffff49e158a in sql_relation2stmt (be=0x7fffd4002b60, r=0x7fffd4032080)
at ../../../../sql/backends/monet5/sql_gencode.c:606
-#10 0x00007ffff49e1793 in backend_dumpstmt (be=0x7fffd4002b60,
mb=0x7fffd40022e0, r=0x7fffd4032080, top=1, add_end=1, query=0x7fffd4002030
"select foo.c from foo where i = 2 or i in (1);") at
../../../../sql/backends/monet5/sql_gencode.c:646 #9 0x00007ffff49e1793
in backend_dumpstmt (be=0x7fffd4002b60, mb=0x7fffd40022e0, r=0x7fffd4032080,
top=1, add_end=1, query=0x7fffd4002030 "select foo.c from foo where i = 2 or i
in (1);") at ../../../../sql/backends/monet5/sql_gencode.c:646
-#11 0x00007ffff49e2057 in backend_dumpproc (be=0x7fffd4002b60,
c=0x7ffff5054368, cq=0x7fffd4001fa0, r=0x7fffd4032080) at
../../../../sql/backends/monet5/sql_gencode.c:776 #10 0x00007ffff49e2057 in
backend_dumpproc (be=0x7fffd4002b60, c=0x7ffff5054368, cq=0x7fffd4001fa0,
r=0x7fffd4032080) at ../../../../sql/backends/monet5/sql_gencode.c:776
-#12 0x00007ffff499cfd3 in SQLparser (c=0x7ffff5054368) at
../../../../sql/backends/monet5/sql_scenario.c:1262 #11 0x00007ffff499cfd3 in
SQLparser (c=0x7ffff5054368) at
../../../../sql/backends/monet5/sql_scenario.c:1262
-#13 0x00007ffff7a09c34 in runPhase (c=0x7ffff5054368, phase=1) at
../../../monetdb5/mal/mal_scenario.c:510 #12 0x00007ffff7a09c34 in runPhase
(c=0x7ffff5054368, phase=1) at ../../../monetdb5/mal/mal_scenario.c:510
-#14 0x00007ffff7a09d3d in runScenarioBody (c=0x7ffff5054368, once=0) at
../../../monetdb5/mal/mal_scenario.c:532 #13 0x00007ffff7a09d3d in
runScenarioBody (c=0x7ffff5054368, once=0) at
../../../monetdb5/mal/mal_scenario.c:532
-#15 0x00007ffff7a09ff9 in runScenario (c=0x7ffff5054368, once=0) at
../../../monetdb5/mal/mal_scenario.c:569 #14 0x00007ffff7a09ff9 in
runScenario (c=0x7ffff5054368, once=0) at
../../../monetdb5/mal/mal_scenario.c:569
-#16 0x00007ffff7a0bd8b in MSserveClient (dummy=0x7ffff5054368) at
../../../monetdb5/mal/mal_session.c:525 #15 0x00007ffff7a0bd8b in
MSserveClient (dummy=0x7ffff5054368) at ../../../monetdb5/mal/mal_session.c:525
-#17 0x00007ffff7a0b84d in MSscheduleClient (command=0x7fffd4000b30 "0A",
challenge=0x7fffef1f7eab "1WFbqUXSprF", fin=0x555557092f60,
fout=0x7fffe8002d60, protocol=PROTOCOL_9, blocksize=8190) at
../../../monetdb5/mal/mal_session.c:403 #16 0x00007ffff7a0b84d in
MSscheduleClient (command=0x7fffd4000b30 "0A", challenge=0x7fffef1f7eab
"1WFbqUXSprF", fin=0x555557092f60, fout=0x7fffe8002d60, protocol=PROTOCOL_9,
blocksize=8190) at ../../../monetdb5/mal/mal_session.c:403
-#18 0x00007ffff7aa6c8d in doChallenge (data=0x555557092f70) at
../../../../monetdb5/modules/mal/mal_mapi.c:270 #17 0x00007ffff7aa6c8d in
doChallenge (data=0x555557092f70) at
../../../../monetdb5/modules/mal/mal_mapi.c:270
-#19 0x00007ffff75df237 in thread_starter (arg=0x555555a35d90) at
../../gdk/gdk_system.c:492 #18 0x00007ffff75df237 in thread_starter
(arg=0x555555a35d90) at ../../gdk/gdk_system.c:492
-#20 0x00007ffff59106db in start_thread (arg=0x7fffef1f8700) at
pthread_create.c:463 #19 0x00007ffff59106db in start_thread
(arg=0x7fffef1f8700) at pthread_create.c:463
-#21 0x00007ffff563988f in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:95 #20 0x00007ffff563988f in
clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
-
-I have compared the following two queries
-mclient -f rowcount -s "select foo.c from foo where i = 2 or i < 2;"
-and
-mclient -f rowcount -s "select foo.c from foo where i = 2 or i in (1);"
-
-I notice that rel2bin_select thinks that for the former query there is only
one (parent) expression associated with the select query: the OR predicate
itself. While for the second query, it thinks there are two associated
expression.
_______________________________________________
checkin-list mailing list
[email protected]
https://www.monetdb.org/mailman/listinfo/checkin-list
